What is a reward model, and how is it trained using human preferences?

What is a reward model, and how is it trained using human preferences?

A reward model predicts how desirable an output is under a particular feedback scheme. In a common human-preference setup, people compare multiple responses to the same prompt. Their choices become training data for a model that learns to score responses.

From comparisons to a scoring function

Suppose an assistant drafts two explanations of a coding error. One is clear and correct; the other is vague and misses the bug. Reviewers select the better response using an evaluation rubric.

The reward model is trained to assign a higher score to the preferred response. Repeating this across many prompts teaches patterns associated with the collected judgments.

The numerical score is a learned preference signal, not a direct measurement of objective truth or a probability that every statement is correct.

How the score supports training

A language model can generate candidate responses, receive reward scores, and be optimized toward outputs that score better. A classic RLHF pipeline uses this arrangement.

A reward model can also support ranking or analysis without necessarily being deployed as the user-facing assistant.

Quality depends on the feedback

If reviewers consistently reward confident wording, excessive length, or agreement with the user, the reward model may learn those shortcuts. Disagreement between reviewers also reveals that preferences can be contextual.

Use clear rubrics, varied examples, and independent evaluations. A score is useful only insofar as it predicts the behavior the application actually wants, especially on responses unlike those in its training comparisons.