What is reinforcement learning with human feedback (RLHF)?

What is reinforcement learning with human feedback (RLHF)?

Reinforcement learning with human feedback, or RLHF, uses human judgments to guide a model's behavior through a reward signal. For language assistants, the feedback often compares candidate responses according to qualities such as helpfulness, relevance, and clarity.

A common language-model pipeline

In a classic setup, a model first receives supervised demonstrations. Human reviewers then compare generated responses, and those comparisons train a reward model.

A reinforcement-learning algorithm updates the assistant to produce responses that score well, often while limiting how far it moves from a reference model. The InstructGPT paper describes a prominent version of this pipeline using PPO.

For example, reviewers might prefer a coding answer that fixes the actual bug and explains the change over one that only restates the error.

Preference training has several forms

The classic sequence is not a universal requirement. Different systems use different feedback sources and optimization methods.

Direct Preference Optimization is an alternative that trains from preference pairs without fitting a separate explicit reward model in the standard DPO procedure. Preference tuning should not automatically be described as the same PPO-based pipeline.

What human feedback can and cannot establish

RLHF can encourage useful response behavior, but a preferred answer is not necessarily a true answer. Reviewers may disagree or miss subtle errors.

Evaluation should therefore check factual accuracy, task completion, and robustness independently of reward scores. Human feedback is a powerful training signal, not a guarantee of correctness or universally shared values.