Why can a reward model be exploited?
It is an imperfect proxy for the intended goal. An optimized model may find response patterns that earn high scores without delivering the desired quality.
Explore This Question
Search for additional explanations using these tools:
More FAQs in What is a reward model, and how is it trained using human preferences?
Not necessarily. A learned reward model can score additional generated responses after being trained on human comparisons, reducing the need for direct human review each time.
They serve different roles: the assistant generates responses, while the reward model estimates a feedback score. Some approaches share foundations, but the functions remain distinct.
Still have questions?
Browse all our FAQs or reach out to our support team