Interactive · Week 4 · Section 7

Watch a policy game its reward.

Pick which reward components are active, then drag the training-stage slider and watch the policy progressively discover ways to score higher without actually doing the task better. The "true quality" score (held out from training) tracks what a thoughtful human would think — and diverges sharply from the reward as the policy learns to hack. Toggle the KL constraint on to see the load-bearing defense in action.

Active reward components

Mitigation

KL constraint to reference caps how far the policy can drift

Training stage Stage 0 · Honest

honest verbose cited structured hacked

user prompt How do I reset my password on the dashboard?

Reward score

—

what training optimizes

True quality

—

held-out human evaluator

Reward components · current stage

Reward vs quality across training

Reward (training signal)

True quality (held out)

What's happening here

Stage outputs are pre-scripted to demonstrate the canonical reward-hacking failure modes — length bias, citation hallucination, sycophancy, format gaming. Real RLHF training discovers these in the same order, just over thousands of steps instead of five clicks. The KL-constraint toggle clamps the slider at stage 2, simulating how a tight KL bound to a sane reference prevents the policy from reaching the most egregious gaming. The "true quality" scoring is hand-calibrated to reflect what a thoughtful evaluator would judge — separate from any of the gameable reward components. Source-of-truth lives as <RewardHackingDemo/> in the curriculum codebase.