Interactive · Week 4 · Section 3

Why DPO works without a reward model.

DPO's loss is a one-line function of the margin between chosen and rejected log-probabilities. The β parameter controls how steeply the loss falls as the margin grows — and how aggressively the gradient saturates. Drag β to see the curve change shape, run training steps, and watch how four preference pairs converge at different rates because of where they started on the curve.

β (beta) 0.30

Mean margin

—

how aligned the policy is

Total DPO loss

—

across all 4 pairs

DPO loss curve

−log σ(β · margin) · current dots show each preference pair

Preference pairs

step 0

What's happening here

Margins evolve via gradient descent on the DPO loss with a fixed learning rate of 0.8 and small Gaussian noise per step (real training is stochastic). Starting margins are deliberately varied to show different convergence rates: some pairs are already partially aligned, others start with the policy preferring the wrong response. The reference model is held fixed throughout (this is the standard DPO setup). Source-of-truth lives as <PreferenceLossExplorer/> in the curriculum codebase.