Interactive · Week 11 · Section 1

Why RLHF models are confidently wrong.

Five training stages, same task, 500 samples each. The reliability diagram plots stated confidence (x-axis) against empirical accuracy in that confidence bin (y-axis). Perfect calibration lies on the diagonal. Watch what happens through the training pipeline: base models are roughly calibrated; SFT shifts confidence up slightly; RLHF cranks confidence to the ceiling regardless of actual correctness; temperature scaling pulls it back; verbal hedging fixes it differently. ECE quantifies the gap. The visceral lesson: if you trust an RLHF model's reported confidence, you're trusting the wrong number.

Pick a training stage

Expected Calibration Error

—

average gap, weighted by bin size

Overall accuracy

—

across 500 samples

Mean stated confidence

—

what the model claims

Overconfidence

—

stated minus actual

Reliability diagram · stated confidence vs empirical accuracy

10 confidence bins · 500 predictions

In the chart

Perfect calibration (y = x)

Empirical accuracy per bin

Calibration curve through bin midpoints

Miscalibration gap

How to read it

When the curve sits below the diagonal, the model is overconfident at that confidence level. When above, underconfident. Total gap weighted by bin size = ECE.

Per-bin breakdown

—

What's happening here

Each variant draws 500 predictions from a synthetic distribution where the model has variable per-question skill. Stated confidence is generated to reproduce the documented training-stage patterns: base models ≈ calibrated; SFT shifts confidence up modestly; RLHF compresses confidence to the high end regardless of actual accuracy; temperature scaling (T ≈ 1.7 fitted to validation data) pulls high-confidence predictions back toward the diagonal; verbal hedging produces a different correction pattern by adding linguistic uncertainty markers that map to lower stated confidence. ECE = Σᵢ (nᵢ/N) · |confᵢ − accᵢ| across 10 bins. Source-of-truth lives as <CalibrationCurveDemo/> in the curriculum codebase.