reward-hacking-lab · live demo

Watch a policy hack its reward model

As you crank up optimization pressure (KL distance from the base policy), the proxy reward the model is trained on keeps climbing — but the true reward peaks and then collapses. That gap is reward overoptimization, a.k.a. Goodhart's law. Drag the slider.

proxy reward (what the RM scores) true / gold reward (what we actually want)

optimization pressure

KL from base policy

—

Proxy reward

—

True reward

—

What stops the collapse?

Same setup, different reward model. A pessimistic ensemble of reward models is far harder to hack — it pushes the achievable peak true-reward much higher before overoptimization sets in.

1-D synthetic RM reproducing the Gao et al. (2022) overoptimization scaling law · policy validated against a closed-form KL-regularized optimum · source & full report →