Goodhart’s Law — when a measure becomes a target, it ceases to be a good measure — is the failure mode that haunts incentive design. This page goes deep on why, and what to do about it.
Four ways a metric goes wrong
“Gaming the metric” is really four distinct failures, and good defenses treat them separately:
Regressional
The metric was only correlated with what you cared about, and optimizing it selects for the correlation rather than the cause. Test scores rise while learning doesn’t.
Extremal
The relationship holds in the normal range but breaks at the extremes — exactly where an optimizer will push. A little of something helps; the maximum is pathological.
Causal
Someone intervenes directly on the metric without touching what it was supposed to reflect. If rankings depend on reported wait times, the fastest fix is to change how wait time is recorded.
Adversarial
Agents actively reshape their behavior to score well, hollowing out the metric on purpose. This is the classic image of gaming — and the least of the four in automated systems, because the others don’t even require bad intent.
Why optimizers make it worse
A human pursuing a flawed metric retains a sense of the actual goal and will eventually notice divergence. An optimizer has no such sense: to the system, the metric is the goal. It will pursue the proxy past every point where a human would have flinched.
A human follows a bad metric until it feels wrong. A machine follows it until the metric is satisfied and the world is ruined — because “feels wrong” was never in the objective.
Defenses that actually help
No metric is un-gameable, so the goal is resilience: measure as close to the real goal as you can afford; use several uncorrelated metrics so disagreement becomes a smoke alarm; cap the extremes; keep a human-legible feedback loop with the authority to intervene; and watch second-order effects, since the damage usually shows up where the metric wasn’t looking. The aim is to make divergence cheap, visible, and reversible rather than expensive, hidden, and locked in.