Alexander Arvidsson talks metrics:
In 1974, the Soviet Union set production quotas for nails measured in tons.
Factories responded rationally. They produced a small number of enormous, completely useless nails. Problem solved. Quota met.
Moscow reconsidered and switched to quotas measured in number of nails.
Factories responded rationally. They produced millions of tiny, flimsy nails—too small to hold anything together.
The Soviet nail industry was, by any measurable standard, thriving. It was also producing almost nothing of value.
Click through for a detailed description of the problem and Alexander’s push for an answer. I typically summarize Goodhart’s Law in my own words as, any system can be gamed. I think Alexander’s tips on good measurement architecture are sound (and he even mentions gaming systems here), but even with all of those in place, you can still end up with a monkey’s paw of a KPI suite. Even the example of “ensure that users who log in are getting measurable value from the product, as evidenced by X” falls apart in the end because we now maximize for X while hollowing out Y and Z, because nobody’s watching those, as we’re focused on X-maximization. Like, say, “ensure that users who long in are getting measurable value from the product, as evidenced by average order value.” Now we maximize for AOV, streamlining the experience for people making large orders and relentlessly pushing more cart items at the user. The other points aim to mitigate the worst of these results, but it’s an off-the-cuff example of how no system is perfect. But some can still be superior.