- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The work mathematically formalizes and analyzes Goodhart's law in a general setting involving, , , , the goal function, proxy function, and discrepancy function respectively. Generalizing on previous works, they do not assume independence between and . They propose a taxonomy of the law, separating it into 4 regimes: none, benign, weak, and strong. Theorems 1 and 2 present two relatively broad settings under which weak and strong Goodhart are impossible while Proposition 3 presents two specific couplings of and and the Goodhart law results. Finally, in the experimental section, they test robustness of their gaussian results as well as cases where the general results do not apply.
Critical strengths
There are two critical strengths of the work: (1) loosening the assumption of independence between goal and proxy, (2) the taxonomy of Goodhart. I elaborate further:
- Loosening assumption. The point made on lines 060-062, that this model does not make the assumption that and are independent and that this original assumption is very unnatural, is extremely compelling. In the supervised learning setting of Goodhart's law (strategic learning/classification) this assumption is not generally made and in a real-world context it is also very obviously strange, so I feel that the authors' contribution to analyze the model without it is very significant. I would recommend, if anything, that they can elaborate on this a bit further, perhaps highlighting specific and easy-to-understand examples where it would obviously not hold or citing references where it does not hold to further highlight the value to the assumption's loosening.
- Taxonomy. Though I take some issue with the presentation of the taxonomy (see example 2 in weaknesses), the taxonomy itself is a very strong contribution, in my opinion. It presents a useful separation of different ways in which Goodhart's law can happen and, I imagine, can open the door to future work on specific parts of the taxonomy.
Critical weaknesses
The primary weakness of this work (and the reason why I have given a weak accept rather than a stronger one) is that though the writing generally makes sense, for a paper whose contribution includes the model/taxonomy itself, there is frequently insufficient narrative explanation and example for various modeling concepts. To give a few concrete examples:
- It's not totally clear to me what conditioning on really means in the narrative of a machine learning algorithm optimizing proxy . Lines 056-057 mention that instead of true optimization authors consider this selection procedure, but it is not clear to me, why this is done or what it means narratively.
- The taxonomy is explained in both formal and qualitative language, but it is still not immediate when/how this might actually happen. Benign Goodhart, as written, is particularly unintuitive. A set of 4 concrete narrative examples would be extremely useful
- Table 2 is extremely useful, but never properly incorporated properly as a reference into the main body text in order to use it for explanation
- Most importantly, the cases described by Thms 1 and 2 are never properly narratively explained. What does it mean in terms of a real ML setting for to be heavy-tailed or to be heavy/light-tailed? To my understanding, these results could provide some useful theoretical backing for intuition about when Goodhart's law might be good/bad in real ML settings, but this is never stated outright.
Questions for authors
My questions are clarifying questions each relating to the concrete examples I provided in the weaknesses section. Provided a good answer to them (and assurance they will be clarified in the final text) I would be open to increasing my weak accept to a strong accept.
- Can the authors please explain more clearly what it means in the narrative of ML policies that they condition on ? Is it a problem that usually we expect a training to maximize rather than just select some model whose is greater than ?
- Can the authors please give concrete (narrative) examples for each of the 4 parts of the Goodhart taxonomies? In particular, I do not quite understand Benign Goodhart: what does it mean that the proxy is decreasing in informativeness and yet somehow the goal can still be maximized?
- Can the authors please explain narratively what Theorems 1 and 2 mean? What does it mean in terms of a real ML setting for to be heavy-tailed or to be heavy/light-tailed and what can we take away (if anything) about how to select an M when training ML systems?
11 comments
Response to Reviewer 6FR9 (2/3)
Q2. Four concrete examples
In each, is what we want and what we measure and select on.
No Goodhart: the measure keeps working. Ranking sellers by recorded sales, when true demand differs by orders of magnitude from one seller to another and the records can be off by a few units. The top sellers in the records are the top sellers, and among them the records still say who is ahead: the error stays small next to the differences among the best. This is what a heavy-tailed goal with a lighter-tailed error means in Theorems 1 and 2. In our RL experiment this is the game without the combo multiplier (): style points are a small bounded bonus, the score-optimal policy is the distance-optimal policy, and among the top 0.01% of episodes by score the ranking by score still agrees with the ranking by distance (Kendall's 0.69).
Benign Goodhart: the bar keeps selecting better candidates, but the measure no longer ranks those who pass. Admission on an entrance score. Raising the bar admits better students on average, so the goal keeps improving. Among the admitted, scores differ by a few points, and these differences say little about who will do best; the differences in ability that the score does not capture now dominate. Nothing was gamed: the measure has been used up as a target. In ML: picking the best of many training runs on a validation score, where the winner is better the more runs are tried, while among the top runs the validation ranking is mostly noise. The Gaussian case of Proposition 3(i) is this regime.
Weak Goodhart: beyond a point, pushing the measure selects luck, not quality. Choosing a trading strategy on its best backtest, or an RL configuration on its best seed, when the evaluation has rare, very large errors. The top of the ranking is filled with the lucky, whose true quality is ordinary. Optimization does not destroy the goal; it stops serving it, and can fall back from an earlier peak. A saturating measure does the same: once all candidates reach the maximum grade, a higher bar selects on noise.
Strong Goodhart: the highest values of the measure are obtained by sacrificing the goal. The boat of the CoastRunners game, rewarded for hitting targets, turns in circles to hit the same targets and never finishes the race (Clark and Amodei, 2016). In our experiment, with a combo multiplier between 1.1 and 2, the score-optimal policy does tricks for the whole episode and covers no distance, although score and distance have correlation above 0.97 under the reference policy.
Response to Reviewer 6FR9 (3/3)
Q3. What Theorems 1 and 2 say, and what to take away when choosing
What the tails are. They describe the population of candidates the optimizer searches, on the scale on which the goal is valued.
- Heavy-tailed goal: a few candidates are worth orders of magnitude more than typical ones. Revenue or audience of items in a recommender, returns of strategies, impact of papers, returns in environments where gains compound.
- Light-tailed goal: accuracy, a rating on a fixed scale, a win rate, the time to finish a task. Most goals in ML are of this kind.
- Light-tailed discrepancy: the errors of the measure are bounded or averaged out. A clipped score, an average over many independent raters, label noise on a large test set.
- Heavy-tailed discrepancy: on rare candidates the measure can be wrong by any amount. Scores with multipliers or bonuses that compound, counts driven by cascades, a learned reward model queried far from its training data, a bug.
The theorems. If exceptional true value exists (heavy-tailed goal) and the errors of the measure are of a smaller order than these exceptional values, in both directions, then an exceptional measured value can only come from exceptional true value. Selecting on the measure is then safe however the errors depend on the goal: they may be adversarially arranged and the conclusion still holds. "In both directions" matters: the measure must not overstate ordinary candidates, and must not miss exceptional ones. A measure that saturates fails the second condition, and we give an example where selection then stalls although the goal is heavy-tailed (reply to MDQy).
Takeaways.
- If what you value is heavy-tailed, use a measure whose errors are lighter-tailed on both sides: do not cap it, and average out its noise. The guarantee then needs no knowledge of how errors and goal depend on each other.
- If what you value is light-tailed (for instance bounded) and the measure is not, which is the common case in ML, no guarantee of this kind exists and the outcome is decided by the dependence between goal and error. The same marginal distributions give no, weak or strong Goodhart depending on the coupling (reply to MDQy). Under independence the worst that can happen is that optimization becomes useless; with dependence it can become harmful.
- An overall correlation between measure and goal says little. In our experiment it is 0.977 to 0.997 in settings where the optimum of the measure has zero true value. What is informative is how the two relate among the top-scoring candidates. This can be estimated on reference samples where both are measured, and it gives a level of the measure up to which pushing it has been checked to help. The new Section 5 tests this, with its limits: such a check cannot see beyond the range of the samples.
- Look at the tail of the measure itself. This needs no true labels. When the measure takes rare, very large values (infinite variance, in our experiment), a high correlation with the goal estimated on samples cannot be trusted, and a guarantee built on it can certify a policy with zero true value (reply to MDQy).
On the independence assumption
Following your suggestion, the introduction now gives two cases where independence plainly fails. A learned reward model is most wrong on outputs far from its training data, which are often outputs of low true quality: the size of the error depends on the goal. And when a bonus and true progress compete for the same budget of time or actions, as in the boat race, a large bonus implies little progress. Strategic classification, which you mention, is a third: the gap between score and true label is produced by agents who respond to the score.
References. Chiang et al., Loss Landscapes Are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent, ICLR 2023. Clark and Amodei, Faulty Reward Functions in the Wild, OpenAI blog, 2016. Mingard et al., Is SGD a Bayesian Sampler? Well, Almost, JMLR 2021.
Response to Reviewer 6FR9 (1/3)
Thank you for the encouraging review and for three precise questions. The revised paper carries a condensed version of each answer in the main text: a paragraph on conditioning in Section 3 (Model), one example per regime next to Table 1, and a short plain reading after Theorems 1 and 2. The full answers below are in a new appendix. Table 2, which compares our results with prior work, is now referenced and discussed in the text of Section 4.
Q1. What does conditioning on M>m mean, and is it a problem that training maximises?
The picture. Consider every model (or policy, or output) the optimizer could return, each weighted by how likely the un-optimized process is to produce it: a random initialisation, or a reference policy before fine-tuning. Training up to level m returns one of the candidates whose metric exceeds m. Which one depends on the algorithm. Conditioning on M>m takes a typical one, and this choice is not arbitrary: among all the ways of guaranteeing that the metric exceeds m, it is the one that departs least from the starting distribution (in KL divergence). It models an optimizer that reaches the level and has no other preference.
Maximizing is the limit, and we study the path. Maximizing M is the limit of m at the top of the range. Real training stops before it: finite compute, early stopping, a penalty towards the initial model. The path matters, because "optimizing helps, then hurts" is a statement about intermediate levels. So it is not a problem; and there is more: any optimizer that favours candidates with a higher metric, in any monotone way, returns an average of selections at different thresholds. This covers Best-of-N sampling and the optimum of KL-regularised RL when transitions are deterministic, the objective used in RLHF. The regimes of our taxonomy, defined on the selection curve m↦E[G∣M>m], carry over to them.
One case where selection is training, exactly. In overparametrised linear regression, gradient descent from a Gaussian initialisation and "draw parameters at random and keep those that fit the data" produce the same distribution over interpolating models. For deep networks this is an empirical finding: drawing random weights until the training loss is low generalises comparably to SGD in small-data settings (Chiang et al., 2023; see also Mingard et al., 2021).
Where the picture is wrong. A trained agent reaches only what it explores, and in a random environment selection also keeps the lucky episodes, which no policy can reproduce. The new RL experiment measures both effects (general response).