acceptodds
Under review as a conference paper at ICLR 2027

The Strong, weak and benign Goodhart's law: An independence-free and paradigm-agnostic formalisation

Abstract

Goodhart's law is a famous adage in policy-making stating that “When a measure becomes a target, it ceases to be a good measure”. As machine learning models and the optimization capacity to train them grow, increasing empirical evidence reinforced the belief in the validity of this law, without however formalizing it. Recently, a few attempts were made to formalize Goodhart's law, either by categorizing variants of it, or by looking at how optimizing a proxy metric affects the optimization of an intended goal. In this work, we relax the simplifying independence assumption, made in previous works, and the assumption on the learning paradigm made in most of them, to study the effect of the coupling between the proxy metric and the intended goal on Goodhart's law. Our results show that in the case where the tail of the goal strictly dominates the tail of the discrepancy, the dependence does not change the nature of Goodhart's effect. However, in the light tailed goal and heavy tailed discrepancy case, we provide an example where over-optimization occurs at a rate inversely proportional to the heavy-tailedness of the discrepancy between the goal and the metric. To the best of our knowledge, this is the first formalization of Goodhart's law that relieved both the assumption on independence and that does not assume knowledge of the learning paradigm.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Goodhart's law, AI alignment, reward misspecification, overoptimization, heavy-tailed distributions, extreme value theory, copulas

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6FR9OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work mathematically formalizes and analyzes Goodhart's law in a general setting involving, GG, MM, ξ\xi, the goal function, proxy function, and discrepancy function respectively. Generalizing on previous works, they do not assume independence between GG and MM. They propose a taxonomy of the law, separating it into 4 regimes: none, benign, weak, and strong. Theorems 1 and 2 present two relatively broad settings under which weak and strong Goodhart are impossible while Proposition 3 presents two specific couplings of GG and ξ\xi and the Goodhart law results. Finally, in the experimental section, they test robustness of their gaussian results as well as cases where the general results do not apply.

Critical strengths

There are two critical strengths of the work: (1) loosening the assumption of independence between goal and proxy, (2) the taxonomy of Goodhart. I elaborate further:

  1. Loosening assumption. The point made on lines 060-062, that this model does not make the assumption that GG and MM are independent and that this original assumption is very unnatural, is extremely compelling. In the supervised learning setting of Goodhart's law (strategic learning/classification) this assumption is not generally made and in a real-world context it is also very obviously strange, so I feel that the authors' contribution to analyze the model without it is very significant. I would recommend, if anything, that they can elaborate on this a bit further, perhaps highlighting specific and easy-to-understand examples where it would obviously not hold or citing references where it does not hold to further highlight the value to the assumption's loosening.
  2. Taxonomy. Though I take some issue with the presentation of the taxonomy (see example 2 in weaknesses), the taxonomy itself is a very strong contribution, in my opinion. It presents a useful separation of different ways in which Goodhart's law can happen and, I imagine, can open the door to future work on specific parts of the taxonomy.
Critical weaknesses

The primary weakness of this work (and the reason why I have given a weak accept rather than a stronger one) is that though the writing generally makes sense, for a paper whose contribution includes the model/taxonomy itself, there is frequently insufficient narrative explanation and example for various modeling concepts. To give a few concrete examples:

  1. It's not totally clear to me what conditioning on M>mM>m really means in the narrative of a machine learning algorithm optimizing proxy MM. Lines 056-057 mention that instead of true optimization authors consider this selection procedure, but it is not clear to me, why this is done or what it means narratively.
  2. The taxonomy is explained in both formal and qualitative language, but it is still not immediate when/how this might actually happen. Benign Goodhart, as written, is particularly unintuitive. A set of 4 concrete narrative examples would be extremely useful
  3. Table 2 is extremely useful, but never properly incorporated properly as a reference into the main body text in order to use it for explanation
  4. Most importantly, the cases described by Thms 1 and 2 are never properly narratively explained. What does it mean in terms of a real ML setting for GG to be heavy-tailed or ξ\xi to be heavy/light-tailed? To my understanding, these results could provide some useful theoretical backing for intuition about when Goodhart's law might be good/bad in real ML settings, but this is never stated outright.
Questions for authors

My questions are clarifying questions each relating to the concrete examples I provided in the weaknesses section. Provided a good answer to them (and assurance they will be clarified in the final text) I would be open to increasing my weak accept to a strong accept.

  1. Can the authors please explain more clearly what it means in the narrative of ML policies that they condition on M>mM>m? Is it a problem that usually we expect a training to maximize MM rather than just select some model whose MM is greater than mm?
  2. Can the authors please give concrete (narrative) examples for each of the 4 parts of the Goodhart taxonomies? In particular, I do not quite understand Benign Goodhart: what does it mean that the proxy is decreasing in informativeness and yet somehow the goal can still be maximized?
  3. Can the authors please explain narratively what Theorems 1 and 2 mean? What does it mean in terms of a real ML setting for GG to be heavy-tailed or ξ\xi to be heavy/light-tailed and what can we take away (if anything) about how to select an M when training ML systems?
11 comments
AuthorsOpenReview ↗

Response to Reviewer 6FR9 (1/3)

Thank you for the encouraging review and for three precise questions. The revised paper carries a condensed version of each answer in the main text: a paragraph on conditioning in Section 3 (Model), one example per regime next to Table 1, and a short plain reading after Theorems 1 and 2. The full answers below are in a new appendix. Table 2, which compares our results with prior work, is now referenced and discussed in the text of Section 4.

Q1. What does conditioning on M>mM>m mean, and is it a problem that training maximises?

The picture. Consider every model (or policy, or output) the optimizer could return, each weighted by how likely the un-optimized process is to produce it: a random initialisation, or a reference policy before fine-tuning. Training up to level mm returns one of the candidates whose metric exceeds mm. Which one depends on the algorithm. Conditioning on M>mM>m takes a typical one, and this choice is not arbitrary: among all the ways of guaranteeing that the metric exceeds mm, it is the one that departs least from the starting distribution (in KL divergence). It models an optimizer that reaches the level and has no other preference.

Maximizing is the limit, and we study the path. Maximizing MM is the limit of mm at the top of the range. Real training stops before it: finite compute, early stopping, a penalty towards the initial model. The path matters, because "optimizing helps, then hurts" is a statement about intermediate levels. So it is not a problem; and there is more: any optimizer that favours candidates with a higher metric, in any monotone way, returns an average of selections at different thresholds. This covers Best-of-NN sampling and the optimum of KL-regularised RL when transitions are deterministic, the objective used in RLHF. The regimes of our taxonomy, defined on the selection curve m↦E[G∣M>m]m\mapsto E[G\mid M>m], carry over to them.

One case where selection is training, exactly. In overparametrised linear regression, gradient descent from a Gaussian initialisation and "draw parameters at random and keep those that fit the data" produce the same distribution over interpolating models. For deep networks this is an empirical finding: drawing random weights until the training loss is low generalises comparably to SGD in small-data settings (Chiang et al., 2023; see also Mingard et al., 2021).

Where the picture is wrong. A trained agent reaches only what it explores, and in a random environment selection also keeps the lucky episodes, which no policy can reproduce. The new RL experiment measures both effects (general response).

AuthorsOpenReview ↗

Response to Reviewer 6FR9 (2/3)

Q2. Four concrete examples

In each, GG is what we want and MM what we measure and select on.

No Goodhart: the measure keeps working. Ranking sellers by recorded sales, when true demand differs by orders of magnitude from one seller to another and the records can be off by a few units. The top sellers in the records are the top sellers, and among them the records still say who is ahead: the error stays small next to the differences among the best. This is what a heavy-tailed goal with a lighter-tailed error means in Theorems 1 and 2. In our RL experiment this is the game without the combo multiplier (ρ=1\rho = 1): style points are a small bounded bonus, the score-optimal policy is the distance-optimal policy, and among the top 0.01% of episodes by score the ranking by score still agrees with the ranking by distance (Kendall's τ\tau 0.69).

Benign Goodhart: the bar keeps selecting better candidates, but the measure no longer ranks those who pass. Admission on an entrance score. Raising the bar admits better students on average, so the goal keeps improving. Among the admitted, scores differ by a few points, and these differences say little about who will do best; the differences in ability that the score does not capture now dominate. Nothing was gamed: the measure has been used up as a target. In ML: picking the best of many training runs on a validation score, where the winner is better the more runs are tried, while among the top runs the validation ranking is mostly noise. The Gaussian case of Proposition 3(i) is this regime.

Weak Goodhart: beyond a point, pushing the measure selects luck, not quality. Choosing a trading strategy on its best backtest, or an RL configuration on its best seed, when the evaluation has rare, very large errors. The top of the ranking is filled with the lucky, whose true quality is ordinary. Optimization does not destroy the goal; it stops serving it, and can fall back from an earlier peak. A saturating measure does the same: once all candidates reach the maximum grade, a higher bar selects on noise.

Strong Goodhart: the highest values of the measure are obtained by sacrificing the goal. The boat of the CoastRunners game, rewarded for hitting targets, turns in circles to hit the same targets and never finishes the race (Clark and Amodei, 2016). In our experiment, with a combo multiplier between 1.1 and 2, the score-optimal policy does tricks for the whole episode and covers no distance, although score and distance have correlation above 0.97 under the reference policy.

AuthorsOpenReview ↗

Response to Reviewer 6FR9 (3/3)

Q3. What Theorems 1 and 2 say, and what to take away when choosing MM

What the tails are. They describe the population of candidates the optimizer searches, on the scale on which the goal is valued.

  • Heavy-tailed goal: a few candidates are worth orders of magnitude more than typical ones. Revenue or audience of items in a recommender, returns of strategies, impact of papers, returns in environments where gains compound.
  • Light-tailed goal: accuracy, a rating on a fixed scale, a win rate, the time to finish a task. Most goals in ML are of this kind.
  • Light-tailed discrepancy: the errors of the measure are bounded or averaged out. A clipped score, an average over many independent raters, label noise on a large test set.
  • Heavy-tailed discrepancy: on rare candidates the measure can be wrong by any amount. Scores with multipliers or bonuses that compound, counts driven by cascades, a learned reward model queried far from its training data, a bug.

The theorems. If exceptional true value exists (heavy-tailed goal) and the errors of the measure are of a smaller order than these exceptional values, in both directions, then an exceptional measured value can only come from exceptional true value. Selecting on the measure is then safe however the errors depend on the goal: they may be adversarially arranged and the conclusion still holds. "In both directions" matters: the measure must not overstate ordinary candidates, and must not miss exceptional ones. A measure that saturates fails the second condition, and we give an example where selection then stalls although the goal is heavy-tailed (reply to MDQy).

Takeaways.

  1. If what you value is heavy-tailed, use a measure whose errors are lighter-tailed on both sides: do not cap it, and average out its noise. The guarantee then needs no knowledge of how errors and goal depend on each other.
  2. If what you value is light-tailed (for instance bounded) and the measure is not, which is the common case in ML, no guarantee of this kind exists and the outcome is decided by the dependence between goal and error. The same marginal distributions give no, weak or strong Goodhart depending on the coupling (reply to MDQy). Under independence the worst that can happen is that optimization becomes useless; with dependence it can become harmful.
  3. An overall correlation between measure and goal says little. In our experiment it is 0.977 to 0.997 in settings where the optimum of the measure has zero true value. What is informative is how the two relate among the top-scoring candidates. This can be estimated on reference samples where both are measured, and it gives a level of the measure up to which pushing it has been checked to help. The new Section 5 tests this, with its limits: such a check cannot see beyond the range of the samples.
  4. Look at the tail of the measure itself. This needs no true labels. When the measure takes rare, very large values (infinite variance, in our experiment), a high correlation with the goal estimated on samples cannot be trusted, and a guarantee built on it can certify a policy with zero true value (reply to MDQy).

On the independence assumption

Following your suggestion, the introduction now gives two cases where independence plainly fails. A learned reward model is most wrong on outputs far from its training data, which are often outputs of low true quality: the size of the error depends on the goal. And when a bonus and true progress compete for the same budget of time or actions, as in the boat race, a large bonus implies little progress. Strategic classification, which you mention, is a third: the gap between score and true label is produced by agents who respond to the score.

References. Chiang et al., Loss Landscapes Are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent, ICLR 2023. Clark and Amodei, Faulty Reward Functions in the Wild, OpenAI blog, 2016. Mingard et al., Is SGD a Bayesian Sampler? Well, Almost, JMLR 2021.

Reviewer_YzHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Goodhart’s law when the true goal and the error in its proxy metric are not assumed to be independent. The authors model optimization as selecting samples whose metric value exceeds an increasingly high threshold. They define different Goodhart regimes and provide theoretical results for several distributional settings. In particular, they show conditions under which severe Goodhart effects cannot occur, analyze a Gaussian setting in which the expected goal improves while conditional correlation vanishes, and construct a dependent setting that produces strong Goodhart behavior. The theoretical results are illustrated using synthetic experiments with different dependence structures.

Critical strengths
  • The paper studies Goodhart’s law without assuming independence between the true goal and the metric error. This is useful because such independence is unlikely to hold in many realistic optimization problems. The theoretical results clarify how dependence and tail behavior can affect the outcome of metric optimization.

  • The paper distinguishes between cases in which optimization makes the true goal worse and cases in which the expected goal improves while the metric becomes less informative. This distinction provides a useful perspective on different forms of Goodhart behavior, and the mathematical results are supported by synthetic experiments.

Critical weaknesses
  • The connection between the theoretical optimization process and actual machine learning is unclear. The paper models optimization as conditioning on increasingly high metric values. Although it provides a neural network example, it does not explain when this selection process is a reasonable representation of model training. This limits the conclusions that can be drawn for real machine learning systems.

  • The interpretation and practical relevance of the results are not sufficiently established. In the Gaussian case, conditional correlation decreases partly because conditioning on high metric values restricts the range of the metric. It is unclear why this mechanical decrease should be interpreted as the metric losing useful information. In addition, the strong Goodhart result depends on a particular dependence structure, and all experiments use synthetic data. The paper therefore does not show how commonly these behaviors occur in realistic applications, despite connecting its conclusions to AI safety and alignment.

Questions for authors
  • Under what assumptions does conditioning on M>mM>m provide a reasonable representation of optimization through machine learning training? Can the authors give a more explicit connection between this abstraction and an actual training process?

  • In the Gaussian setting, how can the paper distinguish a meaningful loss of metric informativeness from a decrease in correlation caused only by range restriction? Would the conclusion remain the same under a more decision-relevant measure of metric usefulness?

  • Is there evidence that the dependence structure used in the strong Goodhart example occurs in realistic machine learning or AI alignment problems? Providing such an example would materially affect my assessment of the practical importance of the result.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YzHJ (1/2)

Thank you for these three questions. The general response describes the new RL experiment (Combo Race), which replaces the synthetic Section 5.

Q1. When does conditioning on M>mM>m represent training?

Take the population of candidates an optimizer could return (parameters, policies, outputs), weighted by how likely the unoptimized process is to produce them: random initialisation, a reference policy. We call this the reference law.

Exact cases.

  1. Selection procedures. Best-of-NN sampling, reranking, random search and model selection on a validation metric choose among draws of the reference law according to the metric. Best-of-NN returns an average of the selection curve m↦E[G∣M>m]m\mapsto E[G\mid M>m] over thresholds.
  2. KL-regularised RL with deterministic transitions (token generation is of this kind). The policy that maximises Eπ[M]−β KL(π ∥ πref)E_\pi[M]-\beta\,\mathrm{KL}(\pi\,\Vert\,\pi_{\rm ref}), the objective used in RLHF, induces on trajectories the exponential tilt of the reference law by the metric. Any reweighting by a nondecreasing function of the metric is an average of conditionings, so the true return of this policy is again an average of the selection curve, and the limits that define our regimes carry over as β→0\beta\to0. Mirror ascent on whole trajectories (natural policy gradient for a softmax policy) visits exactly these tilts: training time is an inverse temperature.
  3. Overparametrised least squares. Gradient descent on the training loss from θ0∼N(0,σ02I)\theta_0\sim\mathcal N(0,\sigma_0^2 I) converges to the minimal-norm interpolator plus the projection of θ0\theta_0 on the kernel of the design matrix. The Gaussian prior conditioned on the training loss being below mm converges, as m→0m\to0, to the same distribution. Selection reproduces the outcome of training exactly, and the implicit bias towards the minimal norm comes out of the conditioning. Only the endpoints coincide; along the way the closest family is the ridge path, which is the soft version of the conditioning.

In all three, conditioning is the least-committal choice: among the distributions under which the metric exceeds mm, the conditional law is the closest to the reference in KL divergence.

Empirical support beyond these cases. For deep networks, Mingard et al. (2021) find that the functions reached by SGD are distributed close to the prior over functions conditioned on fitting the training set, and Chiang et al. (2023) find that "guess and check", drawing random weights until the training loss is low, which is exactly our selection, generalises comparably to SGD in small-data settings.

Where it fails.

  • Random transitions. Conditioning also selects episodes on which the environment was lucky, which no policy can do; the selection curve is then optimistic.
  • Exploration. A trained agent reaches only what it explores. In Combo Race with ρ=1.1\rho=1.1, the KL-regularised optimum uses the exploit for small penalties, but PPO started at πref\pi_{\rm ref} never finds it, even in 6000 iterations. The selection model describes what a strong enough optimizer reaches, not what a given run finds.
  • The conclusions are relative to the reference law, which has to be specified.

A more in-depth analysis of the relationship between conditioning and optimization is a direction for future research, which we consider outside the scope of this paper. The revised model section does, however, include a paragraph on how representative the conditioning model is, to give readers more context.

AuthorsOpenReview ↗

Response to Reviewer YzHJ (2/2)

Q2. Range restriction in the Gaussian case

The decrease in correlation in the Gaussian case does come from range restriction. We do not think this makes it uninteresting, since it illustrates a behaviour that matters in our setting: whether it is "the metric losing useful information" depends on the decision the metric is used for, and the two natural decisions give opposite answers:

  • Keep optimizing? The relevant quantity is the marginal return, the slope of E[G∣M>m]E[G\mid M>m], which tends to Cov(G,M)/Var(M)>0\mathrm{Cov}(G,M)/\mathrm{Var}(M)>0 (0.83 in the example below) and does not decay. Under this measure nothing is lost. That is why the regime is called benign: the goal keeps improving at an undiminished rate.
  • Rank the candidates that passed? The relevant quantity is the probability that, of two selected candidates, the one with the larger metric has the larger goal. This is (1+τ)/2(1+\tau)/2, with τ\tau Kendall's tau among the selected. It tends to 1/21/2: with variances 1 and 0.2 and independence, it is 0.70, 0.64, 0.60 and 0.58 at thresholds 2, 4, 6 and 8 (Pearson correlation 0.62, 0.46, 0.35, 0.28). Among the selected, the ranking by the metric tends to a coin flip. This is the classical reading of "ceases to be a good measure": among admitted students, the entrance score hardly predicts results, although raising the bar did select better students.

So the conclusion holds under a decision-relevant measure of ranking, and does not hold under a measure of marginal return; the taxonomy records exactly this pair, and distinguishes it from weak and strong Goodhart, where the regression of the goal on the metric itself flattens or turns down in the tail.

Q3. Does the dependence structure of the strong Goodhart example occur?

In words, the structure of Proposition 3(ii) is: goal and metric agree over typical candidates (correlation 0.88 for b=5b=5 and 0.99 for b=10b=10; for b≤3b\le3 the discrepancy has infinite variance and the correlation is not defined), and, given the goal, the discrepancy is light-tailed with a scale that grows without bound as the goal decreases. The proxy can be most wrong on the worst candidates, so the largest metric values come from the smallest goals.

In our controlled experiment, the proxy return (game score) and the true return (distance) of Combo Race have correlation 0.977 to 0.997 under the reference policy for multipliers from 1.1 to 2, and yet the score-optimal policy covers no distance. Two ordinary ingredients suffice: a step budget shared between progress and an exploitable bonus, and a bonus that compounds. The dependence is measurable on reference episodes: for multiplier 2, of the top 0.1 % of episodes by score, 24 % are in the top 0.1 % by distance (86 % without the multiplier), and Kendall's tau among them is −0.13.

In documented cases the same shape is reported qualitatively: the highest scores are obtained by behaviours of little or no task value. The CoastRunners boat circles to hit targets and never finishes the race (Clark and Amodei, 2016); an evolution strategy finds a scoring bug in Q*bert (Chrabąszcz et al., 2018); Krakovna et al. (2020) list about 60 such cases, and Dharna et al. (2026) collect 26 first-hand accounts.

In an RLHF-like setting, Gao et al. (2023) optimise a learned reward model and measure a separate "gold-standard" reward model: the gold score first rises, then falls as optimization continues, under best-of-nn sampling (selection on the proxy) and under RL. The coupling of Proposition 3(ii) produces the same hump: for b=5b=5 the expected goal rises from 1 to 6.3 when the top 9×10−49\times10^{-4} is kept, then collapses.

What we do not have is a measurement of this dependence on a production reward model, and we do not claim that the exponential–Pareto coupling describes one. The quantities to measure are the ones above (the selection curve and the agreement of goal and metric among top-scoring samples), on reference samples with gold labels.

One statement of the submission on this example is withdrawn ("the lighter the tail of the discrepancy, the faster the collapse"): see point 4 of the general response.

References. Chiang et al., Loss Landscapes Are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent, ICLR 2023. Chrabąszcz et al., Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari, IJCAI 2018. Clark and Amodei, Faulty Reward Functions in the Wild, OpenAI blog, 2016. Dharna et al., AI Finds A Way, arXiv:2608.23875, 2026. Gao et al., Scaling Laws for Reward Model Overoptimization, ICML 2023. Krakovna et al., Specification Gaming: The Flip Side of AI Ingenuity, DeepMind blog, 2020. Mingard et al., Is SGD a Bayesian Sampler? Well, Almost, JMLR 2021.

AuthorsOpenReview ↗

Call to discussion

Dear reviewer,

We thank you again for your review, we addressed your questions in our revisions of the paper and in our previous comment.

Following the discussion with Reviewer 6FR9, we published a second revision incorporating their remaining concern. Changes in this second revision are shown in orange; those of the first revision remain in blue.

If any of your concerns have not been addressed by our previous comment, we kindly ask you to engage in the discussion so that we can address any potential concern in a further revision of the paper.

If your concerns have been resolved by our first comment, we kindly ask you to reflect that by raising your score and thank you very much for the effort put in this process.

AuthorsOpenReview ↗

Call to discussion (2)

Dear reviewer, As the rebuttal period is coming to a close, we would like to ask whether any concerns remain that our previous response to your review did not address, so that we can address them in the time remaining. For convenience, changes made in response to the reviews are highlighted in blue (first revision) and orange (second revision). We thank you again for your time and feedback.

Reviewer_MDQyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Goodhart’s law through the model M=G+ξM=G+\xi, treating optimization as conditioning on increasingly large proxy values. It extends previous formalizations by allowing dependence between the goal and discrepancy, and distinguishes four outcomes: no, benign, weak, and strong Goodhart. The paper provides sufficient conditions excluding weak and strong Goodhart when the goal’s tail dominates the discrepancy’s, analyzes Gaussian and exponential–Pareto couplings, and presents synthetic experiments with different copulas.

Critical strengths

The paper provides a useful extension of existing mathematical formalizations by relaxing independence between the goal and discrepancy. The distinction between successful optimization with vanishing correlation and genuine deterioration of the goal is valuable, and the four-outcome taxonomy helps organize these behaviors. In particular, the dependence-agnostic guarantees and the example in which dependence changes weak Goodhart into strong Goodhart offer complementary theoretical insights.

Critical weaknesses
  1. Although reward hacking in RL motivates the paper, the experiments only sample prescribed marginal distributions and copulas. They do not examine an actual RL or RLHF system, leaving it unclear whether the assumed tail structures and couplings describe realistic reward-model errors. Conditioning a fixed joint distribution on M>mM>m also abstracts away policy updates and changes in state visitation. Furthermore, the results do not yield an operational diagnostic, stopping criterion, or mitigation procedure. A theoretical contribution need not introduce a new algorithm, but the practical motivation would be better supported by a testable implication or intervention. Relevant prior work already develops theoretically motivated early stopping and robust optimization [2], as well as occupancy-measure regularization with empirical validation [3]. An RL experiment showing how the proposed framework predicts reward hacking or informs these approaches would strengthen the paper.

  2. The general theorems require regular variation, strict tail dominance, and additional moment and left-tail conditions, while the explicit outcomes are demonstrated through particular couplings. Theorems 1 and 2 exclude weak and strong Goodhart but do not distinguish no from benign Goodhart. Although extending tail-dominance results to arbitrary dependence is meaningful, the paper could more clearly explain what is learned beyond the tail-based analysis in prior work [1] and the intuition that extreme proxy values are driven by the dominant tail. Boundary cases, sensitivity to the chosen coupling, and examples clarifying which assumptions are essential would help establish the scope and significance of the results.

[1] El-Mhamdi and Hoang. On Goodhart’s Law, with an Application to Value Alignment, 2024.

[2] Karwowski et al. Goodhart’s Law in Reinforcement Learning, ICLR 2024.

[3] Laidlaw et al. Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking, ICLR 2025.

Questions for authors
  1. Can the authors add a controlled RL experiment with separately measurable proxy and true rewards, and test whether estimated tail and dependence properties predict the onset of reward hacking or the point at which further optimization becomes harmful?

  2. Section 5.1 specifies a Pareto discrepancy with survival probability proportional to x−2x^{-2}, which has infinite second moment. With the positive Pareto marginals used in Figure 2, M=G+ξM=G+\xi also has infinite conditional second moment above any finite threshold. Consequently, the population Pearson correlation Corr⁡(G,M∣M>m)\operatorname{Corr}(G,M\mid M>m) is not well-defined in the usual sense. How should the plotted sample correlations be interpreted, given the finite-variance requirement acknowledged in footnote 2? Was truncation used in the experiments, and would the conclusions persist with finite-variance Pareto marginals?

7 comments
AuthorsOpenReview ↗

Response to Reviewer MDQy (1/4)

Thank you for this review. Both of your questions pointed at real gaps: the paper had no RL experiment, and the correlation panels of Figure 2 were not well defined. The general response describes the new experiment (Combo Race); this reply gives what it answers for your points, block by block. GG is the true return, MM the proxy return, πref\pi_{\rm ref} the reference policy, ρ\rho the combo multiplier that sets the tail of the discrepancy.

Q1 (a). Do estimated tail and dependence properties predict the onset of hacking?

Under selection, yes, within the range the reference samples cover. Table 1 of the general response gives the selection curve estimated on 10710^7 episodes of πref\pi_{\rm ref}, and the fraction qq at which it peaks. Table 3 gives, on the same episodes, the dependence of GG and MM among the episodes with the largest proxy return.

Table 3. Dependence among the top-scoring reference episodes (deterministic variant, 10710^7 episodes of πref\pi_{\rm ref}). "Top qq" is the fraction qq of episodes with the largest proxy return. Kendall's τ\tau among them is the probability that, of two such episodes, the one with the larger proxy return also has the larger true return, minus the probability of the reverse; it needs no moments. The overlap is the share of the top 10−310^{-3} by proxy return that is also in the top 10−310^{-3} by true return. The last column is the NN for which Best-of-NN sampling (keep, among NN episodes, the one with the largest proxy return) gives the largest expected true return, among N=2kN=2^k, k≤22k\le 22. The overall correlation is exact up to ρ=2\rho=2; beyond the finite-variance threshold ρ=5\rho=\sqrt5 it is not a usable quantity (general response, point 1).

ρ\rhoCorr$(G,M)$, all episodesKendall's τ\tau among the top 10−210^{-2}top 10−310^{-3}top 10−410^{-4}overlap of the top 10−310^{-3} by MM and by GGbest NN for Best-of-NN
10.9980.750.700.6986 %none*
1.10.9970.740.670.6285 %none*
1.20.9970.710.610.5683 %none*
1.50.9950.560.330.0170 %2152^{15}
20.9770.14−0.13−0.0624 %282^{8}
2.5not usable (estimate: 0.80)−0.14−0.060.436 %262^{6}
3not usable (estimate: 0.30)−0.05−0.120.386 %252^{5}

none*: the goal is still rising after 2222^{22} draws.

The overall correlation does not separate ρ=1\rho=1 from ρ=2\rho=2; the dependence among the top-scoring episodes does. The estimated selection curve and Best-of-NN give the point beyond which more selection lowers the true return.

For ρ≥2.5\rho\ge2.5, τ\tau is positive again among the top 10−410^{-4} (standard error 0.02), while the selection curve keeps falling (Table 1). These 1000 episodes almost all contain a trick run of the same length (8 tricks for about 80 % of them). Among episodes with the same run length, an earlier run raises both the score and the distance (τ≈0.65\tau\approx0.65). The negative dependence comes from comparing run lengths: a longer run raises the score and lowers the distance. A rank correlation inside a thin slice can therefore hide a reversal that the selection curve shows.

What is not predicted. For ρ=1.1\rho=1.1 and 1.21.2 nothing is visible down to the top 10−510^{-5}, although the proxy-optimal policy has zero true return: the exploit (a run of 60 consecutive tricks) has probability 0.2600.2^{60} under πref\pi_{\rm ref}. Reference data describe the region they sample. This is the "change in state visitation" you point to, and it bounds what any diagnostic computed on πref\pi_{\rm ref} can do. Measuring the true reward on a few episodes of the policy being trained, as Gao et al. (2023) do with a gold reward model, extends the range the data cover beyond πref\pi_{\rm ref}. The number of reference episodes needed to see a reversal is also large: a reversal (the estimated selection curve falls significantly below its maximum, with a margin of 3 standard errors on each side) was detected in 5 of 5 samples of 10510^5 episodes for ρ≥2.5\rho\ge 2.5, and of 10610^6 for ρ=2\rho=2; never for ρ≤1.5\rho\le1.5 at these sizes.

AuthorsOpenReview ↗

Response to Reviewer MDQy (2/4)

Q1 (b). The point at which further optimization becomes harmful

Three results, from the most to the least usable in practice. The first two need only proxy values of sampled episodes, which every RL pipeline records; the third needs the true reward on reference episodes.

(i) Check the tail of the proxy before trusting a correlation-based certificate. The bound of Laidlaw et al. (2025; your [3]) needs the proxy reward to have finite variance under the reference occupancy. Here this is 0.2 ρ2<10.2\,\rho^2<1 in the idealisation without horizon, i.e. ρ<2.24\rho<2.24. Beyond, the correlation that the bound uses exists only because episodes are finite: it tends to 0 as the horizon grows, and its value is set by trick runs that no sample contains. At 60 steps its exact value falls from 0.89 at ρ=2.2\rho=2.2, below the threshold, to 0.31 at 2.352.35 and 0.02 at 2.52.5, while its estimate on 10510^5 reference episodes stays at 0.95, 0.92 and 0.85. With exact parameters the bound holds, as their theorem states, and selects πref\pi_{\rm ref}. With estimated parameters it certifies a gain of order 102310^{23} at ρ=2.5\rho=2.5 and selects a policy with zero true return in the deterministic variant (5 of 5 samples). The finite-variance threshold of the proxy's tail, which can be checked without the true reward, says when the correlation estimate can be trusted.

(ii) Monitor the tail of the proxy return during training. If the true return is bounded by gmax⁡g_{\max}, then P(ξ>y−gmax⁡)≥P(M>y)P(\xi>y-g_{\max})\ge P(M>y) for every yy: a heavy tail of the proxy return under the current policy forces a heavy tail of the discrepancy, and it is observable without the true reward. The alert fires when the Hill index of the proxy return, on the top 5 % of 4096 episodes of the current policy, exceeds 0.2 (it is 0.03 to 0.06 under πref\pi_{\rm ref}; the threshold was set by hand).

  • PPO. The alert comes at or before 9 of the 15 collapses of the deterministic variant (0 or 10 iterations ahead; evaluations every 10 iterations) and all 16 of the random variant (median lead 20 iterations, up to 80), with 3 false alerts in the 309 runs that do not collapse.
  • Exact optimization paths. With random transitions it fires early, while the true return still improves: on the exact natural policy gradient at iterations 2 to 7, for a collapse at iterations 52 to 62. It also fires at ρ=1.1\rho=1.1, where the true return does not collapse. In the deterministic variant it fires one iteration after the peak of the true return, 2 to 4 iterations before the collapse, and never along the KL-regularised path: a hacked deterministic policy has no tail left.

It tells when the policy has entered a heavy-tailed regime, not how much true return will be lost.

(iii) What reference data with the true reward can certify. The selection curve of Q1 (a), estimated on nn labelled reference episodes, is available only down to the top fraction 100/n100/n (at least 100 episodes kept). Keeping the top fraction qq costs log⁡(1/q)\log(1/q) nats of KL divergence, so nn labelled episodes certify a selection pressure of about log⁡(n/100)\log(n/100) nats and say nothing beyond. We tested a stopping rule built on this certified zone: stop before the mean proxy return of the current policy exceeds the proxy level at which the estimated selection curve peaks, or, if no peak is seen, the highest level the data cover. It never selected a policy below πref\pi_{\rm ref} (480 cases on 24 exact paths) and stopped before all 31 PPO collapses. We do not propose it as a stopping rule for RL, for two main reasons.

  1. Its budget is small. With 10510^5 labelled episodes it stops at 4.8 nats in the benign case ρ=1\rho=1 (true return 151.9), where the best policy is at 34 nats (187.7).
  2. It cannot see exploits that πref\pi_{\rm ref} never samples. For ρ=1.1\rho=1.1 and 1.2 it was safe only because the exploit lies beyond the certified range, where benign progress lies too.

In addition, a plain KL budget of the same size also stops before all 31 PPO collapses; the rule keeps more of the gain only because the exploit of our toy inflates the proxy by orders of magnitude, and with random transitions it gets none of the available gain. The figures are in the appendix of the revision.

The rule applies as is to Best-of-NN sampling and reranking, where the selection model is exact. For RL training it states a limit of any diagnostic computed on πref\pi_{\rm ref}: the harmful point can be located from reference data only within about log⁡(n/100)\log(n/100) nats of πref\pi_{\rm ref}. Beyond, what remains is label-free signals such as (ii), or true-reward labels on episodes of the policy being trained.

AuthorsOpenReview ↗

Response to Reviewer MDQy (3/4)

W1. How the framework informs Karwowski et al. (2024), Laidlaw et al. (2025) and the KL penalty

  • Laidlaw et al. (2025), correlation and occupancy divergence: see Q1 (b)(i).
  • Karwowski et al. (2024; your [2]), angle-based early stopping. Given the true angle between projected rewards, it is safe in every case and stops at πref\pi_{\rm ref} as soon as ρ>1\rho>1: the angle exceeds 90° because the Euclidean projection is dominated by rarely visited high-combo states.
  • KL penalty. The coefficient at which the regularised optimum switches to the exploit is predicted by comparing the proxy return of the extreme trajectory (denoted ω\omega instead of the usual τ\tau to avoid confusion with the previously introduced Kendal's τ\tau in this response) with its KL cost, M(ω)/(Hlog⁡5)M(\omega)/(H\log 5): within 25 % in 19 of the 22 (multiplier, horizon) pairs where a switch exists, across 18 orders of magnitude. A fixed coefficient is overcome as the horizon grows whenever 0.99 ρ>10.99\,\rho>1, in line with Kwa et al. (2024).

W1. Conditioning abstracts away policy updates and state visitation

A policy changes state visitation by changing the probability of trajectories. Our fixed joint law is the law of trajectories under πref\pi_{\rm ref}, and policies are reweightings of it. With deterministic transitions, the conditional law given M>mM>m is itself the law of a policy (a Doob transform of πref\pi_{\rm ref}), and the optimum of Eπ[M]−β KL(π ∥ πref)E_\pi[M]-\beta\,\mathrm{KL}(\pi\,\Vert\,\pi_{\rm ref}) is the exponential tilt of the reference law, whose true return is an average of the selection curve over thresholds. Mirror ascent on whole trajectories visits exactly these tilts. Two things are abstracted away, and the experiment measures both: with random transitions, conditioning also selects lucky episodes (the selection curve is optimistic); and an agent reaches only what it explores. PPO from πref\pi_{\rm ref} never finds the exploit at ρ=1.1\rho=1.1, even in 6000 iterations, although the regularised optimum uses it. The selection model describes what a strong enough optimizer reaches, not what a given run finds.

W1. Do the assumed tails and couplings describe realistic reward errors?

We do not have measurements on a production reward model and do not claim them. What the experiment adds is a mechanism. In Proposition 3(ii) the discrepancy given the goal is light-tailed, with a scale that grows without bound as the goal decreases: the proxy can be most wrong on the worst candidates. Combo Race produces this with two ordinary ingredients, a budget shared between progress and an exploitable bonus, and a bonus that compounds. Documented cases of reward hacking (Amodei et al., 2016) have this shape: the boat that circles to hit targets and never finishes (Clark and Amodei, 2016), the Q*bert agent that exploits a scoring bug (Chrabąszcz et al., 2018), the first-hand accounts collected by Dharna et al. (2026). In an RLHF-like setting, Gao et al. (2023) measure a "gold" score that first rises and then falls as a learned reward model is optimized further, under best-of-nn sampling, which is selection on the proxy, and under RL.

W2. What is learned beyond El-Mhamdi and Hoang (2024) and the dominant-tail intuition

  • Under independence, selection cannot hurt: E[G∣M>m]≥E[G]E[G\mid M>m]\ge E[G] for every mm (two lines: nondecreasing functions of the same variable are positively correlated). The independent model can make optimization useless, never harmful. Harm needs dependence. El-Mhamdi and Hoang (2024; your [1]) contain one specific dependent construction, where the expected goal tends to −∞-\infty; it plays the role of our exponential–Pareto example, without parameters. All their general results assume independence, and their concluding section lists "diagnosing strong Goodhart's law" from samples, including dependence, as open.
  • The dominant-tail intuition says which variable is large, not what the goal is. With the same exponential goal and Pareto discrepancy, three couplings give three regimes: comonotone (ξ\xi an increasing function of GG), the goal grows without bound and the proxy ranks perfectly, no Goodhart; independent, the goal returns to its mean, weak; the coupling of Proposition 3(ii), the goal tends to 0, strong. For b=5b=5, E[G∣M>50]E[G\mid M>50] is 15.3, 1.09 and 1.6×10−71.6\times10^{-7}.
  • When the goal's tail dominates, it is not obvious under dependence that large proxy values come from large goals (the discrepancy could cancel large goals). Theorems 1 and 2 show that it holds for every coupling, provided the goal's tail also dominates the left tail of the discrepancy; the next block shows that this condition cannot be dropped.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General response to all reviewers (1/4)

We thank the three reviewers for their careful reading. Three requests recur: a controlled RL experiment with separately measured proxy and true rewards, and an operational implication (MDQy); an explicit link between conditioning on M>mM>m and training (all three); concrete examples, a plain reading of the theorems and evidence that the regimes occur (6FR9, YzHJ). We answer them here and in the individual replies. We also correct one statement of the submission (point 4).

1. A controlled RL experiment replaces Section 5

Combo Race is a tabular MDP modelled on the boat-race example of Clark and Amodei (2016). For 60 steps the agent either advances (true reward: its current speed) or does a trick (style points bρjb\rho^{j} at combo level jj; the combo resets when the agent advances; successful tricks raise the speed for the rest of the episode, with diminishing returns). The goal GG is the discounted distance. The metric MM is the game score: GG plus the discounted style points, so ξ=M−G≥0\xi=M-G\ge 0. The reference policy πref\pi_{\rm ref} does a trick with probability 0.2. In the deterministic variant every action succeeds; in the random variant an advance succeeds with probability 0.6 and a trick with probability 0.9.

What the experiment shows.

  • Strong Goodhart behind a proxy that looks excellent. For 1<ρ≤21<\rho\le 2 the correlation of proxy and true return under πref\pi_{\rm ref} is at least 0.977 (Table 1), and the policy that maximises the proxy has zero true return (Table 2). The structure of Proposition 3(ii), agreement in the bulk with the largest discrepancies reserved for the lowest goals, comes here from a mechanism (a step budget shared between progress and an exploit with compounding payoff), not from a hand-made copula.
  • The selection curve estimated under πref\pi_{\rm ref} locates the onset of hacking under selection, within the range the samples cover (last column of Table 1). Best-of-NN sampling is best at N=215N=2^{15}, 282^{8}, 262^{6} and 252^{5} for ρ=1.5\rho=1.5, 2, 2.5 and 3. For ρ≤1.2\rho\le1.2 nothing is visible down to the top 10−510^{-5}: from reference data, the harmful point can be located only within about log⁡(n/100)\log(n/100) nats of πref\pi_{\rm ref}.
  • The finite-variance threshold predicts where a correlation-based criterion breaks. Beyond α=1/2\alpha=1/2 (ρ≈2.24\rho\approx2.24) the correlation is no longer a usable quantity, although its estimate on reference episodes stays high (Table 1), and the bound of Laidlaw et al. (2025), with its parameters estimated on reference episodes, certifies policies with zero true return (reply to MDQy).
  • The tail of the proxy return flags collapses without the true reward. It precedes most PPO collapses, with a blind spot for deterministic hacks (below).

Limits, stated in the revision: the environment is a toy built to have compounding style gains; for ρ≤1.2\rho\le 1.2 nothing is visible under πref\pi_{\rm ref} although the optimum of the proxy destroys the goal; PPO started at πref\pi_{\rm ref} collapses only under a weak but nonzero KL penalty (β=1\beta=1 or 33 in the deterministic variant).

The rest of this point gives the details: the tail of the discrepancy, what is computed, the two tables, and the diagnostics.

Tail of the discrepancy. We use the convention of the paper (Definition 2): a variable has tail index α\alpha when its survival function is l(y) y−1/αl(y)\,y^{-1/\alpha} with ll slowly varying, so a larger α\alpha means a heavier tail, and the variance is finite if and only if α<1/2\alpha<1/2. Under πref\pi_{\rm ref} the combo level is geometric, and in the deterministic variant the style reward RR of one step satisfies c1 y−1/α≤P(R>y)≤c2 y−1/αc_1\,y^{-1/\alpha}\le P(R>y)\le c_2\,y^{-1/\alpha} for two constants and

α=log⁡ρlog⁡5.\alpha=\frac{\log\rho}{\log 5}.

For ρ=1\rho=1 style points are bounded: the tail is light. The variance becomes infinite at ρ=5≈2.24\rho=\sqrt5\approx 2.24. Two caveats. An episode has 60 steps, so the episode-level discrepancy ξ\xi is bounded, and the power law only holds over the range of values an episode can reach. And the lattice bρjb\rho^{j} makes the survival function oscillate between the two bounds, so it is not exactly regularly varying. Table 1 therefore also gives the Hill estimate of the tail index of ξ\xi itself, on the largest 1 % of 10710^7 reference episodes. It increases with ρ\rho like the one-step index but is lower from ρ=1.2\rho=1.2 on, and, like any Hill estimate on finite data, it is not zero at ρ=1\rho=1 (0.06).

AuthorsOpenReview ↗

# General response to all reviewers (2/4)

1. A controlled RL experiment replaces Section 5 (continued)

Correlation beyond the finite-variance threshold. Because an episode has 60 steps, every reward is bounded, and the correlation of GG and MM exists for every ρ\rho. For ρ>5\rho>\sqrt5 it is nevertheless not a usable quantity. It tends to 0 as the horizon grows: at ρ=2.5\rho=2.5 it is 0.47, 0.13, 0.02 and 0.004 for episodes of 20, 40, 60 and 80 steps. Its value is set by long trick runs that no sample contains: a run of 15 tricks has probability about 10−910^{-9} per episode, and if combos stopped growing after 10 tricks the correlation would be 0.82, close to the 0.80 estimated on 10710^7 episodes. Table 1 therefore gives only the estimate for ρ=2.5\rho=2.5 and 3.

What is computed. Everything about the reference policy is either exact (dynamic programming) or estimated on 10710^7 sampled episodes. The KL-regularised optimum πβ\pi^\beta, which maximises Eπ[M]−β KL(π ∥ πref)E_\pi[M]-\beta\,\mathrm{KL}(\pi\,\Vert\,\pi_{\rm ref}), is computed exactly for 501 values of β\beta, from πref\pi_{\rm ref} (β=∞\beta=\infty) to the policy that maximises E[M]E[M] (β→0\beta\to 0); this is the objective of RLHF, and with deterministic transitions πβ\pi^\beta induces the exponential tilt of the reference law (point 2). PPO with the same KL penalty is trained from πref\pi_{\rm ref} for 2000 iterations of 64 episodes, with 5 seeds and 8 to 10 values of β\beta per ρ\rho (340 runs), and its policy is evaluated exactly every 10 iterations. The tables give the deterministic variant; the random variant is summarised after them.

Table 1. What the reference policy shows (deterministic variant, Eref[G]=116.3E_{\rm ref}[G]=116.3). Column 2 and the correlations up to ρ=2\rho=2 are exact; the other entries use 10710^7 episodes of πref\pi_{\rm ref}. "Top qq" is the fraction qq of episodes with the largest proxy return, so Eref[G∣top q]E_{\rm ref}[G\mid \text{top } q] is the selection curve of the paper, E[G∣M>m]E[G\mid M>m], at the threshold mm exceeded with probability qq.

ρ\rhotail index α\alpha of the step rewardHill estimate of the tail index of ξ\xiCorr$(G,M)$Eref[G∣top 10−2]E_{\rm ref}[G\mid\text{top }10^{-2}]top 10−310^{-3}top 10−410^{-4}qq at which the estimate of Eref[G∣top q]E_{\rm ref}[G\mid\text{top }q] peaks
10 (light tail)0.060.998148.0154.7159.9none: still rising at 10−510^{-5}
1.10.060.060.997148.0154.6159.9none
1.20.110.070.997148.0154.6159.8none
1.50.250.140.995147.7153.6156.310−410^{-4}
20.430.290.977145.2142.2134.95.6×10−35.6\times10^{-3}
2.50.57 (infinite variance)0.41not usable (estimate: 0.80)138.4130.7121.73.2×10−23.2\times10^{-2}
30.68 (infinite variance)0.50not usable (estimate: 0.30)131.5130.0121.65.6×10−25.6\times10^{-2}

Table 2. What optimizers do (deterministic variant, Eref[G]=116.3E_{\rm ref}[G]=116.3). Columns 2 and 3 are exact; columns 4 to 6 are PPO runs. A run collapses when its E[G]E[G] falls below 90 % of Eref[G]E_{\rm ref}[G] after its maximum. The tail alert is defined below. "Best → final" is the mean over 5 seeds of the best E[G]E[G] during training and of E[G]E[G] after 2000 iterations.

ρ\rhoE[G]E[G] of the policy maximising E[M]E[M]best E[G]E[G] on the KL-regularised path (its KL to πref\pi_{\rm ref})PPO, all β\beta: runs that collapsePPO: collapses flagged by the tail alert at or before the collapse (false alerts)PPO, β=1\beta=1: best → final E[G]E[G]
1187.7187.7 (34.2 nats)0 of 40no collapse (0 false alerts)180.1 → 178.6
1.10133.6 (0.96 nats)0 of 50no collapse (0)183.6 → 182.6
1.20116.6 (6×10−46\times10^{-4} nats)5 of 402 of 5 (0)185.2 → 118.5
1.50116.3 (πref\pi_{\rm ref})10 of 407 of 10 (1)177.6 → 4.4

PPO was not run for ρ≥2\rho\ge2; there the exact columns are as for ρ=1.5\rho=1.5 (true return 0 at the maximum of E[M]E[M], and πref\pi_{\rm ref} is the best policy on the KL-regularised path). For ρ≥1.5\rho\ge1.5 the true return decreases as soon as the exact path leaves πref\pi_{\rm ref}: the tilt is driven at once by trick runs that πref\pi_{\rm ref} almost never samples. PPO, which learns from sampled episodes, first exploits the speed boost and climbs far above πref\pi_{\rm ref}. With β=1\beta=1 or 33 it then finds the trick runs and collapses towards the exact optimum, whose true return is 0. Without penalty, or with β≥10\beta\ge10, it never finds them and stays benign (132.8 at ρ=1.5\rho=1.5, β=10\beta=10), although the exact optimum for the same β\beta has true return 0.

AuthorsOpenReview ↗

General response to all reviewers (3/4)

1. A controlled RL experiment replaces Section 5 (continued)

Tail alert, without the true reward. When the true return is bounded, a heavy tail of the proxy return under the current policy forces a heavy tail of the discrepancy, so the tail of MM can be monitored without measuring GG. We estimate its Hill index on the top 5 % of 4096 episodes of the current policy at each evaluation and raise an alert above 0.2 (it is 0.03 to 0.06 under πref\pi_{\rm ref}; the threshold was set by hand). On PPO the alert comes at or before 9 of the 15 collapses of the deterministic variant (at most 10 iterations ahead) and all 16 collapses of the random variant (median lead 20 iterations), with 3 false alerts in the 309 runs that do not collapse. Its blind spot: a hacked deterministic policy has no tail left, so along the exact KL-regularised path of the deterministic variant the alert never fires.

What reference data with the true reward can certify. From nn reference episodes where GG is measured, the selection curve of Table 1 can be estimated only down to the top fraction 100/n100/n (at least 100 episodes kept). Keeping the top fraction qq costs log⁡(1/q)\log(1/q) nats of KL (point 2), so nn labelled episodes certify a selection pressure of about log⁡(n/100)\log(n/100) nats, 6.9 for n=105n=10^5, and say nothing beyond; in the benign case ρ=1\rho=1 the best policy is at 34 nats. We tested a stopping rule built on this certified zone. It never selected a policy below πref\pi_{\rm ref}, but mostly because it acts as a small optimization budget, and it cannot see exploits that πref\pi_{\rm ref} never samples, which are the ones that matter in RL. We therefore do not propose it as a stopping rule for RL training; it applies as is to Best-of-NN sampling and reranking. Details in the reply to MDQy.

Random variant (Eref[G]=66.6E_{\rm ref}[G]=66.6). Failed tricks break the runs, so the hack is weaker. The policy maximising E[M]E[M] has true return 107.5, 96.5, 59.3, 16.0 and 6.0 for ρ=1\rho=1, 1.1, 1.2, 1.5 and 2. The exact KL-regularised path first improves the true return (to 99.0 at ρ=1.1\rho=1.1, 84.1 at ρ=1.2\rho=1.2 and 83.2 for ρ≥1.5\rho\ge1.5) and then collapses for ρ≥1.2\rho\ge1.2; 16 of the 170 PPO runs of this variant collapse. The tail index is log⁡ρ/log⁡(1/0.18)\log\rho/\log(1/0.18), with infinite variance from ρ≈2.36\rho\approx 2.36, and the estimated selection curve peaks at q=1.8×10−3q=1.8\times10^{-3}, 1.8×10−21.8\times10^{-2} and 3.2×10−23.2\times10^{-2} for ρ=2\rho=2, 2.5 and 3.

2. Conditioning on M>mM>m and training

Three statements, each with a short proof in the revision:

  • Among all distributions under which the metric exceeds mm almost surely, the conditional law is the closest to the reference in KL divergence. Selection is the least-committal optimizer that reaches level mm.
  • Every optimizer that reweights the reference law by a nondecreasing function of the metric returns an average of the selection curve m↦E[G∣M>m]m\mapsto E[G\mid M>m] over thresholds. This covers Best-of-NN sampling and, when transitions are deterministic (as in token generation), the optimum of KL-regularised RL, the objective used in RLHF. The limits that define our regimes carry over.
  • In overparametrised least squares, gradient descent from a Gaussian initialisation and the Gaussian prior conditioned on interpolating the data are the same distribution.

Where the link fails is also stated: with random transitions conditioning also selects lucky episodes, which no policy can do; and a trained agent only reaches what it explores.

3. Concrete examples and plain reading

Four narrative examples of the taxonomy and a plain-language reading of Theorems 1 and 2 are given in the reply to 6FR9. Sections 3 and 4 carry a condensed version, and a new appendix carries the full text. Table 2 is now referenced and discussed in the text.

4. A correction

"The lighter the tail of the discrepancy, the faster the collapse." We withdraw this sentence (abstract, introduction of Section 4, Table 2, Section 4.4, Discussion). It holds as a function of the threshold mm, but mm is not a fixed optimization pressure when the tail changes. In the coupling of Proposition 3(ii), E[G∣ξ>y]=P(ξ>y)E[G\mid \xi>y]=P(\xi>y) exactly, and E[G∣M>m]∼P(M>m)E[G\mid M>m]\sim P(M>m): when a fraction qq of candidates is kept, the goal is eventually of order qq whatever the exponent bb (the tail index of ξ\xi is 1/(b−1)1/(b-1)). What bb changes is when the collapse starts. The selection curve first rises, then turns down near the threshold where the tail of ξ\xi overtakes that of GG. For η=1\eta=1 the peak is at kept fraction 0.120.12 for b=3b=3, 9×10−49\times10^{-4} for b=5b=5 and 2×10−122\times10^{-12} for b=10b=10: a lighter tail delays the collapse. The revised statement is: the collapse occurs for every tail index, and its onset is set by where the two tails cross.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.