acceptodds
Under review as a conference paper at ICLR 2027

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

11 views

Abstract

In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reinforcement learning with verifiable rewards, Reward hacking, Verifier errors, Selective control, Auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aSHYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes reward hacking in RLVR with fixed imperfect verifiers using gradient flow. It characterizes when verifier reward increases while correctness decreases and studies the limits of verifier-only feedback. It proposes projected audit correction (PAC), which uses additional correctness labels to suppress hacks while preserving instantaneous acceptance growth under stated conditions. Experiments cover log-linear and neural contextual bandits and Qwen2-0.5B with LoRA on a synthetic digit-substitution task.

Critical strengths
  1. The paper offers an interesting explanation of reward hacking.

  2. The theoretical study of training dynamics in reinforcement learning is a worthwhile research direction.

Critical weaknesses
  1. The paper reduces reward hacking by modifying gradients, but I find the motivation for this approach insufficiently clear. If an audit can already determine whether a response is correct, why not directly adjust the reward using this information, rather than preserve the growth of the original verifier reward? Given that the original verifier is itself flawed, why is it necessary to preserve growth in its acceptance rate? In my view, the paper needs to explain more fully why this constraint is needed and what advantages it offers over directly correcting the reward. The practical benefit of projection is also unclear. PAC achieves higher correctness at some early stages, but its final correctness does not exceed that of raw correction. For example, in N40, PAC achieves 97.2%, compared with 98.4% for raw correction. Moreover, although raw correction initially reduces acceptance, it does not reduce correctness at the recorded evaluation points. Is the constraint that protects acceptance therefore necessary?

  2. The paper uses a linear projection to reduce hacking while preserving the learning of correct responses. However, the model has a large number of parameters, and autoregressive generation and parameter sharing inherently couple the gradients of different responses. Although the projection has mathematical guarantees for exact gradients and local first-order changes, this does not imply that it can separate the effects on correctness and hacking in practical training. In particular, the experiments use sampled gradients, finite step sizes, and AdamW, with the projection computed before the AdamW step. Further validation is needed to establish whether the desired property still holds after the actual parameter update.

  3. The paper decomposes hack growth into hack bias and correctness-to-hack leakage, but the roles of these two mechanisms have not been directly validated in actual LLM training. Deriving a growth condition mathematically does not establish that the condition holds in practice. In my view, the proposed mechanistic explanation for actual LLM training remains closer to a hypothesis. Observing rising reward and declining correctness demonstrates reward hacking, but does not establish that the gradient mechanisms proposed in the paper caused it. The authors should measure these terms directly and show how they evolve during training.

  4. Regarding hack bias, if hacks are initially rare and the model mostly samples correct responses, should learning from those responses not make correct responses easier to produce subsequently? Why would the model eventually become increasingly biased toward hacks? In the paper's LLM experiments, hacks already account for 39.1% and 49.2% of responses at initialization, and SFT includes demonstrations that copy incorrect hints. These experiments therefore primarily show how training further amplifies hacking when it is already prevalent, without adequately explaining how initially rare hacks gradually develop in models under more typical conditions.

  5. The LLM experiments are limited to Qwen2-0.5B with LoRA and a synthetic digit-substitution task. Qwen2 is relatively old and small, and the task deliberately provides incorrect hints that can fool the verifier. The current experiments are insufficient to establish whether the observed phenomena and the proposed method extend to stronger, more recent models or to realistic mathematics and coding tasks.

Questions for authors

Please refer to the weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer aSHY (1/2): why we project the audit correction

Thank you for your review. We are glad you found our explanation of reward hacking interesting and the theoretical study of training dynamics in reinforcement learning a worthwhile direction. Your concerns helped us clarify why our model prescribes projection and where our theory applies. Our response to all reviewers restates our contributions and the scope of our theory and explains why we project the audit correction.

W1. Why Project the Audit Correction

Concern 1. If audits reveal correctness, why not use them to adjust the reward directly?

Response. We agree with the reviewer that audits reveal correctness, and Section 4 shows that a guarantee of selective control requires such information. Adjusting the reward with audits, however, gives raw audit correction. Raw audit correction can achieve selective control in some runs, but even under the assumptions of Theorem 5.1, it cannot guarantee it (see response to all reviewers). We revised Section 5.1 to state this limit clearly and to show how our model prescribes projection.

Formally. Under full audit coverage, raw audit correction changes hacks and correct responses at the rates p˙H=∇pH⊤gR−λ∣∇pH∣2,p˙G=∇pG⊤gR−λ ∇pG⊤∇pH.\dot p_H = \nabla p_H^\top g_R - \lambda\\|\nabla p_H\\|^2, \qquad \dot p_G = \nabla p_G^\top g_R - \lambda\,\nabla p_G^\top\nabla p_H. Selective control holds at a policy when λ∣∇pH∣2>∇pH⊤gR\lambda\\|\nabla p_H\\|^2 > \nabla p_H^\top g_R and λ∇pG⊤∇pH≤∇pG⊤gR\lambda\nabla p_G^\top\nabla p_H \le \nabla p_G^\top g_R. Under the assumptions of Theorem 5.1, the second condition can still fail.

Evidence. In the bandit, raw audit correction first lowers both hacks and correctness (Figs. 5a, 1d).

Concern 2. If the verifier is flawed, why preserve the growth of its acceptance?

Response. We do not preserve the growth of acceptance because we trust the flawed verifier, and this preservation is not an arbitrary design choice. The objective of our paper is a guarantee of selective control. To obtain this guarantee, our model prescribes a correction that gives to correct responses whatever it takes from hacks. Because acceptance is the sum of correct responses and hacks, projecting the audit correction realizes this transfer and, as a consequence, leaves the rate of acceptance unchanged. Under the assumptions of Theorem 5.1, projected audit correction thus guarantees selective control. Raw audit correction, however, changes the rate of acceptance and can lower correctness (Concern 1). Our response to all reviewers and revised Section 5.1 give the full derivation.

Formally. Under full audit coverage, projected audit correction, u=−λP⊥∇pHu = -\lambda P_\perp\nabla p_H, changes hacks and correct responses at the rates p˙H=∇pH⊤gR−λ∣P⊥∇pH∣2,p˙G=∇pG⊤gR+λ∣P⊥∇pH∣2.\dot p_H = \nabla p_H^\top g_R - \lambda\\|P_\perp\nabla p_H\\|^2, \qquad \dot p_G = \nabla p_G^\top g_R + \lambda\\|P_\perp\nabla p_H\\|^2. The correction takes λ∣P⊥∇pH∣2\lambda\\|P_\perp\nabla p_H\\|^2 from hacks and gives it to correct responses. Because p=pG+pHp = p_G + p_H, adding these rates gives p˙=p˙H+p˙G=∣gR∣2\dot p = \dot p_H + \dot p_G = \\|g_R\\|^2, the rate of acceptance under verifier flow. When λ∣P⊥∇pH∣2>∇pH⊤gR\lambda\\|P_\perp\nabla p_H\\|^2 > \nabla p_H^\top g_R, hacks fall and correct responses rise, which is selective control.

Concern 3. If raw audit correction reaches similar final correctness, is projection necessary?

Response. The objective of our paper is to derive conditions that guarantee selective control. Concern 2 shows that projection, under the assumptions of Theorem 5.1, provides this guarantee. What projection adds is this guarantee, not higher final correctness in a given run. This guarantee holds for the population flow, and Appendix A.1 and Section 8 state that it does not extend to the language model, which Section 6.2 uses as a qualitative test of our theory. In this test, however, the trajectory of raw audit correction is consistent with our theory. Raw audit correction can achieve selective control in some runs, but it cannot guarantee selective control (Concern 1).

AuthorsOpenReview ↗

Response to Reviewer aSHY (2/2): how far our theory extends

W2. Projection Under Parameter Sharing and Practical Training

Concern 1. If parameter sharing couples the gradients of correct responses and hacks, can projection separate their effects?

Response. Yes, within our theory. Our results hold for any differentiable policy along the population flow, so parameter sharing enters only through the gradients of correct responses and hacks. Theorem 5.1 holds for any such coupling under its assumptions. Projection separates the effects on correct responses and hacks for any such coupling: whatever it takes from hacks, it gives to correct responses (W1, Concern 2). The neural bandit is our closest test of this claim, because it closely approximates the flow while all responses share the parameters of one network. In this test, projected audit correction lowers hacks while raising correctness (Fig. 1d). Concern 2 addresses practical training of language models.

Concern 2. If training uses sampled gradients, finite steps, and AdamW, does selective control still hold after the actual parameter update?

Response. We agree with the reviewer: our theory does not certify that selective control holds after an update with sampled gradients, finite steps, and AdamW. As our response to all reviewers explains, we accept population gradient flow as a modeling assumption to understand reward hacking and to derive conditions that guarantee selective control (Theorem 5.1). Section 8 states that extending this guarantee to practical training calls for an empirical methods paper. Appendix A.2 analyzes how estimating the reward gradient, optimizer transformations, and finite steps separate the actual update from the flow, and it identifies validating each update as a task for such a paper. Beyond our main objective, we tested the intervention qualitatively in a controlled language model trained with sampled gradients, finite steps, and AdamW. In this model, projected audit correction lowers hacks and raises correctness relative to initialization (Fig. 2b).

W3. Is the Growth Decomposition a Hypothesis in LLM Training?

Response. We agree with the reviewer that the decomposition is not established for LLM training, and the paper does not claim it is. As our response to all reviewers explains, LLM training lies outside our theory. Within our theory, however, the decomposition is not a hypothesis but an identity: for any differentiable policy, Equation (3) holds at every point along the population flow (Fig. 3a,b). We use the decomposition to show that RLVR has no defense of its own against hacks that reinforce their own growth. This vulnerability motivates the intervention that Sections 4 and 5 derive.

W4. Rare Hacks and Amplification

Concern 1. If hacks are initially rare, why would training favor them?

Response. Along the population flow, the growth conditions we derive do not rule out the amplification of initially rare hacks. The hacked share grows whenever the condition of Proposition 3.2 holds, and RLVR does not oppose this growth. This growth is aggregate and can include initially rare hacks. Section 3.3 calls this possibility a latent vulnerability of RLVR. Whether the condition holds for rare hacks in practice depends on how the geometry of the group scores evolves during training, which our theory does not describe. Thus, how rare hacks emerge remains open (Concern 2).

Concern 2. If the language model starts with many hacks, do the experiments show amplification instead of emergence?

Response. We agree with the reviewer that our language model experiments show amplification, not emergence. Section 8 states this boundary: the language model is a controlled test of whether amplification and selective control agree qualitatively with the theory, and it does not test emergence. To test amplification and selective control, we initialize the language model with many hacks. Prior work also seeds hacking behavior before reinforcement learning (MacDiarmid et al., 2025; Khalifa et al., 2026). How rare hacks emerge and become common remains open (Khalifa et al., 2026).

W5. Scale and Realism of the Experiments

Response. We agree that our experiments do not show that the guarantee of projected audit correction extends to larger models or realistic mathematics and coding tasks, and the paper does not claim so. Section 8 states that this extension depends on solving the tasks in Appendix A.2, and our response to all reviewers leaves these tasks to an empirical methods paper. The language model experiment is a controlled test of qualitative agreement with the theory. The digit task makes this test possible, because it labels every sampled response as a correct response, a hack, or a rejected response.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

AuthorsOpenReview ↗

Practical benefit of projection under noisy audits (W1)

We would like to add one result from the revised paper that addresses W1. The reviewer noted that, with exact audits, raw audit correction does not reduce correctness at the recorded evaluation points. With noisy audits, it does. In the language model, projected audit correction satisfies selective control between every pair of recorded rounds, and raw audit correction does not. Held-out correctness is 93.8% and 94.5% under projected audit correction, compared with 56.2% and 35.9% under raw audit correction (Appendix D.3, Ablation III; Figure 9). The neural bandit shows the same pattern (Appendix D.2, Figure 7). Under a simple model of noise, projection therefore shows a potential practical advantage over raw audit correction.

Reviewer_qA2yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the theoretical foundation of the influence of verifier errors in the learning dynamics of RLVR, and proposes a method to improve the RLVR training when there are verifier errors.

Critical strengths

1)The motivation makes sense to me.

2)The presentation is clear.

Critical weaknesses

1)The proposition in section 3 mainly depends on procedural value(or value still depends on/changes during the process, especially Proposition 3.2). The description of the learning dynamics and the theoretical contribution don’t seem sufficient to me.

2)The idea of auditing is very direct, since you can directly expect that adding additional correctness information to rewards will improve the training performance. Also, though the idea of projected audit correction is interesting, do your results show that your method is significantly better than raw audit correction(or other baseline methods)?

Questions for authors

1)Could you please respond to my two concerns in the weaknesses section? Clarifying your key theoretical and methodological contributions beyond the intuitive ideas could be helpful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qA2y (1/2): local conditions and the learning dynamics

Thank you for your review. We appreciate that the motivation made sense to you and that you found the presentation clear. Your concerns helped us clarify our theoretical and methodological contributions. Our response to all reviewers restates our contributions and the scope of our theory and explains why we project the audit correction.

W1. Local Conditions and the Learning Dynamics

Concern 1. The propositions in Section 3, especially Proposition 3.2, depend on quantities that change during training.

Response. We agree with the reviewer. The paper states this dependence after Propositions 3.1 and 3.2 and in Appendix A.1. The revised Section 8 extends this discussion. We use these propositions to show that RLVR has no defense of its own against hacks that reinforce their own growth. Conditions that depend on the current policy suffice for this purpose. Proposition 3.1 needs its condition at only a single time, which produces reward hacking over a short interval of training. Several of our other results, however, do not depend on quantities that change during training. Propositions 4.1 and 4.2, for example, hold for any training procedure that uses only the training record, because their proofs show by induction over training steps that the record is the same under compatible correctness assignments (Appendices C.3.2 and C.3.3). Separately, Theorem 5.1(ii) bounds the hacked share over [0,T][0,T], provided its assumptions hold throughout that interval.

Concern 2. The description of the learning dynamics and the theoretical contribution seem insufficient.

Response. We read this concern as asking whether the description of the learning dynamics and the theoretical contribution suffice for the objective of the paper. They suffice because they answer both research questions posed in Section 2.3. The first question asks when the hacked share grows. Equation (3) answers this question at every policy, and Propositions 3.1 and 3.2 give local necessary and sufficient conditions for reward hacking and for growth of the hacked share. The second question asks whether verifier feedback alone can reduce the hacked share and, if not, what additional information can. Propositions 4.1–4.3 show that the training record alone cannot guarantee detecting hacks, identifying them, or reducing them without also reducing correct responses. The theory then prescribes projected audit correction, whose correction term opposes the hack bias and leakage in Equation (3). Under the assumptions of Theorem 5.1, this correction achieves selective control: lowering the probability of hacks without lowering the probability of correct responses. If the reviewer has a specific aspect of the learning dynamics or the theoretical contribution in mind, we would be glad to address it.

AuthorsOpenReview ↗

Response to Reviewer qA2y (2/2): why audits need projection

W2. Why Audits Need Projection

Concern 1. Auditing is a direct idea, because adding correctness information to the reward should be expected to improve training.

Response. We agree that adding audits to the reward is a direct idea. Section 4 shows that a guarantee of selective control requires information beyond the verifier, such as audits that reveal correctness. Adding audits to the reward, however, gives raw audit correction. Our response to all reviewers shows that raw audit correction can lower the probability of correct responses, so it cannot guarantee selective control. We revised Section 5.1 to state this limit clearly and to show how our model prescribes projection (see Concern 2).

Formally. Under full audit coverage, raw audit correction changes hacks and correct responses at the rates p˙H=∇pH⊤gR−λ∣∇pH∣2,p˙G=∇pG⊤gR−λ ∇pG⊤∇pH.\dot p_H = \nabla p_H^\top g_R - \lambda\\|\nabla p_H\\|^2, \qquad \dot p_G = \nabla p_G^\top g_R - \lambda\,\nabla p_G^\top\nabla p_H. Selective control holds at a policy when λ∣∇pH∣2>∇pH⊤gR\lambda\\|\nabla p_H\\|^2 > \nabla p_H^\top g_R and λ ∇pG⊤∇pH≤∇pG⊤gR\lambda\,\nabla p_G^\top\nabla p_H \le \nabla p_G^\top g_R. Under the assumptions of Theorem 5.1, the second condition can still fail, so raw audit correction can achieve selective control in some runs but cannot guarantee it. Our response to all reviewers and revised Section 5.1 give the full description.

Evidence. In the bandit, raw audit correction first lowers both hacks and correctness (Figs. 5a, 1d).

Concern 2. Do the results show that projected audit correction is significantly better than raw audit correction or other baselines?

Response. We thank the reviewer for finding the idea of projected audit correction interesting. Our experiments do not show that it is significantly better than raw audit correction, and we do not claim so. Our claim is theoretical: under the assumptions of Theorem 5.1, projected audit correction guarantees selective control. To obtain this guarantee, we use our modeling assumptions to understand reward hacking in a tractable model. From this understanding, we derive conditions for selective control. Projected audit correction satisfies these conditions under the assumptions of Theorem 5.1, whereas raw audit correction need not. Raw audit correction can still achieve selective control in some runs, but without a guarantee (Concern 1). Section 5.1 and our response to all reviewers show why projection guarantees selective control. To test the theory, our experiments also include two methods that use only verifier feedback: GRPO and gradient regularization. By Proposition 4.3, neither can guarantee selective control.

Evidence. Their results agree with this limit. In the neural bandit, gradient regularization increases hacks (Fig. 1d). In the language model, which lies outside our theory, GRPO raises acceptance while lowering correctness (Table 5), in qualitative agreement with Proposition 3.1.

Q1. Theoretical and Methodological Contributions

Answer. Theoretically, we give local necessary and sufficient conditions for reward hacking and for growth of the hacked share (Propositions 3.1 and 3.2), limits on what any monitor or controller that uses only the training record can guarantee (Propositions 4.1–4.3), and conditions under which projected audit correction guarantees selective control (Theorem 5.1). Methodologically, our model of the learning dynamics prescribes projected audit correction beyond the intuitive idea of adding audits to the reward, which carries no such guarantee (W2). Our response to all reviewers and W1 (Concern 2) give details.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

Thank you for the clarifications and additional experiments. Although I understand that your claim is theoretical and appreciate the new experimental results, I am still not fully convinced by the overall contribution. First, while I understand that your analysis mainly serves your own goal, the theoretical contribution itself is still relatively limited. Second, your LLM experiments don’t show enough benefits of your projected audit correction over the raw audit correction. Given these concerns, I intend to keep my original rating.

AuthorsOpenReview ↗

Response to Reviewer qA2y

Theoretical contribution

Response. We thank the reviewer for agreeing that our analysis serves the goal of the paper. The reviewer raised this concern in the initial review and again here. The concern does not point to a missing result, an error, or a stronger result that the paper should prove. In our first response, we invited the reviewer to name a specific aspect (W1, Concern 2). We therefore state what our theory establishes and how far each result reaches.

Growth. Equation (3) and Propositions 3.1 and 3.2 hold along the flow for any differentiable policy, not only for bandits. They give local necessary and sufficient conditions for reward hacking and for growth of the hacked share at every policy.

Limits of verifier feedback. Propositions 4.1 and 4.2 hold for any training procedure that uses only the training record of RLVR, including sampled gradients and any optimizer. Their proofs argue by induction over training steps (Appendices C.3.2 and C.3.3). These limits therefore reach beyond our model of the dynamics.

Control. Theorem 5.1 gives conditions under which projected audit correction guarantees selective control, whatever the coupling between the gradients of correct responses and hacks.

The theory also predicts behavior that intuition does not. As the reviewer noted in the initial review, adding audits to the reward is the intuitive intervention. Our theory shows that this raw audit correction carries no guarantee of improving correctness, even with exact audits. It changes the rate of correct responses by −λ∇pG⊤∇pH-\lambda\nabla p_G^\top\nabla p_H, whose sign depends on the coupling between the gradients of correct responses and hacks (Section 5.1). Moreover, when audits are noisy, false alarms lower the rate of correct responses under raw audit correction even when these gradients do not align, whereas noise only weakens projected audit correction (Appendix A.2). Both the neural bandit and the language model agree with these predictions (next response).

If the reviewer sees a specific limitation in our theoretical results, we would be glad to address it, as we offered in our first response.

Benefit of projected audit correction over raw audit correction

Response. We thank the reviewer for this concern. Our response above explains the theoretical benefit: raw audit correction carries no guarantee of selective control, whereas projected audit correction guarantees it under the conditions of Theorem 5.1. The revised paper now also gives empirical evidence of this benefit in the language model.

Language model evidence. We repeated the language model comparison with noisy audits, in which the auditor flags a correct response as a hack with probability α\alpha and misses a hack with probability β\beta. We compare α=β=0.25\alpha=\beta=0.25 with α=0.5, β=0\alpha=0.5,\ \beta=0 (Appendix D.3, Ablation III; Figure 9). In both conditions, projected audit correction satisfies selective control between every pair of recorded rounds, and raw audit correction does not. On held-out responses, PAC reaches 93.893.8\\% and 94.594.5\\% correctness, compared with 56.256.2\\% and 35.935.9\\% for raw audit correction. Under this simple model of noise, projection therefore shows a potential practical advantage over raw audit correction.

Neural bandit. The neural bandit, a neural policy in which correct responses and hacks share parameters, shows the same pattern. Both corrections use the same flagged set and the same gain, so they differ only in the projection. With noisy audits (α=β=0.25\alpha=\beta=0.25), raw audit correction drives correctness to near zero, whereas projected audit correction raises correctness from 0.33 to 0.99 and lowers hacks (Appendix D.2, Ablation III; Figure 7). Our theory predicts this difference: under raw audit correction, false alarms lower the rate of correct responses, whereas noise only weakens projected audit correction (Appendix A.2).

Exact audits. With exact audits, raw audit correction and PAC reach similar final correctness in the language model (Table 5). This outcome agrees with our theory: raw audit correction can achieve selective control in some runs, but it cannot guarantee it (Section 5.1). Noisy audits show where the guarantee matters.

Reviewer_coTEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies RLVR with a fixed, imperfect verifier that could be hacked. Using a gradient-flow model, the authors characterize when reward goes up while correctness goes down ("hack bias" and "correctness-to-hack leakage"), argue that the training signal alone can't detect or fix such hacks, and propose "projected audit correction". Experiments on contextual bandits and a small language model support the analysis.

(I am not an expert in this field tho, especially on the theorm proof part. so please take my review with a grain of salt.)

Critical strengths
  • The paper is generally well scoped and followable (esp. given the intensive math portion). I like that it isolates one question (what does a fixed verifier's error do to learning) and is upfront about its assumptions.

  • The hack bias vs. leakage decomposition gives a nice intuition for why hacking can be self-reinforcing under on-policy training.

  • The projection idea is simple and elegant, and the bandit experiments illustrate the theory well.

Critical weaknesses
  • The hacks and correct responses are treated as fixed sets that the policy just reweights between. My intuition is that the more worrying failure in practice is RL discovering new hacks that weren't covered when the audits were collected. I'd love some discussion on how the analysis (or the fix) would behave in that case.

  • In general I'd encourage the authors to consider adding more discussion on how practical the audit-based fix is. The guarantees assume audits that cover essentially all hacks, which feels close to already having a correct verifier. And if we do have audits, I'd expect the most natural thing people would do is just fold them into the reward (e.g., zero out audited hacks). It looks to me that the paper's own simpler variant already ends up similarly good, with the projection mostly helping earlier in training. It would help to compare against simple reward relabeling, and to discuss where audits would come from in practice, how much they cost, and what happens when they're noisy (e.g., from an LLM judge).

Questions for authors

How does the analysis generalize to emergent reward hacking scenarios? Related - How easily generalizable do you think the assumption on "gradient flow with a fixed verifier" is?

4 comments
AuthorsOpenReview ↗

Response to Reviewer coTE (1/2): emerging hacks and gradient flow

Thank you for your review. We are glad you found the paper well scoped and upfront about its assumptions, the decomposition into hack bias and leakage a useful intuition for why hacking reinforces itself, and the projection simple and elegant. Your concerns helped us clarify the boundaries of our theory and how it prescribes our intervention. Our response to all reviewers restates our contributions and the scope of our theory and explains why we project the audit correction.

W1. Hacks That Emerge During Training

Concern 1. Hacks and correct responses are fixed sets, so the analysis may miss hacks that emerge during training.

Response. The set of hacks HH is a population: it contains every accepted incorrect response in the response space, including responses the policy has not yet sampled. A hack that emerges during training belongs to HH from the start. Its probability is negligible at first, and training can raise it. Our local growth condition characterizes when the hacked share grows.

Formally. Membership in HH comes from the labels R=1R=1 and c=0c=0, which do not depend on the policy parameters θ\theta, so HH stays fixed during RLVR training. The probability pH(θ)p_H(\theta), however, changes with θ\theta: an emerging hack shows up as a contribution to pHp_H that starts near zero and grows. Equation (3) governs the hacked share of this whole population, so the local growth condition of Proposition 3.2 covers emerging hacks in aggregate. How individual families of rare hacks emerge, however, lies outside our population analysis and remains open (Khalifa et al., 2026). We added this discussion to Section 8.

Concern 2. The correction may fail on hacks that emerge after the audits were collected.

Response. We agree with the reviewer: the correction can fail on such hacks. Our theory, however, characterizes how the correction affects such hacks. The effect depends on whether the emerging hack and the audited hacks have aligned gradients, that is, whether pushing down audited hacks also pushes down the emerging hack.

Formally. We split the hacks into audited and undetected sets, H=HA⊔HUH = H_A \sqcup H_U, so ∇pH=∇pHA+∇pHU\nabla p_H = \nabla p_{H_A} + \nabla p_{H_U}. The correction u=−λP⊥∇pHAu = -\lambda P_\perp \nabla p_{H_A} changes the growth of undetected hacks by ∇pHU⊤u=−λcUA\nabla p_{H_U}^\top u = -\lambda c_{UA}, where cUA=(P⊥∇pHU)⊤(P⊥∇pHA)c_{UA} = (P_\perp \nabla p_{H_U})^\top (P_\perp \nabla p_{H_A}) is their projected alignment. When cUA>0c_{UA} > 0, the correction also pushes undetected hacks down; when cUA=0c_{UA} = 0, it leaves them unchanged; and when cUA<0c_{UA} < 0, it shifts probability toward them. The correction changes the growth of all hacks by −λ(∣P⊥∇pHA∣2+cUA)-\lambda(\\|P_\perp \nabla p_{H_A}\\|^2 + c_{UA}), so it still opposes the growth of all hacks when cUA<0c_{UA} < 0, provided ∣P⊥∇pHA∣2>−cUA\\|P_\perp \nabla p_{H_A}\\|^2 > -c_{UA}. Because p˙G=∣gR∣2−p˙H\dot p_G = \\|g_R\\|^2 - \dot p_H under the corrected flow, correct responses increase whenever all hacks decrease. The sign of cUAc_{UA}, however, depends on the current policy and can change during training. We added this analysis to Appendix A.2.

Q1. How does the analysis generalize to emergent reward hacking scenarios?

Answer. Our responses to W1 answer this question in two parts. First, emerging hacks belong to the set of hacks HH from the start, with negligible initial probability, so our analysis covers them in aggregate (W1, Concern 1). Second, projected audit correction pushes down emerging hacks that no audit reveals when their projected gradients align with those of audited hacks, and can push them up otherwise (W1, Concern 2).

Q2. How easily generalizable do you think the assumption on "gradient flow with a fixed verifier" is?

Answer. The answer differs across our results. Propositions 4.1 and 4.2 do not depend on gradient flow: they hold for any training procedure that uses only the training record, because their proofs argue by induction over training steps (Appendices C.3.2 and C.3.3). Selective control, in contrast, is defined through rates along the flow, so Proposition 4.3, like Propositions 3.1 and 3.2 and Theorem 5.1, holds exactly only for gradient flow. Along the flow, these results hold for any differentiable policy, and the neural bandit, which learns its own representation through parameters that correct responses and hacks share, closely approximates this flow (Appendix D.2). Beyond the flow, our results do not certify behavior: discrete updates approach the flow as the step size shrinks (Figure 3d), and the language model is a controlled test of qualitative agreement (response to all reviewers; Section 8 and Appendix A.1). A verifier that changes during training, for example when tests are added, shifts the group probabilities without a policy update, which our dynamics exclude (Appendix A.1).

AuthorsOpenReview ↗

Response to Reviewer coTE (2/2): practicality of audit correction

W2. Practicality of Audit Correction

Concern 1. The guarantees assume audits that cover essentially all hacks, which seems close to having a correct verifier, and source and cost of audits are unclear.

Response. We agree with the reviewer: Theorem 5.1 assumes full audit coverage (A2), so audits reveal every hack, which is close to having a correct verifier. Full coverage, however, is sufficient, not necessary: partial coverage can replace it when the projected gradients of audited hacks and all hacks align (Remark after Theorem 5.1; Appendix A.2). W1 (Concern 2) extends this condition to undetected hacks. Partial coverage also reduces cost: a correct verifier must judge every response, whereas audits judge only a fraction, so the cost of auditing scales with the fraction audited. Audits can come from hidden tests, human review, or monitors of reward hacking (Baker et al., 2025; Wang et al., 2026a). Fewer audits, however, make the correction noisier, so cost trades off against reliability. Choosing which responses to audit under a fixed budget remains open. We added this discussion to Section 8.

Evidence. In the language model, auditing each accepted response with probability ρ=0.25\rho = 0.25 reduced audits from 293 to 72 per run on average, and projected audit correction still reached 95.2% test correctness, compared with 97.2% under full auditing (Table 5 and Figure 8f).

Concern 2. Folding audits into the reward, e.g., by zeroing out audited hacks, seems natural, and raw audit correction reaches similar final correctness.

Response. We agree that folding audits into the reward is natural. Our limits of verifier feedback (Propositions 4.1–4.3) also call for correctness information such as audits. The reviewer’s example, zeroing out audited hacks, is raw audit correction with λ=1\lambda = 1 (Section 5.1). Under full audit coverage, it maximizes correctness directly, so correctness never falls. This case agrees with our response to all reviewers. There, correctness falls when λ∇pG⊤∇pH>∇pG⊤gR\lambda\nabla p_G^\top\nabla p_H > \nabla p_G^\top g_R. At λ=1\lambda = 1, this condition reduces to 0>∣∇pG∣20 > \\|\nabla p_G\\|^2 and never holds. Hacks, however, grow whenever their gradients align with those of correct responses. Under partial coverage, zeroing out also rewards undetected hacks, so correctness can fall as well. Zeroing out thus cannot guarantee the objective of our paper: selective control. Raw audit correction can still succeed in some runs, as in the language model (response to all reviewers).

Formally. Zeroing out replaces RR with R−1HAR - \mathbf{1}_{H_A}, so the expected reward becomes p−pHAp - p_{H_A}. Under full coverage, p−pHA=pGp - p_{H_A} = p_G, so p˙G=∣∇pG∣2\dot p_G = \\|\nabla p_G\\|^2 and p˙H=∇pG⊤∇pH\dot p_H = \nabla p_G^\top\nabla p_H. Under partial coverage, p−pHA=pG+pHUp - p_{H_A} = p_G + p_{H_U}, so p˙G=∣∇pG∣2+∇pG⊤∇pHU\dot p_G = \\|\nabla p_G\\|^2 + \nabla p_G^\top\nabla p_{H_U} can be negative. Selective control also requires p˙H<0\dot p_H < 0, so zeroing out violates it whenever ∇pG⊤∇pH≥0\nabla p_G^\top\nabla p_H \ge 0, despite full audit coverage.

Concern 3. Audits can be noisy, e.g., when an LLM judge provides them.

Response. We agree that audits can be noisy. To study how noise affects selective control, we added a simple noise model to Appendix A.2: audits flag correct responses as hacks at a fixed rate α\alpha (false alarms) and miss hacks at a fixed rate β\beta. With this model, we ask whether noise only weakens the correction or also changes its direction. Noise only weakens projected audit correction, but it can change the direction of raw audit correction. False alarms add a component along the reward gradient, and projection removes it, so on average the projected correction still opposes hacks whenever α+β<1\alpha + \beta < 1. Raw audit correction keeps this component, so false alarms push correct responses down directly.

Formally. We assume full coverage and error rates that do not depend on the response, and we analyze the expected correction at a fixed policy. The flagged responses then have expected gradient (1−α−β)∇pH+αgR(1-\alpha-\beta)\nabla p_H + \alpha g_R. Projection removes αgR\alpha g_R and leaves the correction of Theorem 5.1 with gain λ(1−α−β)\lambda(1-\alpha-\beta). Raw audit correction keeps −λα∇pG-\lambda\alpha\nabla p_G, which subtracts λα∣∇pG∣2\lambda\alpha\\|\nabla p_G\\|^2 from the rate of correct responses (Appendix A.2).

Evidence. We repeated the neural bandit of Figure 1d with noisy audits (λ=6\lambda = 6, 10 seeds). With α=β=0.25\alpha = \beta = 0.25, projected audit correction still raises correctness, whereas raw audit correction drives correctness to near zero, as the penalty from false alarms predicts (Figure 7, Appendix D.2).

Limitations. Selective control under noise that depends on the response remains open.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

Thank you for the clarifications! I appreciate the detailed rebuttal and found it informative. I will maintain my score since it already reflected my positive view on the paper.

AuthorsOpenReview ↗

Response to Reviewer coTE

Thank you for your positive assessment and for engaging with our rebuttal. Your concern about noisy audits (W2, Concern 3) led to one more experiment, which we added to the revised paper. We repeated the language model comparison with noisy audits under two conditions, α=β=0.25\alpha=\beta=0.25 and α=0.5, β=0\alpha=0.5,\ \beta=0 (Appendix D.3, Ablation III; Figure 9). In both conditions, projected audit correction satisfies selective control between every pair of recorded rounds, and raw audit correction does not. On held-out responses, projected audit correction reaches 93.893.8\\% and 94.594.5\\% correctness, compared with 56.256.2\\% and 35.935.9\\% for raw audit correction. Under this simple model of noise, projection therefore shows a potential practical advantage over raw audit correction. These runs show qualitative agreement with the analysis of Appendix A.2 and do not certify the behavior of language models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers: objective, scope, and why we project audits

We thank the reviewers for their careful reviews. In this comment, we restate the objective and contributions of the paper and the scope of our theory, and we answer the question that all three reviewers raised: why we project the audit correction.

Reminder of contribution. We use a population analysis to understand how verifier errors lead to reward hacking in RLVR. From this understanding, we derive conditions for selective control: lowering the probability of hacks without lowering the probability of correct responses. Our contributions are: (1) We give local necessary and sufficient conditions under which hacks reinforce their own growth through hack bias and leakage and lead to reward hacking (Propositions 3.2 and 3.1), so RLVR needs an intervention. (2) This intervention cannot rely on the training record alone: from the record, no monitor or controller is guaranteed to detect whether hacks exist, identify which accepted responses are hacks, or achieve selective control (Propositions 4.1–4.3). Selective control therefore requires information beyond the verifier, such as audits that reveal correctness. (3) Adding audits to the reward, however, can lower the probability of correct responses, because correct responses and hacks share parameters. Our population analysis prescribes a different intervention: projected audit correction, which turns every reduction in hacks into a gain in correctness and thus achieves selective control under the assumptions of Theorem 5.1.

Scope of the theory. We accept population gradient flow as a modeling assumption to understand reward hacking and to derive conditions that guarantee selective control. Along the flow, our results hold for any differentiable policy, and our bandits closely approximate this flow, including the neural bandit, in which correct responses and hacks share parameters. Beyond these two objectives, we test our intervention in a controlled language model. Training this model with sampled gradients and AdamW lies outside our theory, so this test checks qualitative agreement and does not certify the behavior of language models (Section 8 and Appendix A.1). Extending the guarantee to practical training requires solving the problems that Appendix A.2 identifies, which calls for an empirical methods paper that we leave to future work.

Why we project the audit correction (Section 5.1). Section 4 shows that a guarantee of selective control requires information beyond the verifier, and adding audits to the reward is the direct way to use this information. This direct use, raw audit correction, pushes hacks down, but the direction that pushes hacks down can also push correct responses down. Raw audit correction can therefore achieve selective control in some runs, but even under the assumptions of Theorem 5.1, it cannot guarantee the objective of our paper: selective control. Our model prescribes how to obtain this guarantee: a correction that gives to correct responses whatever it takes from hacks. Because acceptance is the sum of correct responses and hacks, projecting the audit correction realizes this transfer and, as a consequence, leaves the rate of acceptance unchanged. The projected correction thus turns every reduction in hacks into a gain in correctness and, under the assumptions of Theorem 5.1, achieves selective control.

Formally. Under full audit coverage, raw audit correction uses u=−λ∇pHu = -\lambda\nabla p_H. Along θ˙=gR+u\dot\theta = g_R + u, hacks and correct responses change at the rates p˙H=∇pH⊤gR−λ∣∇pH∣2,p˙G=∇pG⊤gR−λ ∇pG⊤∇pH.\dot p_H = \nabla p_H^\top g_R - \lambda\\|\nabla p_H\\|^2, \qquad \dot p_G = \nabla p_G^\top g_R - \lambda\,\nabla p_G^\top\nabla p_H. The correction always pushes hacks down, but it also pushes correct responses down whenever their gradients align with those of hacks, ∇pG⊤∇pH>0\nabla p_G^\top\nabla p_H > 0, and correctness falls when λ∇pG⊤∇pH>∇pG⊤gR\lambda\nabla p_G^\top\nabla p_H > \nabla p_G^\top g_R. Raw audit correction can therefore fail, as in the bandit, where it first lowers both hacks and correctness (Figs. 5a, 1d), or succeed, as in the language model (Appendix D.3.2).

Our theory removes this failure. Acceptance is p=pG+pHp = p_G + p_H, so its rate is p˙=p˙G+p˙H\dot p = \dot p_G + \dot p_H and its gradient is gR=∇pG+∇pHg_R = \nabla p_G + \nabla p_H. Projection, u=−λP⊥∇pHu = -\lambda P_\perp\nabla p_H, changes the rates to p˙H=∇pH⊤gR−λ∣P⊥∇pH∣2,p˙G=∇pG⊤gR+λ∣P⊥∇pH∣2.\dot p_H = \nabla p_H^\top g_R - \lambda\\|P_\perp\nabla p_H\\|^2, \qquad \dot p_G = \nabla p_G^\top g_R + \lambda\\|P_\perp\nabla p_H\\|^2. The correction takes λ∣P⊥∇pH∣2\lambda\\|P_\perp\nabla p_H\\|^2 from hacks and gives it to correct responses, whatever the sign of ∇pG⊤∇pH\nabla p_G^\top\nabla p_H. This transfer implies gR⊤u=0g_R^\top u = 0, so acceptance keeps its rate under verifier flow, p˙=∣gR∣2\dot p = \\|g_R\\|^2. When the correction outweighs the growth of hacks driven by the verifier, λ∣P⊥∇pH∣2>∇pH⊤gR\lambda\\|P_\perp\nabla p_H\\|^2 > \nabla p_H^\top g_R, it achieves selective control (Theorem 5.1).

We answer the remaining concerns of each reviewer in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.