- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes reward hacking in RLVR with fixed imperfect verifiers using gradient flow. It characterizes when verifier reward increases while correctness decreases and studies the limits of verifier-only feedback. It proposes projected audit correction (PAC), which uses additional correctness labels to suppress hacks while preserving instantaneous acceptance growth under stated conditions. Experiments cover log-linear and neural contextual bandits and Qwen2-0.5B with LoRA on a synthetic digit-substitution task.
Critical strengths
-
The paper offers an interesting explanation of reward hacking.
-
The theoretical study of training dynamics in reinforcement learning is a worthwhile research direction.
Critical weaknesses
-
The paper reduces reward hacking by modifying gradients, but I find the motivation for this approach insufficiently clear. If an audit can already determine whether a response is correct, why not directly adjust the reward using this information, rather than preserve the growth of the original verifier reward? Given that the original verifier is itself flawed, why is it necessary to preserve growth in its acceptance rate? In my view, the paper needs to explain more fully why this constraint is needed and what advantages it offers over directly correcting the reward. The practical benefit of projection is also unclear. PAC achieves higher correctness at some early stages, but its final correctness does not exceed that of raw correction. For example, in N40, PAC achieves 97.2%, compared with 98.4% for raw correction. Moreover, although raw correction initially reduces acceptance, it does not reduce correctness at the recorded evaluation points. Is the constraint that protects acceptance therefore necessary?
-
The paper uses a linear projection to reduce hacking while preserving the learning of correct responses. However, the model has a large number of parameters, and autoregressive generation and parameter sharing inherently couple the gradients of different responses. Although the projection has mathematical guarantees for exact gradients and local first-order changes, this does not imply that it can separate the effects on correctness and hacking in practical training. In particular, the experiments use sampled gradients, finite step sizes, and AdamW, with the projection computed before the AdamW step. Further validation is needed to establish whether the desired property still holds after the actual parameter update.
-
The paper decomposes hack growth into hack bias and correctness-to-hack leakage, but the roles of these two mechanisms have not been directly validated in actual LLM training. Deriving a growth condition mathematically does not establish that the condition holds in practice. In my view, the proposed mechanistic explanation for actual LLM training remains closer to a hypothesis. Observing rising reward and declining correctness demonstrates reward hacking, but does not establish that the gradient mechanisms proposed in the paper caused it. The authors should measure these terms directly and show how they evolve during training.
-
Regarding hack bias, if hacks are initially rare and the model mostly samples correct responses, should learning from those responses not make correct responses easier to produce subsequently? Why would the model eventually become increasingly biased toward hacks? In the paper's LLM experiments, hacks already account for 39.1% and 49.2% of responses at initialization, and SFT includes demonstrations that copy incorrect hints. These experiments therefore primarily show how training further amplifies hacking when it is already prevalent, without adequately explaining how initially rare hacks gradually develop in models under more typical conditions.
-
The LLM experiments are limited to Qwen2-0.5B with LoRA and a synthetic digit-substitution task. Qwen2 is relatively old and small, and the task deliberately provides incorrect hints that can fool the verifier. The current experiments are insufficient to establish whether the observed phenomena and the proposed method extend to stronger, more recent models or to realistic mathematics and coding tasks.
Questions for authors
Please refer to the weakness
5 comments
Response to Reviewer aSHY (2/2): how far our theory extends
W2. Projection Under Parameter Sharing and Practical Training
Concern 1. If parameter sharing couples the gradients of correct responses and hacks, can projection separate their effects?
Response. Yes, within our theory. Our results hold for any differentiable policy along the population flow, so parameter sharing enters only through the gradients of correct responses and hacks. Theorem 5.1 holds for any such coupling under its assumptions. Projection separates the effects on correct responses and hacks for any such coupling: whatever it takes from hacks, it gives to correct responses (W1, Concern 2). The neural bandit is our closest test of this claim, because it closely approximates the flow while all responses share the parameters of one network. In this test, projected audit correction lowers hacks while raising correctness (Fig. 1d). Concern 2 addresses practical training of language models.
Concern 2. If training uses sampled gradients, finite steps, and AdamW, does selective control still hold after the actual parameter update?
Response. We agree with the reviewer: our theory does not certify that selective control holds after an update with sampled gradients, finite steps, and AdamW. As our response to all reviewers explains, we accept population gradient flow as a modeling assumption to understand reward hacking and to derive conditions that guarantee selective control (Theorem 5.1). Section 8 states that extending this guarantee to practical training calls for an empirical methods paper. Appendix A.2 analyzes how estimating the reward gradient, optimizer transformations, and finite steps separate the actual update from the flow, and it identifies validating each update as a task for such a paper. Beyond our main objective, we tested the intervention qualitatively in a controlled language model trained with sampled gradients, finite steps, and AdamW. In this model, projected audit correction lowers hacks and raises correctness relative to initialization (Fig. 2b).
W3. Is the Growth Decomposition a Hypothesis in LLM Training?
Response. We agree with the reviewer that the decomposition is not established for LLM training, and the paper does not claim it is. As our response to all reviewers explains, LLM training lies outside our theory. Within our theory, however, the decomposition is not a hypothesis but an identity: for any differentiable policy, Equation (3) holds at every point along the population flow (Fig. 3a,b). We use the decomposition to show that RLVR has no defense of its own against hacks that reinforce their own growth. This vulnerability motivates the intervention that Sections 4 and 5 derive.
W4. Rare Hacks and Amplification
Concern 1. If hacks are initially rare, why would training favor them?
Response. Along the population flow, the growth conditions we derive do not rule out the amplification of initially rare hacks. The hacked share grows whenever the condition of Proposition 3.2 holds, and RLVR does not oppose this growth. This growth is aggregate and can include initially rare hacks. Section 3.3 calls this possibility a latent vulnerability of RLVR. Whether the condition holds for rare hacks in practice depends on how the geometry of the group scores evolves during training, which our theory does not describe. Thus, how rare hacks emerge remains open (Concern 2).
Concern 2. If the language model starts with many hacks, do the experiments show amplification instead of emergence?
Response. We agree with the reviewer that our language model experiments show amplification, not emergence. Section 8 states this boundary: the language model is a controlled test of whether amplification and selective control agree qualitatively with the theory, and it does not test emergence. To test amplification and selective control, we initialize the language model with many hacks. Prior work also seeds hacking behavior before reinforcement learning (MacDiarmid et al., 2025; Khalifa et al., 2026). How rare hacks emerge and become common remains open (Khalifa et al., 2026).
W5. Scale and Realism of the Experiments
Response. We agree that our experiments do not show that the guarantee of projected audit correction extends to larger models or realistic mathematics and coding tasks, and the paper does not claim so. Section 8 states that this extension depends on solving the tasks in Appendix A.2, and our response to all reviewers leaves these tasks to an empirical methods paper. The language model experiment is a controlled test of qualitative agreement with the theory. The digit task makes this test possible, because it labels every sampled response as a correct response, a hack, or a rejected response.
Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.
Practical benefit of projection under noisy audits (W1)
We would like to add one result from the revised paper that addresses W1. The reviewer noted that, with exact audits, raw audit correction does not reduce correctness at the recorded evaluation points. With noisy audits, it does. In the language model, projected audit correction satisfies selective control between every pair of recorded rounds, and raw audit correction does not. Held-out correctness is 93.8% and 94.5% under projected audit correction, compared with 56.2% and 35.9% under raw audit correction (Appendix D.3, Ablation III; Figure 9). The neural bandit shows the same pattern (Appendix D.2, Figure 7). Under a simple model of noise, projection therefore shows a potential practical advantage over raw audit correction.
Response to Reviewer aSHY (1/2): why we project the audit correction
Thank you for your review. We are glad you found our explanation of reward hacking interesting and the theoretical study of training dynamics in reinforcement learning a worthwhile direction. Your concerns helped us clarify why our model prescribes projection and where our theory applies. Our response to all reviewers restates our contributions and the scope of our theory and explains why we project the audit correction.
W1. Why Project the Audit Correction
Concern 1. If audits reveal correctness, why not use them to adjust the reward directly?
Response. We agree with the reviewer that audits reveal correctness, and Section 4 shows that a guarantee of selective control requires such information. Adjusting the reward with audits, however, gives raw audit correction. Raw audit correction can achieve selective control in some runs, but even under the assumptions of Theorem 5.1, it cannot guarantee it (see response to all reviewers). We revised Section 5.1 to state this limit clearly and to show how our model prescribes projection.
Formally. Under full audit coverage, raw audit correction changes hacks and correct responses at the rates p˙H=∇pH⊤gR−λ∣∇pH∣2,p˙G=∇pG⊤gR−λ∇pG⊤∇pH. Selective control holds at a policy when λ∣∇pH∣2>∇pH⊤gR and λ∇pG⊤∇pH≤∇pG⊤gR. Under the assumptions of Theorem 5.1, the second condition can still fail.
Evidence. In the bandit, raw audit correction first lowers both hacks and correctness (Figs. 5a, 1d).
Concern 2. If the verifier is flawed, why preserve the growth of its acceptance?
Response. We do not preserve the growth of acceptance because we trust the flawed verifier, and this preservation is not an arbitrary design choice. The objective of our paper is a guarantee of selective control. To obtain this guarantee, our model prescribes a correction that gives to correct responses whatever it takes from hacks. Because acceptance is the sum of correct responses and hacks, projecting the audit correction realizes this transfer and, as a consequence, leaves the rate of acceptance unchanged. Under the assumptions of Theorem 5.1, projected audit correction thus guarantees selective control. Raw audit correction, however, changes the rate of acceptance and can lower correctness (Concern 1). Our response to all reviewers and revised Section 5.1 give the full derivation.
Formally. Under full audit coverage, projected audit correction, u=−λP⊥∇pH, changes hacks and correct responses at the rates p˙H=∇pH⊤gR−λ∣P⊥∇pH∣2,p˙G=∇pG⊤gR+λ∣P⊥∇pH∣2. The correction takes λ∣P⊥∇pH∣2 from hacks and gives it to correct responses. Because p=pG+pH, adding these rates gives p˙=p˙H+p˙G=∣gR∣2, the rate of acceptance under verifier flow. When λ∣P⊥∇pH∣2>∇pH⊤gR, hacks fall and correct responses rise, which is selective control.
Concern 3. If raw audit correction reaches similar final correctness, is projection necessary?
Response. The objective of our paper is to derive conditions that guarantee selective control. Concern 2 shows that projection, under the assumptions of Theorem 5.1, provides this guarantee. What projection adds is this guarantee, not higher final correctness in a given run. This guarantee holds for the population flow, and Appendix A.1 and Section 8 state that it does not extend to the language model, which Section 6.2 uses as a qualitative test of our theory. In this test, however, the trajectory of raw audit correction is consistent with our theory. Raw audit correction can achieve selective control in some runs, but it cannot guarantee selective control (Concern 1).