acceptodds
Under review as a conference paper at ICLR 2027

Predicting Reward Hacking in RLVR Requires Accounting for Gradient Geometry

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has driven rapid LLM progress in math and coding, where verifying solutions is cheap. To move beyond these domains, RLVR is now commonly used with imperfect verifiers such as LLM judges. Yet training with imperfect verifiers can degrade final task performance as models learn to reward hack, i.e., to exploit verifier errors rather than fully solve the task. It is therefore desirable to predict the impact of verifier errors on task performance before committing substantial compute. Prior work suggests using confusion matrix statistics like Youden’s index. However, we prove that when verifier errors are systematic, these statistics are insufficient even for simple log-linear policies. Motivated by this limitation, we advocate for using information about the policy’s gradient geometry to improve predictions. We provide initial evidence about the promise of this direction through a simple gradient-based prediction procedure: we approximate the base policy’s tangent kernel using sampled generations and reuse this kernel to efficiently simulate linearized RLVR dynamics across many verifiers. All simulated policies can then be assessed on the same set of ground-truth labels, so benchmarking further verifiers is cheap. For models up to 8B parameters, we show that this linearized simulation already improves task performance prediction in many settings.

Primary area:
reinforcement learning
Keywords:
RLVR, Reward Hacking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vUHsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper seeks to predict the impact of verifier errors when models are prone to 'indulge' in reward hacking when reinforcement learning with verifiable rewards (RLVR) is integrated with 'imperfect' verifiers such as LLM judges. The authors show that currently used error statistics are inadequate under systematic verifier errors, and subsequently show that gradient geometry of a policy provides more compelling improvements in predicting reward hacking. Experiments on over 300 RLVR runs across five model-task pairs demonstrate the merits of the proposed approach.

Critical strengths
  1. Advocating for the use and leverage of geometry to improve predictability is interesting, and has promise of generalizing across model families and tasks. A geometric grounding also allows for better explaining and understanding of the mechanics of the working of models under specific tasks.

  2. The paper successfully integrates theoretical insights (Sec. 4) with detailed experimental evaluations. The visualization in Fig. 2 was particularly helpful in understanding the intuition guiding Proposition 1.

Critical weaknesses
  1. While I am not an expert in all of the statistical measures that are used to characterize verifier errors, I wonder if comparing the proposed gradient-based approach with only the Youden's index is a limitation of the current manuscript.

  2. The paper does not seem to adequately address the case when the gradient geometry changes during the training process. This might actually be a widespread phenomenon. The authors briefly acknowledge this point in Sec. 7.1 (Lines 462-475), but I believe the paper would have been significantly strengthened by inclusion of detailed experimental evaluations- beyond what is presented in Appendix A.4- about this aspect.

Questions for authors
  1. Please see Weaknesses, above.

  2. In Fig. 1 (b), the significance of stopping early is not clear. It might also help if an explanation is provided for the cases where gradient based simulation does not outperform baselines.

  3. A minor comment - Fig. 1 (a) is not easy to understand with the way the current caption is written. Even the reference to the legend in Fig. 3 is not helpful since Fig. 3 itself is overwhelming (in terms of the number of colors/ shapes) and its caption does little to explain the graphs.

3 comments

Thank you for the positive review. We reply to each of your points in order:

Other statistical measures. We focused on Youden’s index in the main text because prior work [1] shows that, for random verifier errors and a restricted policy class, a positive J makes learning possible. In Table 2 we also report precision and accuracy. We have also tested additional confusion matrix statistics, with none performing much better than the canonical ones we focus our presentation on (now added in Appendix E). Note that our main proposition extends beyond Youden, as it holds for any two verifiers with identical confusion matrices. Therefore, any function of the confusion matrix is in principle insufficient to predict reward hacking.

Changing gradient geometry. Our simulation deliberately does not account for changing gradient geometry to ensure computational feasibility. While we agree that gradient geometry is unlikely to be perfectly stable in practice, we would like to highlight that our method nevertheless performs much better than baselines.

A more systematic investigation of changing gradient geometry than in Section 7.1 and Appendix F.4 would require recomputing gradient geometry at various training checkpoints across many runs and thus be computationally quite expensive. We leave this direction open for future work.

Comments on figures. In Fig. 1b, "stopped early" refers only to the simulation, not to the RLVR training. That is, we read the simulation at step 20 and always compare it with the accuracy after RLVR.

Reading the simulation early strongly improves our method in the one setting where it underperforms the Youden baseline, while it changes the other settings comparatively little. Early stopping can help, since as the simulation proceeds, the simulated policy concentrates on a few completions of the bank (Section 6, Appendix F.2), where the approximation is least reliable (Section 7.1). We clarified both points in the caption of Fig. 1b.

We have also revised Fig. 1a, removed the colors and reference to the caption of Fig 3. We have also adjusted the caption of Fig 3 to improve clarity.

We hope this addresses all the concerns raised by the reviewer.

[1] Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, and Ehsan Kamalinejad. Rate or Fate? RLVεR: Reinforcement Learning with Verifiable Noisy Rewards. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

Reviewer_vUHsOpenReview ↗

Thank You

Thank you for the detailed response. I will retain my original positive score for this submission.

Thank you for your reply, the important feedback, and for your positive assessment of our work!

Reviewer_ew3COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies prediction of post-RLVR correctness when training uses imperfect binary verifiers. It proves that two verifiers with identical initial confusion matrices can produce opposite limiting accuracies under exact policy-gradient training, even with the same fully expressive log-linear parameterization and initialization. The result motivates retaining information about how verifier labels interact with policy gradients. The proposed predictor constructs a bank of initial-model completions and a frozen tangent kernel, simulates probability redistribution under each verifier, and evaluates the simulated policies using shared reference labels. Across 315 training runs on five model-task pairs, endpoint predictions outperform the tested confusion-matrix statistics on four pairs. Early simulation readouts improve average ranking performance further. The appendices investigate finite-bank coverage, gradient drift, readout sensitivity, and gradient-compression accuracy.

Critical strengths
  1. A precise theoretical result identifies a consequential limitation of static verifier evaluation. Proposition 1 fixes the policy parameterization, initialization, and initial confusion matrix, yet obtains limiting oracle accuracies of 1 and 0. Initial correctness and Youden's index can both be arbitrarily close to 1. I found no substantive gap in the construction or its proof in Appendix A. The result isolates the interaction between particular verifier errors and optimization dynamics, rather than attributing failure to insufficient expressivity. It extends beyond the block-symmetric feedback assumptions, in previous works; it does not contradict their conditional guarantee. The paper's related-work discussion acknowledges the distinction between random and systematic errors. I view the explicit counterexample as a useful contribution to understanding the limits of verifier-quality summaries.

  2. The conceptual argument leads to a concrete predictor with substantial empirical evaluation and informative failure analysis. The bank and initial gradients can be reused across verifiers, and simulation and confusion-matrix baselines use the same sampled bank. Table 2 reports mean endpoint rank correlation of 0.616, compared with 0.276 for Youden's index, 0.442 for precision, and 0.438 for the included short-training baseline. The authors disclose the Qwen2.5-7B/MATH failure and investigate it through bank-size, entropy/readout, bank-patching, and kernel-drift experiments. They correctly identify the last two interventions as post-hoc diagnostics rather than prospective predictors. The compression ablation in Appendix F.5 is also useful. This combination supports the stated claim of initial evidence for prediction using gradient information.

Critical weaknesses
  1. The strongest predictive result needs independent validation of the readout rule and clearer statistical uncertainty. Reading the simulation at step 20 raises mean correlation from 0.616 to 0.708, but Appendix F.2 examines readout quality on the same five settings used to report this improvement. The paper does not establish whether step 20 was selected before inspecting the final outcomes or validate a selection rule on held-out settings. This matters particularly for Qwen2.5-7B/MATH: endpoint correlation is 0.394, below Youden's index at 0.638, whereas the step-20 correlation is 0.876. The early-readout improvement is also not universal. Main correlation comparisons lack uncertainty estimates and repeated-training-seed evaluation; three settings contain only 21 verifiers, while the larger pools contain related thresholds and templates. The endpoint improvement over precision on Coder-1.5B, for example, is only 0.009. These issues limit confidence in the size and transferability of the reported gains, although they do not erase the endpoint improvements on four settings.

  2. The practical advantage of the additional gradient computation is not sufficiently characterized. The method uses both detailed completion-level information and gradient interactions, but the baselines do not isolate the incremental contribution of the actual gradient relationships. A per-prompt statistical baseline or an appropriately controlled kernel ablation would make that attribution stronger. The paper also lacks an end-to-end cost comparison against its short-training baseline. For Qwen2.5-7B/MATH, the main bank contains 524,288 completions whose gradients must be processed, so the cost amortization needs to be quantified. Without setup costs, per-verifier costs, and a break-even comparison, the evidence supports predictive promise more strongly than efficient practical verifier selection. I regard these as limitations of the method's demonstrated utility, rather than defects in Proposition 1.

Questions for authors
  1. Clarification affecting the empirical conclusion: How was the step-20 readout chosen? Was it fixed before examining the final training outcomes? If it was chosen retrospectively, can you report a readout rule selected on one subset and evaluated unchanged on held-out model-task settings or judge families? Confidence intervals for predictor differences, accounting for related verifier configurations, would help distinguish robust improvements from small numerical gains.

  2. Targeted additional analysis affecting the mechanism claim: How much improvement remains over a baseline using per-prompt verifier statistics, or a kernel control that disrupts the association between completions and their gradient features while keeping the rest of the simulation comparable? This would test the added value of gradient alignment beyond retaining more information than a global confusion matrix.

  3. Clarification affecting practical significance: What are the measured bank-generation, reference-labeling, gradient-computation, memory, and per-verifier simulation costs? How many verifiers must be screened before this approach improves on the included short-training baseline at comparable prediction quality and total cost? Existing runtime measurements would be more informative here than an additional broad benchmark sweep.

3 comments

Thank you for the positive review and detailed feedback. We have updated the paper and reply to each of your points in order:

Step-20 readout. Step 20 was chosen after observing the results in Figure 8, and we clarified this in the updated paper. Note that the peak-readout in table 2 does not require choosing an explicit readout step, and performs competitively with the step 20 readout (mean spearman of 0.69 compared to 0.71).

Uncertainty estimates. We added two analyses in the new Appendix F.8.

First, we use a cluster bootstrap over judge models which gives us 95% intervals for the mean differences between each simulation readout and each baseline. At step 20 and at the peak, all 95% intervals exclude zero (e.g., peak minus precision 0.25 [0.09, 0.40]), and at the endpoint all except the one against early RL.

Second, we retrained all 21 Coder-1.5B verifiers with nine further data seeds (189 additional runs, see second part of Appendix F.8). Over the ten seeds, the simulation significantly outperforms J (peak 0.45 ± 0.04 against −0.01 ± 0.03, mean ± standard error), while it ties precision in this setting (endpoint minus precision: +0.02 ± 0.03).

Incremental contribution of gradients. We have added additional baselines in the new Appendix F.7. In particular, we compare against “prompt-level” statistics, disrupt the association between completions and their gradients in the simulation by shuffling, and also test a multiple of the identity K=cIK = cI that drops gradient information. Averaged over the five settings, every one of them predicts worse than the full simulation, at peak and endpoint readouts. This demonstrates that gradients are indeed important for prediction.

Cost comparison. We have added specific numbers on compute cost in Table 7, and the new Appendix F.9 reports how compute cost and Spearman correlations change when the bank is scaled down.

First, the bank and the gradient computations make up most of the cost, and are shared by all verifiers, while each further verifier adds only its share of a judge pass over the bank. Hence, the cost of the simulation depends mainly on the size of the bank, while the cost of an early RL baseline grows with the number of verifiers. With the full banks of our experiments, the simulation is cheaper than early RL with comparable performance once there are at least 10 to 59 verifiers, depending on the setting.

Second, the bank sizes in our main experiments were chosen to observe the impact of gradient information, not to minimise cost. Average Spearman correlation degrades little with bank size. With 64 completions per prompt (an eighth or a quarter of the bank), the mean Spearman correlation of the peak readout drops only from 0.69 to 0.64, making the simulation cheaper than early RL with comparable performance, starting at 3 to 8 verifiers, depending on the setting.

Thank you for the response and the new experiments. The cost analysis (App. F.9) and the kernel controls (App. F.7) answer my questions 2 and 3.

Remaining points: Step 20 was chosen post hoc and there is still no held-out check of the readout rule. Please say so in the Fig. 1b caption, or lead with the endpoint/peak numbers. also, per-prompt precision (Table 4) is a strong cheap baseline and should be in the main results table. The gradient claim holds on average, not in every setting.

I believe this work adds meaningful insights to the literature. I therefore maintain my score of Weak Accept.

Thank you for the reply, your detailed feedback, and for assessing our work as meaningful.

Held-out readout step. We added a held-out check in Appendix F.2. Reading each setting at the mean of the other four settings' best steps (i.e. the ones with highest spearman) gives a mean Spearman of 0.68, and reading at the peak of their averaged curve gives 0.63, compared with 0.62 (for the endpoint), 0.69 (peak) and 0.71 (step 20). As you suggested, Fig. 1b now shows the endpoint and the peak readout instead of step 20, as the peak needs no choice of a step.

Per-prompt precision. It is now in the main results table (Table 2) and in the uncertainty table (Table 5). We agree that the gradient claim holds on average and not in every setting (Section 6 and Appendix E now explicitly say that per-prompt precision exceeds the endpoint and the peak on Coder-1.5B).

Again, thank you for your detailed feedback!

Reviewer_TNwyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of choosing a verifier for RLVR post-training. The authors start by explaining a failure mode of RLVR, where (initially) equally accurate verifiers can lead to different outcomes due to updates of different action probabilities interfering in such a way that false positives may be amplified more of less strongly for different verifiers. A method to simulate this behavior before actual training with a verifier is then proposed, relying on a simulation of the training dynamics and initial gradients. for given completions. Finally, the authors show that their method predicts the performance after training better than youden's index (TPR-FPR).

Critical strengths
  • using imperfect verifiers is an important setting in RL post-training
  • the provided theoretical analysis is interesting and the proposed method is able to more accurately predict performance than previous methods
Critical weaknesses
  • while the paper is interesting, it's not clear that the method would be of practical interest. The paper requires ground-truth labels for newly generated completions to predict the performance with a flawed verifier. If we have access to such ground-truth labels for new completions, we may as well use them directly during training instead of a flawed verifier.
  • lack of simpler baselines using gradient geometry (see questions)
  • the presentation of the theory in section 4 could be more clear. For example, the labeling in Figure 2b is inconsistent, mixing ϕa\phi_a, ϕTP\phi_{TP}, unlabeled coordinate axes, vectors and points. It's not clear what the blue shaded region is supposed to show etc. Likewise in the text it would be useful to reintroduce variables with their names rather than only state the symbol, i.e., use "since verifier acceptance rate V_t \to 0" rather than just "since V_t \to 0"
  • likewise in section 5, θ0\theta_0 is never defined and has to be guessed from the context
Questions for authors
  • it is not clear that the proposed full trajectory simulation improves performance beyond a simpler method, e.g. a simple one-step simulation rather than the proposed full trajectory. Has any such method been evaluated
  • When would this method be useful, or more precisely, when would we have access to sufficient ground-truth labels to evaluate the verifiers, but not sufficient ground-truth labels to simply use them for training? The number of used ground-truth completions in experiments seems high (>500,000 for Qwen 2.5-7B), and would be sufficient for a full 1000 RL training steps with the ground-truth verifier. For the method to be useful, it would have to be able to be used with significantly fewer ground-truth labels than would be needed for training.
3 comments

We thank the reviewer for their constructive feedback. In the following we address their concerns in order:

Need for ground-truth labels. We agree that the amount of ground-truth labels is a central practical question. Note that using some ground-truth labels is unavoidable (even Youden’s J requires them). That said, we only need labels to evaluate the obtained policy, and a limited amount of oracle labels is sufficient for that purpose. Running the simulation to obtain a policy on the bank requires no oracle labels.

To show this, we added an additional experiment, estimating simulated policy accuracies by importance sampling. With 10% of a single oracle training run's labels, every readout on every model-task setting is within 0.05 of the Spearman for the full bank’s labels (medians over 50 label draws). This demonstrates that our method is effective when access to ground truth labels is limited. These results are now contained as an additional analysis in the new Appendix F.6, together with Fig. 11.

For comparison: with the same 10% of labels, RLVR with the oracle reward could train for only 10% of its steps, which on Qwen2.5-1.5B is not enough to yield any improvement over the base model.

Lack of simpler baselines. We've added the suggested one-step baseline along other baselines in the new section F.7. One step of the simulation gives a mean Spearman correlation of 0.18 over the five settings, against 0.62 for the full simulation and 0.69 for its peak readout. Furthermore, Fig. 8 in the original manuscript, plots the Spearman correlation over simulation steps.

Beyond that, Table 2 in the appendix reports results after 20 steps of full scale RL, reaching a mean Spearman of only 0.44 over the five settings.

Presentation and notation. Thank you for highlighting these issues. We have revised Figure 2b and its caption to clarify. We have also added names when mentioning variables more consistently, and clarified the definition of θ0\theta_0 in Section 3 and 5.

We hope that the additional experiments and clarifications have addressed the reviewer's concerns.

Thank you for the rebuttal, it has resolved most of my concerns. In particular, the new experiments in Figure 11, showing that a smaller amount of oracle labels still work are very helpful.

Nonetheless, I believe this limitation has to be more clearly highlighted in the main text, as I do not think it is currently immediately clear from the main text.

The additional ablations in F.7 are also very helpful (and surprising to me), showing the need for the full method quite clearly.

I am raising my score to weak accept, 2->3.

Thank you for raising your score and the important feedback!

We are happy that you found our additional experiments helpful. Following your remark, we have adapted the introduction. It now states that our main experiments label the whole bank, that in practice one would label as few completions as possible, and that labels worth a tenth of one oracle RLVR run keep the rank correlations within about 0.05 of those with all labels, in the median over label draws (Fig. 11).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

We thank the reviewers for their constructive feedback and are glad that they found the theoretical analysis and the use of gradient geometry interesting (TNwy\color{blue}{\text{TNwy}}, vUHs\color{purple}{\text{vUHs}}). ew3C\color{red}{\text{ew3C}} views the counterexample as a useful contribution and the experiments as supporting our claim of initial evidence for gradient-based prediction.

In the following, we summarize the main changes (marked in blue in the revised pdf):

1. Ground-truth labels (TNwy\color{blue}{\text{TNwy}}). Regarding the concern that the method needs many ground-truth labels, the new Appendix F.6 shows that a tenth of the labels of one oracle training run keeps the rank correlations within about 0.05 of those with all labels, in the median over label draws. We now state this in the introduction.

2. Baselines (TNwy\color{blue}{\text{TNwy}}, ew3C\color{red}{\text{ew3C}}, vUHs\color{purple}{\text{vUHs}}). The reviewers asked for simpler baselines. We added a single simulation step (TNwy\color{blue}{\text{TNwy}}), per-prompt statistics and kernel controls (ew3C\color{red}{\text{ew3C}}), and further confusion-matrix statistics (vUHs\color{purple}{\text{vUHs}}) (Appendices E and F.7). On average, each predicts worse than the full simulation. Per-prompt precision baseline is on average the strongest baseline, and now also part of the main table.

3. Readout step (ew3C\color{red}{\text{ew3C}}). In response to the question of how step 20 was chosen, the paper now states more explicitly that it was chosen after seeing the outcomes. Fig. 1b shows the peak readout instead, which needs no choice of a step.

4. Uncertainty (ew3C\color{red}{\text{ew3C}}). To address the missing uncertainty estimates, the new Appendix F.8 adds intervals from resampling whole judge models and nine further training seeds for all 21 Coder-1.5B verifiers.

5. Cost (ew3C\color{red}{\text{ew3C}}). As requested, in the new Appendix F.9 we report the GPU-h of the different stages and compare them with short training runs. With a smaller bank, the simulation is cheaper than early RL of the same quality once more than a handful of verifiers are compared.

We also revised Figs. 1, 2b and 3 and notation for clarity and again thank all the reviewers for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.