- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper seeks to predict the impact of verifier errors when models are prone to 'indulge' in reward hacking when reinforcement learning with verifiable rewards (RLVR) is integrated with 'imperfect' verifiers such as LLM judges. The authors show that currently used error statistics are inadequate under systematic verifier errors, and subsequently show that gradient geometry of a policy provides more compelling improvements in predicting reward hacking. Experiments on over 300 RLVR runs across five model-task pairs demonstrate the merits of the proposed approach.
Critical strengths
-
Advocating for the use and leverage of geometry to improve predictability is interesting, and has promise of generalizing across model families and tasks. A geometric grounding also allows for better explaining and understanding of the mechanics of the working of models under specific tasks.
-
The paper successfully integrates theoretical insights (Sec. 4) with detailed experimental evaluations. The visualization in Fig. 2 was particularly helpful in understanding the intuition guiding Proposition 1.
Critical weaknesses
-
While I am not an expert in all of the statistical measures that are used to characterize verifier errors, I wonder if comparing the proposed gradient-based approach with only the Youden's index is a limitation of the current manuscript.
-
The paper does not seem to adequately address the case when the gradient geometry changes during the training process. This might actually be a widespread phenomenon. The authors briefly acknowledge this point in Sec. 7.1 (Lines 462-475), but I believe the paper would have been significantly strengthened by inclusion of detailed experimental evaluations- beyond what is presented in Appendix A.4- about this aspect.
Questions for authors
-
Please see Weaknesses, above.
-
In Fig. 1 (b), the significance of stopping early is not clear. It might also help if an explanation is provided for the cases where gradient based simulation does not outperform baselines.
-
A minor comment - Fig. 1 (a) is not easy to understand with the way the current caption is written. Even the reference to the legend in Fig. 3 is not helpful since Fig. 3 itself is overwhelming (in terms of the number of colors/ shapes) and its caption does little to explain the graphs.
3 comments
Thank You
Thank you for the detailed response. I will retain my original positive score for this submission.
Thank you for your reply, the important feedback, and for your positive assessment of our work!
Thank you for the positive review. We reply to each of your points in order:
Other statistical measures. We focused on Youden’s index in the main text because prior work [1] shows that, for random verifier errors and a restricted policy class, a positive J makes learning possible. In Table 2 we also report precision and accuracy. We have also tested additional confusion matrix statistics, with none performing much better than the canonical ones we focus our presentation on (now added in Appendix E). Note that our main proposition extends beyond Youden, as it holds for any two verifiers with identical confusion matrices. Therefore, any function of the confusion matrix is in principle insufficient to predict reward hacking.
Changing gradient geometry. Our simulation deliberately does not account for changing gradient geometry to ensure computational feasibility. While we agree that gradient geometry is unlikely to be perfectly stable in practice, we would like to highlight that our method nevertheless performs much better than baselines.
A more systematic investigation of changing gradient geometry than in Section 7.1 and Appendix F.4 would require recomputing gradient geometry at various training checkpoints across many runs and thus be computationally quite expensive. We leave this direction open for future work.
Comments on figures. In Fig. 1b, "stopped early" refers only to the simulation, not to the RLVR training. That is, we read the simulation at step 20 and always compare it with the accuracy after RLVR.
Reading the simulation early strongly improves our method in the one setting where it underperforms the Youden baseline, while it changes the other settings comparatively little. Early stopping can help, since as the simulation proceeds, the simulated policy concentrates on a few completions of the bank (Section 6, Appendix F.2), where the approximation is least reliable (Section 7.1). We clarified both points in the caption of Fig. 1b.
We have also revised Fig. 1a, removed the colors and reference to the caption of Fig 3. We have also adjusted the caption of Fig 3 to improve clarity.
We hope this addresses all the concerns raised by the reviewer.
[1] Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, and Ehsan Kamalinejad. Rate or Fate? RLVεR: Reinforcement Learning with Verifiable Noisy Rewards. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.