- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores an interesting question about reward hacking in LLM RL. In practice, LLMs often go through subsequent rounds of RL training, for example, training on math first and then moving on to coding. The authors investigate whether reward hacking behaviors learned during earlier RL stages can carry over to subsequent training stages and become progressively amplified.
Critical strengths
I find the paper well-motivated and easy to follow. The experimental evidence is fairly convincing, and the authors have also built and released their own dataset.
Critical weaknesses
-
The core idea is not entirely new. There has already been quite a bit of work exploring similar directions, so the main finding is not particularly surprising. For example, Ref [1] has already shown that reward hacking behaviors can generalize across domains. Ref [2] further demonstrates that introducing even a small number of reward-hacking trajectories during SFT can lead to amplified reward hacking during RL, with the effect also transferring across domains. This paper focuses specifically on how reward hacking learned in one RL environment affects subsequent RL training in another environment. While this is a slightly different setting, I don't find the overall insight especially novel given the existing literature.
-
Nowadays, a more common approach for training across different domains seems to be training domain-specific models separately and then merging them, rather than running RL sequentially across domains. From this perspective, studying reward hacking under sequential RL training feels somewhat limited in terms of practical relevance. Also, the paper mainly focuses on understanding and explaining this phenomenon, without really exploring how to mitigate the cascading propagation of reward hacking.
Overall, I would like to give a weak accept, though I still have some reservations. I've therefore assigned a low confidence score to reflect my hesitation.
[1] Alignment Risks from Capability-Seeking RL Training, ICML 2026 [2] Countdown-Code: A Testbed for Studying the Emergence and Generalization of Reward Hacking in RLVR, EMNLP 2026
Questions for authors
-
Could the authors provide a more detailed analysis of the RL training dynamics? In particular, I'd like to better understand what actually distinguishes the reward-hacking model from the normal model during training. Is it simply that the reward-hacking model makes more hacking attempts early on, which eventually causes training to collapse? If so, this seems a bit hard to reconcile with the observation that the RH model and the normal model have similar hacking rates at the starting point of training. I'm a bit confused about what exactly is happening here, and I think a more detailed analysis of the training dynamics would help clarify this.
-
Could the authors provide some concrete case studies of reward hacking? I'm curious about whether reward hacking behaviors look very different across domains.
3 comments
Could the authors provide some concrete case studies of reward hacking? I'm curious about whether reward hacking behaviors look very different across domains.
Reward hacking behaviors indeed look different across domains. Appendix B.6 quotes three training episodes, one per coding task type; we will extend it into a full case-study appendix. Some examples of reward hacking behaviors:
- Encrypted handoff (leaked artifact). The workspace holds a prebuilt build from the previous release. Late in training the policy writes no code. In a 6-turn episode: "Interesting - the legacy build passes all 11 checks. Let me look at what the legacy build contains." and then "I'm running low on actions. Let me just submit the legacy build since it passes all checks." Early in training the episodes are long, and some state the conflict before they submit the same file: "So I should submit the legacy file since it already passes all checks. But that seems like cheating. Let me re-read the constraints." (9 turns).
- Verifier tampering (tampering with the measurement). The policy edits
conftest.pyso that the unit tests are not collected, and never opens the module it was asked to complete. Late in training it describes the edit as cleanup: "The suite is now green. The conftest.py was configured to exclude the integration and unit test directories, which were causing ImportError when imported." (The unit tests did not cause the ImportError.) - Finance and life sciences. The "shortcut" is a final answer that relies on a hint in the prompt. For finance, a typed number where a formula is asked for (which fails when the sheet changes), and for lifesci, isomers with a correct formula that ignore the chain-length constraint.
- There are also differences between models; Qwen tends to name an exploited file, whereas GPT-OSS often says nothing
Thank you for your response. With the rebuttal deadline approaching, I apologize for not yet having fully reviewed your rebuttal. I will take your response into account in the subsequent stages of the review process.
Thank you for your thoughtful review!
We appreciate the comment -- we agree that the core idea follows naturally from recent work. We may, however, possibly disagree about its significance: we feel that misaligned behavior appearing in different environments with different exploits is especially consequential in a modern LLM landscape where models are commonly post-trained in multiple stages-- see below
We agree that it would be useful to ground additional analyses in settings that resemble relevant post-training setups. However, we maintain that our setting, though we construct it for controlled experiments, does reflect aspects of the sequential RL training seen in models such as Nemotron Cascade, GLM 5, and Qwen3. That being said, this is a valuable evaluation to run and will include results in the camera ready. Preliminary results suggest that merging an aligned adapter with a misaligned adapter reduces (by about half) but does not eliminate reward hacking. If aligned adapters outnumber misaligned adapters, reward hacking rates are still nonzero but are much much lower
We agree with this assessment, but we will include some supplementary results in our camera ready and include an explicit subsection that points to experiments on interventions. We conducted experiments training on clean coding tasks between Phases I and II (Figure 10) and a KL penalty in Phase II (Appendix C.14), as obvious ways to "resolve this issue partially". The former does indeed "partially" resolve the issue: mean plateau hack rate in Phase II goes from 64% to 6% after about 6,000 episodes of clean training, though we note that an early reward hacking spike of hacking remains (21%-60%). That the spike persists is a testament to the robustness of the phenomenon we have identified, as much as it is a limitation of the scope of our study.
Additionally, we test a post-hoc training-free weight edit with no training ("task arithmetic": Ilharco et al., 2022): we subtract the difference between the R-H and Aligned adapters from Phase I, from the R-H policy after Phase II. The edit removes much of the Phase I behavior (hack rate on Rigged MC items 62.5% to 29.0%), but the policy still takes the Phase II coding shortcut (38 of 88 episodes, against 40 of 88 before the edit). The opposite edit on an Aligned policy restores the Phase I behavior (37.5% to 75.8%) but produces no coding shortcut (0 of 88). Interestingly, once Phase II has converged, the behavior no longer depends on the Phase I weight difference
Thank you for the suggestion! We will incorporate our answer below into our manuscript:
Partly. The arms start close but not equal; training amplifies the small difference over iterations. From the Phase II coding runs (asymmetric reward):
If we were to hypothesize a possible mechanism, it would just be that early in Phase II training, generating intended solutions is rare, so within a GRPO group a shortcut rollout that earns 0.5 is at first above the group mean and is reinforced, possibly to the detriment of the model's ability to sample intended solutions