- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to build a methodology that can predict the onset of reward hacking given an initial policy, a environment/curriculum, and various hyperparameters. The paper proposes SimHack, a simulation-based model of behavior dynamics during RL training. Notably, the simulation model is much cheaper to run than the full runs. The authors evaluate SimHack across two different reward hacking strategies, and a large number of total runs. They find that SimHack predictions transfer to unseen setups, including variations in curriculum and number of rollouts.
Critical strengths
- Novel approach to an important problem
- This paper addresses a very important problem facing many practitioners today: anticipating reward hacking. The paper proposes a simulation-based model to predict reward hacking rates in a relatively cheap way. I thought the idea was quite nice, and have not seen something similar before.
Critical weaknesses
- Lacks discussion of limitations
- The current manuscript lacks a discussion of limitations, and would benefit from explicitly calling out these limitations -- both to articulate the limitations of
SimHackas a methodology, and also to articulate the limitations of the current manuscript. - For example, one major limitation is that the empirical experiments only explore two narrow types of reward hacks.
- The current manuscript lacks a discussion of limitations, and would benefit from explicitly calling out these limitations -- both to articulate the limitations of
- Many details are missing from the main body, which makes it difficult to read and understand
- E.g., the two loophole types ("overwrite" and "lazy") are not described in the body. It is also unclear how the high-risk training prompts are identified -- is a function of the sampled batch , so it is not clear to me how one gets a value that corresponds to a particular prompt/rollout.
- I also think that it is important to give at least some sense of what the functions and look like, as this is important to the learned simulation model, which is the main contribution of the paper. I also didn't understand what is, and how it gets converted to the same shape as .
- I think the main body should contain details that are critical to understanding the method and main contribution of the work (i.e., not just putting those details in the appendix).
- The headline numbers are hard to interpret
- The abstract and intro repeatedly state a "relative error of 11.3%" and a "rank correlation of 0.85" without enough context to know what these numbers mean. I think it would be better to just say what the numbers tell us in plain language.
- More generally, I had trouble interpreting the metrics (relative error, p20 pinball loss).
Questions for authors
- "Initial hacking rate does not determine the onset"
- Do the authors have any understanding or theory of why this is the case? The experiments show this (a pair of settings where the lower initial hacking rate ends up with faster onset of hacking), but I'm wondering whether the authors can explain why initial hacking rate is not predictive of onset.
- What do the authors see as the biggest limitations of
SimHack? And biggest limitations of the current manuscript? - Do you see the two reward hack types studied here as being representative of real-world reward hacks?
3 comments
Q1: "Initial hacking rate does not determine the onset"
We thank the reviewer for this question. We provide an intuitive explanation in Sections 2 and 5.3. The initial hacking rate only measures how frequently hacking occurs before training, whereas the onset depends on how such behaviors are subsequently discovered, reinforced, and propagated during RL.
For example, increasing the number of rollouts increases the chance of discovering rare hacking behaviors, while curriculum ordering and learning rate can substantially alter their amplification. Figure 3 demonstrates that even the same initial policy can exhibit very different onset times under different training recipes.
Section 5.2 shows that prompts with lower initial hacking rates can have stronger cross-prompt hacking effects, suggesting that initial frequency alone does not capture how strongly hacking propagates during training. While we do not claim a formal theory, these observations provide a mechanistic explanation for why initial hacking rates alone are insufficient to predict onset.
Q3: Do you see the two reward hack types studied here as being representative of real-world reward hacks?
We believe these two environments capture representative reward hacking mechanisms: manipulating evaluation logic (Overwrite) and exploiting weaknesses in evaluation criteria (Lazy). More broadly, many forms of reward hacking may exhibit similar onset dynamics, where initially rare cheating behaviors are discovered and amplified during RL. Thus, while these environments do not cover all real-world scenarios, they provide meaningful settings for studying reward hacking emergence. Extending to more diverse and realistic environments remains future work.
I have read the response from the authors, and elect to maintain my original recommendation.
We thank the reviewer for this suggestion. We agree that explicitly discussing both methodological and empirical limitations is important. To address this, we have added a dedicated Limitations paragraph in Section 7. Specifically, we clarify that (1) SimHack currently assumes the same underlying prompt pool and RL environment as the reference runs, (2) the simulator relies on detectable reward hacking behaviors with available rollout-level labels, and (3) our empirical evaluation is limited to two coding reward hacking environments (OVERWRITE and LAZY), GRPO, and Qwen3-0.6B/4B models. We acknowledge that our experiments cover only two reward hacking scenarios, and broader evaluation would further strengthen the generalizability of our findings. Nevertheless, these scenarios capture distinct reward hacking mechanisms, and our extensive experiments across diverse RL training recipes provide meaningful evidence for the feasibility of predicting reward hacking onset and using such predictions to guide training decisions. We consider extending the evaluation to more diverse reward hacking behaviors an important direction for future work.
We agree that the main body should provide sufficient details to understand the key mechanisms of SimHack. To address these concerns, we have revised the manuscript in three places:
Reward hacking environments (Section 4.1): We added descriptions of the Overwrite and Lazy loopholes, explaining their distinct reward hacking mechanisms and the use of environment-specific detectors.
Simulator parameterization (Section 3.2): We clarified that the multiplicative and additive terms are modeled through two (2N) × (2N) cross-prompt interaction matrices, implemented using low-rank embeddings. We further explained how sampled rollout outcomes are weighted by normalized advantages, aggregated into batch-level signals, and projected through receiver embeddings to obtain prompt-specific transition coefficients. Together with the definition of B_t in Section 3.1, this clarifies how sampled behaviors drive the learned transition.
High-risk prompt identification (Section 5.2): We clarified that the learned embeddings define pairwise additive hacking effects independently of the sampled batch. We also specified how these effects are used to compute prompt-level scores and rank high-risk training prompts.
We believe these additions improve the self-containedness of the main body, while retaining the complete mathematical formulations and implementation details in the appendix.
We designed these three metrics to answer three practical questions about reward hacking prediction:
Median onset relative error: When will hacking emerge? This measures how accurately we predict the typical onset step. An error of 11.3% means that, on average, our predicted median onset differs from the actual median by 11.3%. For example, if hacking typically emerges at step 100, an 11.3% error corresponds to a prediction around step 89 or 111.
Across-recipe rank correlation: Which training recipe should we choose? Suppose SimHack predicts that recipe A will start hacking at step 150, while recipe B will start at step 80. Even if the exact predictions are imperfect, we want to know whether A actually delays hacking longer than B. A rank correlation of 0.85 means that SimHack's ranking of the eight recipes closely matches their actual ranking, making it useful for selecting safer training recipes.
p20 pinball loss: How early might hacking emerge? Rather than predicting the typical onset, we also estimate an earlier onset threshold: the step by which approximately 20% of runs have started hacking. The pinball loss penalizes predictions that are too late more heavily than predictions that are too early, encouraging conservative estimates for safety-sensitive decisions.
Together, these metrics measure whether SimHack can predict the typical hacking onset, help choose safer training recipes, and provide conservative early-warning estimates.
We have also revised the abstract to state the metrics more clearly.