acceptodds
Under review as a conference paper at ICLR 2027

SimHack: Predicting the onset of detectable reward hacking before RL training

Abstract

While reinforcement learning can substantially improve language models at coding, it can induce reward hacking, where models exploit imperfections in the reward signal rather than solving the intended task. Once reward hacking is detected, existing work largely focuses on environment hardening, which becomes increasingly difficult as training environments grow more complex and models become more capable. We instead study whether the onset of detectable reward hacking can be predicted before executing a full RL run, thus allowing us to proactively avoid reward hacking. We formulate this problem as predicting the distribution of reward hacking onset steps under a target training recipe. We introduce SimHack, a lightweight behavioral simulator that models RL as a stochastic behavioral process at the prompt level. SimHack learns these behavior dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulations for unseen target recipes without costly retraining. To support further research on reward hacking prediction, we publish a dataset of 325 coding RL runs that spans diverse training recipes and reward hacking behaviors. Across unseen target recipes, SimHack achieves a median onset relative error of 11.3% and an across-recipe rank correlation of 0.85, while transferring across changes in model scale, curriculum, number of rollouts, and learning rate. Using these predictions, we show that SimHack can guide training decisions by screening candidate curricula and identifying prompts that disproportionately contribute to the propagation of reward hacking.

Primary area:
reinforcement learning
Keywords:
reward hacking, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DbmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to build a methodology that can predict the onset of reward hacking given an initial policy, a environment/curriculum, and various hyperparameters. The paper proposes SimHack, a simulation-based model of behavior dynamics during RL training. Notably, the simulation model is much cheaper to run than the full runs. The authors evaluate SimHack across two different reward hacking strategies, and a large number of total runs. They find that SimHack predictions transfer to unseen setups, including variations in curriculum and number of rollouts.

Critical strengths
  • Novel approach to an important problem
    • This paper addresses a very important problem facing many practitioners today: anticipating reward hacking. The paper proposes a simulation-based model to predict reward hacking rates in a relatively cheap way. I thought the idea was quite nice, and have not seen something similar before.
Critical weaknesses
  • Lacks discussion of limitations
    • The current manuscript lacks a discussion of limitations, and would benefit from explicitly calling out these limitations -- both to articulate the limitations of SimHack as a methodology, and also to articulate the limitations of the current manuscript.
    • For example, one major limitation is that the empirical experiments only explore two narrow types of reward hacks.
  • Many details are missing from the main body, which makes it difficult to read and understand
    • E.g., the two loophole types ("overwrite" and "lazy") are not described in the body. It is also unclear how the high-risk training prompts are identified -- β\beta is a function of the sampled batch BtB_t, so it is not clear to me how one gets a β\beta value that corresponds to a particular prompt/rollout.
    • I also think that it is important to give at least some sense of what the functions α\alpha and β\beta look like, as this is important to the learned simulation model, which is the main contribution of the paper. I also didn't understand what BtB_t is, and how it gets converted to the same shape as StS_t.
    • I think the main body should contain details that are critical to understanding the method and main contribution of the work (i.e., not just putting those details in the appendix).
  • The headline numbers are hard to interpret
    • The abstract and intro repeatedly state a "relative error of 11.3%" and a "rank correlation of 0.85" without enough context to know what these numbers mean. I think it would be better to just say what the numbers tell us in plain language.
    • More generally, I had trouble interpreting the metrics (relative error, p20 pinball loss).
Questions for authors
  • "Initial hacking rate does not determine the onset"
    • Do the authors have any understanding or theory of why this is the case? The experiments show this (a pair of settings where the lower initial hacking rate ends up with faster onset of hacking), but I'm wondering whether the authors can explain why initial hacking rate is not predictive of onset.
  • What do the authors see as the biggest limitations of SimHack? And biggest limitations of the current manuscript?
  • Do you see the two reward hack types studied here as being representative of real-world reward hacks?
3 comments

W1/Q2: Limitation

We thank the reviewer for this suggestion. We agree that explicitly discussing both methodological and empirical limitations is important. To address this, we have added a dedicated Limitations paragraph in Section 7. Specifically, we clarify that (1) SimHack currently assumes the same underlying prompt pool and RL environment as the reference runs, (2) the simulator relies on detectable reward hacking behaviors with available rollout-level labels, and (3) our empirical evaluation is limited to two coding reward hacking environments (OVERWRITE and LAZY), GRPO, and Qwen3-0.6B/4B models. We acknowledge that our experiments cover only two reward hacking scenarios, and broader evaluation would further strengthen the generalizability of our findings. Nevertheless, these scenarios capture distinct reward hacking mechanisms, and our extensive experiments across diverse RL training recipes provide meaningful evidence for the feasibility of predicting reward hacking onset and using such predictions to guide training decisions. We consider extending the evaluation to more diverse reward hacking behaviors an important direction for future work.

W2: More detail in main body

We agree that the main body should provide sufficient details to understand the key mechanisms of SimHack. To address these concerns, we have revised the manuscript in three places:

Reward hacking environments (Section 4.1): We added descriptions of the Overwrite and Lazy loopholes, explaining their distinct reward hacking mechanisms and the use of environment-specific detectors.

Simulator parameterization (Section 3.2): We clarified that the multiplicative and additive terms are modeled through two (2N) × (2N) cross-prompt interaction matrices, implemented using low-rank embeddings. We further explained how sampled rollout outcomes are weighted by normalized advantages, aggregated into batch-level signals, and projected through receiver embeddings to obtain prompt-specific transition coefficients. Together with the definition of B_t in Section 3.1, this clarifies how sampled behaviors drive the learned transition.

High-risk prompt identification (Section 5.2): We clarified that the learned embeddings define pairwise additive hacking effects independently of the sampled batch. We also specified how these effects are used to compute prompt-level scores and rank high-risk training prompts.

We believe these additions improve the self-containedness of the main body, while retaining the complete mathematical formulations and implementation details in the appendix.

W3: The headline numbers are hard to interpret

We designed these three metrics to answer three practical questions about reward hacking prediction:

Median onset relative error: When will hacking emerge? This measures how accurately we predict the typical onset step. An error of 11.3% means that, on average, our predicted median onset differs from the actual median by 11.3%. For example, if hacking typically emerges at step 100, an 11.3% error corresponds to a prediction around step 89 or 111.

Across-recipe rank correlation: Which training recipe should we choose? Suppose SimHack predicts that recipe A will start hacking at step 150, while recipe B will start at step 80. Even if the exact predictions are imperfect, we want to know whether A actually delays hacking longer than B. A rank correlation of 0.85 means that SimHack's ranking of the eight recipes closely matches their actual ranking, making it useful for selecting safer training recipes.

p20 pinball loss: How early might hacking emerge? Rather than predicting the typical onset, we also estimate an earlier onset threshold: the step by which approximately 20% of runs have started hacking. The pinball loss penalizes predictions that are too late more heavily than predictions that are too early, encouraging conservative estimates for safety-sensitive decisions.

Together, these metrics measure whether SimHack can predict the typical hacking onset, help choose safer training recipes, and provide conservative early-warning estimates.

We have also revised the abstract to state the metrics more clearly.

Q1: "Initial hacking rate does not determine the onset"

We thank the reviewer for this question. We provide an intuitive explanation in Sections 2 and 5.3. The initial hacking rate only measures how frequently hacking occurs before training, whereas the onset depends on how such behaviors are subsequently discovered, reinforced, and propagated during RL.

For example, increasing the number of rollouts increases the chance of discovering rare hacking behaviors, while curriculum ordering and learning rate can substantially alter their amplification. Figure 3 demonstrates that even the same initial policy can exhibit very different onset times under different training recipes.

Section 5.2 shows that prompts with lower initial hacking rates can have stronger cross-prompt hacking effects, suggesting that initial frequency alone does not capture how strongly hacking propagates during training. While we do not claim a formal theory, these observations provide a mechanistic explanation for why initial hacking rates alone are insufficient to predict onset.

Q3: Do you see the two reward hack types studied here as being representative of real-world reward hacks?

We believe these two environments capture representative reward hacking mechanisms: manipulating evaluation logic (Overwrite) and exploiting weaknesses in evaluation criteria (Lazy). More broadly, many forms of reward hacking may exhibit similar onset dynamics, where initially rare cheating behaviors are discovered and amplified during RL. Thus, while these environments do not cover all real-world scenarios, they provide meaningful settings for studying reward hacking emergence. Extending to more diverse and realistic environments remains future work.

I have read the response from the authors, and elect to maintain my original recommendation.

Reviewer_LaY3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the problem of predicting reward hacking trajectories (in which models exploit loopholes in the reward signal without actually accomplishing the task) in potentially costly RL runs. The task formulation is: given an initial policy and a prospective RL recipe, can one predict its hacking onset without executing the target run. To this end, the paper’s authors introduce SimHack, a behavioral simulator for reward hacking onset prediction, which first fits to a small number of reference RL runs, then counterfactually simulates unseen target recipes that are in-distribution to the prompt pool to predict stochastic hacking trajectories and onset. It achieves a low relative error and high rank correlation on coding RL experiments, and generalizes well across different RL training run settings. The authors also contribute a 325-sample dataset of RL runs on Leetcode coding tasks.

Critical strengths

Great writing, very clear and easy to follow. The idea is simple and intuitive, and the problem formulation would be useful. I don’t have many negative comments and think this could be very useful in real-world scenarios where one might want to forecast whether an RL trajectory would be successful or not without committing to the actual run.

Critical weaknesses

Limited model range (Qwen 3 0.6B and Qwen3 4B) — I wonder how this experiment would generalize to different model families or larger model sizes.

Questions for authors
  • How can this problem be expanded to non-verifiable or difficult-to-verify tasks, e.g. auto research?
  • How did you come up with setting tau=0.5, and have you tried other training step fractions?
3 comments

We sincerely thank the reviewer for the positive assessment of our work. We are glad that the reviewer finds the paper clearly written, the problem formulation intuitive and useful, and the proposed approach potentially valuable in real-world RL training scenarios.

W1: Limited model range

We thank the reviewer for raising this important point. We agree that evaluating additional model families and larger model sizes would further strengthen our conclusions. Our current experiments provide evidence of cross-scale generalization: SimHack is fitted on Qwen3-0.6B reference runs and successfully predicts reward hacking onset for unseen Qwen3-4B target recipes, as demonstrated in Section 4.3. Moreover, SimHack models prompt-level behavioral dynamics rather than model-specific parameters, so its formulation is not inherently tied to the Qwen3 architecture. Nevertheless, we acknowledge that generalization to other model families and substantially larger models remains to be empirically validated. We have explicitly acknowledged this limitation in Section 7 and consider broader cross-model evaluations an important direction for future work.

Q1: expanded to non-verifiable or difficult-to-verify tasks

We thank the reviewer for this question. SimHack is not inherently restricted to verifiable coding tasks, as its prompt-level behavioral simulation framework can accommodate different behavior. For difficult-to-verify tasks such as autonomous research, one possible extension is to use LLM-based evaluators, automated checks, or human feedback to identify behaviors such as fabricated results or unsupported claims. These behavioral signals could then be used to learn training dynamics and predict the emergence of undesirable behaviors before executing target RL runs. Extending SimHack to handle uncertain behavioral labels and non-verifiable tasks is an important direction for future work.

Q2: tau sensitivity

Thanks for the question. To clarify, tau is a threshold on the reward hacking rate for onset definition. The onset t_tau is the first training step at which at least a tau fraction of rollouts exhibit reward hacking. We use tau = 0.5 because it marks the point where hacking becomes the majority behavior.

We also reran the full Table 1 evaluation with tau = 0.25 and tau = 0.75, using the same fitted SimHack model and the same protocol:

tauMethodMedian onset RE (lower is better)p20 pinball (lower is better)Rank corr. (higher is better)
0.255 target RL runs0.2460.0840.695
0.25SimHack0.1230.0640.851
0.505 target RL runs0.2350.0830.667
0.50SimHack0.1130.0790.852
0.755 target RL runs0.2420.0850.635
0.75SimHack0.1440.0810.769

SimHack outperforms the target RL baseline on all three metrics for every tau, so its advantage is not sensitive to this choice.

Thank you for your response and for running the ablation experiments! My concerns have been addressed and I will retain my original (positive) score. Good luck!

Thank you for your positive feedback and for taking the time to consider our additional experiments. We sincerely appreciate your support and encouragement!

Reviewer_Fgy7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates pre-RL prediction of the reward hacking onset distribution p(t50∣R)p(t_{50} | R) (the first step where ≥50%\ge 50\% of rollouts exploit a verifier loophole) for a target recipe R=(π0,C,h)R = (\pi_0, C, h). Showing that pre-RL initial hacking rates of π0\pi_0 fail to predict t50t_{50}, the authors propose SimHack, a prompt-level Markov behavioral simulator tracking a K=3K=3 categorical distribution (correct, hacking, other) Si,t∈Δ2S_{i,t} \in \Delta^2 per prompt i∈{1,…,N}i \in \{1, \dots, N\}. At step tt, SimHack samples rollout counts Bi,t∼Multinomial(Gi,t,Si,t)B_{i,t} \sim \text{Multinomial}(G_{i,t}, S_{i,t}) and updates states via a low-rank sender-receiver affine transition (with momentum MtM_t) fitted via stepwise NLL on 10 Qwen3-0.6B reference runs. Evaluated on a newly released 325-run coding RL benchmark (OVERWRITE and LAZY loopholes), SimHack achieves 11.3% median relative error and 0.852 rank correlation across 8 unseen Qwen3-4B target recipes, and guides curriculum selection to delay t50t_{50} by 96.6%.

Critical strengths
  • Refutation of Static Initial-Policy Risk & Large-Scale Benchmark: The 325-run coding RL dataset (>19.3M rollouts, ~9,000 A100 GPU hours) and Figures 2–3 convincingly demonstrate that t50t_{50} is governed by training dynamics (curriculum ordering, rollout count GG, learning rate) rather than static initial hacking propensity.
  • Validated Proactive Interventions: Beyond trajectory fitting, Section 5 uses SimHack to screen 8 prompt curricula before training (ρ=0.79\rho = 0.79, delaying real t50t_{50} by 96.6% and improving pre-onset clean accuracy by 4.7% relative) and ranks individual "seed" prompts by their learned additive cross-prompt hacking effect β\beta (Table 2).
Critical weaknesses
  • Transductive / Closed-Prompt-Pool Limitation: Because the receiver embeddings VA[i,k],VB[i,k]V_A[i,k], V_B[i,k] and sender weights in Appendix A.2–A.3 are directly indexed by fixed training prompt IDs i∈{1,…,N}i \in \{1, \dots, N\}, SimHack is strictly transductive over the prompt pool. It cannot simulate recipes that introduce new coding prompts not observed during the reference RL runs.
  • Omission of GRPO Zero-Variance Advantage Gating: The underlying RL runs use GRPO via VeRL (Appendix B.4). In GRPO, when all Gi,tG_{i,t} rollouts for prompt ii receive the same reward (e.g., 100% hacking/correct or 100% failing), the group reward standard deviation is zero, producing zero policy gradient update. However, SimHack's transition gt=At⊙Δt+btg_t = A_t \odot \Delta_t + b_t (Appendix A.2) linearly feeds raw behavior counts BtB_t into the sender embeddings without zeroing out zero-variance prompt batches, allowing saturated prompts to continuously drive cross-prompt hacking drift in simulation.
  • Small Evaluation Sample Size (n=5n=5 Ground-Truth Runs Across 8 Recipes): In Section 4.1 (Table 1), each of the 8 target recipes has 10 runs split into 5 baseline runs and only 5 held-out evaluation runs to define the ground-truth t50t_{50} distribution. Estimating the ground-truth 20th percentile (p20p_{20} pinball loss) from 5 runs reduces to the minimum single run, and computing Spearman rank correlation across only 8 target recipes has high finite-sample variance.
Questions for authors
  • Clarification Requests:
    1. Ablation of Cross-Prompt Coupling vs. Diagonal / Scalar Dynamics: Since SimHack requires estimating the target policy's exact initial behavioral state S0S_0 from pre-RL Qwen3-4B rollouts (N=1024N=1024 or 256256 rollouts per prompt, Appendix B.3), how much of the 0.852 rank correlation and 11.3% relative error is retained by a diagonal baseline (where prompt ii only updates itself without cross-prompt sender-receiver matrices U,VU, V)?
    2. Counterfactual Validation of High-β\beta Prompt Removal (§5.2): In Section 5.2 (Table 2), you rank prompts (#1931, #650, #555) by their learned additive hacking effect β\beta. Did you run a real RL experiment removing the top-kk high-β\beta prompts from the training set, and does that delay t50t_{50} more effectively than removing prompts with the highest initial hacking rate Si,0(hack)S_{i,0}^{(\text{hack})}?
  • Suggestions for Future Work:
    1. Incorporating GRPO Variance Weighting & Prompt Features: Weight the sender signal by the empirical reward standard deviation of Bi,tB_{i,t} to match GRPO mechanics, and parameterize U,VU, V as functions of prompt embeddings to generalize to unseen prompts.
2 comments

We sincerely thank the reviewer for recognizing the strengths of our work, particularly the large-scale benchmark demonstrating the importance of training dynamics in reward hacking onset and the practical value of SimHack for proactive interventions through curriculum selection and high-risk prompt analysis.

W1: Transductive / Closed-Prompt-Pool Limitation

We agree with the reviewer that SimHack is currently transductive and cannot directly generalize to unseen training prompts. This scope was already explicitly stated in the Introduction (i.e., "same underlying prompt pool") and Section 2 (i.e., "same prompt set"). To further clarify this constraint, we have also highlighted it in the newly added Limitations paragraph in Section 7. Extending SimHack to unseen prompts remains an important direction for future work.

W2: Omission of GRPO Zero-Variance Advantage Gating

We thank the reviewer for raising this concern. However, SimHack already accounts for zero-variance advantage gating through the GRPO-style reward normalization described in Appendix A.2. Specifically, each rollout is weighted by its prompt-normalized advantage before contributing to the sender signals. When all rollouts from a prompt receive the same reward, their advantages are exactly zero, and the prompt contributes nothing to either sender signal. We will clarify this mechanism in the main text to avoid confusion.

W3: Small Evaluation Sample Size

We acknowledge that the limited number of evaluation runs and target recipes introduces uncertainty in estimating onset distributions, particularly the lower quantiles, and across-recipe rank correlations. However, collecting RL trajectories is computationally expensive. Our benchmark already comprises 325 independent RL runs, requiring approximately 9000 A100 GPU-hours. Given this substantial computational cost, we evaluate 8 target recipes with 10 independent runs each and repeat the evaluation over 100 random splits to reduce sensitivity to any particular split. While this procedure improves the stability of the reported results, we acknowledge that it does not eliminate the statistical uncertainty arising from the limited sample size. We agree that larger-scale evaluations would provide more robust statistical estimates and consider this an important direction for future work.

Q1: Ablation of Cross-Prompt Coupling vs. Diagonal / Scalar Dynamics:

We thank the reviewer for this suggestion and ran this ablation under the Table 1 protocol. In the diagonal variant, prompt i's rollouts update only prompt i's own state; it has the same parameter count, momentum, and initial state S_0 as SimHack. In the shared variant, cross-prompt propagation and the per-prompt S_0 are kept, but the sender/receiver embeddings are shared across prompts.

MethodMedian onset RE ↓p20 pinball ↓Rank corr. ↑Median t50 on 0.6B reference (real: 12.8)
5 target RL runs0.2350.0830.667–
SimHack0.1130.0790.85212.5
Diagonal (self-only)1.2951.372N/A (constant prediction)200 (no onset)
Shared embeddings1.2951.372N/A (constant prediction)10.3

The diagonal model retains none of SimHack's accuracy. It predicts no onset within 200 steps for any target recipe, and it cannot fit even its own reference recipe. The shared variant fits the reference recipe but fails on Qwen3-4B in the same way.

Real RL runsShare of batch at onset stepHack rate during RLHack rate of the same prompts before RL
Qwen3-0.6B reference90%51.8%0.64%
Qwen3-4B targets33%39.7%<0.01%

The real trajectories show why diagonal dynamics cannot work. Prompts that have never been trained on already hack heavily at onset, even though they almost never hacked before RL. Under diagonal dynamics such prompts would stay at their pre-RL state. Rapid onset on not-yet-trained prompts within the fixed prompt pool therefore requires cross-prompt propagation, which SimHack explicitly models.

Q2: Counterfactual Validation of High-beta Prompt Removal (§5.2):

Thank you for the suggestion. We ran this experiment with real RL training. In the Lazy environment, we remove 200 of the 950 training prompts by different strategies: at random, by highest initial hack rate S_{i,0}^(hack), or by highest learned β. We then retrain with the same RL recipe (10 seeds per setting). We report the median onset step t50, where higher means hacking emerges later:

Removed StrategiesMedian t50
None (full data)81.9
Random63.7
Highest initial hack rate S_{i,0}^(hack)91.3
Highest SimHack β133.9
  • Removing the highest-β prompts delays the median onset from 81.9 to 133.9 steps. Removing the same number of random prompts makes hacking emerge earlier.

  • Removing high-β prompts delays hacking more than removing the prompts with the highest S_{i,0}^(hack) (133.9 vs. 91.3), even though the high-β prompts have a lower average initial hack rate (4.0% vs. 7.5%). This indicates that β captures a prompt's influence on the training dynamics, not just how often it is hacked.

We have added these results to Section 5.2 (Table 3), and the detailed setup is in Appendix B.8.

Q3: Incorporating GRPO Variance Weighting & Prompt Features

We thank the reviewer for these constructive suggestions. Regarding GRPO variance weighting, as clarified above, SimHack already incorporates reward standard deviation through the prompt-normalized advantages in Appendix A.2, which also ensures that zero-variance prompt batches contribute no new sender signals.

Regarding prompt features, we agree that parameterizing sender and receiver embeddings as functions of prompt representations is a promising direction for generalizing SimHack to unseen prompts. We acknowledge this as an important extension beyond the current transductive setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.