- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH constructs software-engineering environments with deliberately exposed vulnerabilities, including test-file modification, test-data exploitation, and execution interference. To distinguish reward hacking from genuine task completion, the framework evaluates model-generated code using both a vulnerable evaluator that provides proxy rewards and an independent audit that approximates true task correctness. The authors further control the model's initial hacking tendency by varying the proportion of hacking demonstrations in SFT data and adjust reward difficulty through different test-slice weights. Experiments on Qwen3-4B with GRPO show that reinforcement learning can amplify initial hacking tendencies and that increasing the weight of harder test cases accelerates the emergence of reward hacking. The paper also evaluates several mitigation approaches, including CoT monitoring, χ² regularization, and gradient regularization. The results suggest that CoT monitoring initially suppresses hacking but becomes less effective during training, with additional analyses indicating that code comments can increasingly mislead the monitor.
Critical strengths
- The paper addresses an important problem in coding RL. Reward hacking is a significant challenge for RLVR, particularly when executable evaluation environments contain exploitable vulnerabilities. Developing controllable environments for studying this phenomenon is valuable for understanding the reliability and safety of RL training.
- The experiments examine how initial hacking demonstrations and reward configurations affect the emergence of reward hacking. The analysis of CoT monitoring under continued optimization is particularly interesting, as it highlights the possibility that a mitigation method can become less effective when the policy adapts to its feedback.
Critical weaknesses
- The methodological novelty is relatively limited compared with existing reward-hacking testbeds. The main components of CATCH, including deliberately exploitable evaluation environments, SFT-based initialization of hacking behavior, and comparisons between proxy and true rewards, are closely related to prior work. Although CATCH integrates these components into a controllable coding environment and enables dynamic monitor evaluation, the paper does not sufficiently establish which capabilities or scientific findings are fundamentally unavailable in existing frameworks. A more systematic comparison would help clarify its distinctive contribution.
- The experimental validation is limited in model diversity and statistical reliability. All major RL experiments are conducted on Qwen3-4B, without evaluations on larger models or other model families.
- The gold hacking criterion does not fully characterize reward-hacking behavior. CATCH labels a response as hacking only when both the easy and hard test slices pass while the independent audit fails. However, a model may exploit the evaluator to obtain partial proxy rewards without passing both slices. Such behavior would not be counted as hacking under the proposed definition.
- The environment's representativeness and the evaluation of mitigation methods are limited. CATCH wraps algorithmic problems into standardized software-engineering repositories and uses single-turn GRPO, which differs substantially from realistic multi-step coding agents interacting with complex repositories and testing tools. In addition, the mitigation comparison primarily examines hacking rates over relatively short training horizons, while coding performance is reported only at step 70 on LiveCodeBench v6. This makes it difficult to assess whether the observed trade-offs between hacking suppression and coding ability persist throughout training.
Questions for authors
- Why is reward hacking defined only when both the easy and hard slices pass but the independent audit fails? How frequently do models exploit evaluation loopholes while receiving only partial proxy rewards? Could the authors provide an analysis of such cases and evaluate the agreement between the proposed gold labels and independently verified exploit behaviors?
- How can the authors distinguish deliberate monitor-evasion behavior from incidental changes in code-comment distributions during training? Would the decline in monitor recall persist after controlling for comment length, exploit type, and other output characteristics? Could controlled comment replacement or rewriting experiments help establish the causal role of misleading comments?
- To what extent do the observed reward-hacking dynamics generalize beyond Qwen3-4B and single-turn algorithmic tasks?
5 comments
Response to Reviewer boYu (Part 2/3)
-
Q2 - Controlled comment rewriting experiments are needed to establish the causal role of misleading comments in monitor false negatives.
Thank you for suggesting a controlled experiment to test whether misleading comments contribute to monitor false negatives. We conduct the experiment and found that correcting comments while leaving the exploit code unchanged improved monitor recall in the training run with monitor penalties.
Specifically, we conducted a controlled comment rewriting experiment in which DeepSeek-Flash read the problem and submitted code, then corrected only factual descriptions in existing comments and docstrings, including descriptions of data sources, return values, and execution paths. We preserved comment locations, the number of comment blocks, and the number of comment lines, while keeping each trajectory’s total comment token count within of its original value. The task, exploit implementation, and all other non-comment output features remained unchanged, allowing us to examine the effect of comment content.
We define . Negative values indicate that correcting the comments improves monitor recall. The results are shown in the table below.
Run \ Step 5 20 40 60 80 100 120 140 w/o Penalty −0.15 −0.09 −0.08 −0.02 0.02 0.02 0.01 0.01 w/ Penalty 0.10 −0.16 −0.06 −0.13 −0.32 −0.34 −0.37 −0.49 As training progresses, in the training run without monitor penalties gradually approaches zero, showing a slight upward trend. In the training run with monitor penalties, it remains clearly negative and trends downward in the later stages of training. This indicates that correcting only the comments, without changing the exploit code they describe, makes hacking behavior that previously escaped detection easier for the monitor to identify.
This controlled intervention provides evidence that comment content causally contributes to monitor false negatives. The findings are also consistent with behavioral adaptation under monitor pressure.
For detailed information, please check Appendix E.4.
-
Q3 & W2 & W4 - The findings rely on Qwen3-4B and single-turn algorithmic tasks, leaving their generalization to other models and realistic multi-step coding agents unclear.
Our additional MiniCPM5-2B experiments reproduced the rise in hacking rate and decline in monitor recall under monitor penalties. Below, we present these results and explain the design choices and limitations of CATCH’s task setting.
Cross-model training. We conducted additional experiments on MiniCPM5-2B using the same settings as in the Qwen3-4B experiments. The results are shown in the table below.
Metric \ Step 20 40 60 80 100 120 140 160 180 Monitor recall (%) 46.15 32.31 43.24 26.03 7.83 2.07 1.22 1.18 1.35 Hacking rate (%) 5.91 8.29 15.74 58.63 45.63 56.86 64.31 67.33 58.50 Proxy reward 0.46 0.34 0.45 0.72 0.95 0.99 1.02 0.91 1.02 True reward 0.42 0.25 0.33 0.28 0.52 0.40 0.35 0.21 0.41 As training progressed, the hacking rate increased while monitor recall on gold-labeled hacking samples declined. These results are consistent with our main observations on Qwen3-4B, providing evidence that the same training dynamics occur across model families
For detailed information, please check Appendix H.
Task settings. We agree that CATCH captures only part of the challenges faced by realistic multi-step coding agents. Its multi-file repositories, test scripts, and caching introduce opportunities for hacking beyond algorithm implementation, but single-turn submissions do not reproduce the iterative process of inspecting repositories, invoking tools, running tests, and revising code.
We chose CATCH’s single-turn SWE setting to make the factors driving reward hacking easier to isolate and measure. Known loopholes and the gold monitor allow us to track how initialization, reward design, and monitor penalties affect hacking under controlled conditions. Single-turn interactions keep training costs affordable and reduce variation from tool use and interaction histories. This control supports the analysis, but it also limits how directly our findings apply to multi-step agents.
We acknowledge in the Limitations section that these controlled, SWE-wrapped algorithmic environments do not capture the full complexity of real-world software repositories. Extending our findings to multi-turn SWE agent scenarios remains an important direction for future work.
Response to Reviewer boYu (Part 3/3)
-
W4 - The trade-off between hacking suppression and coding ability needs evaluation across training checkpoints.
Thank you for highlighting the need to assess this trade-off across training checkpoints. Results across multiple checkpoints support the trade-off reported in our paper. CoT monitoring helps preserve coding ability but only transiently suppresses hacking, whereas the penalty consistently suppresses hacking without sustained improvements in coding ability.
We had already evaluated multiple checkpoints on LiveCodeBench v6, although the original manuscript reported only the results at step 70. We now report the pass@1 and pass@2 results across checkpoints below.
-
LiveCodeBench v6 pass@1 (%)
Method \ Step 0 20 40 60 70 Penalty 28.00 29.71 27.71 27.71 27.71 Gradient Regularization 28.00 27.14 25.71 28.00 27.71 LLM CoT Monitor 28.00 27.43 30.86 27.71 30.00 Baseline (no mitigation) 28.00 29.14 28.29 26.86 26.00 -
LiveCodeBench v6 pass@2 (%)
Method \ Step 0 20 40 60 70 Penalty 33.71 37.14 36.00 34.29 33.14 Gradient Regularization 33.71 36.57 32.00 34.86 34.86 LLM CoT Monitor 33.71 32.00 38.86 36.57 36.57 Baseline (no mitigation) 33.71 36.00 34.86 33.14 32.57
The baseline without mitigation initially improves, but its coding performance gradually declines later in training. CoT monitoring outperforms this baseline at several intermediate checkpoints, showing that its benefit to coding performance extends beyond the single checkpoint originally reported.
The two regularization methods exhibit different training trajectories. With the penalty, pass@1 remains relatively stable during the middle and later stages of training, whereas performance under Gradient Regularization initially declines and then recovers. The penalty consistently suppresses hacking but does not yield sustained improvements in coding ability. With the CoT monitor, coding performance remains relatively strong at multiple stages, but the hacking rate rises later in training.
These results clarify how the trade-off evolves within the evaluated training horizon. Whether these patterns persist over longer training remains to be established.
For detailed information, please check Appendix A.4.
-
Response to Reviewer boYu (Part 1/3)
Thank you for your questions about how CATCH differs from prior testbeds, the coverage and reliability of our gold labels, and whether our findings generalize to other models and tasks. We also appreciate your suggestions for controlled comment rewriting experiments and coding evaluations across training checkpoints. All changes in the revised PDF are highlighted in blue.
W1 - The paper should clarify CATCH’s additional capabilities and scientific insights compared with prior reward hacking testbeds.
CATCH combines a more complex, controllable SWE environment with broader loophole coverage, while retaining single-turn interactions for training efficiency.
Specifically, Countdown-Code (Khalifa et al., 2026) is restricted to the Countdown task, which differs considerably from realistic coding scenarios. SteeringRL (Wong et al., 2025) does not enable CoT during training and is restricted to algorithmic problems in a relatively simple environment where the model submits a single file. In CATCH, we manually wrap tasks into controllable SWE repositories where the model can submit multiple files in a single response, and deliberately introduce diverse loopholes. This increases environment complexity and broadens loophole coverage, while our single-turn interaction strategy keeps training efficient.
Q1 & W3 - The gold criterion misses partial reward exploits, whose frequency and implications for label reliability need clarification.
We agree that exploits yielding only partial proxy rewards warrant dedicated analysis. Our audits found that such exploits were rare and that no examples in the manually audited sample were mislabeled.
We used an agent to independently audit the Qwen3-4B training run with a 9.1% SFT toxic ratio. Of the 77k trajectories, 11k passed the easy slice but failed the hard slice, and we sampled trajectories from this subset. Only 4.72% of the sampled trajectories exploited loopholes, corresponding to an estimated 0.68% of all 77k trajectories. These cases occurred predominantly in the later stages of training, after hacking onset. This indicates that the model rarely exploits loopholes to obtain partial proxy rewards.
We also sampled and manually audited a subset of the training trajectories. Within this sample, the gold labels achieved a recall of 1.0, with no mislabeled examples. We additionally used an agent to verify these findings.
For detailed information, please check Appendix D.4.