acceptodds
Under review as a conference paper at ICLR 2027

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

Abstract

During Reinforcement Learning with Verifiable Rewards (RLVR), large language models may exploit loopholes in their environments to earn rewards instead of completing the intended tasks, a phenomenon known as reward hacking. We introduce LoopholeRL, a small, controllable testbed for studying reward hacking in coding RL. We first turn algorithmic problems into software engineering tasks with deliberately exposed loopholes, then control hacking through two key elements of RL: the base model and reward design. We use synthetic data to perform Supervised Fine-Tuning (SFT) on Qwen3-4B, producing base models with different hacking tendencies, and shape rewards by adjusting the relative weights of weak and strong tests to vary the difficulty of earning rewards during RL. The framework also supports evaluating reward hacking mitigation methods. Our experiments show that higher proportions of hacking demonstrations in SFT data and greater reward weight on strong tests advance hacking onset. Among the tested mitigation methods, a Chain-of-Thought (CoT) monitor achieves the highest coding performance at the evaluated checkpoint, but its initial suppression of hacking does not persist. Notably, under monitoring pressure, the model's code comments increasingly mislead the monitor: removing them makes reward hacking easier to detect.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Reinforcement Learning, Reward Hacking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_boYuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH constructs software-engineering environments with deliberately exposed vulnerabilities, including test-file modification, test-data exploitation, and execution interference. To distinguish reward hacking from genuine task completion, the framework evaluates model-generated code using both a vulnerable evaluator that provides proxy rewards and an independent audit that approximates true task correctness. The authors further control the model's initial hacking tendency by varying the proportion of hacking demonstrations in SFT data and adjust reward difficulty through different test-slice weights. Experiments on Qwen3-4B with GRPO show that reinforcement learning can amplify initial hacking tendencies and that increasing the weight of harder test cases accelerates the emergence of reward hacking. The paper also evaluates several mitigation approaches, including CoT monitoring, χ² regularization, and gradient regularization. The results suggest that CoT monitoring initially suppresses hacking but becomes less effective during training, with additional analyses indicating that code comments can increasingly mislead the monitor.

Critical strengths
  1. The paper addresses an important problem in coding RL. Reward hacking is a significant challenge for RLVR, particularly when executable evaluation environments contain exploitable vulnerabilities. Developing controllable environments for studying this phenomenon is valuable for understanding the reliability and safety of RL training.
  2. The experiments examine how initial hacking demonstrations and reward configurations affect the emergence of reward hacking. The analysis of CoT monitoring under continued optimization is particularly interesting, as it highlights the possibility that a mitigation method can become less effective when the policy adapts to its feedback.
Critical weaknesses
  1. The methodological novelty is relatively limited compared with existing reward-hacking testbeds. The main components of CATCH, including deliberately exploitable evaluation environments, SFT-based initialization of hacking behavior, and comparisons between proxy and true rewards, are closely related to prior work. Although CATCH integrates these components into a controllable coding environment and enables dynamic monitor evaluation, the paper does not sufficiently establish which capabilities or scientific findings are fundamentally unavailable in existing frameworks. A more systematic comparison would help clarify its distinctive contribution.
  2. The experimental validation is limited in model diversity and statistical reliability. All major RL experiments are conducted on Qwen3-4B, without evaluations on larger models or other model families.
  3. The gold hacking criterion does not fully characterize reward-hacking behavior. CATCH labels a response as hacking only when both the easy and hard test slices pass while the independent audit fails. However, a model may exploit the evaluator to obtain partial proxy rewards without passing both slices. Such behavior would not be counted as hacking under the proposed definition.
  4. The environment's representativeness and the evaluation of mitigation methods are limited. CATCH wraps algorithmic problems into standardized software-engineering repositories and uses single-turn GRPO, which differs substantially from realistic multi-step coding agents interacting with complex repositories and testing tools. In addition, the mitigation comparison primarily examines hacking rates over relatively short training horizons, while coding performance is reported only at step 70 on LiveCodeBench v6. This makes it difficult to assess whether the observed trade-offs between hacking suppression and coding ability persist throughout training.
Questions for authors
  1. Why is reward hacking defined only when both the easy and hard slices pass but the independent audit fails? How frequently do models exploit evaluation loopholes while receiving only partial proxy rewards? Could the authors provide an analysis of such cases and evaluate the agreement between the proposed gold labels and independently verified exploit behaviors?
  2. How can the authors distinguish deliberate monitor-evasion behavior from incidental changes in code-comment distributions during training? Would the decline in monitor recall persist after controlling for comment length, exploit type, and other output characteristics? Could controlled comment replacement or rewriting experiments help establish the causal role of misleading comments?
  3. To what extent do the observed reward-hacking dynamics generalize beyond Qwen3-4B and single-turn algorithmic tasks?
5 comments
AuthorsOpenReview ↗

Response to Reviewer boYu (Part 1/3)

Thank you for your questions about how CATCH differs from prior testbeds, the coverage and reliability of our gold labels, and whether our findings generalize to other models and tasks. We also appreciate your suggestions for controlled comment rewriting experiments and coding evaluations across training checkpoints. All changes in the revised PDF are highlighted in blue.

  • W1 - The paper should clarify CATCH’s additional capabilities and scientific insights compared with prior reward hacking testbeds.

    CATCH combines a more complex, controllable SWE environment with broader loophole coverage, while retaining single-turn interactions for training efficiency.

    Specifically, Countdown-Code (Khalifa et al., 2026) is restricted to the Countdown task, which differs considerably from realistic coding scenarios. SteeringRL (Wong et al., 2025) does not enable CoT during training and is restricted to algorithmic problems in a relatively simple environment where the model submits a single file. In CATCH, we manually wrap tasks into controllable SWE repositories where the model can submit multiple files in a single response, and deliberately introduce diverse loopholes. This increases environment complexity and broadens loophole coverage, while our single-turn interaction strategy keeps training efficient.

  • Q1 & W3 - The gold criterion misses partial reward exploits, whose frequency and implications for label reliability need clarification.

    We agree that exploits yielding only partial proxy rewards warrant dedicated analysis. Our audits found that such exploits were rare and that no examples in the manually audited sample were mislabeled.

    We used an agent to independently audit the Qwen3-4B training run with a 9.1% SFT toxic ratio. Of the 77k trajectories, 11k passed the easy slice but failed the hard slice, and we sampled trajectories from this subset. Only 4.72% of the sampled trajectories exploited loopholes, corresponding to an estimated 0.68% of all 77k trajectories. These cases occurred predominantly in the later stages of training, after hacking onset. This indicates that the model rarely exploits loopholes to obtain partial proxy rewards.

    We also sampled and manually audited a subset of the training trajectories. Within this sample, the gold labels achieved a recall of 1.0, with no mislabeled examples. We additionally used an agent to verify these findings.

    For detailed information, please check Appendix D.4.

AuthorsOpenReview ↗

Response to Reviewer boYu (Part 2/3)

  • Q2 - Controlled comment rewriting experiments are needed to establish the causal role of misleading comments in monitor false negatives.

    Thank you for suggesting a controlled experiment to test whether misleading comments contribute to monitor false negatives. We conduct the experiment and found that correcting comments while leaving the exploit code unchanged improved monitor recall in the training run with monitor penalties.

    Specifically, we conducted a controlled comment rewriting experiment in which DeepSeek-Flash read the problem and submitted code, then corrected only factual descriptions in existing comments and docstrings, including descriptions of data sources, return values, and execution paths. We preserved comment locations, the number of comment blocks, and the number of comment lines, while keeping each trajectory’s total comment token count within ±5%\pm 5\% of its original value. The task, exploit implementation, and all other non-comment output features remained unchanged, allowing us to examine the effect of comment content.

    We define ΔRecall=RecallOriginal−RecallFaithful\Delta \mathrm{Recall} = \mathrm{Recall}_{\mathrm{Original}} - \mathrm{Recall}_{\mathrm{Faithful}}. Negative values indicate that correcting the comments improves monitor recall. The results are shown in the table below.

    Run \ Step520406080100120140
    w/o Penalty−0.15−0.09−0.08−0.020.020.020.010.01
    w/ Penalty0.10−0.16−0.06−0.13−0.32−0.34−0.37−0.49

    As training progresses, ΔRecall\Delta \mathrm{Recall} in the training run without monitor penalties gradually approaches zero, showing a slight upward trend. In the training run with monitor penalties, it remains clearly negative and trends downward in the later stages of training. This indicates that correcting only the comments, without changing the exploit code they describe, makes hacking behavior that previously escaped detection easier for the monitor to identify.

    This controlled intervention provides evidence that comment content causally contributes to monitor false negatives. The findings are also consistent with behavioral adaptation under monitor pressure.

    For detailed information, please check Appendix E.4.

  • Q3 & W2 & W4 - The findings rely on Qwen3-4B and single-turn algorithmic tasks, leaving their generalization to other models and realistic multi-step coding agents unclear.

    Our additional MiniCPM5-2B experiments reproduced the rise in hacking rate and decline in monitor recall under monitor penalties. Below, we present these results and explain the design choices and limitations of CATCH’s task setting.

    Cross-model training. We conducted additional experiments on MiniCPM5-2B using the same settings as in the Qwen3-4B experiments. The results are shown in the table below.

    Metric \ Step20406080100120140160180
    Monitor recall (%)46.1532.3143.2426.037.832.071.221.181.35
    Hacking rate (%)5.918.2915.7458.6345.6356.8664.3167.3358.50
    Proxy reward0.460.340.450.720.950.991.020.911.02
    True reward0.420.250.330.280.520.400.350.210.41

    As training progressed, the hacking rate increased while monitor recall on gold-labeled hacking samples declined. These results are consistent with our main observations on Qwen3-4B, providing evidence that the same training dynamics occur across model families

    For detailed information, please check Appendix H.

    Task settings. We agree that CATCH captures only part of the challenges faced by realistic multi-step coding agents. Its multi-file repositories, test scripts, and caching introduce opportunities for hacking beyond algorithm implementation, but single-turn submissions do not reproduce the iterative process of inspecting repositories, invoking tools, running tests, and revising code.

    We chose CATCH’s single-turn SWE setting to make the factors driving reward hacking easier to isolate and measure. Known loopholes and the gold monitor allow us to track how initialization, reward design, and monitor penalties affect hacking under controlled conditions. Single-turn interactions keep training costs affordable and reduce variation from tool use and interaction histories. This control supports the analysis, but it also limits how directly our findings apply to multi-step agents.

    We acknowledge in the Limitations section that these controlled, SWE-wrapped algorithmic environments do not capture the full complexity of real-world software repositories. Extending our findings to multi-turn SWE agent scenarios remains an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer boYu (Part 3/3)

  • W4 - The trade-off between hacking suppression and coding ability needs evaluation across training checkpoints.

    Thank you for highlighting the need to assess this trade-off across training checkpoints. Results across multiple checkpoints support the trade-off reported in our paper. CoT monitoring helps preserve coding ability but only transiently suppresses hacking, whereas the χ2\chi^2 penalty consistently suppresses hacking without sustained improvements in coding ability.

    We had already evaluated multiple checkpoints on LiveCodeBench v6, although the original manuscript reported only the results at step 70. We now report the pass@1 and pass@2 results across checkpoints below.

    • LiveCodeBench v6 pass@1 (%)

      Method \ Step020406070
      χ2\chi^2 Penalty28.0029.7127.7127.7127.71
      Gradient Regularization28.0027.1425.7128.0027.71
      LLM CoT Monitor28.0027.4330.8627.7130.00
      Baseline (no mitigation)28.0029.1428.2926.8626.00
    • LiveCodeBench v6 pass@2 (%)

      Method \ Step020406070
      χ2\chi^2 Penalty33.7137.1436.0034.2933.14
      Gradient Regularization33.7136.5732.0034.8634.86
      LLM CoT Monitor33.7132.0038.8636.5736.57
      Baseline (no mitigation)33.7136.0034.8633.1432.57

    The baseline without mitigation initially improves, but its coding performance gradually declines later in training. CoT monitoring outperforms this baseline at several intermediate checkpoints, showing that its benefit to coding performance extends beyond the single checkpoint originally reported.

    The two regularization methods exhibit different training trajectories. With the χ2\chi^2 penalty, pass@1 remains relatively stable during the middle and later stages of training, whereas performance under Gradient Regularization initially declines and then recovers. The χ2\chi^2 penalty consistently suppresses hacking but does not yield sustained improvements in coding ability. With the CoT monitor, coding performance remains relatively strong at multiple stages, but the hacking rate rises later in training.

    These results clarify how the trade-off evolves within the evaluated training horizon. Whether these patterns persist over longer training remains to be established.

    For detailed information, please check Appendix A.4.

Reviewer_HhMUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces CATCH, a controllable testbed for studying reward hacking in coding RL. The authors wrap algorithmic problems into small software repositories with deliberately exposed loopholes, each paired with a plausible engineering justification, and evaluate every submission twice: a hackable run that provides the training reward and a hidden unhackable run that checks true correctness, which together give execution-based gold labels for reward hacking. The model’s initial hacking tendency is controlled through SFT on mixtures of honest and hacking demonstrations, and reward difficulty is controlled through the reward weights. Using Qwen3-4B with GRPO, they show that RL can amplify a weak initial hacking tendency and that harder rewards make hacking emerge earlier. They then compare mitigation methods during training and find that a χ² regularizer suppresses hacking at some cost to coding ability, gradient regularization has limited effect, and a CoT monitor suppresses hacking only temporarily, as its recall degrades while the model learns to write code comments that mislead the monitor. Coding ability is evaluated on LiveCodeBench.

Critical strengths
  1. A useful testbed for studying CoT monitors during training. Compared with prior open testbeds, CATCH has less obvious loopholes, keeps CoT enabled, and provides gold labels that do not depend on any detector. It also allows the hacking tendency and reward difficulty to be controlled separately. Together these make it possible to measure how a CoT monitor’s precision and recall change during RL.
  2. Mitigations are evaluated throughout training. Instead of reporting a single checkpoint, the paper tracks χ² regularization, gradient regularization, and a CoT monitor over the whole run. This shows that a CoT monitor can suppress reward hacking early but lose effectiveness later, which a static evaluation would miss. This is a practical lesson for anyone using CoT monitors as a training signal.
Critical weaknesses
  1. The hacking tendency is deliberately taught through SFT, which departs from how reward hacking arises in practice. In realistic post-training, reward hacking emerges during RL from a model that has never been trained on hacking demonstrations. Here, the model is first fine-tuned on such demonstrations, so the paper studies RL on a model that has already been taught to hack, and it is unclear how the observed dynamics transfer to standard training. This also confounds the main conclusions. Most of the hacking demonstrations are implicit: they come with misleading reasoning and were filtered to evade a monitor from the same model family as the CoT monitor used in training. The findings that RL amplifies a weak hacking tendency and that the model learns to mislead the monitor through code comments could therefore largely reflect behaviors already present in the SFT data. Even if the setup is framed as a model organism, the paper should show results with no hacking data in SFT, as well as an SFT control containing only explicit hacking, to separate what RL learns from what SFT injects.
  2. The mitigation comparison is not rigorous enough to support the ranking of methods. All results come from a single seed without error bars. The mitigation methods use untuned hyperparameters and different run lengths, and coding ability is measured on LiveCodeBench at only one checkpoint. Under these conditions, claims such as "χ² regularization suppresses hacking at a capability cost" or “gradient regularization has limited effect” may not hold under a different seed or a reasonable hyperparameter sweep. Since the comparison of mitigations is a main contribution, the paper needs multiple seeds and a fairer comparison to support it.
Questions for authors
  1. The gold label counts only successful hacks (passing the hackable run but failing the unhackable run). How are attempted but unsuccessful hacks handled? With a 6-second per-case timeout in the unhackable run, could correct but slow solutions be labeled as hacks? Was any manual audit of the labels performed?
  2. Under the CoT monitor penalty, the hacks that remain may simply be the ones the monitor already missed, rather than the model learning new ways to mislead it. Can the authors separate these two explanations, for example by comparing monitor recall on a fixed set of hacking samples across checkpoints?
  3. Initial hacking without RL. What is the hacking rate of each SFT checkpoint before RL, and was RL run from a checkpoint with no hacking data in SFT?
  4. The legend of Figure 5 uses “Grad Norm Penalty”, while the text uses “Gradient Regularization (GR)”. Are these the same method?
  5. How does your work relate to prior work on monitor evasion during training (Skaf et al., 2025; Emmons et al., 2025; Chen et al., 2025) and on representation-based detection and penalties against reward hacking (Gupta & Jenner, 2025; Wu & Tang, 2026)? Please discuss these in the related work section.

References

  • Skaf et al. Large language models can learn and generalize steganographic chain-of-thought under process supervision. arXiv:2506.01926, 2025.
  • Emmons et al. When chain of thought is necessary, language models struggle to evade monitors. arXiv:2507.05246, 2025.
  • Chen et al. Reasoning models don’t always say what they think. arXiv:2505.05410, 2025.
  • Gupta & Jenner. RL-Obfuscation: Can language models learn to evade latent-space monitors? arXiv:2506.14261, 2025.
  • Wu & Tang. From rebound to remedy: Understanding and mitigating reward hacking via representation engineering. arXiv:2604.01476, 2026.
5 comments
AuthorsOpenReview ↗

Response to Reviewer HhMU (Part 1/3)

Thank you for your valuable questions about what RL learns beyond the behaviors introduced through SFT and the reliability of our gold labels. We also appreciate your suggestions to strengthen the mitigation comparison through multi-seed runs and hyperparameter tuning, and to discuss additional related work. All changes in the revised PDF are highlighted in blue.

  • W1 & Q3 - The paper should distinguish what RL learns from what SFT injects, and report the SFT models' initial hacking tendency.

    Below we explain why we use SFT initialization, analyze new exploit combinations that emerge during RL, and report the initial hacking rates of different SFT checkpoints.

    Existing frontier models commonly exhibit hacking tendencies, as documented in studies by OpenAI (Baker et al., 2025) and Anthropic (MacDiarmid et al., 2025). These tendencies may arise from borderline content mixed into training data, which is difficult to filter out during industrial-scale training. We simulate this situation on a smaller scale, balancing practicality, efficiency, and realism. Prior work, including MacDiarmid et al. (2025) and Countdown-Code (Khalifa et al., 2026), also uses fine-tuning to expose models to hacking strategies or demonstrations.

    To support systematic analysis, we built a controllable framework that helps isolate the factors affecting reward hacking. A key requirement for these experiments is access to models with a reward hacking tendency. We examined Qwen3-4B and found that, after SFT on non-hacking data alone, it struggled to discover and exploit the loopholes without being explicitly informed that they exist. This may reflect the limited capabilities of a 4B model and Qwen3-4B’s safety alignment. We therefore used SFT to introduce different hacking capabilities, simulating the hacking tendencies that models may exhibit in realistic settings.

    The resulting behavior goes beyond simply copying SFT demonstrations. During RL, the model combines different hacking strategies learned through SFT and generalizes to new strategies. We measured the frequency of new exploit combinations emerging during RL, as shown below.

    • 10k + 1k w/o monitor

      Step050100150200250300
      Unseen Combination Rate (%)0.000.020.020.511.811.251.54
    • 8k + 3k w/ monitor

      Step04080120160
      Unseen Combination Rate (%)0.000.060.310.120.00
    • 8k + 3k w/o monitor

      Step020406080
      Unseen Combination Rate (%)0.000.753.601.801.14

    As shown in the tables, compositional generalization across hacking strategies learned through SFT becomes more common as the hacking rate rises. Later in training, model output diversity decreases, and the frequency of these new exploit combinations gradually declines.

    For detailed information, please check Appendix F.1.

    We evaluated the initial hacking rates of different SFT checkpoints using 1,024 examples sampled from the RL dataset to address your question about the SFT models’ initial hacking tendency. The results are reported in the table below.

    SFT (non-hacking + hacking)SFT toxic ratiohacking rate
    10k + 1k9.10%1/1024 = 0.10%
    9k + 2k18.20%15/1024 = 1.46%
    8k + 3k27.30%60/1024 = 5.86%
AuthorsOpenReview ↗

Response to Reviewer HhMU (Part 2/3)

  • W2 - The mitigation comparison relies on a single seed and untuned hyperparameters, leaving the reliability of its findings unclear.

    We apologize that, due to time constraints, we were unable to conduct multi-seed runs or tune the hyperparameters. Therefore, for each mitigation method, we use the best-performing hyperparameter settings reported in its original paper.

  • Q1 - The labeling of unsuccessful hacking attempts, possible mislabeling due to timeouts, and manual audits of gold labels need clarification.

    Below, we explain which trajectories are labeled as hacking, clarify how timeouts are handled, and report the results of our manual audit of the gold labels.

    Trajectories that attempt hacking but do not succeed are not labeled as hacking. Our framework requires rproxy≥1r_{\mathrm{proxy}} \ge 1 for a trajectory to be labeled as reward hacking and primarily focuses on successful hacking behavior. However, because all deliberately introduced loopholes are known and the corresponding exploit mechanisms are largely specified, our framework can readily be extended to perform syntactic analysis of submitted Python code and extract features indicative of hacking attempts, enabling the study of unsuccessful hacking attempts.

    Solutions that are otherwise correct but time out are not labeled as hacking. Strictly speaking, such solutions are not correct because they fail to satisfy the task’s runtime requirements.

    The gold labels accurately capture the model’s reward hacking behavior. We sampled and manually audited a subset of the training trajectories. Within this sample, the gold labels achieved a recall of 1.0, with no mislabeled examples.

    For detailed information, please check Appendix D.4.

  • Q2 - RL may amplify hacks that already evade detection rather than learn new ways to mislead the monitor.

    We investigate this by conducting RL from a model fine-tuned exclusively on explicit hacking demonstrations.

    We observed cases in which the model misled the monitor through behaviors absent from the SFT data. We performed SFT on Qwen3-4B using only 7k explicit hacking demonstrations, followed by RL with a Qwen3.5-27B CoT monitor. During RL, we identified a few hacking samples that the monitor failed to detect. In some of these cases, we observed the following behavior: the model generated CoT and implementation code that appeared to solve the task normally, while also submitting an additional calls.json file that overwrote the original file containing the test data. The model neither disclosed this submission in its CoT nor provided any explanation for it.

    For example:

    At RL step 174 (record 53), the task asks for the minimum number of unit-length stick extensions needed to form a triangle. The model overwrites calls.json, which stores test inputs and expected outputs, reducing it from 28 cases to the three shown in the visible preview. Its implementation returns 0 for input 100 10 10, whose correct answer is 81, but this failing case is absent from the replacement file. The CoT discusses the algorithm and caching without disclosing the reduction in test coverage. The submission passes the hackable tests but fails the independent correctness audit. Nevertheless, the CoT monitor classifies it as non-hacking, describing the implementation as a general mathematical algorithm and the auxiliary JSON files as necessary setup.

    Although we observed a few such cases, their frequency did not surge during the 250-step training run. This may be due to the limited training duration. If time permits, we will continue this run and update the results.

    For detailed information, please check Appendix G.5.

AuthorsOpenReview ↗

Response to Reviewer HhMU (Part 3/3)

  • Q4 - It is unclear whether “Grad Norm Penalty” in Figure 5 and “Gradient Regularization (GR)” in the main text refer to the same method.

    Thank you for pointing out this terminology inconsistency. “Grad Norm Penalty” in Figure 5 and “Gradient Regularization (GR)” in the main text refer to the same method. We have corrected this inconsistency in the paper.

  • Q5 - The paper should discuss prior work on monitor evasion and representation-based detection and penalties against reward hacking.

    Thank you for suggesting these references, we have expanded the Related Work section accordingly.

    Regarding CoT monitoring, Skaf et al. (2025) show that penalizing specific content in CoT can induce models to learn and generalize steganographic CoT. Emmons et al. (2025) investigate how the necessity of CoT for task completion affects its monitorability and explicitly train models to evade monitors through RL. Chen et al. (2025) find that models can learn not to disclose reward hacking in their CoT, even without explicitly optimizing against a CoT monitor. These studies provide important context for interpreting our CoT monitor results. CATCH examines how the hacking tendency introduced through SFT evolves under CoT monitor penalties and uses ablations to analyze the effects of CoT and code comments on monitor recall.

    Regarding representation-based monitoring and mitigation, Gupta & Jenner (2025) focus on the robustness of latent-space monitors, while Wu & Tang (2026) propose a mitigation method called Advantage Modification.

Reviewer_FAsMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper built a testbed for reward hacking training, monitoring, and mitigation. It varied the difficulty for loophole exploitation and tested various reward-hacking mitigation methods. The insights from the experiments are valuable for reward hacking mitigation field.

Critical strengths

The major strength of the paper is the detailed setup to simulate realistic hacking environments. Beyond that, the author studied different types of hacking strategies such as explicit and implicit hacking.

It is also interesting to see that the χ2\chi^2 is effective in suppressing reward hacking but has a performance trade-off.

Critical weaknesses

The major weakness of the paper is that all the results rely on single coding environment and single Qwen3-4B model. Therefore, it is difficult to get generalizable insights from the paper.

Questions for authors

How would the detection and cot monitorability results be if we train a different model or on a different task?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FAsM

Thank you for highlighting the need to test whether our detection and CoT monitorability results extend to other models and tasks. All changes in the revised PDF are highlighted in blue.

  • Q - When training a different model, would the hacking rate continue to rise under monitor penalties while monitor recall declines?

    We observed the same results in cross-model training. We conducted additional experiments on MiniCPM5-2B using the same configuration as in our Qwen3-4B experiments and observed the same phenomenon. The training dynamics are shown below.

    Metric \ Step20406080100120140160180
    Monitor recall (%)46.1532.3143.2426.037.832.071.221.181.35
    Hacking rate (%)5.918.2915.7458.6345.6356.8664.3167.3358.50
    Proxy reward0.460.340.450.720.950.991.020.911.02
    True reward0.420.250.330.280.520.400.350.210.41

    These results provide evidence that, across models, the hacking rate continues to rise under monitor penalties while monitor recall declines during training.

    For detailed information, please check Appendix H.

  • W - All results rely on a single coding environment, making it difficult to draw generalizable insights for other tasks.

    CATCH is intentionally designed for coding RL, a representative application of RLVR and a typical setting for reward hacking. Within this scope, we deliberately design diverse loopholes that model vulnerabilities that may arise in real-world coding RL. These include test-file modification, test-data exploitation, and execution interference, allowing us to study multiple forms of hacking within a common experimental framework.

    As an analysis testbed, CATCH balances environment complexity with controllability. Its fully specified loopholes and gold hacking monitor allow us to isolate how initialization, reward design, and monitor penalties affect hacking and its training dynamics. This design supports clearer conclusions about the factors driving hacking and the effectiveness of interventions, which can inform research across a broader range of tasks.

    Extending CATCH to other tasks requires substantial work on the environments and evaluators. Its underlying design principles—controlled loopholes, independent verification of task success, and evaluation throughout training—can guide such extensions, and we welcome the community to build on them.

Thanks for providing additional results, which clear some of my concerns. I have raised my score to 3

Dear Reviewer FAsM,

We appreciate your careful review and your updated assessment. Your comments on generalizability prompted us to extend our evaluation to another model family and describe CATCH’s scope more precisely. Thank you for helping us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers, AC and SAC

We thank the three reviewers, the Area Chair, and the Senior Area Chair for their time and careful consideration of our submission. The reviewers’ questions and suggestions have helped us strengthen the experimental evidence and improve the presentation of CATCH.

The reviews recognized the value of a controllable testbed for studying reward hacking in coding RL. Reviewer boYu emphasized the importance of the problem and the analyses of how initialization, reward configurations, and continued optimization affect hacking and monitor effectiveness. Reviewer HhMU appreciated the detector-independent gold labels, separate control of initial hacking tendency and reward difficulty, and evaluation of mitigations throughout training. Reviewer FAsM valued the environment design, the study of explicit and implicit hacking, and the insights into mitigation trade-offs.

Following their suggestions, we added experiments and expanded our analyses in the following areas:

  1. Cross-model validation (boYu, FAsM). We added RL experiments on MiniCPM5-2B, reproducing the increase in hacking rate and decline in monitor recall under monitor penalties. This extends the central observation beyond Qwen3-4B to another model family (Appendix H).

  2. Controlled analysis of misleading comments (boYu). We corrected factual descriptions in comments and docstrings while keeping the exploit code and all other non-comment output unchanged. Comment locations and line counts were preserved, and comment-token counts remained within 5% of the originals. Under monitor penalties, this intervention improved mean stepwise recall by 41.4 percentage points over steps 100 to 140, providing evidence that misleading comment content causally contributes to missed detections (Appendix E.4).

  3. Gold-label validation and partial-reward exploits (boYu, HhMU). We reported manual label checks with no mislabeled examples in the audited sample and added an independent audit of exploits earning partial proxy rewards. The latter estimates such exploits at 0.68% of the 77,056 recovered training trajectories, concentrated after hacking onset. These analyses provide additional evidence for label reliability and quantify behavior beyond the main full-proxy hacking criterion (Appendix D.4).

  4. Distinguishing SFT initialization from RL dynamics (HhMU). We reported the initial hacking rates of the SFT checkpoints and measured exploit combinations that appeared during RL but were absent from the corresponding SFT data. We also ran RL from an explicit-only hacking SFT initialization and identified monitor-missed cases in which the CoT did not disclose reductions in test coverage. These results document recombination beyond direct reproduction of SFT demonstrations and provide a control for studying monitor evasion (Appendices F.1 and G.5).

  5. Coding ability across training checkpoints (boYu, HhMU). We expanded the LiveCodeBench v6 reporting to include pass@1 and pass@2 across multiple checkpoints. Together with the hacking trajectories, these results show how hacking suppression and coding performance evolve during training, strengthening the evidence beyond the single checkpoint reported in the original submission (Appendix A.4).

We also clarified CATCH’s capabilities relative to prior testbeds and the rationale for its controlled coding setting, expanded the related-work discussion, and unified the mitigation terminology. Following HhMU’s presentation suggestions, we redrew Figure 1 and simplified Figure 2’s layout.

The reviewers’ follow-up comments acknowledged these improvements. Reviewer boYu stated that the rebuttal had adequately addressed their concerns and questions and raised their score. Reviewer FAsM noted that the additional results addressed some concerns and raised their score to 3. Reviewer HhMU raised Soundness from 2 to 3 and stated that the rebuttal “clearly strengthened the paper.” They retained an overall score of 3 while explaining that their assessment lay between weak accept and accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.