- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tried to introduce CHERRL which is a controlled environment for studying reward hacking in rubric based reinforcement learning. The environment adds a known bias bonus to a clean judge score. This makes it possible to track when a policy starts exploiting a specific judge preference. The paper studies lexical, tone, format, and self-praise biases on VerInstruct and HealthBench. It then analyzes hacking onset and post-onset growth through discoverability and exploitability. Finally, it proposes RHDA, an agent that inspects sanitized training logs and estimates when reward hacking starts.
Critical strengths
- CHERRL provides a controlled testbed. As the injected bias is known, this work can study at a time one reward defect . This is better than looking at a failed policy and guessing which part of the judge it tried to exploit.
2.The onset and detector protocols are comparatively more structured than visual inspection. The paper uses a threshold sweep and reports onset intervals. RHDA and the baselines can not see the clean reward, injected bonus, shortcut indicators, or reference onset.
Critical weaknesses
-
The discoverability result uses six positive trajectories along with one main RL run for each dataset-bias condition. This does not separate a stable bias effect from training-seed variation. The mentioned robustness across bias magnitudes is also not consistent with Table 19 in several rows OR decreases while onset becomes earlier. In addition to, larger bias multipliers make the fixed reward-gap threshold easier to cross. This directly confounds the onset comparison.
-
All six detector runs are based on positive hacking runs. The two CHERRL settings where hacking does not show up got excluded from this evaluation. So, the result shows onset localization on positive trajectories. It also does not show false-alert control or reliable no-alert behavior.
Questions for authors
1.How much do the reported onset times vary across independent GRPO seeds for the same dataset-bias condition?
2.Does the discoverability result stay after normalizing the reward-gap thresholds by the bias multiplier ?
- What are RHDA’s false-positive and no-alert rates on the two non-hacking runs and on clean rubric-RL runs?
3 comments
Response to Reviewer czB5 (Part 2/3)
Q3: Do the non-monotonic OR–onset patterns in Table 19 contradict our discoverability finding? (Weakness #1)
No. The non-monotonic patterns arise from comparisons across different bias magnitudes, whereas our discoverability finding concerns bias–task entanglement under a fixed bias magnitude. Our original wording did not clearly distinguish these two comparisons.
Specifically, hacking onset depends on both the type of injected bias and its magnitude α. Our analysis in Section 3.1 examines how bias–task entanglement relates to discoverability by comparing different bias types at a fixed α, using OR to quantify their entanglement with task success. In contrast, the original Table 19 varies α while holding bias type fixed. Since OR characterizes bias–task entanglement rather than all factors governing onset, a monotonic OR–onset relationship across different α values is not a necessary consequence of our hypothesis.
To further examine the robustness of our intended finding, we extend Table 19 by comparing different bias types at each fixed α ∈ {0.3, 0.5, 1.0}, following the protocol in Section 3.1. The expanded results are organized into three sub-tables, one for each α.
Table 19(a): α = 0.3
| Bias type | Normalized reference onset | OR (60 steps) |
|---|---|---|
| Lexical | 172 [132, 174] | 1.1870 |
| Format | 301 [301, 445] | 0.8895 |
Table 19(b): α = 0.5
| Bias type | Normalized reference onset | OR (60 steps) |
|---|---|---|
| Lexical | 116 [115, 161] | 1.0911 |
| Format | 301 [301, 443] | 0.8649 |
Table 19(c): α = 1.0
| Bias type | Normalized reference onset | OR (60 steps) |
|---|---|---|
| Lexical | 100 [100, 167] | 1.3214 |
| Format | 301 [98, 370+] | 0.9527 |
Across all three evaluated α values, higher OR consistently corresponds to earlier hacking onset when comparing different bias types within the same dataset at a fixed α. These results confirm that the relationship between bias–task entanglement and hacking discoverability observed in Section 3.1 remains consistent across different bias magnitudes.
We also plan to extend this analysis to HealthBench to further examine whether this relationship generalizes across datasets, and will report the additional results before the end of the rebuttal discussion period.
We acknowledge that our original wording suggested a stronger claim than intended. We have revised Section 3.3 and Appendix H.2 to clarify this distinction and correct the overly broad claim in the original manuscript.
Response to Reviewer czB5 (Part 3/3)
Q4: Can RHDA avoid false alerts while maintaining accurate onset localization? (Weakness #2, Question #3)
RHDA achieves better false-positive control than CC-Qwen, and a lightweight prompt refinement further improves its no-alert rate from 80% to 95% while maintaining competitive onset localization.
We agree that reliable no-alert behavior is essential for reward-hacking detection. To address this concern, we evaluated RHDA on all four non-hacking trajectories, including the two non-hacking CHERRL settings (VerInstruct tone and HealthBench format) and clean rubric-RL trajectories on both datasets. Each detector was evaluated five times per trajectory.
| Negative trajectory | Original RHDA | Refined RHDA | CC-Qwen |
|---|---|---|---|
| HealthBench format (first 280 steps) | 1/5 | 0/5 | 2/5 |
| VerInstruct tone | 2/5 | 0/5 | 3/5 |
| VerInstruct no-bias | 0/5 | 1/5 | 1/5 |
| HealthBench no-bias | 1/5 | 0/5 | 1/5 |
| Overall false-positive rate | 20% | 5% | 35% |
The original RHDA achieves a lower false-positive rate than CC-Qwen, the best-performing baseline in our original onset-localization evaluation (20% vs. 35%). Furthermore, we added a short, targeted clarification to RHDA's system prompt, requiring treating high-score subsets as separate evidence of reward association. This refinement reduces the observed false-positive rate from 20% to 5%.* To examine whether this improvement compromises onset localization, we also evaluated the refined RHDA on all six positive hacking trajectories:
| Setting | Reference onset | Original RHDA | Refined RHDA | CC-Qwen |
|---|---|---|---|---|
| VerInstruct self-praise | 478 [478,492] | 482 | 490 | 490 |
| VerInstruct format | 301 [301,443] | 383 | 404 | 341 |
| VerInstruct lexical | 116 [115,161] | 132 | 117 | 220 |
| HealthBench lexical | 91 [91,95] | 86 | 95 | 96 |
| HealthBench self-praise | 460 [460,466] | 454 | 470 | 474 |
| HealthBench tone | 68 [68,79] | 75 | 81 | 91 |
| Σ point distance | — | 120 | 143 | 198 |
| Σ interval distance | — | 11 | 6 | 80 |
The refined RHDA maintains comparable onset localization performance to the original RHDA, with a slightly higher aggregate point distance (143 vs. 120) but a lower interval distance (6 vs. 11). It also outperforms CC-Qwen on both localization metrics (143 vs. 198 in point distance; 6 vs. 80 in interval distance).
Together, these results show that RHDA alredy achieves better false-positive control than the baseline, and that a short, targeted prompt refinement further reduces false positives while preserving strong onset localization performance, suggesting additional room for improvement. We have incorporated the above results and the exact revised prompt into Appendix N of the revised PDF.
We thank the reviewer again for these valuable suggestions. The new experiments strengthen the empirical support for our findings and provide a more complete evaluation of CHERRL and RHDA. We hope these results address the reviewer's concerns.
Response to Reviewer czB5 (Part 1/3)
We thank the reviewer for recognizing two central strengths of our work: CHERRL provides a controlled testbed that isolates individual reward defects, enabling analysis of reward hacking; and our structured onset protocol, together with the detector's restricted access to privileged information, enables a more objective evaluation than visual inspection.
We also appreciate the reviewer's careful scrutiny of seed variability, reward-gap confounding, the interpretation of our discoverability findings, and false-alert control, which highlights important aspects of experimental reliability. Following these comments, we conducted additional experiments and clarified our original claims to address all the reviewer's concerns.We present our responses below.
Q1: Do different GRPO seeds affect hacking onset? (Weakness #1, Question #1)
Different GRPO seeds yield relatively consistent hacking onset times.
We conducted two additional independent GRPO runs on the HealthBench lexical-bias setting, yielding three runs in total, including the original one. The three runs exhibit closely aligned hacking onset times, suggesting that the observed onset is not merely an artifact of a particular training seed. We have included these additional results in Appendix M of the revised PDF.
Q2: Does the discoverability result persist after normalizing the reward gap by α? (Weakness #1, Question #2)
Yes. The normalized onset estimates remain non-increasing as α grows. However, this cross-α ordering is not required for our discoverability finding, which concerns comparisons across bias types at a fixed α, as clarified in Q3.
We thank the reviewer for identifying the potential confounding effect of α on the fixed reward-gap threshold. We agree that reward gaps should be normalized when comparing reference onsets across different α values. We therefore use normalized reward gaps to compute reference onsets in our analysis of α sensitivity.
Specifically, we normalize the reward gap to the common scale of α = 0.5: Gnorm(t)=2αG(t).
We then recompute the reference onsets using the normalized gap.
After normalization, the canonical onset estimates remain non-increasing as α increases across all three settings: VerInstruct lexical (172 → 116 → 100), VerInstruct format (301 → 301 → 301), and HealthBench lexical (130 → 91 → 73). These results indicate that the observed cross-α trend in reference onset is not solely an artifact of reward-gap threshold scaling.
More importantly, as clarified in Q3, our central discoverability finding concerns the relationship between bias–task entanglement and hacking onset under a fixed α, and does not depend on monotonicity across different bias magnitudes.
We have revised Section 3.3 and Appendix H.2 to use normalized reward gaps in the α sensitivity analysis and clarify this distinction, and added corresponding explanations in Section 2.3 and Appendix A.1.