acceptodds
Under review as a conference paper at ICLR 2027

Reproducing, Analyzing, and Detecting Reward Hacking in Rubric‑Based Reinforcement Learning

Abstract

Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a Controllable Hacking Environment for Rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent for automatically detecting reward hacking onset from training logs. The resources will be publicly released.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward Hacking, Reinforcement Learning, Rubric-based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_czB5OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tried to introduce CHERRL which is a controlled environment for studying reward hacking in rubric based reinforcement learning. The environment adds a known bias bonus to a clean judge score. This makes it possible to track when a policy starts exploiting a specific judge preference. The paper studies lexical, tone, format, and self-praise biases on VerInstruct and HealthBench. It then analyzes hacking onset and post-onset growth through discoverability and exploitability. Finally, it proposes RHDA, an agent that inspects sanitized training logs and estimates when reward hacking starts.

Critical strengths
  1. CHERRL provides a controlled testbed. As the injected bias is known, this work can study at a time one reward defect . This is better than looking at a failed policy and guessing which part of the judge it tried to exploit.

2.The onset and detector protocols are comparatively more structured than visual inspection. The paper uses a threshold sweep and reports onset intervals. RHDA and the baselines can not see the clean reward, injected bonus, shortcut indicators, or reference onset.

Critical weaknesses
  1. The discoverability result uses six positive trajectories along with one main RL run for each dataset-bias condition. This does not separate a stable bias effect from training-seed variation. The mentioned robustness across bias magnitudes is also not consistent with Table 19 in several rows OR decreases while onset becomes earlier. In addition to, larger bias multipliers make the fixed reward-gap threshold easier to cross. This directly confounds the onset comparison.

  2. All six detector runs are based on positive hacking runs. The two CHERRL settings where hacking does not show up got excluded from this evaluation. So, the result shows onset localization on positive trajectories. It also does not show false-alert control or reliable no-alert behavior.

Questions for authors

1.How much do the reported onset times vary across independent GRPO seeds for the same dataset-bias condition?

2.Does the discoverability result stay after normalizing the reward-gap thresholds by the bias multiplier α\alpha?

  1. What are RHDA’s false-positive and no-alert rates on the two non-hacking runs and on clean rubric-RL runs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer czB5 (Part 1/3)

We thank the reviewer for recognizing two central strengths of our work: CHERRL provides a controlled testbed that isolates individual reward defects, enabling analysis of reward hacking; and our structured onset protocol, together with the detector's restricted access to privileged information, enables a more objective evaluation than visual inspection.

We also appreciate the reviewer's careful scrutiny of seed variability, reward-gap confounding, the interpretation of our discoverability findings, and false-alert control, which highlights important aspects of experimental reliability. Following these comments, we conducted additional experiments and clarified our original claims to address all the reviewer's concerns.We present our responses below.

Q1: Do different GRPO seeds affect hacking onset? (Weakness #1, Question #1)

Different GRPO seeds yield relatively consistent hacking onset times.

We conducted two additional independent GRPO runs on the HealthBench lexical-bias setting, yielding three runs in total, including the original one. The three runs exhibit closely aligned hacking onset times, suggesting that the observed onset is not merely an artifact of a particular training seed. We have included these additional results in Appendix M of the revised PDF.

SeedReference onset
Default91 [91, 95]
42108 [108, 112]
12397 [97, 105]

Q2: Does the discoverability result persist after normalizing the reward gap by α? (Weakness #1, Question #2)

Yes. The normalized onset estimates remain non-increasing as α grows. However, this cross-α ordering is not required for our discoverability finding, which concerns comparisons across bias types at a fixed α, as clarified in Q3.

We thank the reviewer for identifying the potential confounding effect of α on the fixed reward-gap threshold. We agree that reward gaps should be normalized when comparing reference onsets across different α values. We therefore use normalized reward gaps to compute reference onsets in our analysis of α sensitivity.

Specifically, we normalize the reward gap to the common scale of α = 0.5: Gnorm(t)=G(t)2α.G_{\mathrm{norm}}(t)=\frac{G(t)}{2\alpha}.

We then recompute the reference onsets using the normalized gap.

SettingαOriginal onsetNormalized onset
VerInstruct lexical0.3178 [169, >186]172 [132, 174]
VerInstruct lexical0.5116 [115, 161]116 [115, 161]
VerInstruct lexical1.019 [19, 73]100 [100, 167]
VerInstruct format0.3442 [301, 445]301 [301, 445]
VerInstruct format0.5301 [301, 443]301 [301, 443]
VerInstruct format1.098 [98, 370+]301 [98, 370+]
HealthBench lexical0.3133 [133, 141]130 [130, 139]
HealthBench lexical0.591 [91, 95]91 [91, 95]
HealthBench lexical1.070 [70, 81]73 [73, 80]

After normalization, the canonical onset estimates remain non-increasing as α increases across all three settings: VerInstruct lexical (172 → 116 → 100), VerInstruct format (301 → 301 → 301), and HealthBench lexical (130 → 91 → 73). These results indicate that the observed cross-α trend in reference onset is not solely an artifact of reward-gap threshold scaling.

More importantly, as clarified in Q3, our central discoverability finding concerns the relationship between bias–task entanglement and hacking onset under a fixed α, and does not depend on monotonicity across different bias magnitudes.

We have revised Section 3.3 and Appendix H.2 to use normalized reward gaps in the α sensitivity analysis and clarify this distinction, and added corresponding explanations in Section 2.3 and Appendix A.1.

AuthorsOpenReview ↗

Response to Reviewer czB5 (Part 2/3)

Q3: Do the non-monotonic OR–onset patterns in Table 19 contradict our discoverability finding? (Weakness #1)

No. The non-monotonic patterns arise from comparisons across different bias magnitudes, whereas our discoverability finding concerns bias–task entanglement under a fixed bias magnitude. Our original wording did not clearly distinguish these two comparisons.

Specifically, hacking onset depends on both the type of injected bias and its magnitude α. Our analysis in Section 3.1 examines how bias–task entanglement relates to discoverability by comparing different bias types at a fixed α, using OR to quantify their entanglement with task success. In contrast, the original Table 19 varies α while holding bias type fixed. Since OR characterizes bias–task entanglement rather than all factors governing onset, a monotonic OR–onset relationship across different α values is not a necessary consequence of our hypothesis.

To further examine the robustness of our intended finding, we extend Table 19 by comparing different bias types at each fixed α ∈ {0.3, 0.5, 1.0}, following the protocol in Section 3.1. The expanded results are organized into three sub-tables, one for each α.

Table 19(a): α = 0.3

Bias typeNormalized reference onsetOR (60 steps)
Lexical172 [132, 174]1.1870
Format301 [301, 445]0.8895

Table 19(b): α = 0.5

Bias typeNormalized reference onsetOR (60 steps)
Lexical116 [115, 161]1.0911
Format301 [301, 443]0.8649

Table 19(c): α = 1.0

Bias typeNormalized reference onsetOR (60 steps)
Lexical100 [100, 167]1.3214
Format301 [98, 370+]0.9527

Across all three evaluated α values, higher OR consistently corresponds to earlier hacking onset when comparing different bias types within the same dataset at a fixed α. These results confirm that the relationship between bias–task entanglement and hacking discoverability observed in Section 3.1 remains consistent across different bias magnitudes.

We also plan to extend this analysis to HealthBench to further examine whether this relationship generalizes across datasets, and will report the additional results before the end of the rebuttal discussion period.

We acknowledge that our original wording suggested a stronger claim than intended. We have revised Section 3.3 and Appendix H.2 to clarify this distinction and correct the overly broad claim in the original manuscript.

AuthorsOpenReview ↗

Response to Reviewer czB5 (Part 3/3)

Q4: Can RHDA avoid false alerts while maintaining accurate onset localization? (Weakness #2, Question #3)

RHDA achieves better false-positive control than CC-Qwen, and a lightweight prompt refinement further improves its no-alert rate from 80% to 95% while maintaining competitive onset localization.

We agree that reliable no-alert behavior is essential for reward-hacking detection. To address this concern, we evaluated RHDA on all four non-hacking trajectories, including the two non-hacking CHERRL settings (VerInstruct tone and HealthBench format) and clean rubric-RL trajectories on both datasets. Each detector was evaluated five times per trajectory.

Negative trajectoryOriginal RHDARefined RHDACC-Qwen
HealthBench format (first 280 steps)1/50/52/5
VerInstruct tone2/50/53/5
VerInstruct no-bias0/51/51/5
HealthBench no-bias1/50/51/5
Overall false-positive rate20%5%35%

The original RHDA achieves a lower false-positive rate than CC-Qwen, the best-performing baseline in our original onset-localization evaluation (20% vs. 35%). Furthermore, we added a short, targeted clarification to RHDA's system prompt, requiring treating high-score subsets as separate evidence of reward association. This refinement reduces the observed false-positive rate from 20% to 5%.* To examine whether this improvement compromises onset localization, we also evaluated the refined RHDA on all six positive hacking trajectories:

SettingReference onsetOriginal RHDARefined RHDACC-Qwen
VerInstruct self-praise478 [478,492]482490490
VerInstruct format301 [301,443]383404341
VerInstruct lexical116 [115,161]132117220
HealthBench lexical91 [91,95]869596
HealthBench self-praise460 [460,466]454470474
HealthBench tone68 [68,79]758191
Σ point distance—120143198
Σ interval distance—11680

The refined RHDA maintains comparable onset localization performance to the original RHDA, with a slightly higher aggregate point distance (143 vs. 120) but a lower interval distance (6 vs. 11). It also outperforms CC-Qwen on both localization metrics (143 vs. 198 in point distance; 6 vs. 80 in interval distance).

Together, these results show that RHDA alredy achieves better false-positive control than the baseline, and that a short, targeted prompt refinement further reduces false positives while preserving strong onset localization performance, suggesting additional room for improvement. We have incorporated the above results and the exact revised prompt into Appendix N of the revised PDF.

We thank the reviewer again for these valuable suggestions. The new experiments strengthen the empirical support for our findings and provide a more complete evaluation of CHERRL and RHDA. We hope these results address the reviewer's concerns.

Reviewer_eVSqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reward hacking is a critical challenge in RL, and particularly rubrics training. This paper designs a controlled environment for studying reward hacking. They use two judges: clean and biased, where the biased judge is instructed to have a bias for lexical, tone, self-praise, or format. This setting allows for observing hacks (spikes in biased judge) in an analysis showing onset time correlates with bias–task entanglement.

Critical strengths

Overall, this is a good paper. They work on a well-motivated problem (reward hacking in rubric based RL). The discoverability and exploitability is decomposed well into pre-hacking steps for discoverability and instruction-following for exploitability.

The experiments design is very solid and robust with multiple human validations (and reporting agreement stats), variance and cost analysis, and ablating search-budget. The results are presented well and the curves in Figure 3 are very insightful and well-discussed.

Critical weaknesses
  • All results are based on experiments with a single model (Qwen 3 4B), adding cross-family and size-scaling would be very helpful to understand generalizability.
  • There's some circularity in the reference onset as the shortcut detectors used in constructing ground truth are derived from the same injected bias prompts. This affects the detector accuracy scores that are relative to this constructed reference.
Questions for authors

none

2 comments
AuthorsOpenReview ↗

Response to Reviewer eVSq

We thank the reviewer for recognizing the motivation and experimental rigor of our work. We particularly appreciate the positive feedback on our decomposition of reward hacking into discoverability and exploitability, as well as the thorough validation of our experimental methodology, including human agreement checks, variance analysis, and search-budget ablations. We are glad that the training dynamics in Figure 3 provide useful insights into how different judge biases shape reward hacking. We address the two concerns about cross-model generalizability and reference-onset circularity below.

Q1: How well do our findings generalize beyond Qwen3-4B? (Weakness #1)

We are conducting additional experiments with other models, such as MiniCPM5-2B.

Due to limited compute, these runs will take some more time to finish. We will post the results before the end of the discussion period. We thank the reviewer for their patience.

Q2: Does the bias-informed reference onset introduce circularity into detector evaluation? (Weakness #2)

The reference onset is intentionally bias-informed, but detectors remain blind to reference-side information. More importantly, our additional blind human validation supports that the reference onsets correspond to independently observable behavioral changes.

We thank the reviewer for raising this point. We considered this risk when designing the reference, and we agree that the paper should discuss it more explicitly.

  1. The shared definition is intended and stays on the reference side. In CHERRL, reward hacking means that the policy exploits a known injected bias, so any reference for when this exploitation starts has to refer to that bias. For this reason we call it an operational reference onset rather than human ground truth (Section 2.3; Appendix A.1). The bias-derived indicators c(i)c(i) are used only to construct the reference. RHDA and all baselines operate on a judge-blind mirror that excludes c(i)c(i). Every detector therefore has to infer the shortcut from the trajectory itself, with the same information as every other detector.

  2. We add blind human checks. We examine the self-praise run on HealthBench, with a reference onset interval of [460, 466]. Two annotators, blinded to the injected bias, reference onset, and detector outputs, examine 12 stages sampled across the full training trajectory, with six randomly sampled responses per stage, followed by up to 12 additional stages to refine onset estimates. We repeat this procedure with three independent samples, yielding six annotations. Both annotators independently identify the injected self-praise shortcut without prior knowledge of the bias. Five of six annotations locate the onset at steps 465–473, either within or close to the reference interval. Neither annotator identifies comparable behavior in the control run without injected bias. These results support that the reference onset reflects observable behavioral changes rather than detector artifacts.

We thank the reviewer again for these constructive comments. We have strengthened the manuscript with additional blind human validation in Appendix O to address the reference-onset concern, and we are extending our experiments to additional models to investigate generalizability. We hope these efforts help address the reviewer's concerns.

AuthorsOpenReview ↗

Additional Results for Q1: Cross-Model Generalizability

We thank the reviewer for their patience while we conducted the additional experiments. We are now pleased to share our results on cross-model generalizability.

Q1: How well do our findings generalize beyond Qwen3-4B? (Weakness #1)

CHERRL generalizes beyond Qwen3-4B, providing a controllable testbed for studying reward hacking across different model families. Our new experiments on MiniCPM5-2B successfully reproduce reward hacking under both lexical and tone biases, while revealing model-dependent differences in hacking onset and bias discoverability.

1. Reproducing reward hacking on MiniCPM5-2B. To examine cross-model generalizability, we conducted additional GRPO experiments using MiniCPM5-2B on HealthBench, injecting lexical and tone biases separately through CHERRL's dual-judge design (α=0.5\alpha=0.5). Both settings successfully reproduce reward hacking, with the proxy reward diverging from the clean reward during training. The corresponding training dynamics are presented in Appendix Q of the revised PDF.

These results demonstrate that CHERRL can reproduce observable reward hacking beyond Qwen3-4B.

2. Cross-model differences in hacking onset. CHERRL further allows us to quantify and compare hacking onset across models. Applying the same reference-onset procedure, we obtain the following results:

ModelDatasetBiasReference onset
MiniCPM5-2BHealthBenchLexical26 [26, 30]
MiniCPM5-2BHealthBenchTone32 [27, 36]
Qwen3-4BHealthBenchLexical91 [91, 95]
Qwen3-4BHealthBenchTone68 [68, 79]

Two observations emerge. First, MiniCPM5-2B exhibits earlier hacking onsets than Qwen3-4B in both settings. Second, the relative onset ordering differs across models: lexical bias emerges before tone bias in MiniCPM5-2B, whereas the opposite holds for Qwen3-4B.

3. Implications. These results suggest that while reward hacking can be reproduced across model families, its discoverability dynamics may depend on the policy model. One possible explanation for the earlier onsets in MiniCPM5-2B is its lower clean reward during early training (0.29 vs. 0.42 for Qwen3-4B, averaged over the first five steps). This may allow the injected bias bonus to exert greater influence on policy optimization, reinforcing shortcut behaviors earlier. Future work can leverage CHERRL to further investigate this hypothesis and identify the factors underlying these model-dependent differences.

The reversed onset ordering further suggests that the relative discoverability of different biases is not necessarily universal across models, highlighting the need for controlled investigations into the underlying mechanisms. By making reward divergence and hacking onset observable, CHERRL provides such a setting. Our new results further demonstrate CHERRL's applicability across different model families, providing a foundation for studying the mechanisms and mitigations of reward hacking in rubric-based RL across models of different scales and architectures.

The detailed experimental results are provided in Appendix Q of the revised PDF.

We hope these additional experiments address your concern about cross-model generalizability. As the discussion period draws to a close, we would welcome any further questions or feedback you may have. Should additional experimental results become available before the end of the discussion period, we will share them here as well.

Reviewer_4JUQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a test bed for studying reward hacking against an LLM judge-based reward, which introduces artificial biases in a controllable way to help with studying it. To demonstrate the efficacy of this environment, they introduce an agent for automatically detecting when reward hacking is starting from training logs, and they analyze some of the different biases.

Critical strengths
  • The introduced environment seems useful and helpful for further researchers. Good open source environments are an undersupplied public good
  • It's good to better enable research into LLM judge RL rather than just RLVR, the industry is shifting away from RLVR
  • I liked the ability to track the true and biased reward separately, that's a useful analysis tool
  • I liked the careful threshold sweep to determine when hacking starts
Critical weaknesses
  • Binary biases are limited and unrealistic
  • Real biases are often several biases at once
  • The results are only drawn from 6 runs with hacking (though I know RL runs can be expensive, even on a small model)
Questions for authors
  • How easy is cherrl to extend to more complex and interesting biases?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4JUQ (Part 1/2)

We thank the reviewer for recognizing CHERRL's value as an open-source research testbed for reward hacking in LLM-judge-based RL. We particularly appreciate the positive feedback on our dual-judge design and our threshold-sweep protocol for identifying hacking onset. We also appreciate the constructive suggestions on experimental robustness and extending CHERRL to more realistic judge biases.

Q1: How readily can CHERRL accommodate more complex and multiple judge biases? (Weaknesses #1–2, Question #1)

CHERRL can accommodate multiple judge biases through a straightforward extension of Eq. (2). To demonstrate this capability, we conducted an additional mixed-bias experiment to investigate how the simultaneous presence of multiple judge biases affects reward hacking dynamics.

1. Extending CHERRL to multiple biases. Our main experiments deliberately isolate one bias per run to enable controlled comparisons of discoverability and exploitability. However, CHERRL's modular reward design naturally supports multiple biases. Specifically, for a set of target biases Btarget={βk}k=1K\mathcal{B}_{\mathrm{target}} = \{\beta_k\}_{k=1}^{K}, we can generalize Eq. (2) as

Jbiased=Junbiased+α⋅rbonus(Btarget),J_{\mathrm{biased}}=J_{\mathrm{unbiased}}+\alpha\cdot r_{\mathrm{bonus}}(\mathcal{B}_{\mathrm{target}}),

where rbonus∈[0,1]r_{\mathrm{bonus}}\in[0,1] aggregates the detected bias features into a scalar reward. The aggregation function can be flexibly designed to investigate different interactions among biases.

2. New experiment: Competition between two biases. Following your suggestion, we conducted an additional experiment on VerInstruct with lexical and self-praise biases injected simultaneously. We adopt a competitive reward scheme, where rbonus=1r_{\mathrm{bonus}}=1 if either bias is detected, and 00 otherwise. Thus, exploiting both biases provides no additional reward over exploiting just one.

To quantify hacking onset in this mixed-bias setting, we follow the reference-onset procedure in Section 2.3, using different shortcut indicators c(i)c(i). Specifically, selecting the lexical or self-praise indicator separately allows us to estimate the onset of each shortcut within the same mixed-bias run.

We compare the bias-specific onsets in the mixed-bias run with their single-bias counterparts below. The "+" symbol indicates that the interval's upper endpoint was not observed within the available training trajectory.

BiasSingle-bias onsetMixed-bias onset
Lexical116 [115, 161]161 [85, 161]
Self-praise478 [478, 492]439 [439, 497+]

While the lexical bias exhibits an overlapping onset interval, the self-praise bias shows an earlier canonical onset but a later upper bound of its onset interval. To further examine the exploitation dynamics of self-praise, we compare its shortcut incidence in mixed-bias and single-bias training, using a trailing five-step moving average.

Self-praise incidence thresholdMixed-bias trainingSingle-bias training
10%413467
15%445475
20%465477
Steps (10% → 20%)5210

Interestingly, self-praise reaches 10% incidence earlier in the mixed-bias setting, but its subsequent amplification is substantially slower: progressing from 10% to 20% incidence takes 52 steps under mixed-bias training, compared with only 10 steps under single-bias training.

These findings suggest a potential competition effect: the presence of an easily exploitable lexical bias may not prevent the discovery of self-praise, but may slow its subsequent amplification as the policy preferentially exploits the lexical shortcut.

3. Implications. This observation offers a possible explanation for why rubric-based RL often exhibits only a small subset of the many biases documented in LLM judges: optimization may concentrate on readily exploitable biases, limiting the subsequent amplification of other shortcuts. While further experiments are needed to establish the generality of this effect, the results demonstrate that CHERRL can support controlled investigations of bias interactions, beyond reproducing reward hacking induced by isolated biases. We believe this extensibility makes CHERRL a useful foundation for deeper analysis of reward hacking in rubric-based RL.

We thank the reviewer for this valuable suggestion and have incorporated this additional experiment into Appendix P of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 4JUQ (Part 2/2)

Q2: How robust are the experimental findings given the limited number of hacking trajectories? (Weakness #3)

We strengthen the empirical evidence by adding independent GRPO runs, which show relatively consistent hacking onset times across training seeds.

We appreciate the reviewer's recognition of the computational cost of RL experiments. To address the concern about the limited number of training runs, we conducted two additional independent GRPO runs on the HealthBench lexical-bias setting, yielding three runs in total, including the original one.

SeedReference onset
Default91 [91, 95]
42108 [108, 112]
12397 [97, 105]

The three onset point estimates range from 91 to 108, with a maximum difference of 17 steps. These results provide additional evidence that the observed hacking onset is reproducible across independent runs.

The results are in Appendix M of the revised PDF.

We appreciate the reviewer’s insights which helps improve our paper, and warmly look forward to future discussions.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewer 4JUQ

We thank the reviewer again for their constructive feedback. We hope that our additional experiments and clarifications have addressed your concerns. As the discussion period draws to a close, we would welcome any further questions or comments you may have, and would be glad to provide additional clarification.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Correction to Appendix Table 19

We would like to report a correction to the reference onset values in Appendix Table 19 that we identified while rechecking our analysis. We found that the originally reported values for VerInstruct format had been computed using an earlier version of our threshold-sweep procedure, rather than the finalized procedure specified in Appendix A. We have recomputed these values from the original experimental data using the exact procedure already defined in the submitted paper. The corrections are as follows:

αOriginally reported onsetCorrected onset
0.3316 [316,360+]442 [301,445]
1.0262 [262,360+]98 [98,370+]

The underlying training runs and experimental data remain unchanged. In the revised PDF, we have replaced the original Table 19 analysis with an updated analysis using normalized reward gaps to account for the effect of α on onset estimation. The corrected values above are therefore provided for transparency.

We apologize for this oversight and are reporting the correction separately to ensure that all reviewers and the Area Chair are informed.

Summary of Rebuttal and Manuscript Revisions

We sincerely thank the Area Chair, Senior Area Chairs, and all reviewers for their time and constructive feedback.

Recognized Strengths of Our Work

Multiple reviewers recognized CHERRL's value as a controlled testbed for studying reward hacking in rubric-based RL. Reviewer 4JUQ described it as "useful and helpful for further researchers," while Reviewer czB5 appreciated its ability to isolate specific reward defects. Reviewer eVSq described our work as "a good paper" and our experimental design as "very solid and robust."

The reviewers also recognized the strengths of our methodology and analysis. Reviewers 4JUQ and czB5 appreciated our structured approach to tracking reward divergence and identifying hacking onset, while Reviewer eVSq highlighted our discoverability–exploitability decomposition and insightful training dynamics.

Rebuttal Progress and Additional Experiments

During the rebuttal, we conducted extensive additional experiments and revised the manuscript to address most of the reviewers' concerns. Reviewer 4JUQ acknowledged that our experiments addressed the extensibility concern, while Reviewer czB5 confirmed that several original concerns had been addressed and increased their rating. We subsequently provided further clarification and experimental evidence in response to their feedback.

The main additions to the revised PDF are:

  1. Appendix M — Seed robustness: Two additional GRPO runs demonstrate relatively consistent hacking onsets across seeds.
  2. Appendix N — RHDA reliability: Expanded evaluation confirms that the original RHDA outperforms the CC-Qwen baseline in both false-positive rate and onset localization, while prompt refinement further reduces the false-positive rate to 5% without sacrificing its advantage in onset localization.
  3. Appendix O — Reference-onset validation: Blind behavioral checks provide independent evidence supporting the operational reference onset.
  4. Appendix P — Framework extensibility: Mixed-bias experiments demonstrate CHERRL's ability to study interactions between multiple biases and reveal a potential competition effect between lexical and self-praise shortcuts.
  5. Appendix Q — Cross-model generalizability: MiniCPM5-2B experiments reproduce reward hacking beyond Qwen3-4B and reveal model-dependent onset dynamics.

Manuscript Revisions

We also clarified the generality of CHERRL's dual-judge formulation (Section 2.2), the operational reference-onset definition (Section 2.3; Appendix A.1), and the discoverability analysis with normalized reward gaps and cross-dataset validation (Section 3.3; Appendix H.2). We explicitly acknowledged the limitations of our current implementation (Section 6) and separately disclosed a correction to the original Appendix Table 19.

We sincerely thank all reviewers again for their valuable feedback, which has helped us strengthen the empirical evidence, methodological clarity, and scope of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.