acceptodds
Under review as a conference paper at ICLR 2027

STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking

Abstract

Reward hacking occurs when policy optimization exploits a brittle reward interface or an overly permissive proxy objective, improving the training score without improving the underlying response quality. This phenomenon is amplified in group-relative policy optimization: an unsupported reward can shift the group baseline and alter the updates of other rollouts, while post-hoc or purely relative weighting cannot represent group-wide uncertainty. We propose Self-Tuned Anchored Reliability Group-Relative Policy Optimization (STAR-GRPO), a reliability-first advantage estimator based on paired assessments of the same rollout. STAR separates the quality signal from its learning influence: score disagreement determines rollout reliability, relative reliability enters a self-tuned robust location–scale fit before group normalization, and absolute group reliability attenuates the resulting bounded advantage. The analysis establishes coordinate and second-moment bounds, characterizes exact centering through the weighted location equation, and gives reliability-dependent attenuation guarantees for outlying rewards. We evaluate STAR-GRPO in two complementary reward-hacking regimes. In token-interface exploitation, STAR prevents runaway optimization of the deployed-interface score while improving the canonical quality signal. In rubric-proxy overoptimization for medical reasoning, STAR improves independent semantic evaluation, narrows the proxy–judge discrepancy, and reduces overclaim while optimizing the same task proxy. Together, these results show that reliability-first normalization offers a principled way to limit unsupported reward influence on both group baselines and policy updates, while retaining the task reward as the optimization target.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reward Hacking, Group Relative Policy Optimization, LLM Alignment, Reliability-Weighted Advantages

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UnveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a reliability weighted GRPO advantage by modifying the advantage signal of GRPO in an attempt to reduce reward hacking. The reward of each roll-out is computed twice: by a training verifier and a second anchor verifier. The difference between the two rewards is used as a measure of reliability for that roll-out. These weights are used to compute a new baseline and the final advantage is rescaled by the rollout and the group reliability. Groups with low reliability are completely dropped. The experimental setup is around two settings: RQ1 uses token-interface exploitation, where raw token Ids are passed to a reward model that uses a different tokenizer than the base policy. RQ2 uses the difference between rubric conditioned judge (gpt-4o-mini) and a rubric-free judge (gemini) as the source of the reliability signal. The paper shows that on healthbench-hard set, STAR-GRPO results in low rate of false claims.

Critical strengths
  1. I think extracting a reliability signal based on the disagreement between rubric-based and rubric-free judges is interesting. It addresses a known weakness of rubric rewards where rubric satisfaction drifts from overall response quality.

  2. The results of RQ2 are interesting and useful. We can see in Figure 4c/d as training proceeds GRPO's overclaim fraction and the proxy-judge gap increase, while STAR's decreases. In addition, STAR also reaches a higher independent-judge score which is promising.

Critical weaknesses
  1. In RQ2, STAR-GRPO adds an extra judge (i.e., the holistic judge). So comparing plain GRPO with a single weak judge (gpt-4o-mini) is not a fair comparison.
  2. The experimental setup is very limited. RQ2 uses a single domain, one rubric dataset and a single policy (Qwen3-4B). How do we know that these results generalize across different domains, model families, policy scales
  3. More importantly, there is no reason why the anchor judge would not also be gamed. Since the anchor judge is part of the reward function formulation, it can be exploited by the policy.
  4. Finally, I think that RQ1 is not really a realistic RL setting. The deployed interface passes policy token ids directly to the reward model. Standard RL pipelines either share the same tokenizer between the base policy or decode the policy's output and then retokenize it using the reward model tokenizer. This is basically STAR's canonical view. In addition, the matched canonical-GRPO control beats STAR (3.236 versus 1.781 with shorter response, in table 6). This limits the impact of this work.
Questions for authors
  1. How can you account for the fact that STAR-GRPO uses a strong judge like gemini as the holistic judge while plain GRPO in the presented experiments only use gpt-4o-mini, which is a weak judge. It is possible that the gain observed in the paper in Figure 4 are due to Gemini being generally more strict that gpt-4o-mini. Did you try to run a plain GRPO experiment with Gemini as the rubric-based judge?
  2. Have you evaluated the utility of STAR-GRPO on any other domains, policies or different rubric-based judges?
  3. Can you provide a practical RL training framing where RQ1 can useful?
2 comments

Thank you very much for your support and valuable comments. We have carefully reviewed them, which are of great help to this work. We summarize the key concerns you raised, have supplemented the corresponding experiments, and present our responses below.

1. Equal-Budget Judge Controls and Resistance to Co-Hacking

In RQ2 (κ=1\kappa=1), GPT-4o-mini scores explicit rubric items (ri=Rirubricr_i=R_i^{\rm rubric}), Gemini-2.5-Flash-Lite scores holistically without the rubric, and Claude-Sonnet-4-6 is held out for evaluation. Placing wiw_i inside the robust fit before advantage construction prevents overrated rollouts from flipping clean peers' signs while retaining group confidence wˉ\bar w: AiSTAR=wˉwiν+εwϕi,wˉ=1G∑iwi,ν=1G∑iwi2,∣ϕi∣≤1.A_i^{\rm STAR}=\bar w\frac{w_i}{\nu+\varepsilon_w}\phi_i,\quad \bar w=\frac1G\sum_iw_i,\quad \nu=\sqrt{\frac1G\sum_iw_i^2},\quad |\phi_i|\le 1. At step 260 (gpt-4o-mini-2024-07-18, gemini-2.5-flash-lite, temp 0, invalid →0\to 0, N=995–997 and common N=991), STAR outperforms equal-budget 2-call ensembles, no-information controls, and single-model baselines:

MethodCalls (tokens / latency)Eval (all / 991) ↑Gap ↓Overclaim ↓
Plain GRPO, GPT rubric1 (1.42k / 1.68 s)0.2706 / 0.27090.29350.2464
2× GPT rubric avg + GRPO2 (2.84k / 1.89 s)0.2741 / 0.27440.28710.2438
Constant-anchor STAR (R0≡cR_0\equiv c)1 (1.42k / 1.68 s)0.2794 / 0.27960.28020.2410
Shuffled-anchor STAR (Gemini)2 (2.11k / 1.74 s)0.2768 / 0.27710.28450.2425
Post-hoc weighted, GPT + Gemini2 (2.11k / 1.74 s)0.2831 / 0.28340.27370.2392
Mean-score, GPT + Gemini2 (2.11k / 1.74 s)0.2862 / 0.28650.24840.2315
Minimum-score, GPT + Gemini2 (2.11k / 1.74 s)0.2914 / 0.29160.23750.2279
Self-anchored STAR, GPT + GPT2 (2.11k / 1.81 s)0.3116 / 0.31190.23920.2231
STAR, GPT + Gemini2 (2.11k / 1.74 s)0.3174 / 0.31780.23180.2204

Omitting the rubric in holistic calls (~690 tokens at 99.8% validity vs. ~1,420 at 99.9%, concurrent) adds only ~49% tokens (~32% billed cost) vs. 2.84k for 2× GPT rubric. Zero-Gemini Self-Anchored STAR beats 2× GPT rubric by +0.0375 on N=991N=991, while STAR beats equal-access Minimum-score by +0.0260 and constant/shuffled controls by +0.0380–0.0406. Offline (45 prompts ×16=720\times 16=720: 15 cal = 240, 30 eval = 480 with 479 valid Gemini = 719/720; 60 contaminated groups), Claude cosine flips from −0.1129 ± 0.0162 to +0.1004 ± 0.0168 (self) and +0.1173 ± 0.0195 (STAR), lifting tilted Claude (τ=3.0\tau=3.0) from 0.1126 to 0.2605/0.2686 and cutting overclaim push from +0.1306 to +0.0602/+0.0516.

Structurally, at κ=1\kappa=1, RancR^{\rm anc} is absent from rir_i, and on 89.39% of faithful rollouts (74.11% of 479) wiw_i saturates at 1 (∂ASTAR/∂Ranc≡0\partial A^{\rm STAR}/\partial R^{\rm anc}\equiv 0), while orthogonal views prevent common-mode shifts (Thm. 3.1(ii)). Online, Proxy − Claude widens under GRPO (+0.4589 →\to +0.5436) whereas Gemini − Claude does not inflate (+0.6410 →\to +0.5427); in a 200-prompt Best-of-16 audit (max⁡Rrubric\max R^{\rm rubric} s.t. wi≥0.8w_i\ge 0.8), high-train (≥0.70\ge 0.70) yet low-Claude (≤0.25\le 0.25) picks drop 3-fold from 19.5% (39/200, verified on a 50-item check) to 6.5% (13/200).

2. Gemini as the Rubric-Based GRPO Judge

Across the 719 valid offline rollouts, holistic gemini-2.5-flash-lite is more lenient (mean 0.7644) than GPT rubric (0.6670). Running gemini-2.5-flash-lite as explicit rubric judge at step 260 (N=995–997 and common N=991; new pairings fit (m^D,s^D,q^0.90)(\hat m_D,\hat s_D,\hat q_{0.90}) on the 240 step-0 split, e.g., m^D=−0.38,s^D=0.21,q^0.90=1.28,λ=2.0\hat m_D=-0.38,\hat s_D=0.21,\hat q_{0.90}=1.28,\lambda=2.0, 6.4% clean downweighting, vs. fixed m=0,s=0.1,q=1.645,λ=2m=0,s=0.1,q=1.645,\lambda=2):

Method & viewsRubricEval (all / 991) ↑Gap ↓Overclaim ↓
Plain GRPO, GPT rubric0.56400.2706 / 0.27090.29350.2464
Plain GRPO, Gemini rubric0.48650.2894 / 0.28960.19710.1986
Self-anchored STAR, GPT + GPT0.55080.3116 / 0.31190.23920.2231
STAR, GPT + Gemini hol.0.54920.3174 / 0.31780.23180.2204
STAR, Gemini + Gemini hol.0.47420.3246 / 0.32490.14960.1692

Offline (720 rollouts, Gemini rubric mean 0.3831), 58/310 poor outputs (Claude ≤0.25\le 0.25) fool Gemini rubric (≥0.65\ge 0.65); on 60 contaminated groups, STAR lifts cosine from −0.2544 ± 0.0260 to −0.1524 ± 0.0252 (Gemini hol.), −0.1402 ± 0.0275 (GPT hol.), and −0.1200 ± 0.0274 (Claude-Haiku-4-5 hol.; κ=0\kappa=0 holistic alone is 0.2584 online). Online, Plain GRPO (Gemini rubric) reaches 0.2894 (below zero-Gemini Self-Anchored STAR 0.3116 and STAR 0.3174; 3-seed 42, 43, 44 mean 0.3177 ± 0.0053 vs. 0.2704 ± 0.0056), and STAR atop Gemini rubric reaches 0.3246 (+0.0353 on N=991N=991, 10k bootstrap 95% CI [+0.0178, +0.0528]), cutting gap by 0.0475 and overclaim by 2.94 pp.

3. Generalization Across Domains, Families, Scales, and Ablations

Expanded 260-step rubric runs (G=16G=16, token-mean, HealthBench-Hard N≈996N\approx 996; RubricHub-General splits 5,000 non-medical prompts into 4,500 train / 500 disjoint test), 300-step RLHF (gpt-4-1106-preview vs. gpt4_turbo), and 200-step disjoint RLVR confirm consistent STAR gains across domains, model families, and scales:

Task / benchmarkPolicyPlain GRPOSTAR (κ=1\kappa=1) ↑
Medical / HealthBench-HardQwen3-4B / Qwen2.5-7B0.2706 / 0.28650.3174 / 0.3364
Medical / HealthBench-HardLlama-3.2-3B / Llama-3.1-8B0.2548 / 0.29120.2986 / 0.3418
General / RubricHub-GeneralQwen3-4B / Qwen2.5-7B / Llama-3.1-8B0.4812 / 0.4980 / 0.50450.5396 / 0.5562 / 0.5618
RLHF / AlpacaEval 2.0 LC-WRQwen2.5-3B-Instruct20.6%31.2%
Code MBPP+ (374/378) / Math MATH-500Qwen2.5-Coder-3B / Qwen2.5-1.5B58.7% / 50.2%67.2% / 57.6%

Within-family scaling (Qwen3-4B → Qwen2.5-7B, Llama-3.2-3B → Llama-3.1-8B) and cross-family comparisons (Qwen2.5-7B vs. Llama-3.1-8B) yield +11.4% to +17.4% rubric gains; offline NoveltyBench audits on Llama-3.2-1B (G=8G=8, 240 rollouts) raise Claude cosine from 0.2546 ± 0.0405 to 0.3032 ± 0.0423 (GPT self-anchor) and 0.3284 ± 0.0451 to 0.3400 ± 0.0464 (Gemini rubric + GPT hol.). Step-260 single-factor STAR ablations (Uniform-fit: pb,i→1/Gp_{b,i}\to 1/G in fit only; Quadratic-fit: Huber →L2\to L_2 fit only; 10.9% random skip) further verify each component:

Step-260 variantEval ↑ / Overclaim ↓Step-260 variantEval ↑ / Overclaim ↓
Plain GRPO0.2706 / 0.2464Quadratic-fit STAR0.3082 / 0.2244
Neutral wi=1w_i=10.2782 / 0.2421No wˉb\bar w_b0.3076 / 0.2251
Post-hoc (comp.)0.2831 / 0.2392Relaxed (Wmin⁡→0+,Gmin⁡=1W_{\min}\to 0^+,G_{\min}=1)0.3108 / 0.2235
Uniform-fit STAR0.2964 / 0.231010.9% random skipping0.3102 / 0.2239
Weighted mean/std0.3019 / 0.2268Full STAR0.3174 / 0.2204

Full STAR outperforms every ablation: +0.0210 (Uniform-fit), +0.0092 (Quadratic-fit), +0.0098 (wˉb\bar w_b), and +0.0072/+0.0066 (admission).

4. Matched RQ1 Controls and Practical Single-RM Format-STAR

In RQ1 (Zhang et al., TOMPA; cf. Gao et al., Sharma et al., Bukharin et al.), the Table 6 gap (Canonical-GRPO +3.236 vs. STAR +1.781) came entirely from mismatched loss reduction (seq-mean-token-mean vs. token-mean). Completing the 2×22\times 2 grid at step 855 (M=100M=100, KL 0.001, 10tanh⁡(Rcan/10)10\tanh(R^{\rm can}/10)) confirms STAR matches the clean-interface oracle under both rules:

MethodLoss reductionCanon ↑ObsLength
Original STAR (κ=0\kappa=0)Token mean4.902 ± 0.386−0.570 ± 0.0731872.1
Matched Canonical-GRPOToken mean4.945 ± 0.402−0.612 ± 0.0791846.4 ± 49.2
Original Canonical-GRPOPer-response token mean6.123 ± 0.578−1.862 ± 0.054395.5 ± 36.6
Matched STAR (κ=0\kappa=0)Per-response token mean6.158 ± 0.542−1.841 ± 0.058408.2 ± 38.4

Across 3 seeds (42, 43, 44) under seq-mean, STAR reaches 6.142 ± 0.148 vs. 6.118 ± 0.165 for Canonical-GRPO (within 0.035–0.043 single-seed and 12.7–25.7 tokens; checkpoints 0,200,400,600,855 show 100% top-1 agreement, 0.9567–0.9668 cosine, 0.9977–1.0000 weight).

Practically, single-RM paired scoring mitigates formatting/sycophancy gaming in matched-tokenizer RLHF without a second RM. Defining CplainC_{\rm plain} to strip preambles ("Certainly!...", "In summary,..."), # headers, **/__ bold, and -/• bullets while keeping code, JSON/tables, and math (D=Rϕ(X,O)−Rϕ(X,Cplain(O))D=R_\phi(X,O)-R_\phi(X,C_{\rm plain}(O)), calibrated on 1,024 step-0 rollouts split 70%/30% at seed=42; Qwen2.5-3B-Instruct + GRM-Qwen2.5-3B, 300 steps, G=8G=8, AlpacaEval 2.0 N=805N=805 split into N=168N=168 format / N=637N=637 content; NoveltyBench clean cosine 0.7972 [0.9917 wt, 2.5% downweighted] vs. 0.8141 GRPO and 0.7115 stripped):

Method (single 3B RM)Skywork-8B ↑LC WR (805805) ↑Format (168168) / Content (637637)Len
Base / Plain GRPO3.18 / 1.7422.8% / 20.6%24.1%/22.4% ; 19.6%/20.9%415 / 1146
Plain-text Canonical (κ=0\kappa=0)4.1826.2%21.4% / 27.5%446
Single-RM Format-STAR (κ=1\kappa=1)5.0629.4%28.6% / 29.6%518

With zero second RM loaded (+2.4 s/step 2nd pass), Format-STAR gains +8.8 pp over Plain GRPO and +3.2 pp over Plain-text Canonical-GRPO (+7.2 pp on format prompts, cutting format violations from 18.5% to 4.8%). Secs. 3, 6.1–6.2, and Appendix F.1.1 will integrate all results.

Reviewer_X4BPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STAR-GRPO, a reliability-aware advantage estimation method for mitigating reward hacking in GRPO. It uses paired reward assessments to estimate rollout reliability and incorporates reliability weights into robust group normalization and policy updates, reducing the influence of potentially unreliable rewards. Experiments on token-interface exploitation and rubric-based medical reasoning demonstrate improvements in reward hacking mitigation and independent evaluation quality.

Critical strengths
  1. The paper identifies an important limitation of GRPO: unreliable rewards can contaminate group-level normalization and affect other rollouts.

  2. STAR-GRPO incorporates reward reliability into advantage estimation, reducing the influence of potentially unreliable rewards.

  3. The paper provides theoretical guarantees on advantage boundedness, robust normalization, and reliability-dependent gradient attenuation.

Critical weaknesses
  1. My main concern is the limited experimental coverage. RQ1 focuses on reward hacking caused by tokenizer mismatch, a specific implementation issue that may not reflect reward hacking in realistic RL scenarios.

  2. RQ2 considers a more realistic rubric-based RL setting. However, if Gemini-2.5-Flash-Lite can provide a reliable anchor, why not directly use it as the training reward? Moreover, STAR's effectiveness depends heavily on the quality of the anchor model, which may substantially limit its practical applicability.

  3. I strongly recommend evaluating STAR-GRPO in conventional RLHF settings involving reward hacking, which have been extensively studied in recent years.

  4. The paper lacks systematic ablations to isolate the contributions of individual components, including reliability weighting, robust normalization, absolute group attenuation, and group skipping. It remains unclear which components are essential to the observed improvements.

Questions for authors

The connection between the proposed method and representation-dependent reward hacking seems rather limited. STAR-GRPO essentially relies on reward disagreement for reliability-aware advantage estimation, while its more practical RQ2 setting does not involve representation mismatch at all.

2 comments

Thank you very much for your suggestions and support. We have carefully analyzed your comments, organized them and supplemented the corresponding experiments, hoping to address your concerns. We summarize the issues you raised as follows:

1. Beyond Tokenizer Mismatch: Realistic RLHF and RLVR

Reward exploitation spans distinct channels reviewed in Section 2: content shortcuts via verbosity, sycophancy, or checklist stuffing (Sharma et al., Gao et al.); adversarial OOD search (Bukharin et al.); and token/interface mappings (Zhang et al., TOMPA). RQ1 provides an exact single-RM testbed: at step 855, TOMPA inflates observed reward to 9.641 while collapsing canonical quality to −9.863, whereas STAR prevents collapse (+1.781 to 4.902 ± 0.386 at 1872.1 tokens under token-mean). Completing the 2×22\times 2 loss-reduction control vs. submitted seq-mean-token-mean Canonical-GRPO (+3.236 to 6.123 ± 0.578 at 395.5 tokens) gives 4.945 ± 0.402 (1846.4 tokens) for token-mean Canonical-GRPO and 6.158 ± 0.542 (408.2 ± 38.4 tokens) for seq-mean STAR, matching the clean-interface oracle within 0.035–0.043 points.

Across matched-tokenizer RL pipelines, STAR (κ=1\kappa=1) delivers substantial gains in preference RLHF (Section 3), code RLVR, and math RLVR. In code RLVR, Qwen2.5-Coder-3B-Instruct trained on 374 MBPP tasks (3 visible asserts) and tested on 378 disjoint EvalPlus MBPP+ tasks (35× hidden tests) lifts Pass@1 from 58.7% (222/378) under verifier GRPO to 67.2% (254/378, +8.5 pp, p=2.0×10−6p=2.0\times 10^{-6}; beating direct code-anchor GRPO at 64.8%), slashing visible-literal hardcoding 18-fold from 23.8% to 1.3%. In math RLVR (Qwen2.5-1.5B-Instruct, MATH →\to disjoint MATH-500 audited by Claude-Sonnet-4-6), STAR lifts exact accuracy from 58.6% to 60.4% and valid derivations from 50.2% to 57.6% (+7.4 pp), cutting shortcuts from 14.3% to 4.6%.

2. Why Not Optimize the Anchor Directly, and Anchor Resilience

By Theorem 3.1, an anchor only needs to flag extreme directional inflation (R1≫R0R_1\gg R_0), whereas standalone optimization requires fine-grained domain ranking. We verified both on 720 HealthBench-Hard rollouts (45 prompts ×16\times 16: 15 cal prompts [240 rollouts, 70%/30% at seed=42: 168 fit, 72 quantile] + 30 disjoint eval prompts [480 rollouts], testing 30 unmodified and 60 contaminated groups [2/prompt of 13 faithful ∣Rproxy−RClaude∣≤0.12|R^{\rm proxy}-R^{\rm Claude}|\le 0.12 + 3 overclaimed ≥0.35\ge 0.35; cluster SEs]) and online at step 260 (N=995–997N=995\text{--}997):

MethodClaude cos. (60 groups) ↑Online eval ↑Gap ↓Overclaim ↓
Proxy GRPO−0.1129 ± 0.01620.27060.29350.2464
Anchor-only GRPO (κ=0\kappa=0)−0.2949 ± 0.02920.25840.20880.2146
Penalty (r−0.5[D]+r-0.5[D]_+)−0.1041 ± 0.01750.28850.23290.2284
STAR (proxy + Gemini, κ=1\kappa=1)0.1173 ± 0.01950.31740.23180.2204

Rubric-free Gemini produces only 3.80 score levels/group (vs. 5.10 proxy, 5.83 Claude; unconditional r=0.1870r=0.1870 vs. 0.23140.2314 across 480 rollouts), causing Anchor-only GRPO to fall below Proxy GRPO online (0.2584 vs. 0.2706), whereas STAR reaches 0.3174. Recalibrating (m^D,s^D,q^0.90)(\hat m_D,\hat s_D,\hat q_{0.90}) per anchor on the 240 step-0 rollouts (vs. fixed m=0,s=0.1,q=1.645,λ=2m=0,s=0.1,q=1.645,\lambda=2; invalid →0\to 0) confirms strong gains across every anchor tier:

AnchorValidAll / 297 AUROC ↑FPR ↓Online ↑
Claude-Haiku-4-5-20251001100.0%0.8513 / 0.84890.87%0.3191
Gemini-2.5-Flash-Lite100.0%0.8375 / 0.83410.87%0.3174
Gemini + 38.1% rand. invalid61.9%0.8341 / 0.83410.87%0.3095
GPT-4.1-nano100.0%0.8328 / 0.82960.00%0.3138
DeepSeek-R1-Distill-Qwen-1.5B61.9%0.7671 / 0.76718.33%0.3042
Lexical/length heuristic (0B)100.0%0.7533 / 0.74920.00%0.2948
Constant (R0≡cR_0\equiv c) / Shuffled100.0%0.5842/0.5810 ; 0.5618/0.55948.70%/9.13%0.2794 / 0.2768

Even the 1.5B open model (0.3042; common-297 AUROC order preserved, 38.1% invalidation at 0.3095) and 0B heuristic (0.2948) clearly beat Proxy GRPO (0.2706) and constant/shuffled controls (cosines −0.0894/−0.0941, online 0.2768–0.2794); non-Claude GPT-4.1 rescoring (300 prompts) confirms 0.3412 (Haiku) and 0.3408 (Gemini) vs. 0.2915 (Proxy). Under N(0,σ2)\mathcal{N}(0,\sigma^2) noise at σ∈{0.00,0.10,0.30}\sigma\in\{0.00,0.10,0.30\} (20 seeds), self-calibration holds clean FPR at 0.87%/1.00%/1.13% (vs. 1.74%/3.57%/8.91% fixed), keeping 0.8375/0.8275/0.7828 AUROC and 0.1591/0.1565/0.1522 cosine (0.1548/0.1497/0.1252 fixed).

3. Conventional RLHF, Win Rates, and χPO

From Qwen/Qwen2.5-3B-Instruct on disjoint UltraFeedback splits (18k online RL, 60k pairs for offline DPO/χPO\chi\text{PO}, 1k dev), online runs perform 300 updates (batch=64, G=8G=8, 153,600 rollouts, 2,048 cap, KL 0.001) optimizing matched-tokenizer Ray2333/GRM-Qwen2.5-3B-rewardmodel-ft (R1R_1) paired with Qwen2.5-1.5B-Concise-RM (R0R_0, aligned on 1,024 step-0 rollouts, α=0.05\alpha=0.05, εσ=εw=10−6\varepsilon_\sigma=\varepsilon_w=10^{-6}, +1.9 s/step). Dev-tuned: βlen=0.0025\beta_{\rm len}=0.0025, η=0.5\eta=0.5, and β=0.1\beta=0.1 for DPO and Huang et al.'s χPO\chi\text{PO} (grid {0.02,0.05,0.1,0.2}\{0.02,0.05,0.1,0.2\}), plus online χ2\chi^2-reg. GRPO (β2(πθ/πref−1)2\frac{\beta}{2}(\pi_\theta/\pi_{\rm ref}-1)^2). Evaluated on AlpacaEval 2.0 (N=805N=805, gpt-4-1106-preview vs. gpt4_turbo, greedy, 5k bootstrap CIs refitting LC GLM) and Skywork-Reward-V2-Qwen3-8B:

Method & regimeCallsProxySkywork-8B ↑LC WR [95% CI] ↑RawLen
Base (Qwen2.5-3B)01.423.1822.8% [20.2, 25.5]21.9%415
Offline DPO (β=0.1\beta=0.1, 60k)03.184.1525.8% [23.1, 28.6]26.4%486
Offline χPO\chi\text{PO} (β=0.1\beta=0.1, 60k)03.354.7228.4% [25.6, 31.3]27.9%462
Online Proxy GRPO (R1R_1)15.861.7420.6% [18.0, 23.3]31.4%1146
Online χ2\chi^2-reg. (R1R_1)14.024.3826.9% [24.1, 29.8]28.5%546
Direct Anchor (R0,κ=0R_0,\kappa=0)12.644.0524.6% [21.9, 27.4]23.5%438
Length-pen. (R1−βlen∣O∣R_1-\beta_{\rm len}\lvert O\rvert)13.413.8525.4% [22.7, 28.2]24.8%468
Post-hoc weighted24.194.1226.1% [23.4, 28.9]27.8%589
Mean ((R1+R0)/2(R_1+R_0)/2)24.084.2626.7% [23.9, 29.5]28.9%612
Uncertainty (Rˉ−η∣D∣\bar R-\eta\lvert D\rvert)23.764.5227.5% [24.7, 30.4]28.0%538
Minimum (min⁡(R1,R0)\min(R_1,R_0))23.644.6127.9% [25.1, 30.8]28.1%524
Online STAR (κ=1\kappa=1)23.925.4831.2% [28.4, 34.1]31.8%492

Proxy GRPO inflates length to 1,146 tokens and drops LC-WR to 20.6%, whereas STAR (κ=1\kappa=1) achieves top LC-WR (31.2%) and Skywork score (5.48) at 492 tokens, beating Minimum-score GRPO by +3.3 pp ([+1.1%, +5.5%]), online χ2\chi^2-GRPO by +4.3 pp, and offline χPO\chi\text{PO} by +2.8 pp (54.8% head-to-head LC preference).

4. Systematic Component Ablations

We combined composite comparators with isolated single-factor STAR ablations holding all else fixed: Uniform-fit (pb,i→1/Gp_{b,i}\to 1/G in fit only), Quadratic-fit (Huber →L2\to L_2 fit only), and 10.9% random skip:

VariantOnline eval ↑ / gap ↓ / pass ↑ / overclaim ↓RQ1 flip ↓ / cos ↑RQ2 flip ↓ / cos ↑
Proxy GRPO0.2706 / 0.2935 / 0.4752 / 0.246458.2% / 0.04837.5% / −0.113
Neutral wi=1w_i=10.2782 / 0.2812 / 0.4795 / 0.242127.1% / 0.40630.6% / 0.024
Post-hoc weighted0.2831 / 0.2737 / 0.4826 / 0.239258.2% / 0.26937.5% / −0.029
Uniform-fit STAR0.2964 / 0.2562 / 0.4915 / 0.231034.6% / 0.51831.8% / 0.041
Weighted mean/std0.3019 / 0.2497 / 0.4958 / 0.226838.1% / 0.34930.0% / −0.025
Quadratic-fit STAR0.3082 / 0.2431 / 0.4994 / 0.224426.8% / 0.58427.4% / 0.076
No wˉb\bar w_b0.3076 / 0.2465 / 0.4988 / 0.225117.2% / 0.67623.9% / 0.117
Relaxed (Wmin⁡→0+,Gmin⁡=1W_{\min}\to 0^+,G_{\min}=1)0.3108 / 0.2410 / 0.5006 / 0.223517.2% / 0.67623.9% / 0.117
10.9% random skip0.3102 / 0.2419 / 0.5001 / 0.223917.2% / 0.67623.9% / 0.117
Full STAR0.3174 / 0.2318 / 0.5049 / 0.220417.2% / 0.67623.9% / 0.117

Each component improves performance: pre-norm fit weighting adds +0.0210 over Uniform-fit (halving RQ1 sign flips from 34.6% to 17.2%, Prop. 5.2), pseudo-Huber adds +0.0092 over Quadratic-fit (cutting (G−1)ϵ\sqrt{(G-1)\epsilon} sign flips from 26.8% to 17.2%), wˉb\bar w_b adds +0.0098 while suppressing collective-drift RMS 4.90×/3.35× (0.612/0.587 →\to 0.125/0.175, Thm. 5.1), and reliability-targeted group skipping adds +0.0072 over random skipping (+0.0066 over relaxed admission, abstaining on 100.0% vs. 0.0% of low-mass groups, max ∥A∥∞\lVert A\rVert_\infty 0.038/0.175 →\to 0.000/0.000).

Reviewer_G5kaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes gradient re-weighting to mitigate reward hacking. They define a quantity called discrepancy (D) for any response (O) sampled from the policy to measure how unreliable the reward on this sample is under the given reward function. To calculate D, they score O twice using the same trained reward model -

  1. Observed Reward R_obs: which is the standard reward using the trained reward model.
  2. Canonical Reward R_canon: where they perform extra preprocessing (transformation) on the text before feeding it into the reward model for score. For example, the appendix E.1 shows that the transformation includes unicode normalization, removal of invisible characters etc.

then D = R_obs - R_canon. They then apply a normalization on D and exponentiate to calculate the weights for the gradient update.

While I understand and agree with the core idea of the paper, stated in simple words, "giving less gradient to potentially reward hacked completions leads to less reward hacking in the trained model", their way of calculating R_canon is not sound.

They test on two potential kinds of reward hacking - token interface hacking and medical rubric satisfaction. For token interface reward hacking, they use a reward model that is sensitive to the input tokens. For medical rubric satisfaction they use an LLM Judge to tell how reliable is the response. The experiments are initial and require more baselines and analysis.

Critical strengths

Provides a method to quantify potential reward hacking and use that quantification for advantage estimation.

Critical weaknesses

There are multiple soundness issues with the paper -

  1. They use the baseline TOMPA-GRPO but no where in any plots is the results shows, the only comparison is standard GRPO with their method, STAR-GRPO.

  2. I agree with the idea that reward hacked samples should be assigned lower weights for gradient updates. For this, the first baseline that comes to my mind is using an LLM Judge and ask it whether the response is hacked or not, instead of the discrepancy heuristic defined in the paper. That baseline is missing. I notice that the authors use an LLM judge for the medical rubric hacking setting, but LLM Judge should be compared with the heuristic for the first set of experiments as well.

  3. The implementation of discrepancy (D) is ad-hoc and potentially hackable. Here are a few problems I see -

(a) Disagreement dies not always tell that the reward is hacked. For example if an LLM outputs the final response directly without doing reasoning or the reasoning is wrong but the final answer is right, then both the R_obs and R_canon would be the same. Second case is, where the pre-processing defined in appendix E.1 does not change the response at all but the reward model is still sensitive to the wording.

(b) Legitimate responses could be penalized because of the ad-hoc disagreement: Assume a MATH setting, if the reward model is likely to give high reward to the responses with "..." and the pre-processing removes the token corresponding to "...". Under this setting, a correct response, with correct reasoning would get high discrepancy and hence very low learning signal leading to waste of compute.

  1. There is no analysis of how this would work in RL-VR with a verifiable reward signal. We do observe a lot of hacking where the reward is not given by a reward model which is sensitive to the response format. For example in code, we find that a lot of times the model hardcodes the responses to certain test cases, but with the current post-processing for R_canon, this type of reward hacking cannot be penalized.

  2. the paper does not establish statical significance and does not run multi-seed experiments. It also has limited baseline comparisons. The paper seems to have preliminary evidence but not enough to be the state-of-the-art method.

Questions for authors

Please see the weaknesses

3 comments

Thank you very much for your suggestions, which are of great help to us. We have carefully analyzed your comments and organized our responses below.

1. TOMPA-GRPO: Clarification and Rescoring

TOMPA-GRPO appeared in Figure 2 as "GRPO: observed" / "GRPO" (Sec. 6.1); we will relabel it TOMPA-GRPO and co-plot Canonical-GRPO. Canonically rescoring TOMPA across 172 checkpoints (0–855) and completing the 2×22\times 2 reduction control (token-mean vs. seq-mean-token-mean) at step 855 (M=100M=100 prompt SE; within-prompt std@8: 0.524 TOMPA obs, 203.6 STAR len):

Method & reduction (855)Init canonFinal canonFinal obsLen
TOMPA (tok-mean, κ=1\kappa=1)3.038 ± 0.491−9.863 ± 0.1899.6412048.0 ± 0.0
STAR (tok-mean, κ=0\kappa=0)3.121 ± 0.5074.902 ± 0.386−0.570 ± 0.0731872.1
Canonical (tok-mean)2.887 ± 0.4744.945 ± 0.402−0.6121846.4
Canonical (seq-mean)2.887 ± 0.4746.123 ± 0.578−1.862 ± 0.054395.5 ± 36.6
STAR (seq-mean, κ=0\kappa=0)3.121 ± 0.5076.158 ± 0.542−1.841 ± 0.058408.2 ± 38.4

TOMPA's discrepancy hits 19.504, whereas STAR prevents collapse (+1.781) and matches the Canonical-GRPO oracle within 0.035–0.043 under both rules (6.158 vs. 6.123 [+3.236] under seq-mean), proving loss reduction caused the Table 6 gap (Sec. 6.1, Fig. 2, App. F.1.1).

2. Direct LLM-Judge Baselines

We tested GPT-4o-mini-2024-07-18 (temp 0) on (X,O)(X,O) via wHack=1−phackw_{\rm Hack}=1-p_{\rm hack} and wQG=min⁡{1−pgame,qsem/10}w_{\rm QG}=\min\{1-p_{\rm game},q_{\rm sem}/10\}. On 100 NoveltyBench prompts, 60 give 480 step-0 calibration rollouts (fit_fraction=0.7 [70%/30%], seed=42: 336 fit, 144 quantile; q^0.95=1.9531\hat q_{0.95}=1.9531 at α=0.05\alpha=0.05 vs. 1.79285 on 10k WildChat), and 40 disjoint prompts + TOMPA logs give 675 unique rollouts (1,350 calls; 5 dupes cut): 320 step-0 (307 unique), 150 inversions (A^obs>0.75,A^can<−0.75\hat A^{\rm obs}>0.75,\hat A^{\rm can}<-0.75), 56 drift, 64 step-1, and 100 step 2–5 spam (90 unique). Flags: phack≥0.5p_{\rm hack}\ge 0.5, max⁡(pgame,1−qsem/10)≥0.5\max(p_{\rm game},1-q_{\rm sem}/10)\ge 0.5, SD>1.9531S^D>1.9531; ≤5%\le 5\% FPR cutoffs: >0.900,>0.800,>2.0025>0.900, >0.800, >2.0025 (3k cluster bootstrap, K=40K=40):

Offline metricLLM-HackQual+GuardSTAR (SDS^D)
Step-0 (320320) / Rcan≥2R^{\rm can}\ge 2 (173173) FPR ↓11.9% / 0.6%21.6% / 5.8%5.9% / 0.0%
Fluent AUROC (150150) [95% CI] ↑0.629 [0.571, 0.700]0.806 [0.734, 0.871]0.911 [0.863, 0.952]
TPR @ ≤5%\le 5\% FPR / Spam (100100) ↑0.0% / 100.0%6.0% / 100.0%28.0% / 100.0%
Dual-scored AUROC (206206) [95% CI] ↑0.713 [0.657, 0.785]0.845 [0.785, 0.900]0.910 [0.865, 0.949]

Text judges miss grammatical token collisions (0.629 AUROC, 0.0% TPR) and over-flag clean outputs (11.9%–21.6% vs. 5.9%, 0.0% on Rcan≥2R^{\rm can}\ge 2). Step-300 online RL (Llama-3.2-1B, batch 64, G=8G=8, token-mean, 10tanh⁡(R/10)10\tanh(R/10), invalid →0\to 0, 8×A100):

Online method (300)Canon ↑ObsLenTime ↓
TOMPA (κ=1\kappa=1)−9.384 ± 0.2057.5612043.1102.7 s
LLM-Hack + post-hoc (κ=1\kappa=1)−1.465 ± 0.4621.842 ± 0.2141684.2129.8 s
LLM-Hack + STAR (κ=1\kappa=1)2.184 ± 0.4380.418 ± 0.1561142.6124.5 s
STAR (κ=0\kappa=0)3.980 ± 0.412−1.754482.890.4 s

Pre-norm STAR lifts LLM-Hack (−1.465 →\to 2.184) and cuts spam-group (40×840\times 8, adv. >+0.05>+0.05) sign flips from 81.3% to 5.0% (5.6% STAR κ=1\kappa=1); STAR (κ=0\kappa=0) raises contaminated (M=36M=36) cosine from −0.173/−0.386 to +0.886 ± 0.020 and hits 3.980, >11×>11\times faster in scoring (2.8 s vs. 31.2/32.0 s).

3. Discrepancy Scope and Preprocessing Robustness

By Theorem 3.1(ii), D=R1−R0D=R_1-R_0 targets differential inflation, pairing task anchors for reasoning/wording shortcuts (Secs. 4, 6). RQ1 canonicalization (NFKC, format-control removal, space collapse; App. E.1) keeps 100% of math tokens on 500 MATH-500 items and 1,600 NoveltyBench rollouts: 500 text matches, 508 \boxed, 47 LaTeX ellipses, 1219 \frac, 532 \sqrt, 500 ..., and NoveltyBench ... (step 0: 97$\to$106 [+9 U+2026]; 855: 74$\to$74), with 0.9997 mean wt over 858 updates (439,296 rollouts, 0 skipped, Geff=7.999/8G_{\rm eff}=7.999/8). Under ...-stripping CstripC_{\rm strip} on MATH-500 (clean N(0.05,0.322)\mathcal{N}(0.05,0.32^2), ellipsis bonus N(2.35,0.282)\mathcal{N}(2.35,0.28^2) on 44.5%, attacked N(5.40,0.952)\mathcal{N}(5.40,0.95^2); 300 clean split 210/90, seed=42, α=0.05\alpha=0.05: m^D=1.0718,s^D=1.2376,q^0.95=1.3452\hat m_D=1.0718,\hat s_D=1.2376,\hat q_{0.95}=1.3452), calibration keeps 98.1% mean clean wt while catching 99.0% of attacks (Thm. 5.3):

Rule200 clean (flag / wt)89 ellipsis (flag / wt)200 attacked (flag / wt)
Raw (D>0.80D>0.80)44.5% (89/200) / 0.6568100.0% (89/89) / 0.2288100.0% (200/200) / 0.0142
STAR (SD>q^0.95S^D>\hat q_{0.95})10.0% (20/200) / 0.981222.5% (20/89) / 0.957799.0% (198/200) / 0.1425

4. RLVR with Task-Specific Anchors

STAR (κ=1\kappa=1) keeps RverR^{\rm ver} as target while pairing lightweight anchors against shortcuts and hardcoding. Math: Qwen2.5-1.5B-Instruct (200 steps, G=8G=8, MATH disjoint from MATH-500) pairs \boxed{} verification with a GPT-4o-mini derivation anchor (no ground truth); MATH-500 steps are audited by Claude-Sonnet-4-6 (temp 0):

MATH-500 (500500)Exact ↑Valid deriv. ↑Shortcut ↓
Base / Verifier GRPO52.4% (262) / 58.6% (293)47.8% (239) / 50.2% (251)8.8% (23) / 14.3% (42)
Direct (κ=0\kappa=0) / STAR (κ=1\kappa=1)53.8% (269) / 60.4% (302)51.6% (258) / 57.6% (288)4.1% (11) / 4.6% (14)

STAR cuts shortcuts >3-fold (14.3% →\to 4.6%) and gains +1.8 exact pp (60.4%) and +7.4 valid-derivation pp (57.6%) over verifier GRPO. Code: Qwen2.5-Coder-3B-Instruct (200 steps, G=8G=8, 374 MBPP tasks, 3 visible asserts →\to 378 disjoint MBPP+ tasks, 35× hidden tests) uses Ranc=RverϕAST/fuzz(O)R^{\rm anc}=R^{\rm ver}\phi_{\rm AST/fuzz}(O) (ast literal check + 3 fuzz inputs; 1.6% [6/378] clean ref. FPR):

Code (378378)VisibleMBPP+ ↑Hardcode ↓Flip ↓
Base / Verifier GRPO73.6% / 88.1%62.4% (236) / 58.7% (222)0.5% (2) / 23.8% (90)N/A / 64.2%
Direct (κ=0\kappa=0) / Post-hoc80.2% / 78.6%64.8% (245) / 63.5% (240)1.1% (4) / 8.5% (32)8.2% / 59.4%
STAR (κ=1\kappa=1)81.5%67.2% (254)1.3% (5)4.5%

Verifier GRPO drops below Base on MBPP+ (58.7% vs. 62.4%) via 64.2% sign flips; STAR cuts hardcoding 18-fold (23.8% →\to 1.3%) and flips to 4.5%, adding +8.5 pp (+32 tasks; 39 STAR-only vs. 7 GRPO-only, McNemar p=2.0×10−6p=2.0\times 10^{-6}) and +9 tasks over κ=0\kappa=0.

5. Multi-Seed Replication and Statistical Significance

Three independent seeds (42, 43, 44, mean ± SD, ddof=1) confirm tight stability and zero overlap with baselines:

SettingSeeds (42 / 43 / 44)mean ± SD ↑Obs / gap ↓Clip / overclaim ↓
RQ1 TOMPA (855)−9.863 / −10.041 / −9.682−9.862 ± 0.1809.613 ± 0.1851.000 ± 0.000
RQ1 STAR (855)4.902 / 5.084 / 4.7624.916 ± 0.162−0.576 ± 0.0620.919 ± 0.020
RQ2 GRPO (260)0.2706 / 0.2647 / 0.27590.2704 ± 0.00560.2943 ± 0.01170.2468 ± 0.0052
RQ2 STAR (260)0.3174 / 0.3231 / 0.31260.3177 ± 0.00530.2315 ± 0.00970.2202 ± 0.0042

Every RQ2 seed improves judge score (+0.0469, +0.0584, +0.0367, mean +0.0473), narrows gap (−0.0617, −0.0749, −0.0518, −0.0628), and cuts overclaim (−0.0260, −0.0312, −0.0226, −0.0266). On per-prompt logs (N=993N=993 common prompts, 10k paired bootstrap; seed 42 stored run means on 995/997), 95% CIs are strictly positive on seeds 43 ([+0.0398, +0.0770]) and 44 ([+0.0185, +0.0549]).

6. Broader Baselines and Conventional RLHF

At RQ2 step 260 (token-mean, invalid →0\to 0, N=995–997 and common N=991 within ±0.0004), STAR beats reward-shaping, post-hoc, moment-weighted, self-anchored, and Gemini-rubric baselines:

RQ2 method (260)CallsProxyEval ↑Gap ↓Pass ↑Overclaim ↓
Std (GPT) / Plain (Gemini-2.5-Flash-Lite)1 / 10.5640 / 0.48650.2706 / 0.28940.2935 / 0.19710.4752 / 0.48910.2464 / 0.1986
Anchor (κ=0\kappa=0) / Mean-std (εσ=10−6\varepsilon_\sigma=10^{-6})1 / 20.4672 / 0.55160.2584 / 0.30190.2088 / 0.24970.4618 / 0.49580.2146 / 0.2268
Post-hoc (wiA^iw_i\hat A_i) / Penalty (r−0.5[D]+r-0.5[D]_+)2 / 20.5568 / 0.52140.2831 / 0.28850.2737 / 0.23290.4826 / 0.48690.2392 / 0.2284
Self-anchored STAR (GPT + GPT)20.55080.31160.23920.50120.2231
STAR (GPT+Gem.) / STAR (Gem.+Gem.)2 / 20.5492 / 0.47420.3174 / 0.32460.2318 / 0.14960.5049 / 0.51140.2204 / 0.1692

Anchor-only (κ=0\kappa=0) loses rubric detail (0.2584), while STAR achieves top quality under both GPT (0.3174; 0.3116 with zero Gemini calls) and Gemini rubrics (0.3246). On UltraFeedback (Qwen/Qwen2.5-3B-Instruct, 300 steps, 64×864\times 8, 2,048 cap, Ray2333/GRM-Qwen2.5-3B-rewardmodel-ft + Qwen2.5-1.5B-Concise-RM aligned on 1,024 step-0 rollouts, α=0.05\alpha=0.05; AlpacaEval 2.0 N=805N=805, gpt-4-1106-preview vs. gpt4_turbo, 5k bootstrap CIs, Skywork-Reward-V2-Qwen3-8B), STAR gains +10.6, +6.6, and +3.3 LC-WR pp over standard, κ=0\kappa=0, and min⁡(R1,R0)\min(R_1,R_0) GRPO:

Metric (805805)BaseStd GRPODirect κ=0\kappa=0Min-scoreSTAR κ=1\kappa=1
Len / Raw WR415 / 21.9%1146 / 31.4%438 / 23.5%524 / 28.1%492 / 31.8%
LC WR [95% CI] ↑22.8%20.6% [18.0, 23.3]24.6%27.9% [25.1, 30.8]31.2% [28.4, 34.1]
Skywork-8B ↑3.181.744.054.615.48

Secs. 3, 6.1–6.2, and Apps. E.1, F.1–F.2 will include all new results.

Reviewer_G5kaOpenReview ↗

Major concerns still unaddressed

Thanks for the reply.

For future purposes, I would request the authors to be brief with their rebuttal. I don't think that reviewers are asking to see so many numbers without much explanation. In all honesty, this is a clearly AI generated rebuttal which is very hard to read.

My major concern still remans that the implementation of D remains adhoc and hackable. I believe that the paper requires a major revision for this. Hence, I keep my score.

Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.