- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a reliability weighted GRPO advantage by modifying the advantage signal of GRPO in an attempt to reduce reward hacking. The reward of each roll-out is computed twice: by a training verifier and a second anchor verifier. The difference between the two rewards is used as a measure of reliability for that roll-out. These weights are used to compute a new baseline and the final advantage is rescaled by the rollout and the group reliability. Groups with low reliability are completely dropped. The experimental setup is around two settings: RQ1 uses token-interface exploitation, where raw token Ids are passed to a reward model that uses a different tokenizer than the base policy. RQ2 uses the difference between rubric conditioned judge (gpt-4o-mini) and a rubric-free judge (gemini) as the source of the reliability signal. The paper shows that on healthbench-hard set, STAR-GRPO results in low rate of false claims.
Critical strengths
-
I think extracting a reliability signal based on the disagreement between rubric-based and rubric-free judges is interesting. It addresses a known weakness of rubric rewards where rubric satisfaction drifts from overall response quality.
-
The results of RQ2 are interesting and useful. We can see in Figure 4c/d as training proceeds GRPO's overclaim fraction and the proxy-judge gap increase, while STAR's decreases. In addition, STAR also reaches a higher independent-judge score which is promising.
Critical weaknesses
- In RQ2, STAR-GRPO adds an extra judge (i.e., the holistic judge). So comparing plain GRPO with a single weak judge (gpt-4o-mini) is not a fair comparison.
- The experimental setup is very limited. RQ2 uses a single domain, one rubric dataset and a single policy (Qwen3-4B). How do we know that these results generalize across different domains, model families, policy scales
- More importantly, there is no reason why the anchor judge would not also be gamed. Since the anchor judge is part of the reward function formulation, it can be exploited by the policy.
- Finally, I think that RQ1 is not really a realistic RL setting. The deployed interface passes policy token ids directly to the reward model. Standard RL pipelines either share the same tokenizer between the base policy or decode the policy's output and then retokenize it using the reward model tokenizer. This is basically STAR's canonical view. In addition, the matched canonical-GRPO control beats STAR (3.236 versus 1.781 with shorter response, in table 6). This limits the impact of this work.
Questions for authors
- How can you account for the fact that STAR-GRPO uses a strong judge like gemini as the holistic judge while plain GRPO in the presented experiments only use gpt-4o-mini, which is a weak judge. It is possible that the gain observed in the paper in Figure 4 are due to Gemini being generally more strict that gpt-4o-mini. Did you try to run a plain GRPO experiment with Gemini as the rubric-based judge?
- Have you evaluated the utility of STAR-GRPO on any other domains, policies or different rubric-based judges?
- Can you provide a practical RL training framing where RQ1 can useful?
2 comments
3. Generalization Across Domains, Families, Scales, and Ablations
Expanded 260-step rubric runs (, token-mean, HealthBench-Hard ; RubricHub-General splits 5,000 non-medical prompts into 4,500 train / 500 disjoint test), 300-step RLHF (gpt-4-1106-preview vs. gpt4_turbo), and 200-step disjoint RLVR confirm consistent STAR gains across domains, model families, and scales:
| Task / benchmark | Policy | Plain GRPO | STAR () ↑ |
|---|---|---|---|
| Medical / HealthBench-Hard | Qwen3-4B / Qwen2.5-7B | 0.2706 / 0.2865 | 0.3174 / 0.3364 |
| Medical / HealthBench-Hard | Llama-3.2-3B / Llama-3.1-8B | 0.2548 / 0.2912 | 0.2986 / 0.3418 |
| General / RubricHub-General | Qwen3-4B / Qwen2.5-7B / Llama-3.1-8B | 0.4812 / 0.4980 / 0.5045 | 0.5396 / 0.5562 / 0.5618 |
| RLHF / AlpacaEval 2.0 LC-WR | Qwen2.5-3B-Instruct | 20.6% | 31.2% |
| Code MBPP+ (374/378) / Math MATH-500 | Qwen2.5-Coder-3B / Qwen2.5-1.5B | 58.7% / 50.2% | 67.2% / 57.6% |
Within-family scaling (Qwen3-4B → Qwen2.5-7B, Llama-3.2-3B → Llama-3.1-8B) and cross-family comparisons (Qwen2.5-7B vs. Llama-3.1-8B) yield +11.4% to +17.4% rubric gains; offline NoveltyBench audits on Llama-3.2-1B (, 240 rollouts) raise Claude cosine from 0.2546 ± 0.0405 to 0.3032 ± 0.0423 (GPT self-anchor) and 0.3284 ± 0.0451 to 0.3400 ± 0.0464 (Gemini rubric + GPT hol.). Step-260 single-factor STAR ablations (Uniform-fit: in fit only; Quadratic-fit: Huber fit only; 10.9% random skip) further verify each component:
| Step-260 variant | Eval ↑ / Overclaim ↓ | Step-260 variant | Eval ↑ / Overclaim ↓ |
|---|---|---|---|
| Plain GRPO | 0.2706 / 0.2464 | Quadratic-fit STAR | 0.3082 / 0.2244 |
| Neutral | 0.2782 / 0.2421 | No | 0.3076 / 0.2251 |
| Post-hoc (comp.) | 0.2831 / 0.2392 | Relaxed () | 0.3108 / 0.2235 |
| Uniform-fit STAR | 0.2964 / 0.2310 | 10.9% random skipping | 0.3102 / 0.2239 |
| Weighted mean/std | 0.3019 / 0.2268 | Full STAR | 0.3174 / 0.2204 |
Full STAR outperforms every ablation: +0.0210 (Uniform-fit), +0.0092 (Quadratic-fit), +0.0098 (), and +0.0072/+0.0066 (admission).
4. Matched RQ1 Controls and Practical Single-RM Format-STAR
In RQ1 (Zhang et al., TOMPA; cf. Gao et al., Sharma et al., Bukharin et al.), the Table 6 gap (Canonical-GRPO +3.236 vs. STAR +1.781) came entirely from mismatched loss reduction (seq-mean-token-mean vs. token-mean). Completing the grid at step 855 (, KL 0.001, ) confirms STAR matches the clean-interface oracle under both rules:
| Method | Loss reduction | Canon ↑ | Obs | Length |
|---|---|---|---|---|
| Original STAR () | Token mean | 4.902 ± 0.386 | −0.570 ± 0.073 | 1872.1 |
| Matched Canonical-GRPO | Token mean | 4.945 ± 0.402 | −0.612 ± 0.079 | 1846.4 ± 49.2 |
| Original Canonical-GRPO | Per-response token mean | 6.123 ± 0.578 | −1.862 ± 0.054 | 395.5 ± 36.6 |
| Matched STAR () | Per-response token mean | 6.158 ± 0.542 | −1.841 ± 0.058 | 408.2 ± 38.4 |
Across 3 seeds (42, 43, 44) under seq-mean, STAR reaches 6.142 ± 0.148 vs. 6.118 ± 0.165 for Canonical-GRPO (within 0.035–0.043 single-seed and 12.7–25.7 tokens; checkpoints 0,200,400,600,855 show 100% top-1 agreement, 0.9567–0.9668 cosine, 0.9977–1.0000 weight).
Practically, single-RM paired scoring mitigates formatting/sycophancy gaming in matched-tokenizer RLHF without a second RM. Defining to strip preambles ("Certainly!...", "In summary,..."), # headers, **/__ bold, and -/• bullets while keeping code, JSON/tables, and math (, calibrated on 1,024 step-0 rollouts split 70%/30% at seed=42; Qwen2.5-3B-Instruct + GRM-Qwen2.5-3B, 300 steps, , AlpacaEval 2.0 split into format / content; NoveltyBench clean cosine 0.7972 [0.9917 wt, 2.5% downweighted] vs. 0.8141 GRPO and 0.7115 stripped):
| Method (single 3B RM) | Skywork-8B ↑ | LC WR () ↑ | Format () / Content () | Len |
|---|---|---|---|---|
| Base / Plain GRPO | 3.18 / 1.74 | 22.8% / 20.6% | 24.1%/22.4% ; 19.6%/20.9% | 415 / 1146 |
| Plain-text Canonical () | 4.18 | 26.2% | 21.4% / 27.5% | 446 |
| Single-RM Format-STAR () | 5.06 | 29.4% | 28.6% / 29.6% | 518 |
With zero second RM loaded (+2.4 s/step 2nd pass), Format-STAR gains +8.8 pp over Plain GRPO and +3.2 pp over Plain-text Canonical-GRPO (+7.2 pp on format prompts, cutting format violations from 18.5% to 4.8%). Secs. 3, 6.1–6.2, and Appendix F.1.1 will integrate all results.
Thank you very much for your support and valuable comments. We have carefully reviewed them, which are of great help to this work. We summarize the key concerns you raised, have supplemented the corresponding experiments, and present our responses below.
1. Equal-Budget Judge Controls and Resistance to Co-Hacking
In RQ2 (κ=1), GPT-4o-mini scores explicit rubric items (ri=Rirubric), Gemini-2.5-Flash-Lite scores holistically without the rubric, and Claude-Sonnet-4-6 is held out for evaluation. Placing wi inside the robust fit before advantage construction prevents overrated rollouts from flipping clean peers' signs while retaining group confidence wˉ: AiSTAR=wˉν+εwwiϕi,wˉ=G1∑iwi,ν=G1∑iwi2,∣ϕi∣≤1. At step 260 (
gpt-4o-mini-2024-07-18,gemini-2.5-flash-lite, temp 0, invalid →0,N=995–997and commonN=991), STAR outperforms equal-budget 2-call ensembles, no-information controls, and single-model baselines:Omitting the rubric in holistic calls (~690 tokens at 99.8% validity vs. ~1,420 at 99.9%, concurrent) adds only ~49% tokens (~32% billed cost) vs. 2.84k for 2× GPT rubric. Zero-Gemini
Self-Anchored STARbeats 2× GPT rubric by +0.0375 on N=991, whileSTARbeats equal-access Minimum-score by +0.0260 and constant/shuffled controls by +0.0380–0.0406. Offline (45 prompts ×16=720: 15 cal = 240, 30 eval = 480 with 479 valid Gemini = 719/720; 60 contaminated groups), Claude cosine flips from −0.1129 ± 0.0162 to +0.1004 ± 0.0168 (self) and +0.1173 ± 0.0195 (STAR), lifting tilted Claude (τ=3.0) from 0.1126 to 0.2605/0.2686 and cutting overclaim push from +0.1306 to +0.0602/+0.0516.Structurally, at κ=1, Ranc is absent from ri, and on 89.39% of faithful rollouts (74.11% of 479) wi saturates at 1 (∂ASTAR/∂Ranc≡0), while orthogonal views prevent common-mode shifts (Thm. 3.1(ii)). Online,
Proxy − Claudewidens under GRPO (+0.4589 → +0.5436) whereasGemini − Claudedoes not inflate (+0.6410 → +0.5427); in a 200-prompt Best-of-16 audit (maxRrubric s.t. wi≥0.8), high-train (≥0.70) yet low-Claude (≤0.25) picks drop 3-fold from 19.5% (39/200, verified on a 50-item check) to 6.5% (13/200).2. Gemini as the Rubric-Based GRPO Judge
Across the 719 valid offline rollouts, holistic
gemini-2.5-flash-liteis more lenient (mean 0.7644) than GPT rubric (0.6670). Runninggemini-2.5-flash-liteas explicit rubric judge at step 260 (N=995–997and commonN=991; new pairings fit (m^D,s^D,q^0.90) on the 240 step-0 split, e.g., m^D=−0.38,s^D=0.21,q^0.90=1.28,λ=2.0, 6.4% clean downweighting, vs. fixed m=0,s=0.1,q=1.645,λ=2):Offline (720 rollouts, Gemini rubric mean 0.3831), 58/310 poor outputs (Claude ≤0.25) fool Gemini rubric (≥0.65); on 60 contaminated groups, STAR lifts cosine from −0.2544 ± 0.0260 to −0.1524 ± 0.0252 (Gemini hol.), −0.1402 ± 0.0275 (GPT hol.), and −0.1200 ± 0.0274 (Claude-Haiku-4-5 hol.; κ=0 holistic alone is 0.2584 online). Online,
Plain GRPO (Gemini rubric)reaches 0.2894 (below zero-GeminiSelf-Anchored STAR0.3116 andSTAR0.3174; 3-seed42, 43, 44mean 0.3177 ± 0.0053 vs. 0.2704 ± 0.0056), and STAR atop Gemini rubric reaches 0.3246 (+0.0353 on N=991, 10k bootstrap 95% CI[+0.0178, +0.0528]), cutting gap by 0.0475 and overclaim by 2.94 pp.