- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how LLM grader randomness travels through GRPO in rubric-based RL, where group normalization ties a group's advantages together and the policy's gradients decide how changed advantages move the update. SetGRPO calibrates a joint uncertainty set over each group's advantages from two grader scorings, keeps or shrinks a group's advantages according to a worst-case projection of the resulting update, and picks one acceptance threshold that bounds how often the update direction reverses. Experiments on three backbones and ten medical, writing and instruction-following benchmarks give the best macro average among the compared training methods.
Critical strengths
The paper looks at grader noise where the optimizer actually feels it. Rather than treating judge scores as fixed or simply averaging them, it follows a noisy score through the normalization to the advantages and then to the update. I think this is a useful way to look at rubric RL with sampled judges, and since the worst case over an ellipsoid has a closed form, the test is cheap to add to GRPO.
The ablations include the right control. Besides beating plain GRPO on all three backbones, SetGRPO is compared with a variant that randomly drops the same number of groups, and it stays clearly ahead. This separates targeted screening from simply training on fewer groups, and to me it is the most convincing result in the paper that the screening itself helps.
Critical weaknesses
The guarantee does not cover what training needs. Coverage is defined against a second draw from the same noisy grader, so avoiding a reversal only means two noisy estimates agree, not that the update moves toward a higher expected score. The set and threshold are fixed on groups sampled before training, and exchangeability is lost once the policy moves. The set's shape is also doubtful. The normalization Jacobian vanishes along the advantage vector, exactly where two normalized scorings differ, so coverage seems to rest on the small ridge term, not on the coupling that motivates the method.
It is unclear where the gain comes from. Calibration already pays for a second scoring on more groups than training uses, yet no GRPO run uses the average of the two scores at equal grader cost. Dr. GRPO (Liu et al., 2025), which drops the standard deviation behind part of this coupling, and the unbiased reward correction of El Mansouri et al. (2026) would also be natural comparisons. The main results come without variance, though the margin over RubricHub is one or two points, and the reported reversal risk at the chosen threshold varies across the paper.
References
Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik, and Salem Lahlou. Noise-corrected GRPO: From noisy rewards to unbiased gradients. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, 2026.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. In Second Conference on Language Modeling (COLM), 2025.
Questions for authors
- The normalization Jacobian is singular along the advantage vector. How, then, is the reported coverage radius obtained, and how sensitive is it to the ridge term?
- What is the "reward space" in which the recovery path is computed, and how does it differ from projecting in the full space?
- How many runs are averaged in the main results, and what are the standard deviations?
- The qualitative examples in the appendix seem to come from HealthBench, WritingBench and IFEval, while the paper says the evaluation benchmarks were not used for training or calibration. Could the authors say where these examples come from?
- Recalibrating on freshly sampled groups during training, and comparing with score averaging at equal grader cost, would show whether the gain really comes from the proposed screening.
12 comments
The Response to Reviewer RCgS (Part1/5)
In addition to the stability of the calibration radius during training, the reviewer also asked how the shape matrix of the coverage set , behaves along the singular directions of the Jacobian. Specifically, for the group normalization mapping , translation invariance and positive scale invariance imply that the Jacobian satisfies and along the all ones direction and the unit radial direction . Along these two null space directions, provides no additional scale. The geometric scale of the full ellipsoid is therefore explicitly determined by the ridge term , so that the ellipsoid has a semi-axis of along any unit null space direction. Projecting the paired residual onto the positive scaling direction , and letting the coefficient cosine similarity be , the squared Mahalanobis distance satisfies . Therefore, for a sample to lie inside the ellipsoid (), it must satisfy
Under the formal configuration and , the necessary cosine lower bounds for the three backbones are , , and , respectively.
To illustrate the effect of the ridge term on the scale of the null space directions, we fix the backbone-specific calibration radii and compare the zero-space semi-axis and necessary cosine lower bound under in Table R3-1. At the same , decreasing shrinks the zero-space semi-axis proportionally to and correspondingly increases the necessary cosine lower bound. This shows that, in our ellipsoid construction, characterizes the coupled within-group perturbation structure in the nonzero subspace, while explicitly controls the tolerance scale of the set along the Jacobian null space directions and the strength of the associated cosine constraint.
Table R3-1: Analytic ridge sensitivity for under the reported backbone-specific radii.
| Backbone () | Zero-space semi-axis | Necessary lower bound | |
|---|---|---|---|
| Qwen3.5-4B (1.35) | 0.9995227 | ||
| Qwen3.5-4B (1.35) | 0.9999523 | ||
| Qwen3.5-4B (1.35) | 0.9999952 | ||
| Qwen3.5-9B (1.42) | 0.9994980 | ||
| Qwen3.5-9B (1.42) | 0.9999498 | ||
| Qwen3.5-9B (1.42) | 0.9999950 | ||
| Llama3.2-3B (1.36) | 0.9995192 | ||
| Llama3.2-3B (1.36) | 0.9999519 | ||
| Llama3.2-3B (1.36) | 0.9999952 |
We further checked this necessary condition on the formal coverage-calibration split. The cosine quantiles are ,, and for the three backbones, respectively, and all exceed their corresponding necessary lower bounds. Separately, Figure 12(a) in the original paper evaluates empirical coverage under the full ellipsoidal constraint on the independent risk-calibration split. The observed coverage is broadly consistent with the nominal 95% target. Together with the sensitivity results above, these findings show that the ridge scale, cosine constraint, and coverage results are mutually consistent under the reported configuration.
The Response to Reviewer RCgS (Part2/5)
W2: The source of the gains is unclear Calibration already incurs the cost of a second score evaluation on more groups than those used during training, yet no GRPO baseline uses the average of two scores at the same evaluation cost. Dr. GRPO (Liu et al., 2025), which removes the standard-deviation term underlying this coupling, and the unbiased reward correction of El Mansouri et al. (2026) are also natural baselines. The main results report no variance estimates, the improvement over RubricHub is only 1–2 points, and the reported reversal risk at the selected threshold is inconsistent across the paper.
A-W2: We thank the reviewer for this constructive comment. In the experimental design of the original manuscript (Section 4.1 and Table 1), we primarily controlled the online training scoring cost. Specifically, each response group was scored only once during online training, exactly as in the original GRPO, RubricHub, and RuscaRL settings. This design establishes that, without increasing the online training cost, update-level geometric screening can make more effective use of a single scoring signal than existing rubric-based RL methods.
However, we fully agree that controlling only the online training cost is not sufficient to determine whether SetGRPO still outperforms averaging two scores when the second score used for offline calibration is included in the total cost. It also does not directly establish how SetGRPO compares with removing the group standard deviation term in Dr. GRPO, applying unbiased reward correction in Noise-corrected GRPO, or accounting for variation across random seeds. To address these four specific concerns, we add the following four sets of experiments and comparative analyses.
(1) Concern about comparisons with recent GRPO variants (Dr. GRPO and Noise-corrected GRPO).
To the best of our knowledge, Dr. GRPO primarily addresses the optimization bias introduced by group standard deviation normalization, whereas Noise-corrected GRPO directly corrects noisy rewards. Neither method determines whether the joint training coefficients induced by score discrepancies will change the actual update direction under the current policy’s gradient geometry. To compare these mechanisms under the same backbone and downstream evaluation protocols, we report the following results in Table R3-2.
Table R3-2: Comparison with related GRPO methods.
| Method | Qwen3.5-4B | Qwen3.5-9B | Llama3.2-3B | Overall macro avg. |
|---|---|---|---|---|
| GRPO | 55.41 | 58.69 | 44.72 | 52.94 |
| Dr. GRPO | 58.14 | 61.42 | 47.39 | 55.65 |
| Noise-corrected GRPO | 58.62 | 61.88 | 47.86 | 56.12 |
| SetGRPO (Ours) | 60.26 | 63.51 | 49.57 | 57.78 |
The results in Table R3-2 show that Dr. GRPO and Noise-corrected GRPO improve the overall Macro Avg. over the original GRPO by and points, respectively. However, their scores remain lower than those of SetGRPO on all three backbones. SetGRPO achieves an average improvement of points over Dr. GRPO and points over Noise-corrected GRPO. This performance gap reflects the limitations of the two methods at different stages of the training process. First, although Dr. GRPO removes the group standard deviation term from the advantage function to reduce scale bias, group centering still allows a scoring perturbation for one response to affect the other responses in the same group. Dr. GRPO also does not calibrate or filter grader noise. Second, although Noise-corrected GRPO provides an unbiased correction at the reward level, it does not account for the possibility that residual errors, after being multiplied by the current policy’s response-level gradients , may still cause local update direction reversals. In contrast, SetGRPO jointly calibrates the within-group coefficient shifts induced by shared normalization in coefficient space and further uses the current policy’s gradient geometry to filter high-risk updates that may reverse the update direction. It therefore achieves higher downstream scores on all three backbones.
The Response to Reviewer RCgS (Part1/5)
W1: Theoretical guarantee does not cover the property needed for training. The coverage guarantee is defined for a second draw from the same noisy scorer. Avoiding direction reversal therefore only ensures consistency between two noisy estimates; it does not imply that the update increases the expected score. Moreover, the sets and threshold are fixed using pre-training groups, so exchangeability may fail once the policy changes. The set geometry is also questionable: the normalized Jacobian vanishes along the advantage direction, which is precisely where the two normalized scores may differ. Thus, the coverage guarantee appears to rely mainly on the small ridge term rather than on the coupling structure motivating the method.
A1: We thank the reviewer for pointing out that the coverage and risk guarantees in this work do not directly imply an improvement in downstream expected reward, and for highlighting the applicability limits introduced by fixed calibration and Jacobian singularity. As described in the main text, SetGRPO formalizes and controls two well-defined statistics. The first is the marginal coverage of the joint coefficient set in Section 3.1, Eq. 5, with proof in Appendix A.2. Using the coverage-calibration split, it guarantees that the reference training coefficient ax⋆ induced by the second score falls inside the ellipsoidal set Cx constructed from the first score with probability at least 1−α: Pr(ax⋆∈Cx)≥1−α. The second is the conditional direction-reversal risk of the accepted set in Section 3.3, Eq. 9, with proof in Appendix A.6. Using the independent risk-calibration split, it controls the conditional probability that an accepted nonzero update (Ax,τ=1) has a negative inner product with the reference update: Rτsel=Pr(⟨uxfull(ax(ηx⋆(τsel))),uxfull(ax⋆)⟩<0∣Ax,τsel=1)≤β. These guarantees characterize the coverage of the coefficient perturbation set and the local consistency of update directions under score perturbations. They are not theoretical lower bounds on improvements in the true downstream expected reward.
We fully agree with the reviewer’s point that exchangeability is not automatically preserved after the policy distribution changes. The marginal coverage and conditional risk guarantees for the joint coefficient set hold only when the calibration groups and target groups are exchangeable. Once the policy evolves during training, the fixed Qα and τsel obtained before training no longer retain their formal guarantees unless newly sampled groups are recalibrated. During training, we keep (Qα,τsel) fixed, while recomputing the local update projections Ψxfull and Ψxreward at every step using the latest response-level gradients Gxq from the current policy. We also conduct an empirical risk diagnostic across checkpoints at training steps t∈{0,100,250,500} in Appendix F.4, Figure 13 of the original manuscript. The diagnostic shows that, as the policy evolves, the empirical direction-reversal rate of the accepted groups and its one-sided upper bound remain below the target threshold β=0.08. In addition, Section (3) of A-W2 below compares this fixed-calibration protocol with Fresh recalibration, which resamples calibration groups during training. The comparison shows that the pretraining fixed values of (Qα,τsel) capture most of the attainable performance gain in practice.