acceptodds
Under review as a conference paper at ICLR 2027

SetGRPO: Calibrating Grader-Induced Policy Optimization

Abstract

Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families. The code is available at https://anonymous.4open.science/r/SetGRPO-73EB.

Primary area:
reinforcement learning
Keywords:
Rubric-based Reinforcement Learning, GRPO, Grader Uncertainty, Conformal Prediction, Selective Risk Control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RCgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how LLM grader randomness travels through GRPO in rubric-based RL, where group normalization ties a group's advantages together and the policy's gradients decide how changed advantages move the update. SetGRPO calibrates a joint uncertainty set over each group's advantages from two grader scorings, keeps or shrinks a group's advantages according to a worst-case projection of the resulting update, and picks one acceptance threshold that bounds how often the update direction reverses. Experiments on three backbones and ten medical, writing and instruction-following benchmarks give the best macro average among the compared training methods.

Critical strengths

The paper looks at grader noise where the optimizer actually feels it. Rather than treating judge scores as fixed or simply averaging them, it follows a noisy score through the normalization to the advantages and then to the update. I think this is a useful way to look at rubric RL with sampled judges, and since the worst case over an ellipsoid has a closed form, the test is cheap to add to GRPO.

The ablations include the right control. Besides beating plain GRPO on all three backbones, SetGRPO is compared with a variant that randomly drops the same number of groups, and it stays clearly ahead. This separates targeted screening from simply training on fewer groups, and to me it is the most convincing result in the paper that the screening itself helps.

Critical weaknesses

The guarantee does not cover what training needs. Coverage is defined against a second draw from the same noisy grader, so avoiding a reversal only means two noisy estimates agree, not that the update moves toward a higher expected score. The set and threshold are fixed on groups sampled before training, and exchangeability is lost once the policy moves. The set's shape is also doubtful. The normalization Jacobian vanishes along the advantage vector, exactly where two normalized scorings differ, so coverage seems to rest on the small ridge term, not on the coupling that motivates the method.

It is unclear where the gain comes from. Calibration already pays for a second scoring on more groups than training uses, yet no GRPO run uses the average of the two scores at equal grader cost. Dr. GRPO (Liu et al., 2025), which drops the standard deviation behind part of this coupling, and the unbiased reward correction of El Mansouri et al. (2026) would also be natural comparisons. The main results come without variance, though the margin over RubricHub is one or two points, and the reported reversal risk at the chosen threshold varies across the paper.

References

Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik, and Salem Lahlou. Noise-corrected GRPO: From noisy rewards to unbiased gradients. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, 2026.

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. In Second Conference on Language Modeling (COLM), 2025.

Questions for authors
  1. The normalization Jacobian is singular along the advantage vector. How, then, is the reported coverage radius obtained, and how sensitive is it to the ridge term?
  2. What is the "reward space" in which the recovery path is computed, and how does it differ from projecting in the full space?
  3. How many runs are averaged in the main results, and what are the standard deviations?
  4. The qualitative examples in the appendix seem to come from HealthBench, WritingBench and IFEval, while the paper says the evaluation benchmarks were not used for training or calibration. Could the authors say where these examples come from?
  5. Recalibrating on freshly sampled groups during training, and comparing with score averaging at equal grader cost, would show whether the gain really comes from the proposed screening.
12 comments
AuthorsOpenReview ↗

The Response to Reviewer RCgS (Part1/5)

W1: Theoretical guarantee does not cover the property needed for training. The coverage guarantee is defined for a second draw from the same noisy scorer. Avoiding direction reversal therefore only ensures consistency between two noisy estimates; it does not imply that the update increases the expected score. Moreover, the sets and threshold are fixed using pre-training groups, so exchangeability may fail once the policy changes. The set geometry is also questionable: the normalized Jacobian vanishes along the advantage direction, which is precisely where the two normalized scores may differ. Thus, the coverage guarantee appears to rely mainly on the small ridge term rather than on the coupling structure motivating the method.

A1: We thank the reviewer for pointing out that the coverage and risk guarantees in this work do not directly imply an improvement in downstream expected reward, and for highlighting the applicability limits introduced by fixed calibration and Jacobian singularity. As described in the main text, SetGRPO formalizes and controls two well-defined statistics. The first is the marginal coverage of the joint coefficient set in Section 3.1, Eq. 5, with proof in Appendix A.2. Using the coverage-calibration split, it guarantees that the reference training coefficient ax⋆a_x^\star induced by the second score falls inside the ellipsoidal set Cx\mathcal C_x constructed from the first score with probability at least 1−α1-\alpha: Pr⁡(ax⋆∈Cx)≥1−α\Pr(a_x^\star \in \mathcal C_x) \ge 1-\alpha. The second is the conditional direction-reversal risk of the accepted set in Section 3.3, Eq. 9, with proof in Appendix A.6. Using the independent risk-calibration split, it controls the conditional probability that an accepted nonzero update (Ax,τ=1A_{x,\tau}=1) has a negative inner product with the reference update: Rτsel=Pr⁡(⟨uxfull(ax(ηx⋆(τsel))),uxfull(ax⋆)⟩<0∣Ax,τsel=1)≤βR_{\tau_{\mathrm{sel}}} = \Pr\bigl(\langle u_x^{\mathrm{full}}(a_x(\eta_x^\star(\tau_{\mathrm{sel}}))), u_x^{\mathrm{full}}(a_x^\star)\rangle \lt 0 \mid A_{x,\tau_{\mathrm{sel}}} = 1\bigr) \le \beta. These guarantees characterize the coverage of the coefficient perturbation set and the local consistency of update directions under score perturbations. They are not theoretical lower bounds on improvements in the true downstream expected reward.

We fully agree with the reviewer’s point that exchangeability is not automatically preserved after the policy distribution changes. The marginal coverage and conditional risk guarantees for the joint coefficient set hold only when the calibration groups and target groups are exchangeable. Once the policy evolves during training, the fixed QαQ_\alpha and τsel\tau_{\mathrm{sel}} obtained before training no longer retain their formal guarantees unless newly sampled groups are recalibrated. During training, we keep (Qα,τsel)(Q_\alpha,\tau_{\mathrm{sel}}) fixed, while recomputing the local update projections Ψxfull\Psi_x^{\mathrm{full}} and Ψxreward\Psi_x^{\mathrm{reward}} at every step using the latest response-level gradients GxqG_x^q from the current policy. We also conduct an empirical risk diagnostic across checkpoints at training steps t∈{0,100,250,500}t\in\{0,100,250,500\} in Appendix F.4, Figure 13 of the original manuscript. The diagnostic shows that, as the policy evolves, the empirical direction-reversal rate of the accepted groups and its one-sided upper bound remain below the target threshold β=0.08\beta=0.08. In addition, Section (3) of A-W2 below compares this fixed-calibration protocol with Fresh recalibration, which resamples calibration groups during training. The comparison shows that the pretraining fixed values of (Qα,τsel)(Q_\alpha,\tau_{\mathrm{sel}}) capture most of the attainable performance gain in practice.

AuthorsOpenReview ↗

The Response to Reviewer RCgS (Part1/5)

In addition to the stability of the calibration radius QαQ_\alpha during training, the reviewer also asked how the shape matrix of the coverage set Cx\mathcal C_x, Sxcoef=JxVxJx⊤+λIKS_x^{\mathrm{coef}}=J_xV_xJ_x^\top+\lambda I_K behaves along the singular directions of the Jacobian. Specifically, for the group normalization mapping a=(r−rˉ1)/sa=(r-\bar r\mathbf{1})/s, translation invariance and positive scale invariance imply that the Jacobian satisfies J1=0J\mathbf{1}=0 and Jv=0Jv=0 along the all ones direction 1\mathbf{1} and the unit radial direction v=a^/Kv=\hat a/\sqrt{K}. Along these two null space directions, JVJ⊤JVJ^\top provides no additional scale. The geometric scale of the full ellipsoid is therefore explicitly determined by the ridge term λ\lambda, so that the ellipsoid has a semi-axis of QαλQ_\alpha\sqrt{\lambda} along any unit null space direction. Projecting the paired residual onto the positive scaling direction vv, and letting the coefficient cosine similarity be c=a^⊤a⋆/Kc=\hat a^\top a^\star/K, the squared Mahalanobis distance satisfies q2≥K(1−c)2λq^2\ge\frac{K(1-c)^2}{\lambda}. Therefore, for a sample to lie inside the ellipsoid (q≤Qαq\le Q_\alpha), it must satisfy

c≥1−QαλK.c\ge 1-Q_\alpha\sqrt{\frac{\lambda}{K}}.

Under the formal configuration K=8K=8 and λ=10−8\lambda=10^{-8}, the necessary cosine lower bounds for the three backbones are 0.99995230.9999523, 0.99994980.9999498, and 0.99995190.9999519, respectively.

To illustrate the effect of the ridge term λ\lambda on the scale of the null space directions, we fix the backbone-specific calibration radii QαQ_\alpha and compare the zero-space semi-axis and necessary cosine lower bound under λ∈{10−6,10−8,10−10}\lambda\in\{10^{-6},10^{-8},10^{-10}\} in Table R3-1. At the same QαQ_\alpha, decreasing λ\lambda shrinks the zero-space semi-axis proportionally to λ\sqrt{\lambda} and correspondingly increases the necessary cosine lower bound. This shows that, in our ellipsoid construction, JxVxJx⊤J_xV_xJ_x^\top characterizes the coupled within-group perturbation structure in the nonzero subspace, while λ\lambda explicitly controls the tolerance scale of the set along the Jacobian null space directions and the strength of the associated cosine constraint.

Table R3-1: Analytic ridge sensitivity for K=8K=8 under the reported backbone-specific radii.

Backbone (QαQ_\alpha)λ\lambdaZero-space semi-axisNecessary cc lower bound
Qwen3.5-4B (1.35)10−610^{-6}1.3500×10−31.3500\times10^{-3}0.9995227
Qwen3.5-4B (1.35)10−810^{-8}1.3500×10−41.3500\times10^{-4}0.9999523
Qwen3.5-4B (1.35)10−1010^{-10}1.3500×10−51.3500\times10^{-5}0.9999952
Qwen3.5-9B (1.42)10−610^{-6}1.4200×10−31.4200\times10^{-3}0.9994980
Qwen3.5-9B (1.42)10−810^{-8}1.4200×10−41.4200\times10^{-4}0.9999498
Qwen3.5-9B (1.42)10−1010^{-10}1.4200×10−51.4200\times10^{-5}0.9999950
Llama3.2-3B (1.36)10−610^{-6}1.3600×10−31.3600\times10^{-3}0.9995192
Llama3.2-3B (1.36)10−810^{-8}1.3600×10−41.3600\times10^{-4}0.9999519
Llama3.2-3B (1.36)10−1010^{-10}1.3600×10−51.3600\times10^{-5}0.9999952

We further checked this necessary condition on the formal coverage-calibration split. The cosine quantiles p5/median/p95p_5/\mathrm{median}/p_{95} are 0.999956/0.999975/0.9999890.999956/0.999975/0.999989,0.999954/0.999972/0.9999870.999954/0.999972/0.999987, and 0.999955/0.999974/0.9999880.999955/0.999974/0.999988 for the three backbones, respectively, and all exceed their corresponding necessary lower bounds. Separately, Figure 12(a) in the original paper evaluates empirical coverage under the full ellipsoidal constraint on the independent risk-calibration split. The observed coverage is broadly consistent with the nominal 95% target. Together with the sensitivity results above, these findings show that the ridge scale, cosine constraint, and coverage results are mutually consistent under the reported configuration.

AuthorsOpenReview ↗

The Response to Reviewer RCgS (Part2/5)

W2: The source of the gains is unclear Calibration already incurs the cost of a second score evaluation on more groups than those used during training, yet no GRPO baseline uses the average of two scores at the same evaluation cost. Dr. GRPO (Liu et al., 2025), which removes the standard-deviation term underlying this coupling, and the unbiased reward correction of El Mansouri et al. (2026) are also natural baselines. The main results report no variance estimates, the improvement over RubricHub is only 1–2 points, and the reported reversal risk at the selected threshold is inconsistent across the paper.

A-W2: We thank the reviewer for this constructive comment. In the experimental design of the original manuscript (Section 4.1 and Table 1), we primarily controlled the online training scoring cost. Specifically, each response group was scored only once during online training, exactly as in the original GRPO, RubricHub, and RuscaRL settings. This design establishes that, without increasing the online training cost, update-level geometric screening can make more effective use of a single scoring signal than existing rubric-based RL methods.

However, we fully agree that controlling only the online training cost is not sufficient to determine whether SetGRPO still outperforms averaging two scores when the second score used for offline calibration is included in the total cost. It also does not directly establish how SetGRPO compares with removing the group standard deviation term in Dr. GRPO, applying unbiased reward correction in Noise-corrected GRPO, or accounting for variation across random seeds. To address these four specific concerns, we add the following four sets of experiments and comparative analyses.

(1) Concern about comparisons with recent GRPO variants (Dr. GRPO and Noise-corrected GRPO).

To the best of our knowledge, Dr. GRPO primarily addresses the optimization bias introduced by group standard deviation normalization, whereas Noise-corrected GRPO directly corrects noisy rewards. Neither method determines whether the joint training coefficients induced by score discrepancies will change the actual update direction under the current policy’s gradient geometry. To compare these mechanisms under the same backbone and downstream evaluation protocols, we report the following results in Table R3-2.

Table R3-2: Comparison with related GRPO methods.

MethodQwen3.5-4BQwen3.5-9BLlama3.2-3BOverall macro avg.
GRPO55.4158.6944.7252.94
Dr. GRPO58.1461.4247.3955.65
Noise-corrected GRPO58.6261.8847.8656.12
SetGRPO (Ours)60.2663.5149.5757.78

The results in Table R3-2 show that Dr. GRPO and Noise-corrected GRPO improve the overall Macro Avg. over the original GRPO by +2.71+2.71 and +3.18+3.18 points, respectively. However, their scores remain lower than those of SetGRPO on all three backbones. SetGRPO achieves an average improvement of +2.13+2.13 points over Dr. GRPO and +1.66+1.66 points over Noise-corrected GRPO. This performance gap reflects the limitations of the two methods at different stages of the training process. First, although Dr. GRPO removes the group standard deviation term from the advantage function to reduce scale bias, group centering (rx−rˉx1)(r_x-\bar r_x\mathbf{1}) still allows a scoring perturbation for one response to affect the other responses in the same group. Dr. GRPO also does not calibrate or filter grader noise. Second, although Noise-corrected GRPO provides an unbiased correction at the reward level, it does not account for the possibility that residual errors, after being multiplied by the current policy’s response-level gradients gx,ig_{x,i}, may still cause local update direction reversals. In contrast, SetGRPO jointly calibrates the within-group coefficient shifts induced by shared normalization in coefficient space and further uses the current policy’s gradient geometry to filter high-risk updates that may reverse the update direction. It therefore achieves higher downstream scores on all three backbones.

Reviewer_KuiKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how stochastic rubric-grader scores propagate through GRPO's shared reward normalization into policy updates. SetGRPO constructs an ellipsoidal uncertainty set for the joint coefficient vector using a normalization Jacobian and conformal calibration. It then searches a path from a minimum-update-norm anchor toward the nominal coefficients, screening candidates through projected gradient geometry.

Critical strengths
  1. The uncertainty target is well motivated. Shared group normalization couples response coefficients, and coefficient differences can interact with gradient geometry. Modeling this chain and evaluating risk after selection addresses a meaningful limitation of assessing grader disagreement solely through reward differences.

  2. The experimental design attempts several relevant controls. The paper includes multiple backbones, ten benchmarks, a matched random-discard baseline, module ablations, and an explicit overhead comparison. These are appropriate ingredients for testing whether selective recovery offers benefits beyond simply reducing the number of updates.

Critical weaknesses
  1. The reported residual correlation structure is incompatible with the stated centered coefficients. Equation (40) and Algorithm 1 imply that every valid coefficient vector has zero sum. Therefore, the paired residual δ=a⋆−a^\delta=a^\star-\hat a also satisfies 1⊤δ=0\mathbf{1}^{\top}\delta=0 for every group. Its covariance matrix must satisfy Σ1=0\Sigma\mathbf{1}=0, and in particular

    =\sum_i\operatorname{Var}(\delta_i) +2\sum_{i<j}\operatorname{Cov}(\delta_i,\delta_j).$$ Nevertheless, Figure 11(a) reports strictly positive correlations for every off-diagonal entry, ranging from 0.05 to 0.66. With nonzero coordinate variances, this cannot be the signed correlation matrix of the defined residuals. Appendix F.2 interprets these entries as positive correlations and uses them to support the joint uncertainty model. If the figure instead concerns absolute residuals, absolute correlations, or another transformation, that distinction changes its interpretation and must be stated. As presented, a central diagnostic does not validate the quantity the method claims to model.
  2. The reported calibration radius and recovered coefficient norms appear incompatible with the specified Jacobian ellipsoid. Write s=∥HKr∥2/Ks=\sqrt{\|H_Kr\|^2/K}. Equation (40) gives ∥a^∥2=K\|\hat a\|_2=\sqrt K and

    \qquad J\hat a=0.$$ With $S=JJ^{\top}+\lambda I$, the eigenvalue in the direction of $\hat a$ is therefore exactly $\lambda$. If $c$ is the cosine between the two normalized coefficient vectors, the calibration score must obey $$\rho\geq\frac{\sqrt K(1-c)}{\sqrt\lambda}.$$ For $K=8$, $\lambda=10^{-8}$ in Appendix C.6, and $Q_\alpha=1.35$ in Table 6, coverage requires $c\geq0.999952$. This requires reconciliation with the substantial coefficient disagreement in the preliminary diagnostics and the claimed 95% coverage in Figure 12. The entire recovery path stays in $C_x$, yet Figure 5 reports recovered coefficient norms around 1.080–1.216 without specifying a transformation or aggregation that explains this difference. These discrepancies concern the core calibration and recovery procedure, so the reported performance cannot presently establish that the stated algorithm works as claimed.
Questions for authors
  1. Clarification of Figure 11: Is this the signed Pearson correlation matrix of the residual vectors defined in Section 3.1, or a transformed quantity? Please provide the exact definition and the corresponding signed covariance matrix, including a check of Σ1=0\Sigma\mathbf{1}=0.

  2. Clarification of calibration and implementation: Are Equation (40), λ=10−8\lambda=10^{-8}, the ordinary inverse in Equation (5), and the Table 6 radii all used together in the reported runs? Please report the calibration coefficient-cosine distribution and verify the lower bound on ρ\rho above. Any pseudoinverse, eigenvalue truncation, or additional rescaling would need to be specified and reconciled with the coverage argument.

  3. Clarification of recovery measurements: What exactly is averaged or normalized in Figure 5, and are rejected zero updates included? Please reconcile its values with the minimum norm of coefficients on the stated recovery path and explain whether correcting either discrepancy changes the calibration, acceptance, or downstream results.

9 comments
AuthorsOpenReview ↗

The Response to Reviewer KuiK (Part1/5)

W1: Incompatibility between the residual correlations and the centered-coefficient definition: Equation (40) and Algorithm 1 imply that every valid coefficient vector sums to zero. Therefore, the paired residual δ\delta also satisfies 1⊤δ=0\mathbf{1}^{\top}\delta=0, and its covariance matrix must satisfy 1⊤Σδ1=0\mathbf{1}^{\top}\Sigma_{\delta}\mathbf{1}=0. When the coordinate variances are nonzero, the signed off-diagonal covariances cannot all be positive. However, Figure 11(a) reports positive correlations for all off-diagonal entries, ranging from 0.050.05 to 0.660.66. If the figure uses absolute residuals or another transformation, this must be stated explicitly; otherwise, the diagnostic does not validate the quantity claimed to be modeled.

A-W1: We thank the reviewer for the careful and professional assessment. As the reviewer suspected, the original Figure 11(a) computed the correlation matrix of the paired residual magnitudes ∣δx∣|\delta_x|, rather than the signed correlation matrix of the residuals without taking absolute values. We apologize for the ambiguity caused by not explicitly labeling this transformation in Appendix F.2 of the original manuscript. Specifically, we used the absolute residuals ∣δx∣|\delta_x| in the original figure to examine the within-group co-fluctuation of error magnitudes. Because GRPO normalizes each response group using shared mean and scale statistics, a large scoring discrepancy for one response can cause the coefficient deviation magnitudes of other responses in the same group to increase simultaneously. This produces the positive correlations ranging from 0.050.05 to 0.660.66. The result shows that the magnitudes of coefficient perturbations are coupled within a group rather than arising as isolated response-level noise. However, we fully agree with the reviewer that magnitude correlations only characterize the co-occurrence of perturbation magnitudes. They do not directly establish the signed linear correlation structure or the zero sum covariance property of the centered residual vector δx\delta_x.

To directly address this concern, we provide the signed Pearson correlation matrix, the signed covariance matrix Σ^δ\widehat\Sigma_\delta, a check of the zero-sum constraint (Σ^δ1=0\widehat\Sigma_\delta\mathbf{1}=\mathbf{0}), and the eigenspectrum of the signed correlation matrix. These quantities are computed from the signed centered residuals without the absolute-value transformation, where δx=ax⋆−a^x\delta_x=a_x^\star-\hat a_x.

According to Eq. (40) and Algorithm 1, the paired residual is defined as δx=ax⋆−a^x\delta_x=a_x^\star-\hat{a}_x, and each residual as well as its theoretical covariance matrix satisfies the zero sum constraint:

1⊤δx=0,Σδ1=0.\mathbf{1}^{\top}\delta_x=0,\quad \Sigma_{\delta}\mathbf{1}=\mathbf{0}.

Let B∈RK×(K−1)B\in\mathbb{R}^{K\times(K-1)} be an orthonormal basis satisfying B⊤B=IK−1B^{\top}B=I_{K-1} and B⊤1=0B^{\top}\mathbf{1}=\mathbf{0}. Defining zx=B⊤δxz_x=B^{\top}\delta_x, the residual covariance can be written as Σδ=B Cov⁡(zx) B⊤\Sigma_{\delta}=B\,\operatorname{Cov}(z_x)\,B^{\top}. As an isotropic reference, when all directions in the zero sum subspace have equal variance, the covariance matrix is Σ0=IK−1K11⊤\Sigma_0=I_K-\frac{1}{K}\mathbf{1}\mathbf{1}^{\top}. For K=8K=8, its theoretical off diagonal correlation coefficients are all −1/(K−1)=−1/7≈−0.143-1/(K-1)=-1/7\approx-0.143. In the general heteroscedastic case with within-group structural coupling, however, the zero sum constraint does not require every off diagonal correlation coefficient to be negative. It requires instead that the entries in every row and every column of the covariance matrix sum exactly to zero, with positive and negative covariance terms canceling each other.

AuthorsOpenReview ↗

The Response to Reviewer KuiK (Part1/5)

Under this precise definition, Table R2-1 reports the signed Pearson correlation matrix of the paired coefficient residuals from 500 groups. The off diagonal entries range from −0.59-0.59 to 0.600.60, with 18 positive and 38 negative entries. Their mean is −0.13-0.13, which is close to the theoretical isotropic reference value of −1/7≈−0.143-1/7\approx-0.143. At the same time, the nonuniform values show substantial heteroscedastic coupling beyond the isotropic reference.

Table R2-1: Signed Pearson correlation matrix of centered coefficient residuals δx\delta_x.

δ1\delta_1δ2\delta_2δ3\delta_3δ4\delta_4δ5\delta_5δ6\delta_6δ7\delta_7δ8\delta_8
δ1\delta_11.000.10-0.54-0.440.24-0.100.20-0.07
δ2\delta_20.101.00-0.20-0.340.08-0.350.21-0.50
δ3\delta_3-0.54-0.201.000.60-0.48-0.27-0.59-0.17
δ4\delta_4-0.44-0.340.601.00-0.52-0.25-0.55-0.19
δ5\delta_50.240.08-0.48-0.521.00-0.030.28-0.05
δ6\delta_6-0.10-0.35-0.27-0.25-0.031.000.030.32
δ7\delta_70.200.21-0.59-0.550.280.031.00-0.07
δ8\delta_8-0.07-0.50-0.17-0.19-0.050.32-0.071.00

We further compute the signed empirical covariance matrix from the same residuals, denoted by Σ^δ\widehat\Sigma_\delta, as

Σ^δ=1N−1∑x=1N(δx−δˉ)(δx−δˉ)⊤,\widehat\Sigma_\delta = \frac{1}{N-1}\sum_{x=1}^N (\delta_x - \bar\delta)(\delta_x - \bar\delta)^\top,

where δˉ=1N∑x=1Nδx\bar\delta = \frac{1}{N}\sum_{x=1}^N \delta_x. The result is reported in Table R2-2, with values given in units of 10−410^{-4}.

Table R2-2: Signed empirical covariance matrix Σ^δ\widehat{\Sigma}_{\delta} (×10−4\times 10^{-4}).

δ1\delta_1δ2\delta_2δ3\delta_3δ4\delta_4δ5\delta_5δ6\delta_6δ7\delta_7δ8\delta_8
δ1\delta_10.91270.1117-0.6385-0.59440.1915-0.08570.1673-0.0646
δ2\delta_20.11171.3860-0.2925-0.56680.0782-0.36840.2164-0.5647
δ3\delta_3-0.6385-0.29251.52511.0438-0.4971-0.2983-0.6405-0.2020
δ4\delta_4-0.5944-0.56681.04381.9875-0.6149-0.3156-0.6819-0.2579
δ5\delta_50.19150.0782-0.4971-0.61490.7002-0.02280.2053-0.0405
δ6\delta_6-0.0857-0.3684-0.2983-0.3156-0.02280.79480.02310.2728
δ7\delta_70.16730.2164-0.6405-0.68190.20530.02310.7697-0.0593
δ8\delta_8-0.0646-0.5647-0.2020-0.2579-0.04050.2728-0.05930.9162

At full floating-point precision before rounding, the maximum absolute row sum of Σ^δ\widehat\Sigma_\delta is approximately 2.7×10−202.7\times10^{-20}. Thus, Σ^δ1=0\widehat\Sigma_\delta\mathbf{1}=\mathbf{0} holds to numerical precision, and 1⊤Σ^δ1≈0\mathbf{1}^\top\widehat\Sigma_\delta\mathbf{1}\approx 0 up to numerical precision. Consistent with the original Figure 11(b), we also perform an eigenspectral decomposition of the signed correlation matrix reported in Table R2-1, with the results shown in Table R2-3. Because the zero sum constraint restricts the residuals to a seven dimensional subspace, the variance contribution of the eighth principal direction is exactly 0.00%0.00\%. The first three principal directions account for 70.58%70.58\% of the variation in the signed correlation structure. If the eigenspectral decomposition is instead applied directly to the covariance matrix Σ^δ\widehat\Sigma_\delta, the first three principal directions account for 74.99%74.99\% of the total variance. Both values are substantially higher than the isotropic reference value of 3/7≈42.86%3/7\approx42.86\% for the zero sum subspace. These results show that the signed residuals retain substantial anisotropic coupling within the zero sum subspace, even when the zero sum constraint is satisfied exactly. This directly supports the use of a joint coefficient ellipsoid in JRSC. In the revised manuscript, we will explicitly distinguish residual magnitude correlations from signed residual correlations in Appendix F.2 and update the signed correlation matrix, covariance matrix, and eigenspectral decomposition accordingly.

Table R2-3: Eigenvalue spectrum and cumulative explained variance of the signed correlation matrix (Table R2-1).

Principal directionExplained varianceCumulative explained variance
136.67%36.67%
222.81%59.47%
311.11%70.58%
49.45%80.03%
58.11%88.14%
67.57%95.71%
74.29%100.00%
80.00%100.00%
AuthorsOpenReview ↗

The Response to Reviewer KuiK (Part2/5)

W2: Inconsistency between the calibration radius and recovered coefficient norms. Under the normalized Jacobian, the all-ones direction lies in its null space. Thus, the relevant coefficient disagreement is not controlled by the unregularized ellipsoid; with ridge regularization, its radius is determined primarily by the ridge term rather than the coefficient covariance. The calibration radius in Table 6 and the constants in Appendix C.6 therefore need to be reconciled with the large coefficient disagreement observed in the pilot diagnostics and the claimed 95% coverage in Fig. 12. Moreover, although the recovery path is constrained to the zero-sum coefficient subspace, Fig. 5 reports coefficient norms of 1.080–1.216 without explaining the transformation or aggregation used. These inconsistencies leave the calibration and recovery claims insufficiently supported.

A-W2: We thank the reviewer for pointing out that the calibration radius, coverage results, and recovered coefficient norms should be mutually consistent. To address this concern, we first reviewed the implementation settings already specified in the manuscript (Appendix C.5, Table 6, and Appendix C.6, Eqs. 40 and 41, and Algorithms 1 to 4). We then added three targeted validation experiments and organized the discussion into four parts: (1) implementation details, which verify that the calibration distance and safe-anchor recovery use the same positive-definite ellipsoidal matrix; (2) a new sensitivity experiment on the Jacobian null space and ridge scale, which fixes the calibration radius reported in Table 6 and evaluates the zero-space semi-axis and necessary cosine lower bound under different values of λ\lambda; (3) a new cross-split cosine-boundary experiment, which evaluates the distributions of paired cosine similarities on the coverage-calibration, risk-calibration, and evaluation-test splits to verify whether the observed samples satisfy the necessary cosine bound and are consistent with the 95% coverage reported in Figure 12; and (4) a new comparison of two aggregation schemes for the recovered coefficient norms in Figure 5, which compares the accepted-group mean with the all-group mean that includes rejected groups whose written-back coefficients are zero, thereby explaining why the recovered coefficient norms in the zero sum subspace range from 1.080 to 1.216.

(1) Implementation details.

As specified in Appendix C.6 of the original manuscript (Page 23, Eqs. 40 and 41, and Algorithm 2), all formal runs use Vx=IKV_x=I_K, λ=10−8\lambda=10^{-8}, and the shape matrix

Sxcoef=JxVxJx⊤+λIK.S_x^{\mathrm{coef}}=J_xV_xJ_x^{\top}+\lambda I_K.

The Mahalanobis distance used during calibration (Eq. 5 in the main text) is computed by directly solving a linear system. The safe anchor (Eq. 8 in the main text and Algorithm 4) is obtained using the same positive-definite ellipsoidal matrix. No pseudoinverse, eigenvalue truncation, or additional rescaling is used at any stage. Therefore, the calibration radius QαQ_\alpha reported in the original Table 6, the coverage rate shown in Figure 12, and the recovery constraints in Algorithm 4 all use exactly the same definitions of the shape matrix and its inverse.

Reviewer_VcfSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addressed the problem of grader stochasticity in rubric-based GRPO training of reinforcement learning algorithms and proposing a new training method called SetGRPO, in which multiple existing techniques were combined to construct a joint uncertainty set in the training-coefficient space, calibrate possible coefficient shifts, recover acceptable training signals, and calibrated the risk of accepted updates. Experiment results on ten open-ended task benchmarks in three domains show that SetGRPO achieved the best macro-average performance among all compared training methods on the Qwen-3.5 and Llama-3.2 model families.

Critical strengths
  • GRPO is a popular RL algorithm with many applications, and the work proposed an improvement of GRPO
  • Experimental evidence was given to motivate the proposed idea first in Section 2
  • The proposed method was carefully designed to ensure robustness, and ablation study showed that all the three techniques (coefficient-set construction, local geometry and recovery, and threshold selection and update control) are effective.
  • Experment results are encouraging.
Critical weaknesses
  • The technical ideas appear to be mainly based on existing work. In this sense, the main novelty lies in applying them to GRPO.
  • The paper includes a lot of technical details but doesn't provide intuitive explanation of why the proposed methods are expected to work.
Questions for authors
  • Are there other methods from existing work that could also be used to address the problem of grader stochasticity? The related work part didn't seem to directly address this question. In other words, which existing work is closest to your work?
5 comments
AuthorsOpenReview ↗

The Response to Reviewer VcfS (Part1/3)

W1: Limited technical novelty: The technical ideas appear to build largely on existing work. In this sense, the main novelty lies in applying these techniques to GRPO.

A-W1: We thank the reviewer for their interest in the novelty of our method. Our contribution is not simply to combine existing mathematical tools. Rather, we are the first to systematically investigate how rubric score discrepancies propagate through GRPO and alter policy updates. To the best of our knowledge, existing rubric-based RL studies have focused primarily on scoring quality, evaluator consistency, or final training performance, without treating the propagation of score discrepancies to policy updates as a research problem in its own right.

To address this problem, we formalize this propagation process as the training-coefficient-driven policy-update problem and propose SetGRPO. SetGRPO conformally calibrates the joint coefficient space used by GRPO and combines policy-update geometry with selective risk gating. This design limits deviations in update direction while preserving effective training progress. Thus, our contribution is not to introduce a standalone mathematical technique, but to formulate a problem specific to this propagation mechanism in GRPO and develop a corresponding training method.

The design, which targets the propagation mechanism, yields consistent performance gains across all three models. In terms of training behavior, SetGRPO keeps the conditional direction-reversal risk of accepted updates between 5.49%5.49\% and 5.54%5.54\%. The corresponding one-sided confidence bounds range from 5.74%5.74\% to 5.82%5.82\%, all below the prespecified 8.0%8.0\% threshold (Table 6). SetGRPO also retains an average recovery ratio between 76%76\% and 80%80\% (Figure 3).In terms of downstream performance, Table 1 reports results on ten benchmarks spanning the medical, writing, and instruction-following domains. SetGRPO achieves the highest Macro Avg. among the compared training methods on all three backbones: 60.26\textbf{60.26} on Qwen3.5-4B, 63.51\textbf{63.51} on Qwen3.5-9B, and 49.57\textbf{49.57} on Llama3.2-3B. These scores exceed those of standard rubric-based GRPO by 4.85\textbf{4.85}, 4.82\textbf{4.82}, and 4.85\textbf{4.85} points, respectively; they also exceed RubricHub by 1.611.61, 1.401.40, and 1.401.40 points, and RuscaRL by 3.123.12, 3.183.18, and 2.722.72 points. Together, these results demonstrate the practical value of modeling this propagation mechanism.

AuthorsOpenReview ↗

The Response to Reviewer VcfS (Part 2/3)

W2: The paper contains extensive technical detail but does not provide an intuitive explanation of why the proposed method is expected to work.

A-W2: We thank the reviewer for raising this concern. SetGRPO addresses the distinct effects of score discrepancies, joint training coefficients, and actual policy updates. We therefore provide the calibration targets, selection criteria, and implementation details needed to make the method reproducible. These components are not redundant technical additions; each addresses a different stage through which grader randomness affects GRPO training.

Intuitively, SetGRPO has three functions: it calibrates the joint coefficient perturbations induced by score discrepancies, assesses whether candidate coefficients still produce directionally consistent updates under the current policy, and controls the direction-reversal risk among accepted updates. To illustrate why this mechanism is effective, we provide visual diagnostics and case studies in the main text and appendix, alongside theoretical guarantees.

For the visual and qualitative analyses, the pilot study (Figure 1 and Appendix F.1, Figure 10) illustrates how a discrepancy in one score can shift the coefficients of an entire response group and, through the policy gradients, reverse the update direction. Appendix D.2 (Figure 6) and Appendix F.8 (Figure 18) visualize the projection distributions of accepted and rejected groups in the full and reward spaces, as well as the recovery trajectories that interpolate from safe anchors toward the training coefficients. Appendix F.3 (Figure 12) and Appendix F.5 (Figure 14) show the empirical coverage of the calibrated sets, the separation in direction-reversal risk between accepted and rejected groups (0.0350.035 versus 0.1180.118), and the trade-off involved in threshold selection. Finally, Appendix F.9 (Figures 19, 20, and 21) presents examples from writing, medical, and instruction-following tasks, illustrating the recovery-path decisions for accepted, rejected, and recovery-infeasible groups.

For the theoretical guarantees, Appendix A provides mathematical proofs for the three modules. These include the finite-sample coverage guarantee for the joint coefficient set (Appendix A.2), the closed-form derivation of the local worst-case projection and its direction-consistency guarantee (Appendix A.4), the existence of a safe anchor and the feasibility of the recovery path (Appendix A.5), and simultaneous conditional risk control under threshold selection and its monotonicity (Appendices A.6 and A.7).

AuthorsOpenReview ↗

The Response to Reviewer VcfS (Part 2/3)

Furthermore, we extended the six ablation and control variants in the main paper’s Table 2, which uses Qwen3.5-4B, to all three backbones (see Table R1-1). The ablation variants D#1–D#6 use a single standard run with a fixed default random seed, whereas the full SetGRPO row reports the mean over three runs from Table 1. The results are consistent with the expected roles of the components: for all three models, removing or replacing any component yields a lower score than full SetGRPO, whose baseline macro average is 57.78. Specifically, (a) replacing JRSC’s joint coefficient ellipsoid with an isotropic spherical set (D#1 Spherical set), which cannot capture coupled within-group coefficient shifts, lowers the overall mean by 3.65 points; (b) ignoring the current-policy gradient geometry in GGSR (D#2 Identity geometry), retaining only the conservative anchor without path recovery (D#3 Safe-anchor-only), or using only the reward-space projection criterion (D#4 Reward-only recovery) lowers the overall mean by 5.17, 3.32, and 2.52 points, respectively; and (c) replacing SRGO’s risk calibration with a fixed threshold (D#5 Fixed-threshold) lowers the overall mean by 2.15 points, while a random-discard control matched for the number of updates (D#6 Matched random discard) lowers it by 5.40 points. This indicates that the gains come from targeted geometric screening rather than simply reducing the number of updates. Taken together, the theoretical analysis, visualization diagnostics, and ablation comparisons support the design rationale for each module and their combined effect.

Table R1-1: Complete ablation and control variants across three backbones (extending Table 2 in the main text).

D#DerivativeVariant GroupQwen3.5-4BQwen3.5-9BLlama3.2-3BMacro avg.
1Spherical set(a) Coefficient-set construction56.6259.8445.9254.13
2Identity geometry(b) Local geometry and recovery55.0858.3544.4152.61
3Safe-anchor-only(b) Local geometry and recovery56.9460.1846.2554.46
4Reward-only recovery(b) Local geometry and recovery57.7860.9647.0555.26
5Fixed-threshold(c) Threshold selection and update control58.1161.3647.4255.63
6Matched random discard(c) Threshold selection and update control54.8358.1244.1852.38
-SetGRPO (Ours)Full Model60.2663.5149.5757.78

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and All Reviewers

Thank you to the Area Chair and the three reviewers ( VcfS , KuiK , and RCgS ) for the careful reviews and constructive feedback. We have responded point by point to every Weakness and Question raised by the three reviewers, and addressed each issue through additional experiments, clarifications, or direct revisions to the original manuscript (see the responses under each reviewer). The main additions are summarized below:

1) Problem formulation, module intuition, local update representations, and multi-backbone ablations ( VcfS , RCgS ): We clarify that our core contribution is to formalize and model how rubric score discrepancies propagate through GRPO's shared normalization and current-policy gradient geometry to alter policy updates, supported by both theoretical guarantees (Appendix A) and visual/case diagnostics. We also specify the distinct screening roles of the D=256D=256 full update and reward space representations (Appendix C.6, Table 10) and extend all six ablation and control variants (Spherical set, Identity geometry, Safe-anchor-only, Reward-only recovery, Fixed-threshold, and Matched random discard) across Qwen3.5-4B, Qwen3.5-9B, and Llama3.2-3B, confirming consistent drops (−2.15-2.15 to −5.40-5.40 Macro Avg.) when any module is removed or replaced (Appendix D.3, Table 17).

2) Related GRPO variants and equal-budget comparisons ( VcfS , RCgS ): Across all three backbones, SetGRPO (57.7857.78 Macro Avg.) outperforms GRPO (+4.84+4.84), Dr. GRPO (+2.13+2.13), Noise-corrected GRPO (+1.66+1.66), and Score averaging (+0.70+0.70) (Section 6; Appendix B; Appendix D.6, Table 21). Under an equal incremental budget of 54,458 group-level (435,664 response-level) grader calls and the analytic error-variance relationship (0.3830+0.6170ρ)σ2(0.3830+0.6170\rho)\sigma^2, Frozen SetGRPO lowers reversal risk from 8.42%–8.85% (Score averaging) to 5.49%–5.54% (+0.59+0.59 to +0.83+0.83 Macro Avg.), while multi-checkpoint online Fresh recalibration adds only +0.27+0.27 to +0.38+0.38 (Appendix D.7, Tables 22–24).

3) Signed residual covariance structure and zero-sum check ( KuiK ): We clarify that Figure 11(a) originally plotted residual-magnitude correlations and update Appendix F.2 (Figure 11, Tables 25–27) with the signed Pearson correlation matrix of δx\delta_x, the signed empirical covariance Σ^δ\widehat\Sigma_\delta, an exact zero-sum verification (Σ^δ1=0\widehat\Sigma_\delta\mathbf{1}=\mathbf{0}, with maximum absolute row sum 2.7×10−202.7\times 10^{-20}), and the eigenspectrum (the top-3 directions explain 70.58%70.58\% correlation variation and 74.99%74.99\% covariance variance, well above the 3/7≈42.86%3/7\approx 42.86\% zero-sum isotropic benchmark).

4) Implementation consistency, Jacobian null-space geometry, and ridge sensitivity ( KuiK , RCgS ): We clarify the scope of our finite-sample guarantees under exchangeability (Sections 3.1, 3.3) and confirm that calibration and anchor recovery use the exact same positive-definite matrix Sxcoef=JxVxJx⊤+λIKS_x^{\mathrm{coef}}=J_xV_xJ_x^\top+\lambda I_K via direct linear solve and ordinary inversion without pseudoinverses, truncation, or rescaling (Appendix C.6). We derive the null-space semi-axis QαλQ_\alpha\sqrt{\lambda} and necessary cosine lower bound 1−Qαλ/K1-Q_\alpha\sqrt{\lambda/K}, test λ∈{10−6,10−8,10−10}\lambda\in\{10^{-6},10^{-8},10^{-10}\}, and report paired cosine quantiles and margins across coverage-calibration, risk-calibration, and evaluation-test, confirming strictly positive p5p_5 margins (Appendix C.5, Tables 6–9).

5) Cross-seed stability and clarifications on recovered norms, risk denominators, and case sources ( KuiK , RCgS ): Across n=3n=3 seeds on all ten benchmarks, SetGRPO achieves the highest mean on 23/30 pairs and its Macro Avg. margins over RubricHub (+1.40+1.40 to +1.61+1.61) exceed cross-seed SDs (0.420.42 to 0.500.50) (Table 1; Appendix D.6, Tables 19–20). We also provide Step 0–500 trajectories and accepted-only (1.0935,1.17761.0935, 1.1776) vs. all-group (0.8657,0.93230.8657, 0.9323) mean comparisons to reconcile recovered coefficient norms with the nominal norm K=8≈2.828\sqrt{K}=\sqrt{8}\approx 2.828 (Appendix D.2, Tables 14–15), align the reversal-risk denominators between the 27,229-group risk-calibration split (5.49%–5.54%) and the Qwen3.5-4B diagnostic subset (4.7% in Section 5.1, Table 2, Appendix F.3), and confirm that the nine cases in Appendix F.9 (Table 28) come from a post-hoc read-only diagnostic pass.

All of the above additions and revisions have been incorporated into the updated manuscript (highlighted in red). Thank you again to the Area Chair and all reviewers for your valuable feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.