- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Range-GRPO is a semi-supervised policy optimization method that uses conformally calibrated reward intervals for unlabeled responses. It constructs group-relative advantages from pairwise comparisons of interval centers, weighting each comparison by the widths of both intervals. Unlike scalarized rewards, this pairwise weighting can change an advantage's sign. The intervals use a judge-based scorer and shift-aware calibration, and the method reduces to Dr.GRPO when intervals have zero width. Experiments on a synthetic task and reasoning benchmarks report improvements over the evaluated semi-supervised baselines.
Critical strengths
- Pairwise interval weighting can change an advantage's sign when scalar reward reduction or positive reweighting cannot, giving uncertainty a direct role in learning credit assignment.
- The method reduces to Dr.GRPO for zero-width intervals and uses shift-aware calibration to construct intervals as the policy changes.
- Results on a synthetic task and reasoning benchmarks are promising, and the paper reports response budgets and training time alongside accuracy.
Critical weaknesses
- The main results do not isolate the benefit of pairwise interval weighting. There is no midpoint Dr.GRPO control using the same intervals and training setup, while the UARM comparison uses a different reward-modeling pipeline. The matched comparisons are limited to the synthetic task.
- The experiments do not empirically validate the calibration quality of the reward intervals under the actual target setting.
- The paper does not establish a coverage guarantee for its implemented on-policy interval procedure, which uses estimated and clipped density ratios and a normalized nonconformity score. Standard weighted conformal guarantees under ideal covariate-shift assumptions do not automatically transfer to this setting.
- The proposed pairwise weight is heuristic. The paper does not provide a statistical derivation or compare it with alternative monotone uncertainty-weighting functions.
- is only tuned on the toy task, while the main experiments report only , without a sensitivity analysis of downstream accuracy, advantage-sign changes, or training stability.
- The main results appear to use one run per configuration, making it hard to assess whether improvements are stable across high temperature sampling or different seeds.
- The normalized nonconformity score in Appendix C.3 is a substantive modification of the interval-construction procedure in Section 3.2, yet its effect is not isolated.
Questions for authors
- Can the authors measure empirical interval coverage and width on a held-out labeled target-like set at multiple stages of training?
- How do performance and advantage signs change with or an alternative monotone width weight?
- Which MMLU-Pro splits supply unlabeled training prompts and OOD evaluation?
11 comments
W2 & Q1. Empirical coverage and width of reward intervals across training stages
We evaluated all responses generated for the unlabeled MMLU-Pro prompts during Range-GRPO policy training. We consider the active phase after the labeled warmup, where unlabeled prompts are used for training (steps 121–195). The correctness labels of these responses were withheld throughout and used only for post-hoc evaluation; they played no role in policy training, judging, scorer fitting, or density-ratio estimation.
Table R2. Shift-aware calibration consistently improves empirical target coverage across training stages. Each cell reports target coverage (%) / mean interval width (α = 0.1). Values in parentheses denote the coverage gain over No DRE in percentage points. No DRE recomputes the intervals for the same responses without accounting for the distribution shift. Coverage gains are computed from unrounded values.
| Training stage | Shift-aware calibration (ours) | No DRE |
|---|---|---|
| Early (steps 121–145) | 88.63 (+3.98) / 0.6129 | 84.65 / 0.5597 |
| Middle (steps 146–170) | 87.30 (+5.83) / 0.6008 | 81.47 / 0.5270 |
| Late (steps 171–195) | 86.30 (+10.34) / 0.7137 | 75.97 / 0.5544 |
| Full trajectory | 87.42 (+6.70) / 0.6423 | 80.72 / 0.5471 |
With shift-aware calibration, coverage ranges from 86.30% to 88.63% across stages, close to the nominal 90%. In contrast, without accounting for the distribution shift, coverage falls to 75.97% in the late stage, and the gap between the two grows to 10.34 points. Shift-aware calibration achieves this by widening the intervals on average to account for the distribution shift. We have added these analyses of coverage, interval width, and calibration ablations to Appendix E.2 (Table 19) of the revised manuscript.
W3. Coverage analysis of the implemented interval procedure
We added a theoretical analysis of per-iteration coverage in Appendix B.1. The analysis follows the reward interval construction described in Appendix D.3.
- Theorem 1 establishes nominal marginal coverage when the judgment state preserves the conditional reward distribution and the density ratio is exact.
- Theorem 2 shows that, with estimated states and weights, coverage is at least under the stated assumptions. Here, bounds the residual conditional reward distribution mismatch, while bounds the density-ratio approximation error, including estimation and clipping effects.
For practical efficiency, we reuse a fixed calibration set throughout training. As the theoretical guarantees rely on conditional independence assumptions, we complement the analysis with an empirical coverage evaluation of the implemented training loop. Observed coverage remains relatively stable across training stages (86.30–88.63%; Table R2), close to the nominal 90% level.
W4 & Q2. Statistical justification of the pairwise weight
We added a statistical derivation of the proposed pairwise weight in Appendix B.2 (Proposition 3) of the revised manuscript. The observed midpoint difference is decomposed into the true pairwise difference and estimation error , such that . Under the stated assumptions, Proposition 3 establishes that the proposed weight uniquely minimizes the conditional mean squared error. Consequently, alternative monotone uncertainty-weighting functions cannot achieve a lower conditional MSE in the same setting.
We thank the reviewer for recognizing the strength of pairwise interval weighting in credit assignment and acknowledging our reported efficiency analyses. We address each concern below.
W1. Isolating the contribution of pairwise interval weighting
To isolate the contribution of pairwise interval weighting on the real tasks, we conducted a new controlled experiment. Every method shares the reward-interval construction procedure and training setup of Range-GRPO. This includes the labeled-warmup checkpoint, judge, scorer, calibration procedure, data partitions, and hyperparameters. Only the way advantages are computed from the intervals differs. We report this three-seed controlled comparison in Appendix E.1 (Table 18) of the revised manuscript.
Table R1. Range-GRPO achieves the highest mean ID, OOD, and overall accuracies under matched experimental conditions. Results are reported as mean ± sample standard deviation across three active-phase training seeds. For each seed, we report the checkpoint with the highest validation accuracy. Overall is the mean of the ID and OOD averages.
Across three active-phase training seeds, Range-GRPO achieves the highest mean ID, OOD, and overall accuracies. It shows gains over UARM-style weighting, with lower variability in OOD and overall performance, while achieving modest improvements over midpoint Dr.GRPO. These results provide empirical support for pairwise interval weighting under matched conditions.
Importantly, shift-aware calibration is another component of Range-GRPO that improves the empirical reliability of reward intervals under distribution shift. While Table R1 isolates the contribution of pairwise weighting, the overall results in Table 2 reflect the performance of the complete framework, combining uncertainty-aware pairwise credit assignment with shift-aware calibration.