acceptodds
Under review as a conference paper at ICLR 2027

Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals

Abstract

As the use of large language models (LLMs) expands, post-training has become increasingly important for adapting them to downstream tasks. However, obtaining reliable supervision remains costly, especially in domains without reference answers or executable verifiers. LLM-as-a-Judge provides scalable pseudo-rewards for unlabeled responses, but a single point score does not explicitly represent reward uncertainty. This motivates representing pseudo-rewards as conformally calibrated reward ranges. We propose Range-GRPO, a semi-supervised post-training framework that combines limited labeled data with unlabeled prompts. In Group Relative Policy Optimization (GRPO), learning signals depend on relative reward comparisons within each rollout group. The proposed objective compares reward ranges pairwise rather than reducing them to point rewards, allowing interval uncertainty to affect both the magnitude and direction of these signals. Our theoretical analysis characterizes this distinction and shows that the proposed objective recovers the Dr.GRPO advantage when all reward ranges collapse to points. Empirically, Range-GRPO achieves the highest in-distribution and out-of-distribution average performance among the evaluated semi-supervised methods while using a smaller total response budget than the compared baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Group Relative Policy Optimization, Conformal Prediction, Reward Uncertainty, Semi-supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ttJBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Range-GRPO is a semi-supervised policy optimization method that uses conformally calibrated reward intervals for unlabeled responses. It constructs group-relative advantages from pairwise comparisons of interval centers, weighting each comparison by the widths of both intervals. Unlike scalarized rewards, this pairwise weighting can change an advantage's sign. The intervals use a judge-based scorer and shift-aware calibration, and the method reduces to Dr.GRPO when intervals have zero width. Experiments on a synthetic task and reasoning benchmarks report improvements over the evaluated semi-supervised baselines.

Critical strengths
  • Pairwise interval weighting can change an advantage's sign when scalar reward reduction or positive reweighting cannot, giving uncertainty a direct role in learning credit assignment.
  • The method reduces to Dr.GRPO for zero-width intervals and uses shift-aware calibration to construct intervals as the policy changes.
  • Results on a synthetic task and reasoning benchmarks are promising, and the paper reports response budgets and training time alongside accuracy.
Critical weaknesses
  • The main results do not isolate the benefit of pairwise interval weighting. There is no midpoint Dr.GRPO control using the same intervals and training setup, while the UARM comparison uses a different reward-modeling pipeline. The matched comparisons are limited to the synthetic task.
  • The experiments do not empirically validate the calibration quality of the reward intervals under the actual target setting.
  • The paper does not establish a coverage guarantee for its implemented on-policy interval procedure, which uses estimated and clipped density ratios and a normalized nonconformity score. Standard weighted conformal guarantees under ideal covariate-shift assumptions do not automatically transfer to this setting.
  • The proposed pairwise weight ωij=λ/(hi+hj+λ)\omega_{ij}=\lambda/(h_i+h_j+\lambda) is heuristic. The paper does not provide a statistical derivation or compare it with alternative monotone uncertainty-weighting functions.
  • λ\lambda is only tuned on the toy task, while the main experiments report only λ=1\lambda=1, without a sensitivity analysis of downstream accuracy, advantage-sign changes, or training stability.
  • The main results appear to use one run per configuration, making it hard to assess whether improvements are stable across high temperature sampling or different seeds.
  • The normalized nonconformity score in Appendix C.3 is a substantive modification of the interval-construction procedure in Section 3.2, yet its effect is not isolated.
Questions for authors
  • Can the authors measure empirical interval coverage and width on a held-out labeled target-like set at multiple stages of training?
  • How do performance and advantage signs change with λ\lambda or an alternative monotone width weight?
  • Which MMLU-Pro splits supply unlabeled training prompts and OOD evaluation?
11 comments

We thank the reviewer for recognizing the strength of pairwise interval weighting in credit assignment and acknowledging our reported efficiency analyses. We address each concern below.

W1. Isolating the contribution of pairwise interval weighting

To isolate the contribution of pairwise interval weighting on the real tasks, we conducted a new controlled experiment. Every method shares the reward-interval construction procedure and training setup of Range-GRPO. This includes the labeled-warmup checkpoint, judge, scorer, calibration procedure, data partitions, and hyperparameters. Only the way advantages are computed from the intervals differs. We report this three-seed controlled comparison in Appendix E.1 (Table 18) of the revised manuscript.

  • Midpoint Dr.GRPO uses the interval midpoints as scalar rewards. It was the strongest scalar reduction in the synthetic experiment (Table 1).
  • UARM-style weighting applies response-level weighting based on Eq. 4 to the same midpoint-based advantages. It uses the same interval construction instead of the quantile reward model proposed in the original UARM.

Table R1. Range-GRPO achieves the highest mean ID, OOD, and overall accuracies under matched experimental conditions. Results are reported as mean ± sample standard deviation across three active-phase training seeds. For each seed, we report the checkpoint with the highest validation accuracy. Overall is the mean of the ID and OOD averages.

Advantage constructionID Avg.OOD Avg.Overall
Midpoint Dr.GRPO42.0 ± 1.1356.0 ± 0.9049.0 ± 0.22
Response-level weighting (UARM-style)41.9 ± 0.5454.7 ± 2.7348.3 ± 1.64
Range-GRPO42.2 ± 0.7156.1 ± 0.6749.1 ± 0.45

Across three active-phase training seeds, Range-GRPO achieves the highest mean ID, OOD, and overall accuracies. It shows gains over UARM-style weighting, with lower variability in OOD and overall performance, while achieving modest improvements over midpoint Dr.GRPO. These results provide empirical support for pairwise interval weighting under matched conditions.

Importantly, shift-aware calibration is another component of Range-GRPO that improves the empirical reliability of reward intervals under distribution shift. While Table R1 isolates the contribution of pairwise weighting, the overall results in Table 2 reflect the performance of the complete framework, combining uncertainty-aware pairwise credit assignment with shift-aware calibration.

W2 & Q1. Empirical coverage and width of reward intervals across training stages

We evaluated all responses generated for the unlabeled MMLU-Pro prompts during Range-GRPO policy training. We consider the active phase after the labeled warmup, where unlabeled prompts are used for training (steps 121–195). The correctness labels of these responses were withheld throughout and used only for post-hoc evaluation; they played no role in policy training, judging, scorer fitting, or density-ratio estimation.

Table R2. Shift-aware calibration consistently improves empirical target coverage across training stages. Each cell reports target coverage (%) / mean interval width (α = 0.1). Values in parentheses denote the coverage gain over No DRE in percentage points. No DRE recomputes the intervals for the same responses without accounting for the distribution shift. Coverage gains are computed from unrounded values.

Training stageShift-aware calibration (ours)No DRE
Early (steps 121–145)88.63 (+3.98) / 0.612984.65 / 0.5597
Middle (steps 146–170)87.30 (+5.83) / 0.600881.47 / 0.5270
Late (steps 171–195)86.30 (+10.34) / 0.713775.97 / 0.5544
Full trajectory87.42 (+6.70) / 0.642380.72 / 0.5471

With shift-aware calibration, coverage ranges from 86.30% to 88.63% across stages, close to the nominal 90%. In contrast, without accounting for the distribution shift, coverage falls to 75.97% in the late stage, and the gap between the two grows to 10.34 points. Shift-aware calibration achieves this by widening the intervals on average to account for the distribution shift. We have added these analyses of coverage, interval width, and calibration ablations to Appendix E.2 (Table 19) of the revised manuscript.

W3. Coverage analysis of the implemented interval procedure

We added a theoretical analysis of per-iteration coverage in Appendix B.1. The analysis follows the reward interval construction described in Appendix D.3.

  • Theorem 1 establishes nominal marginal coverage when the judgment state preserves the conditional reward distribution and the density ratio is exact.
  • Theorem 2 shows that, with estimated states and weights, coverage is at least 1−α−ΔZ,t−Δw,t1-\alpha-\Delta_{Z,t}-\Delta_{w,t} under the stated assumptions. Here, ΔZ,t\Delta_{Z,t} bounds the residual conditional reward distribution mismatch, while Δw,t\Delta_{w,t} bounds the density-ratio approximation error, including estimation and clipping effects.

For practical efficiency, we reuse a fixed calibration set throughout training. As the theoretical guarantees rely on conditional independence assumptions, we complement the analysis with an empirical coverage evaluation of the implemented training loop. Observed coverage remains relatively stable across training stages (86.30–88.63%; Table R2), close to the nominal 90% level.

W4 & Q2. Statistical justification of the pairwise weight

We added a statistical derivation of the proposed pairwise weight in Appendix B.2 (Proposition 3) of the revised manuscript. The observed midpoint difference d=ci−cjd=c_i-c_j is decomposed into the true pairwise difference δ\delta and estimation error ε\varepsilon, such that d=δ+εd=\delta+\varepsilon. Under the stated assumptions, Proposition 3 establishes that the proposed weight ωij=λ/(λ+hi+hj)\omega_{ij}=\lambda/(\lambda+h_i+h_j) uniquely minimizes the conditional mean squared error. Consequently, alternative monotone uncertainty-weighting functions cannot achieve a lower conditional MSE in the same setting.

Reviewer_WAxQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Range-GRPO, a semi-supervised post-training method that constructs calibrated reward intervals from limited labeled data and LLM judge assessments. It forms advantages by weighting pairwise midpoint differences according to both intervals' widths, which allows uncertainty to change the direction of an update. The paper is well written, and the analytical results provide a useful characterization of this mechanism. The contribution is potentially useful, with promising reasoning results. My major concern is on the main experiments. They do not isolate the proposed advantage rule and do not establish interval reliability under the evaluated distribution shifts.

Critical strengths
  1. This paper provides a useful approach to uncertain credit assignment in RL. Weighting individual comparisons addresses a limitation of response-level advantage rescaling. It addresses the problem that an uncertain response can distort the learning signals assigned to other responses. The proposed rule accounts for uncertainty on both sides of each comparison. Its simplicity makes it easy to understand and integrate.

  2. In Table 2, Range-GRPO improves the ID/OOD averages from 40.6/52.9 for reproduced TraPO to 42.7/56.6, achieving the best semi-supervised result on 8 of 9 benchmarks. The controlled synthetic experiment also favors Range-GRPO when methods share the same interval-construction procedure.

  3. The paper explicitly accounts for scorer fitting, calibration, density-ratio estimation, and judge calls. It reports an 18.2% smaller response budget than TraPO and comparable or lower training time on the reported hardware.

Critical weaknesses
  1. The controlled comparison against midpoint, lower-bound, upper-bound, and matched UARM advantages is limited to the synthetic task. In the reasoning experiments, UARM uses a different reward estimator and learning rate, while TraPO uses a different supervision pipeline. The improvements cannot be confidently attributed to pairwise interval weighting. Can the authors train these alternative advantage rules using the same judge, scorer, intervals, data partitions, and tuning budget?

  2. The controlled comparison of advantage rules is confined to a synthetic task. The main experiments involve mathematics and multiple-choice reasoning, the reference answers in those tasks allow binary correctness rewards. Although withholding these answers creates a valid semi-supervised setting, it does not fully test the motivating scenario of learning from uncertain LLM-judge scores. Maybe evaluation on open-ended tasks with rubric-based judge rewards would better establish the method's practical value.

  3. All main experiments use Qwen2.5-Math-7B. It is an older base model with relatively weak initial performance on the evaluated benchmarks. The paper does not establish whether the gains persist with stronger, more recent instruction-tuned or reasoning models. Stronger models' reward distributions and failure modes may differ substantially. This raises the concern of whether the method offers benefits beyond a relatively weak starting policy. Evaluation on at least one stronger backbone might strengthen the evidence for broader applicability.

Questions for authors

Could increasing the rollout group size recover some of the benefits of Range-GRPO? Larger groups may provide more stable baseline estimates, while standard GRPO already scales centered rewards by the within-group reward standard deviation.

4 comments

We thank the reviewer for the thoughtful review and for recognizing that weighting individual comparisons addresses a limitation of response-level advantage rescaling. We provide our responses below.

W1. Isolating the advantage construction and interval reliability on the real tasks

W1-1. Controlled comparison of advantage constructions

Following the reviewer's suggestion, we conducted a controlled comparison of three advantage constructions on the real tasks. All variants start from the same labeled-warmup checkpoint and share the same judge, scorer, reward interval construction, data partitions, and training configuration, including the learning rate and entropy coefficient. Only the advantage construction for unlabeled responses differs. We report these results in Appendix E.1 (Table 18) of the revised manuscript.

We use midpoint Dr.GRPO as the scalar baseline, since it performed best among the scalar reductions (lower bound, midpoint, and upper bound) in the synthetic experiment (Table 1). For response-level weighting, we apply the UARM weighting to the same midpoint-based advantages.

Table R1. Range-GRPO achieves the highest mean ID, OOD, and overall accuracies under matched experimental conditions. Results are reported as mean ± sample standard deviation across three active-phase training seeds. For each seed, we report the checkpoint with the highest validation accuracy. Overall is the mean of the ID and OOD averages.

Advantage constructionID Avg.OOD Avg.Overall
Midpoint Dr.GRPO42.0 ± 1.1356.0 ± 0.9049.0 ± 0.22
Response-level weighting (UARM-style)41.9 ± 0.5454.7 ± 2.7348.3 ± 1.64
Range-GRPO42.2 ± 0.7156.1 ± 0.6749.1 ± 0.45

Across three active-phase training seeds, Range-GRPO achieves the highest mean ID, OOD, and overall accuracies. It shows gains over UARM-style weighting, with lower variability in OOD and overall performance, while achieving modest improvements over midpoint Dr.GRPO. No method-specific hyperparameter tuning was performed in this comparison. These results provide empirical support for pairwise interval weighting under matched conditions.

W1-2. Hyperparameter-matched comparison

We also evaluated Range-GRPO with the same learning rate (1×10−61\times10^{-6}) and entropy coefficient (0.01) as the compared baselines to assess its performance under matched optimization settings.

Table R2. Range-GRPO achieves the highest OOD and overall average accuracies under matched learning-rate settings. All methods use a learning rate of 1×10−61\times10^{-6} and an entropy coefficient of 0.01. Overall is the mean of the reported ID and OOD averages. Bold and underlined values indicate the best and second-best results, respectively.

MethodID Avg.OOD Avg.Overall
TraPO (entropy=0.01)38.051.7‾\underline{51.7}44.85‾\underline{44.85}
UARM39.150.044.55
Range-GRPO38.8‾\underline{38.8}53.646.20

Under this setting, Range-GRPO achieves the highest OOD and overall average accuracies, while ranking second on ID. These results further support the effectiveness of Range-GRPO in leveraging unlabeled responses for policy optimization under matched optimization settings.

W1-3. Reliability of reward intervals under distribution shift

We evaluated reward intervals on unlabeled OOD policy-training responses, using correctness labels only for post-hoc evaluation. The intervals achieve 87.42% empirical coverage with a mean width of 0.6423, close to the nominal 90% level (α=0.1\alpha=0.1). Without DRE, coverage drops to 80.72%. We have added analyses of stage-wise coverage, calibration ablations, and judgment-state reliability to the revised manuscript in Appendix E.2 (Table 19 and Figure 3).

Together, these results provide empirical support for the contribution of pairwise advantage construction under matched conditions, while demonstrating the benefit of shift-aware calibration for reward interval reliability.

Q1. Effect of larger rollout groups and standard deviation normalization

Increasing the group size can stabilize group statistics, while standard deviation normalization rescales group-relative advantages. However, neither explicitly accounts for the reward uncertainty of individual responses, which Range-GRPO directly addresses.

Larger groups. Increasing the group size can improve the stability of the group mean estimate, but individual rewards inferred from judgment states remain uncertain. In contrast, Range-GRPO down-weights pairwise comparisons involving responses with wider reward intervals. Larger groups also require more rollouts per prompt, whereas efficient learning under a limited response budget is another key motivation of our work.

Standard deviation normalization. Standard GRPO divides the centered rewards in a group by a common standard deviation. This rescales the advantages without changing their signs or relative ordering. Similarly, Proposition 2 establishes that positive response-level reweighting preserves advantage signs, whereas Range-GRPO allows uncertainty to change the update direction. Furthermore, normalization by a small standard deviation can amplify small reward differences, potentially including those caused by estimation noise. This is related to the question-level difficulty bias identified in prior work (Liu et al., 2025).

In our controlled synthetic experiment (Table 1), the GRPO variants achieve 23.97–27.63%, compared with 29.83–31.77% for the corresponding Dr.GRPO variants using the same scalar reward reductions. Range-GRPO further improves accuracy to 39.40%. These analyses highlight the distinction between group-level normalization and uncertainty-aware pairwise credit assignment.

References

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-Like Training: A Critical Perspective. In Conference on Language Modeling (COLM), 2025.

To summarize, we conducted controlled comparisons of advantage constructions across three seeds and evaluated reward interval reliability under distribution shift. We also examined matched hyperparameter settings and clarified how increasing the rollout group size differs from uncertainty-aware pairwise credit assignment. We thank the reviewer again for the constructive feedback, which helped strengthen the empirical evaluation and clarify the contributions of Range-GRPO.

Thank you again for your constructive feedback. We would appreciate it if you could review our responses and additional analyses and let us know whether they address your concerns.

Reviewer_5ncaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Range-GRPO, a semi-supervised reinforcement learning framework that addresses the uncertainty of LLM-generated pseudo-rewards. Instead of relying on scalar reward estimates, the method constructs conformally calibrated reward intervals and introduces a pairwise comparison mechanism to compute group-relative advantages. By incorporating the uncertainty of both responses in each comparison, Range-GRPO can adjust not only the magnitude but also the direction of policy updates.

Critical strengths
  1. Clear and Well-Motivated Problem. The paper addresses an important limitation of conventional GRPO: scalar pseudo-rewards from LLM judges may be unreliable, potentially introducing misleading optimization signals.

  2. Interesting Methodological Contribution. The proposed pairwise interval-based advantage formulation incorporates reward uncertainty directly into credit assignment, allowing uncertainty to influence both the magnitude and direction of policy updates rather than merely rescaling existing advantages.

Critical weaknesses
  1. Insufficient Validation of Judge Reliability and Reward Interval Quality. The proposed framework relies heavily on reward intervals derived from judgment features extracted by Qwen3-4B-Instruct. However, the reliability of these judgment features is not sufficiently established. In particular, the judge evaluates answer correctness and reasoning quality without access to reference answers, making its assessments susceptible to reasoning errors and domain-specific biases.

  2. Additional Computational Overhead and Incomplete Efficiency Evaluation. Range-GRPO requires an additional Qwen3-4B-Instruct judge to extract judgment features during reinforcement learning, introducing extra model inference, memory requirements, and system complexity. A more comprehensive efficiency analysis, including GPU-hours, peak memory usage, and compute-matched comparisons, would be valuable for determining whether the performance improvements justify the additional computational overhead.

Questions for authors

See Weakness

5 comments

We thank the reviewer for highlighting the importance of validating the reliability of judgment features. We address each question below.

W1. Reliability of judgment features and reward intervals

Range-GRPO does not directly use individual judge assessments as rewards. Instead, it learns a reward scorer from labeled source responses and calibrates reward intervals using a separate labeled calibration set. We examine whether judgment states remain informative about response correctness and whether calibrated reward intervals reliably cover the ground-truth rewards.

W1-1. Reliability of judgment states

To examine whether the judgment states remain informative without reference answers, we compared correctness within the same judgment state ZZ between the fixed ID calibration set (source) and the unlabeled OOD responses generated during Range-GRPO training (target). Target correctness labels were used solely for this evaluation. We report the states that are among the three most frequent in either domain, which together cover 50.3% of target responses.

Table R1. Judgment states distinguish correct and incorrect responses across source and target domains. Conditional accuracy (%) is reported for each judgment state ZZ. Source (ID) accuracy is computed on calibration responses, while target (OOD) accuracy is computed on policy-training responses across the early (steps 121–145), middle (146–170), and late (171–195) stages of active training.

ZZSourceTarget (early)Target (middle)Target (late)
(1,1,1,1,1)93.5581.9486.1288.95
(1,0.5,1,1,1)78.1261.8345.9749.41
(0,0,0,0,0)7.948.426.105.35
(0,0.5,0,0,0)5.176.494.586.59

In both domains and at every training stage, the highest-scoring state predominantly identifies correct responses, the two low-scoring states predominantly identify incorrect responses, and the intermediate state remains between them. Although conditional accuracy differs across domains for some states, the judgment states remain informative about response correctness even without reference answers. We also added an analysis of judgment-state reliability over the entire active training phase in Appendix E.2 (Figure 3).

W1-2. Reward interval quality

We then evaluated the coverage of calibrated reward intervals against ground-truth correctness labels (α=0.1\alpha = 0.1). Across OOD policy-training responses, the intervals achieve 87.42% coverage with a mean width of 0.6423, close to the nominal 90%. Shift-aware calibration improves empirical coverage, which drops to 80.72% without DRE. These analyses of target-domain coverage, interval width, and the effects of DRE and weight clipping are included in Appendix E.2 (Table 19) of the revised manuscript.

Together, these analyses provide empirical support for the reliability of judgment states and the quality of calibrated reward intervals for policy training. We also added Theorems 1 and 2 in Appendix B.1, which establish nominal coverage under ideal conditions and a coverage lower bound accounting for conditional reward shift and density-ratio weight discrepancies, respectively.

W2. Computational overhead and efficiency

Appendix D.4 reports the response budget (Table 11) and wall-clock training time (Table 17) of the main experiments. All methods are trained for the same number of epochs with the same group size (G = 8). Below, we summarize these results.

W2-1. Total response budget including LaaJ evaluation

Table R2. Range-GRPO uses the smallest total response budget, including LaaJ evaluation. This table summarizes Table 11 by category. For UARM, auxiliary operations cover reward-model training, calibration, and processing. For Range-GRPO, they cover scorer fitting, calibration, density-ratio estimation, and LaaJ evaluation.

CategoryTraPOUARMRange-GRPO
Policy rollouts (labeled + unlabeled)245,760133,120129,280
Auxiliary operations0112,64071,744
Total245,760245,760201,024

Range-GRPO uses 18.2% fewer response units than TraPO and UARM even with the LaaJ evaluation included. As reported in Table 2, Range-GRPO also achieves the highest ID and OOD average accuracies among the evaluated semi-supervised methods.

W2-2. Wall-clock training time including LaaJ evaluation

Table R3. Range-GRPO maintains comparable wall-clock training time despite the additional LaaJ phase. Each run consists of 10 warmup epochs followed by 5 active epochs. Phase durations and total training times are independently rounded to one decimal place.

MethodActive GPUsWarmup (h)Active (h)Training (h)
TraPO (entropy=0.01)412.810.623.4
TraPO (entropy=0.001)412.67.019.7
UARM56.59.916.4
Range-GRPO (ours)46.310.016.3

Range-GRPO completes training in 16.3 hours, compared with 19.7 hours for TraPO and 19.0 hours for UARM considering its additional reward-model preparation. We further analyzed the overhead of Range-GRPO using training logs. LaaJ execution takes 0.9 hours, while generating source, calibration, and DRE target responses takes 2.3 hours. Together, these stages account for 3.2 hours (19.6%) of the total training time. Range-GRPO shares the same four GPUs between policy training and LaaJ evaluation, without requiring a dedicated GPU for the judge.

Overall, Range-GRPO achieves higher ID and OOD average accuracies than both baselines with a smaller response budget and shorter observed wall-clock time, despite the additional reward-range construction. These results support the practical efficiency of the complete Range-GRPO framework.

To sum up, we have incorporated empirical analyses of judgment-state reliability and calibrated reward-interval quality into the revised manuscript. We also provided a more detailed analysis of the computational efficiency of Range-GRPO. We thank the reviewer again for the constructive feedback, which helped strengthen the reliability and efficiency evaluations of our framework.

Reviewer_PucLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies semi-supervised policy optimization using an LLM judge to provide learning signals for unlabeled responses. It constructs reward intervals through low-dimensional judgment states, a reward scorer, and weighted conformal calibration.

Range-GRPO computes pairwise differences between interval midpoints, downweights each comparison according to the combined uncertainty of both responses, and aggregates the weighted comparisons into response-level advantages. Because weighting occurs before aggregation, interval widths can change the sign of the resulting advantage rather than merely rescaling it.

The paper shows that the advantage reduces to Dr.GRPO when intervals have zero width and demonstrates conditions under which interval widths cause sign reversals. Evaluation includes a synthetic task and mathematical and cross-domain reasoning benchmarks. The real-task system additionally incorporates judge-based feature extraction, a fixed scorer, dynamic density-ratio estimation, and calibration.

Critical strengths

1.The optimization rule is simple and meaningfully different from positive rescaling of an already computed advantage. Pairwise weighting changes the relative contributions of individual comparisons and can therefore change the sign of the aggregate advantage. This capability is not provided by multiplying a completed advantage by a positive response-level coefficient. The method is easy to understand and implement, and its additional advantage computation is modest.

2.The synthetic experiment is targeted, and the real-task results are promising. The synthetic experiment shares the interval-construction pipeline across methods, compares several scalar reductions and advantage rules, and reports results across three seeds. In the main experiments, Range-GRPO reaches ID and OOD average accuracies of 42.7 and 56.6, compared with 40.6 and 52.9 for the reproduced TraPO baseline. The runtime measurements also establish that the complete pipeline is practically executable.

Critical weaknesses
  1. The real-task experiments do not isolate the contribution of the proposed pairwise weighting rule. Range-GRPO uses a Qwen3 judge, five-dimensional judgment states, an XGBoost scorer, dynamic density-ratio estimation, and weighted calibration. The main UARM baseline instead uses a quantile head on a frozen backbone and a different calibration procedure. The systems therefore differ in substantially more than advantage construction.

    The decisive comparison would hold the judge, scorer, intervals, data partitions, and training configuration fixed while changing only the advantage rule among midpoint Dr.GRPO, response-level weighting, and Range-GRPO. The paper performs a similar controlled comparison on the synthetic task but not on the real tasks. Furthermore, the synthetic results have substantial variability: Range-GRPO reports a mean accuracy of 39.40% with a standard deviation of 7.70, while midpoint Dr.GRPO reports 31.77% with a standard deviation of 14.11. The main results do not report uncertainty across training seeds. The evidence therefore supports the complete system more strongly than it supports the specific optimization contribution emphasized in the title.

  2. The connection between interval coverage and reliable advantage directions remains insufficiently validated. The calibration procedure assumes that the relationship between judgment states and correctness remains approximately stable across source and target distributions. This is a consequential assumption when moving from mathematics to MMLU-Pro and then updating the policy. Density-ratio estimation can adjust changes in the distribution of judgment states, but it cannot automatically correct changes in correctness conditional on those states. Classical weighted conformal guarantees depend on appropriate distributional assumptions and do not automatically cover all approximations and adaptive effects in this pipeline. Tibshirani et al., 2019

    The implementation also uses estimated density ratios, percentile-based weight clipping, and a repeatedly reused calibration set, yet the real-task experiments do not report coverage throughout training. More fundamentally, even valid interval coverage does not establish that midpoint comparisons or the resulting weighted advantage signs are correct. Proposition 2 proves that signs can change, while Table 3 provides only one favorable example. Neither establishes that these changes improve alignment with oracle learning signals overall.

Questions for authors

1.Critical ablation request: On the real tasks, how much improvement remains when Range-GRPO, midpoint Dr.GRPO, and response-level weighting share exactly the same judge, scorer, calibrated intervals, and training settings? Are the differences stable across training seeds?

2.Calibration validation: Using withheld correctness labels strictly for evaluation, can you report target-domain coverage, mean interval width, and changes in correctness conditional on judgment states throughout training? How are these quantities affected by weight clipping and removing density-ratio estimation?

3.Mechanism validation: How frequently do advantage-sign reversals occur across all responses? Among those reversals, how often does the change move the signal closer to the oracle reward advantage, and how often does it move it farther away? This aggregate evidence would be more decisive than a single positive example.

6 comments

We sincerely thank the reviewer for the careful and constructive review. We address each question below.

W1 & Q1. Controlled comparison of advantage constructions on the real tasks

Following the reviewer's suggestion, we conducted a controlled comparison of three advantage constructions on the real tasks. All variants start from the same labeled-warmup checkpoint and share the same judge, scorer, interval-construction procedure, data partitions, and training configuration. The results are reported in Appendix E.1 (Table 18).

As described in the paper, these methods differ in whether and how they incorporate interval uncertainty into advantage computation. Midpoint Dr.GRPO computes advantages using interval midpoints as scalar rewards. Response-level weighting first computes the same midpoint-based advantages and then adjusts their magnitudes using the UARM uncertainty weighting scheme. In contrast, Range-GRPO incorporates uncertainty into individual pairwise comparisons before aggregating them into advantages.

Table R1. Range-GRPO achieves the highest mean ID, OOD, and overall accuracies under matched experimental conditions. Results are reported as mean ± sample standard deviation across three active-phase training seeds. For each seed, we report the checkpoint with the highest validation accuracy. Overall is the mean of the ID and OOD averages.

Advantage constructionID Avg.OOD Avg.Overall
Midpoint Dr.GRPO42.0 ± 1.1356.0 ± 0.9049.0 ± 0.22
Response-level weighting (UARM-style)41.9 ± 0.5454.7 ± 2.7348.3 ± 1.64
Range-GRPO42.2 ± 0.7156.1 ± 0.6749.1 ± 0.45

This controlled comparison isolates the effect of advantage construction under matched experimental conditions. Across three seeds, Range-GRPO achieves the highest mean ID, OOD, and overall accuracies. Range-GRPO shows gains over UARM-style weighting, with lower variability in OOD and overall performance, and modest improvements over midpoint Dr.GRPO. These results provide additional empirical support for pairwise interval weighting on real reasoning tasks.

We further examined robustness to the reference width λ∈{0.5,1,2}\lambda \in \{0.5, 1, 2\} and rollout sampling temperature. Under these settings, Range-GRPO still outperforms both the reproduced TraPO and UARM baselines in ID, OOD, and overall average accuracy. These additional analyses are provided in Appendix E.3 (Table 20) and Appendix E.4 (Table 22).

W1. Seed-level variability in the synthetic experiment

Table R2 reports per-seed test accuracy in the synthetic experiment. Midpoint Dr.GRPO reaches 48.0% in Seed 6202, compared with 22.4% and 24.9% in the other seeds, contributing to its larger standard deviation.

Table R2. Range-GRPO ranks first in two seeds and second in the remaining seed. Test accuracy (%) per training seed in the synthetic experiment. Bold and underlined values indicate the best and second-best results in each column, respectively.

MethodSeed 6200Seed 6201Seed 6202Mean ± Std
Range-GRPO47.031.639.6‾\underline{39.6}39.40 ± 7.70
Midpoint Dr.GRPO22.424.948.031.77±14.11‾\underline{31.77 \pm 14.11}
LB Dr.GRPO27.124.839.130.33 ± 7.68
UB Dr.GRPO33.7‾\underline{33.7}31.4‾\underline{31.4}24.429.83 ± 4.84
LB GRPO30.826.825.327.63 ± 2.84
UB GRPO27.723.422.724.60 ± 2.71
Midpoint GRPO17.422.232.323.97 ± 7.61
UARM-style16.523.621.220.43 ± 3.61

All methods use the same interval construction procedure and achieve mean empirical coverage above the nominal 90%90\%, as shown in Table 1. Across seeds, different policy rollouts produce different reward intervals and affect interval-dependent learning signals. These differences influence subsequent policy updates, leading to seed-dependent training trajectories. Thus, the observed variability does not necessarily indicate unreliable interval calibration.

Despite this variability, Range-GRPO ranks first or second in every seed and achieves the highest mean accuracy, supporting the robustness of its relative performance across the evaluated seeds. We include this analysis in Appendix C.4 (Table 9).

W2 & Q2. Calibration of reward intervals on the target domain throughout training

Following the reviewer's suggestion, we evaluated calibration on all unlabeled OOD responses generated during policy-training in the reported Range-GRPO run. Target correctness labels were used solely for this evaluation and played no role in policy training, judging, scorer fitting, or DRE. In summary, DRE improves empirical target coverage throughout training, while judgment states continue to distinguish predominantly correct from incorrect responses across domains. Detailed results are provided in Appendix E.2 (Table 19 and Figure 3).

Q2-1. Coverage and interval width

Table R3 reports target coverage and mean interval width (α=0.1\alpha = 0.1) over three stages of unlabeled training.

Table R3. DRE keeps target coverage close to the nominal level throughout training. Each cell reports target coverage (%) / mean interval width. Values in parentheses denote coverage gains over No DRE in percentage points, computed from unrounded values. Both ablations recompute the intervals for the same responses, changing only the calibration weights.

Training stageDRE + clipping (ours)DRE, no clippingNo DRE
Early (steps 121–145)88.63 (+3.98) / 0.612988.50 / 0.616284.65 / 0.5597
Middle (steps 146–170)87.30 (+5.83) / 0.600887.03 / 0.599981.47 / 0.5270
Late (steps 171–195)86.30 (+10.34) / 0.713786.46 / 0.729375.97 / 0.5544
Full trajectory87.42 (+6.70) / 0.642387.34 / 0.648380.72 / 0.5471

With DRE, coverage remains close to the nominal 90% throughout training. DRE improves overall coverage by 6.70 percentage points over the No DRE setting, with larger gains in later stages. Weight clipping limits extreme calibration weights, yielding slightly narrower intervals overall while maintaining comparable empirical coverage.

Q2-2. Correctness conditional on judgment states

Table R4 compares correctness within the same judgment state ZZ between the fixed calibration set (source) and the target responses over training. We report the states that are among the three most frequent in either domain, which together cover 50.3% of target responses.

Table R4. Judgment states distinguish correct and incorrect responses across source and target domains. Conditional accuracy (%) is reported for each judgment state ZZ. Source (ID) accuracy is computed on calibration responses, while target (OOD) accuracy is computed on policy-training responses across the early (steps 121–145), middle (146–170), and late (171–195) stages of active training.

ZZSourceTarget (early)Target (middle)Target (late)
(1,1,1,1,1)93.5581.9486.1288.95
(1,0.5,1,1,1)78.1261.8345.9749.41
(0,0,0,0,0)7.948.426.105.35
(0,0.5,0,0,0)5.176.494.586.59

The judgment states preserve the separation between correct and incorrect responses across domains and throughout training. Z=(1,1,1,1,1)Z=(1,1,1,1,1), the most frequent state, identifies mostly correct responses in both source (93.55%) and target (81.94–88.95%), and its target correctness increases during training, narrowing the gap to the source rate from 11.6 to 4.6 points. We acknowledge that conditional correctness rates may differ between the source and target domains for some judgment states. To address this limitation, we added Theorems 1 and 2 in Appendix B.1. Theorem 1 establishes nominal coverage under ideal conditions, while Theorem 2 provides a coverage lower bound accounting for conditional reward shift and density-ratio weight discrepancies under its stated assumptions. We reuse a fixed calibration set throughout training as a practical design choice. Despite this choice, empirical target coverage remains close to the nominal 90% level, as shown in Table R3.

These results provide empirical support for the practical effectiveness of judgment-state calibration and DRE under limited labeled supervision.

Comments on the paper

1 comment

We sincerely thank Reviewers PucL, 5nca, WAxQ, and ttJB for their thoughtful and constructive feedback. We have done our utmost to address the reviewers’ suggestions and strengthen the empirical and theoretical support for Range-GRPO. Below, we summarize the strengths highlighted by the reviewers and the main additional results. Detailed responses are provided in the individual comments.

1. Strengths Highlighted by the Reviewers

  • [PucL, 5nca, WAxQ, ttJB] Simple and meaningful uncertainty-aware credit assignment. The reviewers recognize that weighting individual pairwise comparisons incorporates the uncertainty of both responses into advantage construction. Because weighting occurs before aggregation, interval widths can change advantage signs, providing a capability beyond positive response-level advantage rescaling.

  • [PucL, WAxQ, ttJB] Promising synthetic and real-task results. Range-GRPO achieves ID/OOD average accuracies of 42.7%/56.6%, compared with 40.6%/52.9% for reproduced TraPO, and achieves the best semi-supervised results on eight of nine benchmarks.

  • [WAxQ, ttJB] Practical efficiency with improved accuracy. The reported results show that Range-GRPO achieves higher average accuracy with an 18.2% smaller total response budget and comparable or shorter training time than the reproduced baselines, even with judge evaluation and reward range construction included.

2. Summary of New Results and Revisions

  • [PucL, WAxQ, ttJB] Controlled comparisons of advantage constructions across three training seeds. We compare midpoint Dr.GRPO, UARM-style response-level weighting, and Range-GRPO using the same warmup checkpoint, judge, scorer, interval-construction procedure, data partitions, and training configuration. Across three active-phase training seeds, Range-GRPO achieves the highest mean ID, OOD, and overall accuracies, with an overall accuracy of 49.1 ± 0.45%. These averages are close to the Table 2 results, supporting robustness across seeds. See Appendix E.1, Table 18.

  • [WAxQ] Additional comparison under matched optimization settings. We evaluate Range-GRPO using the same learning rate (1×10−61\times10^{-6}) and entropy coefficient (0.01) as the compared TraPO and UARM configurations. Range-GRPO achieves an OOD average of 53.6%, compared with 51.7% for TraPO and 50.0% for UARM. It also achieves the highest overall average while ranking second on ID. See our response to WAxQ (W1).

  • [PucL, 5nca, WAxQ, ttJB] Validation of judgment states and reward intervals throughout training. Using correctness labels solely for post-hoc evaluation, we analyze coverage, interval width, and judgment-state reliability on unlabeled OOD policy-training responses. Density-ratio estimation improves overall empirical coverage from 80.72% to 87.42%, toward the nominal 90% level. These analyses provide empirical evidence about interval quality under the distribution shifts encountered during training. See Appendix E.2, Table 19 and Figure 3.

  • [PucL, ttJB] Additional theoretical coverage analysis and statistical motivation for the pairwise weight. We add a per-iteration coverage analysis that establishes nominal coverage under ideal conditions and a coverage lower bound accounting for conditional reward shift and density-ratio weight discrepancies under the stated assumptions. We also derive the proposed pairwise weight through conditional mean squared error minimization under an auxiliary estimation model. See Appendix B.

  • [PucL] Learning signal alignment with oracle advantages. We compare advantage constructions on the same recorded responses and reward intervals. When Range-GRPO and midpoint Dr.GRPO assign opposite advantage signs, Range-GRPO produces signals closer to the oracle in 61.5% of synthetic cases and 58.3% of main-experiment cases. Across all analyzed main-experiment responses, Range-GRPO also achieves a lower MAE relative to oracle advantages: 0.1725, compared with 0.1829 for midpoint Dr.GRPO and 0.1886 for UARM-style weighting. See Appendix E.3, Table 21 and the accompanying analysis.

  • [PucL, ttJB] Robustness to reference width λ\lambda and sampling temperature. We separately examine the reference width λ∈{0.5,1,2}\lambda \in \{0.5, 1, 2\} and rollout sampling temperatures of 1.0 and 1.5. Range-GRPO retains higher ID and OOD average accuracies than the reported TraPO and UARM baselines across these tested settings. We also analyze how varying λ\lambda affects advantage signs on fixed logged responses. See Appendix E.3, Table 20 and the accompanying analysis, and Appendix E.4, Table 22.

We thank the reviewers again for their constructive feedback, which helped us improve the experimental evaluation, clarify the theoretical foundations, and better explain the scope of the evidence for Range-GRPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.