- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates policy optimization with black-box LLM-as-a-judge rewards in RLAIF when the evaluator outputs multi-tier discrete scores (such as 1–10 rubric ratings). To motivate the problem, Section 2 (along with Appendix A) shows that querying the same auto-rater repeatedly on a fixed set of prompt-response pairs yields surprisingly low ranking concordance—median Kendall's stays around across several policy, judge, and dataset combinations—pointing to substantial reward stochasticity that can distort group-relative advantage estimates.
To mitigate this sensitivity, the authors propose ODRPO (Section 3), which decomposes a discrete scalar reward into nested ordinal binary indicators , normalizes each threshold bin independently within the rollout group, and sums the resulting threshold-level advantages. ODRPO is designed as a lightweight, plug-and-play wrapper around existing group-based estimators (GRPO, MaxRL, and OTB), with an optional Gini-weighted variant (ODRPO-Gini, Section 3.2) that upweights thresholds with higher within-group variance.
Empirically, the method is evaluated on UltraFeedback, WildChat, and a 9-tier GSM8K setup across multiple policy models (Qwen2.5-7B-Instruct, Qwen3-4B-Instruct, OLMo-2-7B-Instruct) and judges (Qwen3-30B-A3B-Instruct, GPT-OSS-120B). Across seven downstream benchmarks in Table 1, ODRPO and ODRPO-Gini yield positive mean relative gains of over vanilla GRPO and MaxRL (and up to with OTB in Appendix Table 4) with negligible overhead in wall-clock time per step. Section 5 and Appendix C complement the experiments with a vector-field analysis arguing that multi-tier GRPO/MaxRL updates violate the curl-free condition required for a global scalar objective, whereas summing binary ordinal decompositions preserves integrability. Overall, the motivation is practical and the method is appealingly simple, though several missing baselines, unspecified edge cases in the algorithm, and gaps in both statistical reporting and the theoretical treatment of Gini weighting hold back the current version.
Critical strengths
1. Convincing empirical diagnosis of auto-rater ranking instability in group-based RLAIF.
Section 2 and Appendix A make a compelling practical point: when an LLM judge scores responses on a 1–10 scale, repeated evaluations of the exact same completions show poor rank agreement (Kendall's medians between and ). Because estimators like GRPO and MaxRL normalize rewards across a small rollout group (), a single outlier rubric score shifts the group mean and standard deviation, corrupting the relative advantage of every response in that group. Grounding the method in this specific failure mode makes the paper immediately relevant to practitioners running RLAIF pipelines.
2. A dead-simple, zero-overhead wrapper that shows consistent gains across diverse setups.
ODRPO operates purely on the scalar integer outputs of a black-box judge—requiring no access to token logits, internal confidence scores, or multiple judge calls (and in Table 2, 1-sample ODRPO-Gini even outperforms 32-sample majority-voted MaxRL on AlpacaEval at less than half the step time). Across two base policies, three estimators (GRPO, MaxRL, OTB), two judges, and seven evaluation benchmarks (Tables 1 and 4), adding ODRPO consistently improves mean relative performance without slowing down training.
Critical weaknesses
1. Lack of training-seed uncertainty or statistical significance for the main benchmark claims.
While Figure 2 reports evaluation variance across 5 seeds on the smaller GSM8K setup, the central results in Table 1 and Appendix Table 4 appear to come from single training runs (only Time per Step includes a standard deviation). Several individual benchmark deltas are quite modest—for example, on Qwen3-4B-Instruct with MaxRL, IFEval moves from to () for ODRPO and () for ODRPO-Gini, and HellaSwag gains are around under GRPO. Given the inherent variance of RL post-training and LLM-based evaluation (such as AlpacaEval), single-run point estimates without training-seed standard deviations or bootstrap confidence intervals make it difficult to tell which per-benchmark gains are statistically distinguishable from noise. This directly affects my rating because the paper's headline contribution rests on reliable empirical improvements across benchmarks.
2. Missing natural reward-shaping baselines and underspecified algorithmic/theoretical details.
To isolate why ODRPO works, the experiments only compare against raw GRPO/MaxRL/OTB (plus a batch-normalization ablation in Appendix C.4.1 / Table 4). If the core issue is sensitivity to cardinal outliers in scores, a practitioner would naturally try simple within-group rank or percentile advantages, winsorized/clipped z-scores, or ordinal decomposition without separate per-threshold variance normalization (which reduces to a linear rescaling unless per-bin is doing the heavy lifting). Without these baselines, it is unclear whether the gains come from the cumulative threshold structure itself or simply from bounding outlier leverage. Compounding this, the paper leaves out how threshold divisions are resolved when (which happens constantly when and ), and glosses over how sample-dependent Gini weights affect the binomial expectations in the Appendix C.4 objective derivation.
Questions for authors
-
Clarification (Degenerate threshold bins): With rollouts and reward levels, any threshold lying below (including always) has , and any threshold above has . In both cases, GRPO's bin normalizer is zero (and MaxRL's normalizer is zero when ). How are these bins handled in your code—are constant thresholds masked out (), or do you add an in the denominator? Also, on average, how many of the thresholds are non-constant per prompt group during UltraFeedback training?
-
Clarification (Theory for ODRPO-Gini): In Appendix C.4 (p. 27), you state that for Gini weighting "the objective function derivation remains the same as above just with different values." However, depends on the realized sample mean of the group rather than being a fixed constant. While a 1D function of the suffix probability remains integrable (curl-free), doesn't multiplying by inside the expectation alter and in Eqs. (48)–(50) rather than simply rescaling in the objective of Eq. (2)? Please clarify the exact scalar objective for ODRPO-Gini, and also clarify whether the finite-sample curl analysis assumes leave-one-out statistics (as in Appendix C.3) or full-group statistics (as written in Section 3.1).
-
Assessment-critical experiment (Statistical reliability & simple baselines): Can you report multi-seed training variance (e.g., independent RL runs) or paired bootstrap confidence intervals for the core Table 1 configurations, and compare ODRPO against a simple outlier-robust group baseline such as within-group rank/percentile normalization?
5 comments
Rebuttal (Multi-Response)
Leave-One-Out vs. Full-Group Statistics
We use leave-one-out counts in the derivation, as in Davis and Recht (https://arxiv.org/pdf/2510.13651), but this does not mean that the current response is excluded from the group statistics. We condition on the other responses and then include the current response when computing the mean and normalization, as shown in equations 38–39. So, the statistics are computed over the full group, consistent with Section 3.1. For Figure 5, we used equation 35, which is the finite-group curl condition.
The finite-group expression for binary-reward GRPO is provided on page 8 of Davis and Recht’s work, where they define as group size. To get the finite-group expression for discrete rewards (and continuous rewards by virtue of integration), plug the finite-group binary-reward expressions from Davis and Recht’s paper in Eq. 1 with . Equation (2) uses the large-group expression, which we will state explicitly in the revision. We omitted the finite-group expressions for brevity, but we will include them in the revision for better clarity.
Training-Seed Uncertainty
We used a single training seed for each experiment and we acknowledge that the evaluation seeds do not capture variability across independent training runs. Because of compute constraints, we are unable to run multiple training seeds for each experiment. However, we conducted experiments across different policies, auto-raters, datasets, estimators, ablations, and majority-voting settings to study the impact of ODRPO. On the out-of-distribution evaluation benchmarks, ODRPO and ODRPO-Gini provided mean relative improvements ranging from 3.33% to 5.39% in Table 1 and 2.29% to 6.96% in Table 4, and relative improvements ranging from 5.89% to 9.00% in Table 2. Some of the modest individual gains highlighted in the reviews are from these OOD evaluations, which measure transfer from training on UltraFeedback or WildChat. On the in-distribution GSM8K test set, the improvements are substantially larger, with mean relative improvements across the nine ablation settings of 19.01% for ODRPO and 20.20% for ODRPO-Gini. We believe these results provide encouraging evidence of ODRPO's promise.
We thank the reviewer again for their time and constructive feedback. We will revise our work accordingly.
We would like to kindly remind the reviewer that the deadline for the author-reviewer discussion period is approaching. If you have any further questions or comments, please let us know so we can address them before the deadline.
Rebuttal (Multi-Response)
Thank you for the review. Please find our responses to your questions. Note that the rebuttal is multi-response.
Missing Reward-Shaping Baselines
We also ran some more baselines on GSM8K across the nine temperature and granularity settings in Fig. 2. We ran GOPO (https://arxiv.org/pdf/2602.03876), Clipped Z-Score, Squared Reward Shaping (r→r2), and Square-Root Reward Shaping (r→r). For Clipped Z-Score, we use c=1 and apply clip(A,−c,c) to the GRPO advantages. For squared and square-root rewards, we first apply the reward transformation and then compute GRPO advantages from the transformed rewards. The table reports the mean relative improvement over GRPO across the nine settings.
All provide improvement over GRPO, but ODRPO variants take the lead in mean relative improvement. This indicates that ordinal decomposition improves over the tested simpler reward-shaping and rank-based methods in this setting. We would also like to reinforce that ODRPO is an augmentation to current group-based estimators, so rather than being a separate method on its own, it is a principled approach to discrete and continuous reward settings.
Additionally, Dr. GRPO (https://arxiv.org/pdf/2503.20783) removes the group standard-deviation normalization that you suggest. Applying ODRPO to Dr. GRPO as the base estimator, i.e. removing per-bin normalization with uniform weighting, recovers the same Dr. GRPO advantage estimator. Because of this equivalence, we didn’t consider Dr. GRPO as the base estimator.
Degenerate Bins and Non-Constant Thresholds
For numerical stability, we have +ϵ in the denominator. You are correct that the bins at or below rmin and above rmax will have zero advantage values, thus contributing nothing to the aggregation (even in the weighted case). We do not have to explicitly mask them out because they are zero by virtue of the formulation. We kept K=5,10,20 and applied weighting to each bin in our implementation for simplicity but we also provided a formula in equation 16 that allows readers to implement ODRPO in an arbitrary reward setting without concerning about different granularity settings. Mathematically, Eq. (16) and the formula provided in section 3.1 are identical in the discrete reward setting we worked on. Regarding the number of non-constant thresholds, we didn’t keep track of the average per prompt and can't infer it from the dataset-wide reward range because it depends on the minimum and maximum scores within each group.
Objective for Sample-Dependent Gini Weights
The scalar-objective guarantee for the expected update in Appendix C.4 also covers empirical Gini weighting at finite group size, before final batch normalization, as it depends only on its own ordinal bin’s empirical success fraction without any cross-bin term.
The Gini weight as a function of the mean q is
wm(q)=m[δ+4q(1−q)].
For finite group size, the weightings are applied to the success and failure functions in Eq. (46) as follows:
tmGini(s)=wm!(Ms+1)t(s),umGini(s)=wm!(Ms)u(s).
The rest of the derivation follows as before because there is no cross-bin dependence. These changes consequently affect the functions α and β in equation 56.
In the large-group limit, the empirical success fraction converges to the suffix probability:
μ(m)⟶Pm
The weighting therefore acts as a probability-dependent change in the reward-spacing coefficient in the expected update:
wm(Pm)=m[δ+4Pm(1−Pm)],Δm(Pm)=Δmwm(Pm).
Using the same overall normalization as Eq. (57), the gradient becomes
∇θJGini(θ)=Ex∼Q[π1∑m=2KPm(1−Pm)Δm(Pm)∇θPm]
Integrating this weighted update gives
JGini(θ)=Ex∼Q![π1∑m=2KΔmm((2δ+1)arcsinPm−(1−2Pm)Pm(1−Pm))]
We will include the Gini derivation for clarity in our revision.