acceptodds
Under review as a conference paper at ICLR 2027

ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization

Abstract

The alignment of Large Language Models (LLMs) increasingly relies on Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains such as long-form question answering and open-ended instruction following. These domains often use LLM-based auto-raters to provide granular, multi-tiered discrete rewards (e.g., – rubrics) that can be stochastic due to prompt sensitivity and sampling randomness. We consider the black-box setting in which the optimizer observes only the returned discrete reward and does not assume access to the auto-rater's parameters, logits, hidden states, or auxiliary confidence estimates. We empirically characterize this stochasticity across multiple auto-rater configurations. Such variability enters standard advantage estimators such as GRPO and MaxRL through group-normalization statistics, allowing a noisy reward sample to alter the update assigned to the entire group. Repeatedly sampling and aggregating rewards may reduce this noise, but doing so substantially increases evaluation cost. To address this bottleneck, we introduce rdinal ecomposition for obust olicy ptimization (), a framework that decomposes discrete rewards into a sequence of ordinal binary indicators. By independently computing and accumulating advantages across progressively challenging success thresholds, ODRPO localizes score variability to the thresholds crossed by a score change while supporting flexible weighting schemes and an implicit ordinal curriculum. Empirically, across GRPO, MaxRL, and OTB, three policy models, and seven evaluation benchmarks, all main-experiment configurations ODRPO and its variance-aware variant achieve positive mean relative gains ranging from to . Ablations over auto-rater temperature and reward granularity further show consistent improvements over GRPO, with finer reward resolution producing the clearest trend. These gains incur no material per-step training overhead. ODRPO further attains the noise robustness sought by repeated auto-rater sampling, surpassing MaxRL with -sample majority voting on AlpacaEval while using a single reward sample. Finally, our theoretical analysis establishes that the ODRPO formulation retains a well-defined global scalar objective in arbitrary discrete reward spaces when built on estimators with valid binary objectives.

Primary area:
reinforcement learning
Keywords:
Ordinal Decomposition, Robust Policy Optimization, Advantage Estimation, RLAIF

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mrnoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates policy optimization with black-box LLM-as-a-judge rewards in RLAIF when the evaluator outputs multi-tier discrete scores (such as 1–10 rubric ratings). To motivate the problem, Section 2 (along with Appendix A) shows that querying the same auto-rater repeatedly on a fixed set of prompt-response pairs yields surprisingly low ranking concordance—median Kendall's WW stays around 0.279–0.4000.279\text{--}0.400 across several policy, judge, and dataset combinations—pointing to substantial reward stochasticity that can distort group-relative advantage estimates.

To mitigate this sensitivity, the authors propose ODRPO (Section 3), which decomposes a discrete scalar reward ri∈{1,…,K}r_i \in \{1, \dots, K\} into nested ordinal binary indicators ri(k)=I{ri≥k}r_i^{(k)} = \mathbb{I}\{r_i \ge k\}, normalizes each threshold bin independently within the rollout group, and sums the resulting threshold-level advantages. ODRPO is designed as a lightweight, plug-and-play wrapper around existing group-based estimators (GRPO, MaxRL, and OTB), with an optional Gini-weighted variant (ODRPO-Gini, Section 3.2) that upweights thresholds with higher within-group variance.

Empirically, the method is evaluated on UltraFeedback, WildChat, and a 9-tier GSM8K setup across multiple policy models (Qwen2.5-7B-Instruct, Qwen3-4B-Instruct, OLMo-2-7B-Instruct) and judges (Qwen3-30B-A3B-Instruct, GPT-OSS-120B). Across seven downstream benchmarks in Table 1, ODRPO and ODRPO-Gini yield positive mean relative gains of 3.33%–5.39%3.33\%\text{--}5.39\% over vanilla GRPO and MaxRL (and up to 6.96%6.96\% with OTB in Appendix Table 4) with negligible overhead in wall-clock time per step. Section 5 and Appendix C complement the experiments with a vector-field analysis arguing that multi-tier GRPO/MaxRL updates violate the curl-free condition required for a global scalar objective, whereas summing binary ordinal decompositions preserves integrability. Overall, the motivation is practical and the method is appealingly simple, though several missing baselines, unspecified edge cases in the algorithm, and gaps in both statistical reporting and the theoretical treatment of Gini weighting hold back the current version.

Critical strengths

1. Convincing empirical diagnosis of auto-rater ranking instability in group-based RLAIF.
Section 2 and Appendix A make a compelling practical point: when an LLM judge scores responses on a 1–10 scale, repeated evaluations of the exact same completions show poor rank agreement (Kendall's WW medians between 0.2790.279 and 0.4000.400). Because estimators like GRPO and MaxRL normalize rewards across a small rollout group (G=5G=5), a single outlier rubric score shifts the group mean and standard deviation, corrupting the relative advantage of every response in that group. Grounding the method in this specific failure mode makes the paper immediately relevant to practitioners running RLAIF pipelines.

2. A dead-simple, zero-overhead wrapper that shows consistent gains across diverse setups.
ODRPO operates purely on the scalar integer outputs of a black-box judge—requiring no access to token logits, internal confidence scores, or multiple judge calls (and in Table 2, 1-sample ODRPO-Gini even outperforms 32-sample majority-voted MaxRL on AlpacaEval at less than half the step time). Across two base policies, three estimators (GRPO, MaxRL, OTB), two judges, and seven evaluation benchmarks (Tables 1 and 4), adding ODRPO consistently improves mean relative performance without slowing down training.

Critical weaknesses

1. Lack of training-seed uncertainty or statistical significance for the main benchmark claims.
While Figure 2 reports evaluation variance across 5 seeds on the smaller GSM8K setup, the central results in Table 1 and Appendix Table 4 appear to come from single training runs (only Time per Step includes a standard deviation). Several individual benchmark deltas are quite modest—for example, on Qwen3-4B-Instruct with MaxRL, IFEval moves from 85.5085.50 to 85.7485.74 (+0.28%+0.28\%) for ODRPO and 86.1186.11 (+0.71%+0.71\%) for ODRPO-Gini, and HellaSwag gains are around 1.3%–1.6%1.3\%\text{--}1.6\% under GRPO. Given the inherent variance of RL post-training and LLM-based evaluation (such as AlpacaEval), single-run point estimates without training-seed standard deviations or bootstrap confidence intervals make it difficult to tell which per-benchmark gains are statistically distinguishable from noise. This directly affects my rating because the paper's headline contribution rests on reliable empirical improvements across benchmarks.

2. Missing natural reward-shaping baselines and underspecified algorithmic/theoretical details.
To isolate why ODRPO works, the experiments only compare against raw GRPO/MaxRL/OTB (plus a batch-normalization ablation in Appendix C.4.1 / Table 4). If the core issue is sensitivity to cardinal outliers in 1–101\text{--}10 scores, a practitioner would naturally try simple within-group rank or percentile advantages, winsorized/clipped z-scores, or ordinal decomposition without separate per-threshold variance normalization (which reduces to a linear rescaling unless per-bin N(k)N^{(k)} is doing the heavy lifting). Without these baselines, it is unclear whether the gains come from the cumulative threshold structure itself or simply from bounding outlier leverage. Compounding this, the paper leaves out how 0/00/0 threshold divisions are resolved when μ(k)∈{0,1}\mu^{(k)} \in \{0, 1\} (which happens constantly when G=5G=5 and K=10K=10), and glosses over how sample-dependent Gini weights wGini(k)(μ(k))w_{\mathrm{Gini}}^{(k)}(\mu^{(k)}) affect the binomial expectations in the Appendix C.4 objective derivation.


Questions for authors
  1. Clarification (Degenerate threshold bins): With G=5G=5 rollouts and K=10K=10 reward levels, any threshold kk lying below min⁡iri\min_i r_i (including k=1k=1 always) has μ(k)=1\mu^{(k)}=1, and any threshold above max⁡iri\max_i r_i has μ(k)=0\mu^{(k)}=0. In both cases, GRPO's bin normalizer N(k)=μ(k)(1−μ(k))N^{(k)} = \sqrt{\mu^{(k)}(1-\mu^{(k)})} is zero (and MaxRL's normalizer is zero when μ(k)=0\mu^{(k)}=0). How are these bins handled in your code—are constant thresholds masked out (Ai(k)=0A_i^{(k)} = 0), or do you add an ϵ\epsilon in the denominator? Also, on average, how many of the KK thresholds are non-constant per prompt group during UltraFeedback training?

  2. Clarification (Theory for ODRPO-Gini): In Appendix C.4 (p. 27), you state that for Gini weighting "the objective function derivation remains the same as above just with different Δm\Delta_m values." However, wGini(m)=m(δ+4μ(m)(1−μ(m)))w_{\mathrm{Gini}}^{(m)} = \sqrt{m}(\delta + 4\mu^{(m)}(1-\mu^{(m)})) depends on the realized sample mean μ(m)\mu^{(m)} of the group rather than being a fixed constant. While a 1D function of the suffix probability PmP_m remains integrable (curl-free), doesn't multiplying by μ(m)(1−μ(m))\mu^{(m)}(1-\mu^{(m)}) inside the expectation alter β(Pm)\beta(P_m) and α(Pm)\alpha(P_m) in Eqs. (48)–(50) rather than simply rescaling Δm\Delta_m in the arcsin⁡(Pm)\arcsin(\sqrt{P_m}) objective of Eq. (2)? Please clarify the exact scalar objective for ODRPO-Gini, and also clarify whether the finite-sample curl analysis assumes leave-one-out statistics (as in Appendix C.3) or full-group statistics (as written in Section 3.1).

  3. Assessment-critical experiment (Statistical reliability & simple baselines): Can you report multi-seed training variance (e.g., ≥3\ge 3 independent RL runs) or paired bootstrap confidence intervals for the core Table 1 configurations, and compare ODRPO against a simple outlier-robust group baseline such as within-group rank/percentile normalization?

5 comments
AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Thank you for the review. Please find our responses to your questions. Note that the rebuttal is multi-response.

Missing Reward-Shaping Baselines

We also ran some more baselines on GSM8K across the nine temperature and granularity settings in Fig. 2. We ran GOPO (https://arxiv.org/pdf/2602.03876), Clipped Z-Score, Squared Reward Shaping (r→r2r \to r^2), and Square-Root Reward Shaping (r→rr \to \sqrt{r}). For Clipped Z-Score, we use c=1c=1 and apply clip⁡(A,−c,c)\operatorname{clip}(A,-c,c) to the GRPO advantages. For squared and square-root rewards, we first apply the reward transformation and then compute GRPO advantages from the transformed rewards. The table reports the mean relative improvement over GRPO across the nine settings.

MethodGSM8K Mean Relative Improvement (%)
GRPO0.00
GOPO2.18
Clipped Z-Score6.70
Squared Reward7.51
Square-Root Reward4.57
ODRPO19.01
ODRPO-Gini20.20

All provide improvement over GRPO, but ODRPO variants take the lead in mean relative improvement. This indicates that ordinal decomposition improves over the tested simpler reward-shaping and rank-based methods in this setting. We would also like to reinforce that ODRPO is an augmentation to current group-based estimators, so rather than being a separate method on its own, it is a principled approach to discrete and continuous reward settings.

Additionally, Dr. GRPO (https://arxiv.org/pdf/2503.20783) removes the group standard-deviation normalization that you suggest. Applying ODRPO to Dr. GRPO as the base estimator, i.e. removing per-bin normalization with uniform weighting, recovers the same Dr. GRPO advantage estimator. Because of this equivalence, we didn’t consider Dr. GRPO as the base estimator.

Degenerate Bins and Non-Constant Thresholds

For numerical stability, we have +ϵ+\epsilon in the denominator. You are correct that the bins at or below rmin⁡r_{\min} and above rmax⁡r_{\max} will have zero advantage values, thus contributing nothing to the aggregation (even in the weighted case). We do not have to explicitly mask them out because they are zero by virtue of the formulation. We kept K=5,10,20K=5,10,20 and applied weighting to each bin in our implementation for simplicity but we also provided a formula in equation 16 that allows readers to implement ODRPO in an arbitrary reward setting without concerning about different granularity settings. Mathematically, Eq. (16) and the formula provided in section 3.1 are identical in the discrete reward setting we worked on. Regarding the number of non-constant thresholds, we didn’t keep track of the average per prompt and can't infer it from the dataset-wide reward range because it depends on the minimum and maximum scores within each group.

Objective for Sample-Dependent Gini Weights

The scalar-objective guarantee for the expected update in Appendix C.4 also covers empirical Gini weighting at finite group size, before final batch normalization, as it depends only on its own ordinal bin’s empirical success fraction without any cross-bin term.

The Gini weight as a function of the mean qq is

wm(q)=m[δ+4q(1−q)].w_m(q)=\sqrt m\left[\delta+4q(1-q)\right].

For finite group size, the weightings are applied to the success and failure functions in Eq. (46) as follows:

tmGini(s)=wm!(s+1M)t(s),umGini(s)=wm!(sM)u(s).t_m^{\mathrm{Gini}}(s) = w_m\\!\left(\frac{s+1}{M}\right)t(s), \qquad u_m^{\mathrm{Gini}}(s) = w_m\\!\left(\frac{s}{M}\right)u(s).

The rest of the derivation follows as before because there is no cross-bin dependence. These changes consequently affect the functions α\alpha and β\beta in equation 56.

In the large-group limit, the empirical success fraction converges to the suffix probability:

μ(m)⟶Pm\mu^{(m)} \longrightarrow P_m

The weighting therefore acts as a probability-dependent change in the reward-spacing coefficient in the expected update:

wm(Pm)=m[δ+4Pm(1−Pm)],Δ~m(Pm)=Δmwm(Pm).w_m(P_m)=\sqrt m\left[\delta+4P_m(1-P_m)\right], \qquad \widetilde{\Delta}_m(P_m)=\Delta_m w_m(P_m).

Using the same overall normalization as Eq. (57), the gradient becomes

∇θJGini(θ)=Ex∼Q[1π∑m=2KΔ~m(Pm)Pm(1−Pm)∇θPm]\nabla_{\theta} J_{\mathrm{Gini}}(\theta) = \mathbb{E}_{x\sim Q}\left[ \frac{1}{\pi}\sum_{m=2}^{K} \frac{\widetilde{\Delta}_{m}(P_m)}{\sqrt{P_m(1-P_m)}}\nabla_{\theta} P_m \right]

Integrating this weighted update gives

JGini(θ)=Ex∼Q![1π∑m=2KΔmm((2δ+1)arcsin⁡Pm−(1−2Pm)Pm(1−Pm))]J_{\mathrm{Gini}}(\theta) = \mathbb E_{x\sim Q}\\!\left[ \frac1\pi\sum_{m=2}^K\Delta_m\sqrt m \left( (2\delta+1)\arcsin\sqrt{P_m} -(1-2P_m)\sqrt{P_m(1-P_m)} \right) \right]

We will include the Gini derivation for clarity in our revision.

AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Leave-One-Out vs. Full-Group Statistics

We use leave-one-out counts in the derivation, as in Davis and Recht (https://arxiv.org/pdf/2510.13651), but this does not mean that the current response is excluded from the group statistics. We condition on the other responses and then include the current response when computing the mean and normalization, as shown in equations 38–39. So, the statistics are computed over the full group, consistent with Section 3.1. For Figure 5, we used equation 35, which is the finite-group curl condition.

The finite-group expression for binary-reward GRPO is provided on page 8 of Davis and Recht’s work, where they define MM as group size. To get the finite-group expression for discrete rewards (and continuous rewards by virtue of integration), plug the finite-group binary-reward expressions from Davis and Recht’s paper in Eq. 1 with t=Pmt=P_m. Equation (2) uses the large-group expression, which we will state explicitly in the revision. We omitted the finite-group expressions for brevity, but we will include them in the revision for better clarity.

Training-Seed Uncertainty

We used a single training seed for each experiment and we acknowledge that the evaluation seeds do not capture variability across independent training runs. Because of compute constraints, we are unable to run multiple training seeds for each experiment. However, we conducted experiments across different policies, auto-raters, datasets, estimators, ablations, and majority-voting settings to study the impact of ODRPO. On the out-of-distribution evaluation benchmarks, ODRPO and ODRPO-Gini provided mean relative improvements ranging from 3.33% to 5.39% in Table 1 and 2.29% to 6.96% in Table 4, and relative improvements ranging from 5.89% to 9.00% in Table 2. Some of the modest individual gains highlighted in the reviews are from these OOD evaluations, which measure transfer from training on UltraFeedback or WildChat. On the in-distribution GSM8K test set, the improvements are substantially larger, with mean relative improvements across the nine ablation settings of 19.01% for ODRPO and 20.20% for ODRPO-Gini. We believe these results provide encouraging evidence of ODRPO's promise.

We thank the reviewer again for their time and constructive feedback. We will revise our work accordingly.

We would like to kindly remind the reviewer that the deadline for the author-reviewer discussion period is approaching. If you have any further questions or comments, please let us know so we can address them before the deadline.

Reviewer_fHXiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ODRPO, a framework for improving policy optimization under noisy discrete rewards from LLM judges. It decomposes scalar rewards into ordinal binary indicators, computes advantages independently for each threshold, and aggregates them to guide policy updates. Experiments demonstrate consistent performance improvements across multiple models, advantage estimators, and benchmarks, with negligible additional computational overhead.

Critical strengths
  1. ODRPO is easy to implement and can be integrated with existing advantage estimators without requiring additional LLM judge calls.

  2. The method demonstrates consistent performance improvements across multiple policy models, LLM judges, advantage estimators, and evaluation benchmarks.

Critical weaknesses
  1. The paper lacks theoretical guarantees or direct empirical evidence (e.g., controlled noise injection experiments or numerical analyses of advantage perturbations) demonstrating that ODRPO effectively mitigates noise from LLM judges.

  2. The improvements may primarily stem from adaptive reward reweighting, similar to reward shaping, rather than genuine noise robustness. The paper does not sufficiently distinguish between these two explanations. The paper should compare ODRPO against simpler alternatives, such as nonlinear reward shaping and rank-based normalization, to determine whether the improvements genuinely arise from ordinal decomposition or more general reward reweighting mechanisms.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Thank you for the review. Please find our responses to your questions. Note that the rebuttal is multi-response.

Evidence of Noise Robustness

We would like to clarify that we do not claim that ODRPO mitigates the auto-rater’s noise, but rather its effect on advantage estimates. Considering that the auto-rater is treated as a black box, simple advantage-level algorithms can neither control its behavior nor access its internal information to improve optimization.

ODRPO reduces the sensitivity to the auto-rater’s stochasticity. We tested it under a fixed-response setting, similar to how we generated the scores for Kendall’s WW analysis, for granularity K=5,10,20K=5,10,20, group size G=5G=5, batch size 80 (for the batch normalization case), P=1000P=1000, M=8M=8, and N=16N=16 (defined in lines 135–136). The batch and group sizes are chosen to match the training configuration.

For a given prompt, we first choose five random responses out of eight and keep them fixed. For those five responses, we construct two independent reward vectors by randomly choosing one of the 16 scores for each response and then compute advantage values. We repeat this process R=500R=500 times for each prompt.

For batch normalization, we chunk the 1000 prompts into batches of 80. For each sampling pair, we apply batch normalization across the prompts and responses in the batch (not across the sampling axis), separately for the two reward vectors. So, for each estimator, we get A1A_1 and A2A_2 of shape (P,R,G)(P,R,G). Then we compute the sensitivity as the relative RMS difference as follows:

Numerator=∑p,r,g(A1,prg−A2,prg)2PRG\text{Numerator} = \sqrt{ \frac{\sum_{p,r,g}(A_{1,prg}-A_{2,prg})^2}{PRG} }

Denominator=∑p,r,g(A1,prg2+A2,prg2)2PRG\text{Denominator} = \sqrt{ \frac{\sum_{p,r,g}(A_{1,prg}^2+A_{2,prg}^2)}{2PRG} }

S=Sensitivity=NumeratorDenominatorS=\text{Sensitivity} = \frac{\text{Numerator}}{\text{Denominator}}

Relative improvement over Base’s sensitivity=100SBase−SmethodSBase.\text{Relative improvement over Base's sensitivity} = 100\frac{S_{\text{Base}}-S_{\text{method}}}{S_{\text{Base}}}.

Each cell reports the percentage reduction in sensitivity (with batch normalization / without batch normalization).

ODRPO’s Improvement over GRPO

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback12.2 / 12.219.4 / 18.618.9 / 18.0
Qwen3-4B / Qwen3-30B / UltraFeedback8.6 / 8.510.7 / 11.110.4 / 10.8
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback2.9 / 3.010.5 / 10.814.4 / 14.7
OLMo-2-7B / GPT-OSS-120B / WildChat5.7 / 5.99.5 / 9.79.5 / 9.7

ODRPO-Gini’s Improvement over GRPO

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback15.2 / 15.024.1 / 23.124.2 / 23.2
Qwen3-4B / Qwen3-30B / UltraFeedback8.8 / 8.912.5 / 13.112.6 / 13.2
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback3.8 / 4.011.6 / 11.915.7 / 16.1
OLMo-2-7B / GPT-OSS-120B / WildChat7.0 / 7.411.2 / 11.511.5 / 11.8

ODRPO’s Improvement over MaxRL

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback6.1 / 6.88.1 / 9.09.1 / 10.2
Qwen3-4B / Qwen3-30B / UltraFeedback4.8 / 5.65.8 / 6.96.5 / 7.7
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback1.5 / 1.72.4 / 2.53.0 / 3.1
OLMo-2-7B / GPT-OSS-120B / WildChat2.2 / 2.23.4 / 3.54.2 / 4.3

ODRPO-Gini’s Improvement over MaxRL

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback10.3 / 11.613.2 / 14.714.6 / 16.3
Qwen3-4B / Qwen3-30B / UltraFeedback6.8 / 7.88.2 / 9.49.1 / 10.4
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback4.4 / 4.65.5 / 5.66.0 / 6.2
OLMo-2-7B / GPT-OSS-120B / WildChat4.7 / 5.06.1 / 6.46.9 / 7.2

This shows that ODRPO’s ordinal decomposition lowers the sensitivity to the stochasticity of the auto-rater. Gini weighting provides further improvements, but the main driving force remains ordinal decomposition. Also, batch normalization has a limited empirical effect on relative sensitivity in these settings. Changing the judge’s temperature and reward granularity does change the reward distribution, but within the same setting, ODRPO provides improved performance with minimal overhead as shown in the ablation studies.

The relative reduction in sensitivity generally increases with increasing KK. Our practical question is whether, with the same black-box auto-rater, we should use GRPO or MaxRL alone, or their ODRPO variants. The relative comparison directly addresses this question.

AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Reward-Shaping and Rank-Based Baselines

The previous section already discusses robustness to noise. We also ran some more baselines on GSM8K across the nine temperature and granularity settings in Fig. 2. We ran GOPO (https://arxiv.org/pdf/2602.03876), Clipped Z-Score, Squared Reward Shaping (r→r2r \to r^2), and Square-Root Reward Shaping (r→rr \to \sqrt{r}). For Clipped Z-Score, we use c=1c=1 and apply clip⁡(A,−c,c)\operatorname{clip}(A,-c,c) to the GRPO advantages. For squared and square-root rewards, we first apply the reward transformation and then compute GRPO advantages from the transformed rewards. The table reports the mean relative improvement over GRPO across the nine settings.

MethodGSM8K Mean Relative Improvement (%)
GRPO0.00
GOPO2.18
Clipped Z-Score6.70
Squared Reward7.51
Square-Root Reward4.57
ODRPO19.01
ODRPO-Gini20.20

All provide improvement over GRPO, but ODRPO variants take the lead in mean relative improvement. This indicates that ordinal decomposition improves over the tested simpler reward-shaping and rank-based methods in this setting. We would also like to reinforce that ODRPO is an augmentation to current group-based estimators, so rather than being a separate method on its own, it is a principled approach to discrete and continuous reward settings.

We thank the reviewer again for their time and constructive feedback. We will revise our work accordingly.

Thank you for the additional experiments. However, my central concern remains unresolved. Mathematically, ODRPO is essentially a group-dependent nonlinear reward reweighting scheme. The paper does not provide a sufficiently convincing theoretical or intuitive justification for why this particular reweighting should improve upon standard GRPO, under what conditions it is beneficial, or why its induced optimization objective is preferable to that of GRPO. In particular, reduced advantage sensitivity does not necessarily imply better policy optimization. Therefore, the additional empirical results do not resolve my methodological concern.

Thank you for your response. Regarding your concern about “ODRPO being nonlinear reward reweighting,” we agree that ODRPO can be viewed this way. The process of computing the advantages separately at each discrete level rather than using single estiamtion across the scalar rewards provide a principled extension of group-based methods like GRPO or MaxRL from binary to discrete settings. While we do not have a rigorous proof that reduced advantage sensitivity implies better policy optimization, the downstream evaluations provide separate evidence that ODRPO improves upon GRPO and the additional baselines you requested under the same compute budget. Even compared with MaxRL using more auto-rater samples, ODRPO with a single auto-rater sample provides better results. Regarding the objective, our analysis shows that GRPO's expected update does not admit a global scalar objective for discrete rewards where as ODRPO admits one under the stated assumptions. This theoretical distinction makes it explicit what ODRPO's expected update optimizes. Empirical results and the theoretical contributions together provides evidence of ODRPO's benefits. Finally, regarding "what conditions it is beneficial," our experiments identify several settings in which ODRPO improves upon GRPO. A theoretical characterization of the conditions under which these improvements occur remains an open question.

Reviewer_cZBHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Ordinal Decomposition for Robust Policy Optimization (ODRPO), a framework for learning from stochastic, discrete scores produced by black-box LLM judges. It represents each score as nested binary threshold indicators, computes advantages separately at each threshold, and aggregates them for policy optimization. An optional Gini-based weighting scheme emphasizes thresholds that distinguish among sampled responses while moderately favoring higher reward levels.

The authors first document substantial variability in repeated judge evaluations. They then evaluate ODRPO with GRPO, MaxRL, and OTB across several policy models, two auto-raters, and seven benchmarks, reporting positive average relative gains with little additional per-step cost. Further experiments examine reward granularity, judge temperature, and repeated-score aggregation. The theoretical analysis identifies nonconservative expected update fields for conventional estimators on multi-level rewards and derives a scalar-objective construction for ordinal decomposition under suitable binary estimators. The central contribution is a practical transformation of advantage estimation, although the evidence connecting its gains specifically to noise reduction and the scope of its theoretical guarantees require clarification.

Critical strengths

A simple and practically useful estimator transformation. ODRPO operates on the returned score alone, requires no judge internals or additional reward samples, and can be incorporated into several existing estimators. Its low implementation burden and comparable reported runtimes make the contribution potentially useful for resource-constrained RLAIF pipelines.

Encouraging empirical breadth. The experiments extend beyond a single policy–judge pairing and include GRPO, MaxRL, and token-level OTB, with evaluation covering grounding, preferences, reasoning, and instruction following. Positive aggregate gains across these configurations strengthen the case that ordinal decomposition has practical value beyond one narrowly tuned setting.

Critical weaknesses

The central noise-robustness explanation is not sufficiently isolated, and statistical reliability remains unclear. Restricting a score perturbation to the thresholds it crosses does not itself establish lower variance in the aggregated advantage or policy gradient: normalization within each affected threshold still couples the group’s responses. Temperature and granularity changes also alter the reward distribution and effective objective, so their ablations do not isolate noise reduction. Moreover, the main performance tables lack uncertainty across independent training runs; the five seeds in the GSM8K ablation are explicitly evaluation seeds. These gaps limit confidence in both the proposed mechanism and the reproducibility of the reported gains.

The theoretical claims exceed the demonstrated guarantees. Section 5 states that recovering a scalar objective ensures reliable convergence, but existence of a potential alone does not establish convergence of the implemented stochastic optimization procedure. Equation (2) is presented without the large-group qualification stated in Appendix C.4, despite experiments using small groups. The extension to empirical Gini weights and batch normalization also needs a more explicit expected-update analysis: treating a realized batch scaling as a learning-rate adjustment does not automatically preserve a global potential after expectation. These issues are consequential because theoretical grounding is presented as a central contribution.

Questions for authors

Experimental clarification: How many independent training seeds underlie Tables 1, 2, and 4? Can you provide uncertainty across training runs, separately from evaluation randomness, for the comparisons supporting the main conclusions?

Mechanism clarification: Do you have measurements of advantage or gradient variance under repeated scoring of fixed response groups, or controlled corruption experiments that separate noise reduction from objective reshaping? Such evidence would materially affect my assessment of the robustness claim.

Theoretical clarification: Please distinguish the exact finite-group result from the large-group expression in Equation (2), and specify whether the scalar-objective guarantee covers empirical Gini weighting and batch normalization as implemented. What additional assumptions or results support the convergence statement in Section 5?

3 comments
AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Thank you for the review. Please find our responses to your questions. Note that the rebuttal is multi-response.

Sensitivity to Auto-Rater Noise

ODRPO reduces the sensitivity to the auto-rater’s stochasticity. We tested it under a fixed-response setting, similar to how we generated the scores for Kendall’s WW analysis, for granularity K=5,10,20K=5,10,20, group size G=5G=5, batch size 80 (for the batch normalization case), P=1000P=1000, M=8M=8, and N=16N=16 (defined in lines 135–136). The batch and group sizes are chosen to match the training configuration.

For a given prompt, we first choose five random responses out of eight and keep them fixed. For those five responses, we construct two independent reward vectors by randomly choosing one of the 16 scores for each response and then compute advantage values. We repeat this process R=500R=500 times for each prompt.

For batch normalization, we chunk the 1000 prompts into batches of 80. For each sampling pair, we apply batch normalization across the prompts and responses in the batch (not across the sampling axis), separately for the two reward vectors. So, for each estimator, we get A1A_1 and A2A_2 of shape (P,R,G)(P,R,G). Then we compute the sensitivity as the relative RMS difference as follows:

Numerator=∑p,r,g(A1,prg−A2,prg)2PRG\text{Numerator} = \sqrt{ \frac{\sum_{p,r,g}(A_{1,prg}-A_{2,prg})^2}{PRG} }

Denominator=∑p,r,g(A1,prg2+A2,prg2)2PRG\text{Denominator} = \sqrt{ \frac{\sum_{p,r,g}(A_{1,prg}^2+A_{2,prg}^2)}{2PRG} }

S=Sensitivity=NumeratorDenominatorS=\text{Sensitivity} = \frac{\text{Numerator}}{\text{Denominator}}

Relative improvement over Base’s sensitivity=100SBase−SmethodSBase.\text{Relative improvement over Base's sensitivity} = 100\frac{S_{\text{Base}}-S_{\text{method}}}{S_{\text{Base}}}.

Each cell reports the percentage reduction in sensitivity (with batch normalization / without batch normalization).

ODRPO’s Improvement over GRPO

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback12.2 / 12.219.4 / 18.618.9 / 18.0
Qwen3-4B / Qwen3-30B / UltraFeedback8.6 / 8.510.7 / 11.110.4 / 10.8
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback2.9 / 3.010.5 / 10.814.4 / 14.7
OLMo-2-7B / GPT-OSS-120B / WildChat5.7 / 5.99.5 / 9.79.5 / 9.7

ODRPO-Gini’s Improvement over GRPO

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback15.2 / 15.024.1 / 23.124.2 / 23.2
Qwen3-4B / Qwen3-30B / UltraFeedback8.8 / 8.912.5 / 13.112.6 / 13.2
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback3.8 / 4.011.6 / 11.915.7 / 16.1
OLMo-2-7B / GPT-OSS-120B / WildChat7.0 / 7.411.2 / 11.511.5 / 11.8

ODRPO’s Improvement over MaxRL

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback6.1 / 6.88.1 / 9.09.1 / 10.2
Qwen3-4B / Qwen3-30B / UltraFeedback4.8 / 5.65.8 / 6.96.5 / 7.7
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback1.5 / 1.72.4 / 2.53.0 / 3.1
OLMo-2-7B / GPT-OSS-120B / WildChat2.2 / 2.23.4 / 3.54.2 / 4.3

ODRPO-Gini’s Improvement over MaxRL

Policy / auto-rater / datasetK=5K=5K=10K=10K=20K=20
Qwen2.5-7B / Qwen3-30B / UltraFeedback10.3 / 11.613.2 / 14.714.6 / 16.3
Qwen3-4B / Qwen3-30B / UltraFeedback6.8 / 7.88.2 / 9.49.1 / 10.4
Qwen2.5-7B / GPT-OSS-120B / UltraFeedback4.4 / 4.65.5 / 5.66.0 / 6.2
OLMo-2-7B / GPT-OSS-120B / WildChat4.7 / 5.06.1 / 6.46.9 / 7.2

This shows that ODRPO’s ordinal decomposition lowers the sensitivity to the stochasticity of the auto-rater. Gini weighting provides further improvements, but the main driving force remains ordinal decomposition. Also, batch normalization has a limited empirical effect on relative sensitivity in these settings. Changing the judge’s temperature and reward granularity does change the reward distribution, but within the same setting, ODRPO provides improved performance with minimal overhead as shown in the ablation studies.

AuthorsOpenReview ↗

Rebuttal (Multi-Response)

Convergence Claims

We would like to clarify that our central contribution is ODRPO, for which we provide both empirical and theoretical support.

We acknowledge that we may have used strong statements regarding convergence. Recovering a scalar objective provides a foundation for studying optimization dynamics and convergence, however, we agree that having a scalar objective alone does not establish convergence. We will revise the wording in Section 5.

Finite-Group vs. Large-Group Expressions

The finite-group expression for binary-reward GRPO is provided on page 8 of Davis and Recht’s work (https://arxiv.org/pdf/2510.13651), where they define MM as group size. To get the finite-group expression for discrete rewards (and continuous rewards by virtue of integration), plug the finite-group binary-reward expressions from Davis and Recht’s paper in Eq. 1 with t=Pmt=P_m. Equation (2) uses the large-group expression, which we will state explicitly in the revision. We omitted the finite-group expressions for brevity, but we will include them in the revision for better clarity.

Objective for Empirical Gini Weighting

The scalar-objective guarantee for the expected update in Appendix C.4 also covers empirical Gini weighting at finite group size, before final batch normalization, as it depends only on its own ordinal bin’s empirical success fraction without any cross-bin term.

The Gini weight as a function of the mean qq is

wm(q)=m[δ+4q(1−q)].w_m(q)=\sqrt m\left[\delta+4q(1-q)\right].

For finite group size, the weightings are applied to the success and failure functions in Eq. (46) as follows:

tmGini(s)=wm!(s+1M)t(s),umGini(s)=wm!(sM)u(s).t_m^{\mathrm{Gini}}(s) = w_m\\!\left(\frac{s+1}{M}\right)t(s), \qquad u_m^{\mathrm{Gini}}(s) = w_m\\!\left(\frac{s}{M}\right)u(s).

The rest of the derivation follows as before because there is no cross-bin dependence.

In the large-group limit, the empirical success fraction converges to the suffix probability:

μ(m)⟶Pm\mu^{(m)} \longrightarrow P_m

The weighting therefore acts as a probability-dependent change in the reward-spacing coefficient in the expected update:

wm(Pm)=m[δ+4Pm(1−Pm)],Δ~m(Pm)=Δmwm(Pm).w_m(P_m)=\sqrt m\left[\delta+4P_m(1-P_m)\right], \qquad \widetilde{\Delta}_m(P_m)=\Delta_m w_m(P_m).

Using the same overall normalization as Eq. (57), the gradient becomes

∇θJGini(θ)=Ex∼Q[1π∑m=2KΔ~m(Pm)Pm(1−Pm)∇θPm]\nabla_{\theta} J_{\mathrm{Gini}}(\theta) = \mathbb{E}_{x\sim Q}\left[ \frac{1}{\pi}\sum_{m=2}^{K} \frac{\widetilde{\Delta}_{m}(P_m)}{\sqrt{P_m(1-P_m)}}\nabla_{\theta} P_m \right]

Integrating this weighted update gives

JGini(θ)=Ex∼Q![1π∑m=2KΔmm((2δ+1)arcsin⁡Pm−(1−2Pm)Pm(1−Pm))]J_{\mathrm{Gini}}(\theta) = \mathbb E_{x\sim Q}\\!\left[ \frac1\pi\sum_{m=2}^K\Delta_m\sqrt m \left( (2\delta+1)\arcsin\sqrt{P_m} -(1-2P_m)\sqrt{P_m(1-P_m)} \right) \right]

We will include the Gini derivation for clarity in our revision.

Scope of the Guarantee under Batch Normalization

We applied batch normalization to ensure the advantages after aggregation stay well-behaved. The advantages of binary GRPO are bounded in magnitude for a fixed group size, but the bound grows for ODRPO because of aggregation and weighting. Since per-bin advantages are centered, the overall aggregation stays centered, so the mean advantage for the batch remains zero and the effect is equivalent to dividing the policy gradient by the standard deviation. For a plain policy-gradient update, we view this as data-adaptive learning-rate scaling. Our scalar-objective analysis applies before the final batch normalization, while the ablation in C.4.1 indicates that adding this normalization has little empirical effect in the evaluated settings. Apart from batch normalization, one can use data-independent scaling, but we chose batch normalization because it can be applied easily to other estimators without having to design estimator-specific scaling.

Training-Seed Uncertainty

We used a single training seed for each experiment and we acknowledge that the evaluation seeds do not capture variability across independent training runs. Because of compute constraints, we are unable to run multiple training seeds for each experiment. However, we conducted experiments across different policies, auto-raters, datasets, estimators, ablations, and majority-voting settings to study the impact of ODRPO. On the out-of-distribution evaluation benchmarks, ODRPO and ODRPO-Gini provided mean relative improvements ranging from 3.33% to 5.39% in Table 1 and 2.29% to 6.96% in Table 4, and relative improvements ranging from 5.89% to 9.00% in Table 2. Some of the modest individual gains highlighted in the reviews are from these OOD evaluations, which measure transfer from training on UltraFeedback or WildChat. On the in-distribution GSM8K test set, the improvements are substantially larger, with mean relative improvements across the nine ablation settings of 19.01% for ODRPO and 20.20% for ODRPO-Gini. We believe these results provide encouraging evidence of ODRPO's promise.

We thank the reviewer again for their time and constructive feedback.

We would like to kindly remind the reviewer that the deadline for the author-reviewer discussion period is approaching. If you have any further questions or comments, please let us know so we can address them before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We would like to thank the reviewers for their constructive reviews. To make it easier for the reviewers and AC to go through our rebuttal, we summarize the concerns we addressed below. For details, please see our responses to each reviewer.

  • Noise Robustness (asked by cZBH and fHXi): We did a fixed-response analysis to assess ODRPO's robustness to the auto-rater's stochasticity. Our empirical analysis shows that ODRPO reduces the relative sensitivity of advantages by up to 19.4% compared with the base estimators, with ODRPO-Gini providing reductions of up to 24.2%.
  • Reward Shaping Baselines (asked by fHXi and mrno): We did further GSM8K experiments using GOPO (rank-based advantages), Clipped Z-Score, Squared Reward, and Square-Root Reward across the nine temperature and granularity settings in Fig. 2. Against GRPO, the new baselines provide mean relative improvements ranging from 2.18% to 7.51%, while ODRPO and ODRPO-Gini provide 19.01% and 20.20%, respectively.
  • Theory Clarifications (asked by cZBH and mrno): We provided the objective function for Gini weighting and clarified the finite-group and full-group statistics used in the analysis. We also clarified that the scalar-objective analysis applies before final batch normalization. We will revise our work to include the finite-group expressions, Gini derivation, and batch-normalization details, and clarify the wording regarding convergence.
  • Empirical Reliability (asked by cZBH and mrno): We explained the consistency of improvements across policies, auto-raters, datasets, estimators, and both in-distribution and out-of-distribution evaluations. We acknowledged the absence of training-seed variation and explained our compute constraints in the responses.

We thank the AC and reviewers again for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.