acceptodds
Withdrawn from ICLR 2027

StableMax-PPO: Best-of- RLHF with Ordinal Reward Distributions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many language-model systems use Best-of- sampling at deployment, while standard reinforcement learning from human feedback (RLHF) optimizes the expected reward of a single response. Existing Best-of- training objectives often reduce each response to a scalar and discard the per-rating probabilities that determine the group maximum. This matters because two responses with the same predicted mean can have different probabilities of receiving a high rating. We introduce StableMax-PPO, which directly optimizes the expected best latent rating among candidates using the predictive distribution of a frozen ordinal reward model trained from human ratings. Its variance is retained as a model-based diagnostic of annotator disagreement, not an optimization bonus. A one-sided CDF ambiguity set with a fixed radius gives the exact worst-case expected latent rating maximum and an exact response-level policy credit for PPO. To preserve typical responses and limit policy drift, we impose lower bounds on the predicted mean rating and on an independently trained quality score, and limit changes from the reference policy using a standard KL measure of policy change with rollback. Across three seeds at , StableMax-PPO achieves a competitive robust maximum with lower held-out KL than Mean-PPO, group relative policy optimization (GRPO), and scalar max-PPO baselines.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from human feedback, Best-of-N sampling, reward modeling, calibration robustness, policy optimization, PPO, KL regularization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_WR3JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StableMax-PPO, which uses a frozen ordinal reward model to predict rating probabilities and optimize the expected maximum latent rating within a Best-of-N candidate group. The method constructs a robust objective and response-level credits through a one-sided CDF ambiguity set, and incorporates quality floors, KL control, and complete-state rollback into PPO training. The experiments report lower held-out KL, but do not establish a clear advantage in selected-response quality or the performance–drift trade-off.

Critical strengths
  • The research question has merit. Incorporating full rating distributions into Best-of-N optimization captures differences that response means alone cannot represent, offering a potentially useful perspective on multi-candidate training.
  • Under the stated assumptions, including conditional independence, the theoretical construction is relatively clear. The paper provides closed-form expressions for the robust objective and marginal credits, with efficient computation of the core statistics. It also transparently reports unfavorable results and the limits of its guarantees.
Critical weaknesses

1、The connection between the core objective and practical selection benefits remains insufficiently established, and robustness with respect to true ratings is unverified. The method optimizes the expected maximum latent rating across candidates, whereas common Best-of-N deployments are concerned with the quality of the response ultimately returned by a selector. These quantities are not equivalent. In principle, differences in rating distributions can increase the expected group maximum even when the candidates’ expected ratings remain unchanged. An improvement in the group-level metric therefore does not, by itself, demonstrate better selection quality. The independent evaluations also fail to establish a clear quality advantage for selected responses, leaving a substantive gap between the proposed objective and practical deployment benefits. Furthermore, interpreting the robust objective as a lower bound on the true expected maximum requires conditional CDF coverage for every candidate at every threshold. Population-average calibration does not establish this pointwise condition. The recorded experiments retain the radius obtained from the earlier calibration rule, without reporting new training or evaluation using a radius derived from the corrected rule. The paper also states that the original cluster-level calibration data and report are not included in the release, preventing verification of the sample size and split independence from the released materials. This does not invalidate the exact ambiguity-set solution or the gradient derivation, but the robustness guarantee for true ratings remains without empirical support.

2、The experiments do not establish an advantage for the proposed method or justify the necessity of its complex design. Although the methods share the same KL control rules, their measured held-out KL values differ substantially. Without comparisons at matched KL levels or a complete quality–KL curve, lower KL cannot be directly interpreted as a better performance–drift trade-off. The possibility that comparable results could be obtained by adjusting the regularization strength of existing baselines remains unresolved. More importantly, StableMax-PPO’s improvement in the robust expected maximum over Mean-PPO is not statistically significant. Its robust expected maximum is also lower than those of GRPO and Scalar max-PPO, with the reported paired confidence intervals for both differences lying entirely below zero. Independent model evaluations of selected responses do not identify a clear quality difference, and the predeclared overall scientific-success criterion is not met. The experiments are further limited to one policy model, a fixed N=32, and closely related data domains, without fresh human evaluation of generated outputs. These results provide insufficient support for broader claims of effectiveness. The method also involves multiple evaluators, adaptive constraints, and complete-state rollback. However, the provided PDF does not offer sufficient comparisons of training time, memory usage, or tuning costs. The current evidence neither establishes a better performance–drift trade-off nor adequately demonstrates that the benefits of these system components justify their implementation complexity.

Questions for authors

Please see the Weaknesses above.

Reviewer_GhYtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

StableMax-PPO targets Best-of-N use by maximizing the expected maximum latent rating among N=32 candidates. A frozen Qwen2.5-14B ordinal reward model trained on HelpSteer2 gives each response a distribution over ratings 0–4. The CDFs are shifted up by a fixed radius ε to define a worst-case group value, and each response receives its exact leave-one-out marginal credit. These credits are combined with dual-weighted floors on the predicted mean and on a separate Quality RM, and optimized with PPO, a KL controller and full-state rollback. With Qwen2.5-1.5B-Instruct and three seeds, StableMax-PPO reaches a robust maximum of 3.458 at held-out KL 0.034, against 3.424/0.070 for Mean-PPO, 3.517/0.054 for scalar max-PPO and 3.555/0.094 for GRPO.

Critical strengths
  1. The starting observation is sound: two responses with the same mean rating can differ in their chance of a top rating, and a scalar max@k objective cannot see that. The threshold-product form gives the group value and every leave-one-out credit in O(N∣R∣)O(N|R|) without sampling ratings, and the derivations are correct.

  2. The reporting is unusually candid. Seeds are shown individually, the failed success gates are reported rather than dropped, the error in the calibration radius is disclosed, and proxy evaluations are included even though they show parity. This makes the paper easy to assess.

Critical weaknesses
  1. The objective does not correspond to a deployment target. It is the expected maximum of independent draws of annotator ratings, taken before any response is selected, whereas a deployed selector sees model scores, not realized ratings. Under assumption A1, a response with a wide rating distribution earns a high expected maximum even with a lower mean, and 32 copies of one polarizing answer would score well because their rating noise is treated as independent. The objective therefore rewards rating dispersion, which the paper says it does not use as a bonus, and the mean floor only limits this until it is satisfied. The selected-response proxies show no benefit over Mean-PPO (preference 0.494 and 0.502, Table 4).

  2. The experiments do not support a method contribution. The primary endpoint is the training objective scored by the training reward model. On it, StableMax-PPO is worse than GRPO and scalar max-PPO (paired intervals [−0.134, −0.061] and [−0.102, −0.010]), its difference from Mean-PPO spans zero ([−0.069, 0.130]), and the Gaussian EV variant in the ablations reaches 3.558. What remains is lower KL. Under a shared controller with hard rollback, which rejects 82% of GRPO updates, KL depends on how credit is scaled and concentrated as much as on the objective, so without a matched-KL comparison the lower drift is not shown to be an advantage. Training uses 300 steps of two prompts with at most 64 response tokens. The robustness argument rests on a radius computed with the wrong concentration term, and even the corrected bound is a population average that does not give the per-response coverage Theorem 3 requires.

Questions for authors
  1. At matched predicted mean, do StableMax-PPO's responses have higher predicted rating variance than Mean-PPO's? If so, how much of the gain in expected maximum comes from dispersion alone?

  2. Can you compare methods at equal held-out KL, for example by sweeping the KL target or the number of steps for Mean-PPO and scalar max-PPO?

  3. Is there any evidence at realistic response lengths, with human or strong-judge evaluation of the response actually returned by the selector?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.