- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StableMax-PPO, which uses a frozen ordinal reward model to predict rating probabilities and optimize the expected maximum latent rating within a Best-of-N candidate group. The method constructs a robust objective and response-level credits through a one-sided CDF ambiguity set, and incorporates quality floors, KL control, and complete-state rollback into PPO training. The experiments report lower held-out KL, but do not establish a clear advantage in selected-response quality or the performance–drift trade-off.
Critical strengths
- The research question has merit. Incorporating full rating distributions into Best-of-N optimization captures differences that response means alone cannot represent, offering a potentially useful perspective on multi-candidate training.
- Under the stated assumptions, including conditional independence, the theoretical construction is relatively clear. The paper provides closed-form expressions for the robust objective and marginal credits, with efficient computation of the core statistics. It also transparently reports unfavorable results and the limits of its guarantees.
Critical weaknesses
1、The connection between the core objective and practical selection benefits remains insufficiently established, and robustness with respect to true ratings is unverified. The method optimizes the expected maximum latent rating across candidates, whereas common Best-of-N deployments are concerned with the quality of the response ultimately returned by a selector. These quantities are not equivalent. In principle, differences in rating distributions can increase the expected group maximum even when the candidates’ expected ratings remain unchanged. An improvement in the group-level metric therefore does not, by itself, demonstrate better selection quality. The independent evaluations also fail to establish a clear quality advantage for selected responses, leaving a substantive gap between the proposed objective and practical deployment benefits. Furthermore, interpreting the robust objective as a lower bound on the true expected maximum requires conditional CDF coverage for every candidate at every threshold. Population-average calibration does not establish this pointwise condition. The recorded experiments retain the radius obtained from the earlier calibration rule, without reporting new training or evaluation using a radius derived from the corrected rule. The paper also states that the original cluster-level calibration data and report are not included in the release, preventing verification of the sample size and split independence from the released materials. This does not invalidate the exact ambiguity-set solution or the gradient derivation, but the robustness guarantee for true ratings remains without empirical support.
2、The experiments do not establish an advantage for the proposed method or justify the necessity of its complex design. Although the methods share the same KL control rules, their measured held-out KL values differ substantially. Without comparisons at matched KL levels or a complete quality–KL curve, lower KL cannot be directly interpreted as a better performance–drift trade-off. The possibility that comparable results could be obtained by adjusting the regularization strength of existing baselines remains unresolved. More importantly, StableMax-PPO’s improvement in the robust expected maximum over Mean-PPO is not statistically significant. Its robust expected maximum is also lower than those of GRPO and Scalar max-PPO, with the reported paired confidence intervals for both differences lying entirely below zero. Independent model evaluations of selected responses do not identify a clear quality difference, and the predeclared overall scientific-success criterion is not met. The experiments are further limited to one policy model, a fixed N=32, and closely related data domains, without fresh human evaluation of generated outputs. These results provide insufficient support for broader claims of effectiveness. The method also involves multiple evaluators, adaptive constraints, and complete-state rollback. However, the provided PDF does not offer sufficient comparisons of training time, memory usage, or tuning costs. The current evidence neither establishes a better performance–drift trade-off nor adequately demonstrates that the benefits of these system components justify their implementation complexity.
Questions for authors
Please see the Weaknesses above.