- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
UPAO applies different update rules to positive- and negative-advantage tokens to improve exploration and training stability. Its positive branch uses a self-anchored ratio whose gradient is equivalent to advantage-weighted SFT, while its negative branch retains clipping and dual clipping. The paper introduces a Probability Capacity analysis and reports improvements across multiple model sizes, architectures, and modalities.
Critical strengths
-
The method is simple and easy to integrate into existing training pipelines, making the reported improvements practically relevant.
-
The experiments cover multiple model configurations and relevant baselines, examining both performance and stability under increasing policy staleness.
Critical weaknesses
-
The incremental contribution over prior work is unclear. For fixed advantages and normalization, the positive branch is equivalent to advantage-weighted SFT, closely related to weighted rejection-sampling fine-tuning. TOPR [1] already combines the same positive-gradient form with controlled negative updates. UPAO pairs this positive rule with the negative branch of the paper’s DAPO [2] baseline incorporating dual clipping [3]. Although these methods are cited and evaluated, the narrative mainly develops UPAO from DAPO’s limitations without sufficiently explaining the substantive advance over TOPR.
-
The source of the gains is not sufficiently isolated. The experiments do not clearly separate the contributions of positive-only weighted SFT, the negative clipping rule, and implementation differences. A positive-only weighted SFT baseline and a controlled comparison of TOPR’s and UPAO’s negative branches would be particularly informative. The existing “without asymmetry” ablation retains unrestricted negative updates; its instability does not establish an advantage over simply discarding negative samples.
[1] Nicolas Le Roux et al. Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs. 2025. arXiv:2503.14286.
[2] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025. arXiv:2503.14476.
[3] Deheng Ye et al. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. AAAI 2020. arXiv:1912.09729.
Questions for authors
-
Given that TOPR already uses the same positive-gradient form, what is the specific additional mechanism introduced by UPAO? Can the Probability Capacity analysis explain its advantage over TOPR?
-
Under matched sampling budgets, advantage computation, and normalization, how does positive-only weighted SFT perform? With the positive branch held fixed, how much improvement comes from replacing TOPR’s negative update rule with UPAO’s?