acceptodds
Under review as a conference paper at ICLR 2027

UPAO: Unbounded Positive Asymmetric Optimization for Breaking the Exploration–Stability Dilemma in RL

Abstract

Reinforcement learning (RL) has become the standard paradigm for enhancing the complex logical reasoning capabilities of large language models (LLMs). Modern RL frameworks improve sample efficiency by employing importance sampling (IS) to evaluate policy updates from historical trajectories. However, IS-based RL inherently suffers from an exploration-stability dilemma: pure IS is vulnerable to gradient explosion and catastrophic training instability, while clipping severely limits the model's exploration capacity. By formalizing the concept of Probability Capacity (Cap), we uncover a fundamental structural bottleneck: clipping constrains the update budget of high-reward but low-confidence reasoning paths, prematurely truncating gradients and stifling exploration. To address this challenge, we propose Unbounded Positive Asymmetric Optimization (UPAO). UPAO applies an asymmetric scheme: it removes clipping and uses a self-anchored ratio via the stop-gradient operator for tokens with positive advantages, while retaining clipping for tokens with non-positive advantages to maintain stability. Experiments against 13 RL baselines show that UPAO improves reasoning performance across model families (Qwen3 and Olmo 3), architectures (Dense and MoE), and modalities (language and multimodal) without compromising training stability. These advantages persist as policy staleness increases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UMUMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

UPAO applies different update rules to positive- and negative-advantage tokens to improve exploration and training stability. Its positive branch uses a self-anchored ratio whose gradient is equivalent to advantage-weighted SFT, while its negative branch retains clipping and dual clipping. The paper introduces a Probability Capacity analysis and reports improvements across multiple model sizes, architectures, and modalities.

Critical strengths
  1. The method is simple and easy to integrate into existing training pipelines, making the reported improvements practically relevant.

  2. The experiments cover multiple model configurations and relevant baselines, examining both performance and stability under increasing policy staleness.

Critical weaknesses
  1. The incremental contribution over prior work is unclear. For fixed advantages and normalization, the positive branch is equivalent to advantage-weighted SFT, closely related to weighted rejection-sampling fine-tuning. TOPR [1] already combines the same positive-gradient form with controlled negative updates. UPAO pairs this positive rule with the negative branch of the paper’s DAPO [2] baseline incorporating dual clipping [3]. Although these methods are cited and evaluated, the narrative mainly develops UPAO from DAPO’s limitations without sufficiently explaining the substantive advance over TOPR.

  2. The source of the gains is not sufficiently isolated. The experiments do not clearly separate the contributions of positive-only weighted SFT, the negative clipping rule, and implementation differences. A positive-only weighted SFT baseline and a controlled comparison of TOPR’s and UPAO’s negative branches would be particularly informative. The existing “without asymmetry” ablation retains unrestricted negative updates; its instability does not establish an advantage over simply discarding negative samples.

[1] Nicolas Le Roux et al. Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs. 2025. arXiv:2503.14286.

[2] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025. arXiv:2503.14476.

[3] Deheng Ye et al. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. AAAI 2020. arXiv:1912.09729.

Questions for authors
  1. Given that TOPR already uses the same positive-gradient form, what is the specific additional mechanism introduced by UPAO? Can the Probability Capacity analysis explain its advantage over TOPR?

  2. Under matched sampling budgets, advantage computation, and normalization, how does positive-only weighted SFT perform? With the positive branch held fixed, how much improvement comes from replacing TOPR’s negative update rule with UPAO’s?

Reviewer_krUFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the training stability in IS-based RL for LLM reasoning. Clipping zeroes the gradient of positive-advantage tokens once πθ>(1+ϵ)πold\pi_\theta > (1+ \epsilon)\pi_{old}, which limits updates to low-probability tokens; the authors formalize this as “Probability Capacity.” The proposed method, UPAO, uses a self-anchored ratio $\pi_\theta/|\pi_\theta| $ without clipping for positive-advantage tokens, and keeps DAPO’s clipping for the rest. Experiments on several model families, architectures, and modalities show consistent gains over 13 baselines, and UPAO’s accuracy degrades less than DAPO’s as policy staleness increases.

Critical strengths
  1. The paper is well written and easy to follow. he method is simple: it changes one line of the loss and adds no compute.
  2. The empirical evaluation is broad, with 13 baselines, dense and MoE models, two model families, and both text and vision-language settings. Improvements are consistent across all of them.
  3. The ablations are clean. Both an unclipped IS ratio and a symmetric unclipped objective collapse, which shows that both components are needed.
Critical weaknesses
  1. Limited novelty and missing related work: The positive branch A∇logπA\nabla log \pi is the same as TOPR’s positive branch. It is also REC-OneSide-NoIS [1] with ϵ→∞\epsilon\to \infty AsymRE [2] already argues that off-policy REINFORCE should emphasize positive samples. The paper does not clearly explain how UPAO differs from these works.
  2. The theory is weak and some claims go too far: with /sg(πθ)/sg(\pi_\theta), the loss is a totally biased gradient estimator for RL loss. The paper does not discuss this.
  3. Key ablation are missing: One important ablation is a positive-only variant, such as online sft/RAFT++/Reinforce-Rej.
  4. Entropy rises without explanation. On Qwen3-14B, entropy jumps from about 0.8 to 1.5 late in training. The paper reports no response length or degeneration analysis.

[1] Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends

[2] Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards

Questions for authors
  1. see weakness
  2. Since all training is on math, a domain that is already heavily optimized for Qwen-family models (Appendix F also notes that agentic tasks are untested), do the authors have results on coding or multi-turn agentic tasks, where longer horizons and more off-policy rollouts would better test UPAO’s stability?
Reviewer_kNuQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the well-known dilemma between training stability and exploration in RL methods. By simply replacing /pi_{old} with sg(/pi_theta), the authors manage to solve this dilemma. I appreciate that the improvement goes to the core of the problem and is based on solid theory, which is worth encouraging. However, I still see some weaknesses, mainly in the experimental coverage and the comparison with existing methods.

Critical strengths

This paper well justified the core dilemma between training stability and exploration in RL methods. By designing a Probability Capacity, this paper reviews the reason why exploration is limited by the current clip-based method. The improvement is well-designed and very simple, which is cool.

Critical weaknesses
  1. In Table 1, the authors should state clearly which baseline UPAO is directly compared against. My understanding is that UPAO is not tied to one algorithm: it should apply to any method built on the same importance-sampling design, such as GRPO, DAPO, and Dr.GRPO.

  2. If the gains in Table 1 are built on top of DAPO, then the failure on MATH500 deserves a proper failure-mode analysis. If they are not, then reporting against the direct baseline would show much more clearly what UPAO itself contributes.

  3. Same, results over several baselines (GRPO+UPAO, Dr.GRPO+UPAO, and DAPO+UPAO) are necessary.

  4. Does the improved stability come from keeping the lower-side clipping? Needs to be discussed.

  5. Neither the dilemma the paper focuses on nor the sg(/pi_theta) substitution is novel on its own, but I do believe their combination solves the problem. To make the advantage of UPAO convincing, the authors should compare in more depth with how earlier methods handle this same dilemma.

Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.