- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the premature policy concentration problem in reinforcement learning with verifiable rewards (RLVR), which limits the exploration of alternative reasoning trajectories. It introduces token-probability-based confidence to characterize policy concentration and analyzes how reinforcing likely tokens and suppressing unlikely alternatives contribute to this process. Based on this analysis, the paper proposes -Policy Optimization (-PO), a timing-aware exploration control method that selectively suppresses concentration-inducing token updates when the estimated exploration space falls below a desired margin. An annealing mechanism progressively relaxes this intervention to facilitate the transition from exploration to exploitation. Experiments across four backbone models and five reasoning benchmarks demonstrate improvements in both Pass@1 and Pass@16 over the evaluated baselines, with training dynamics and ablation studies further supporting the effectiveness of the proposed method.
Critical strengths
- Novel perspective on exploration control. The paper distinguishes premature policy concentration from beneficial consolidation, motivating a timing-aware approach to balancing exploration and exploitation in RLVR.
- Principled optimization mechanism. The proposed -PO selectively suppresses concentration-inducing token updates based on rollout confidence, while an annealing strategy progressively shifts optimization toward exploitation.
- Comprehensive empirical validation. Experiments across four backbone models and five reasoning benchmarks demonstrate improvements in both Pass@1 and Pass@16. Training dynamics and ablation studies further support the effectiveness of the proposed method.
Critical weaknesses
- Limited novelty and insufficient comparison with prior work. Several closely related studies have explored entropy collapse and exploration enhancement in RLVR. The authors should provide a systematic comparison with these works to clarify the novelty of the proposed method.
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Rethinking Entropy Regularization in Large Reasoning Models
- Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
- Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
- EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
- Unclear model selection rationale. Why do the authors use Llama-3.2-3B-Instruct rather than a base model, unlike the other experimental settings? The rationale for this choice and its implications for experimental comparability should be clarified.
- Limited performance gains and insufficient analysis of model scaling. The improvements are modest, with gains of only 1.15 for the 1.7B Base model and 0.39 for the 8B Base model over the baseline. Do these results suggest that the method becomes less effective as model size increases, and can the authors provide further analysis of this trend?
- Unclear validity of the confidence metric. Why is confidence defined based on token probabilities, and can this metric reliably capture the available exploration space? The authors should provide theoretical or empirical evidence supporting this design choice.
- Insufficient justification for token-update suppression and annealing. Why does suppressing specific token updates expand the exploration space, and could this intervention undermine effective learning? Is annealing necessary to mitigate such negative effects, and can the authors provide evidence supporting this design choice?
Questions for authors
Explain the concerns in Weaknesses Part.