acceptodds
Withdrawn from ICLR 2027

-Policy Optimization: Timing-Aware Confidence-Based Exploration Control in RLVR

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves language model reasoning, but reward-driven optimization can prematurely concentrate policies on successful behaviors, limiting exploration of alternative reasoning trajectories. Existing methods emphasize how to preserve exploration, with less attention to when it is needed. Since concentration also enables exploitation, exploration control must distinguish premature concentration from useful consolidation. We introduce confidence, defined through token probabilities, to connect policy concentration with token-level optimization dynamics. Our analysis identifies two mechanisms that increase concentration: reinforcing likely tokens and suppressing unlikely alternatives. Based on that, we propose -Policy Optimization (-PO), a timing-aware exploration control method that estimates exploration space from rollout confidence and intervenes only when it falls below a desired margin . It selectively suppresses token updates that contribute to concentration, otherwise retaining the original objective to minimize intervention. And annealing progressively relaxes intervention to balance exploration and exploitation throughout training. Across four backbone models and five reasoning benchmarks, -PO achieves higher average Pass@1 and Pass@16 than the evaluated baselines. Training dynamics and ablations support its effectiveness in delaying early concentration while allowing later exploitation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reasoning, Exploration, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hdzhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the premature policy concentration problem in reinforcement learning with verifiable rewards (RLVR), which limits the exploration of alternative reasoning trajectories. It introduces token-probability-based confidence to characterize policy concentration and analyzes how reinforcing likely tokens and suppressing unlikely alternatives contribute to this process. Based on this analysis, the paper proposes ε\varepsilon-Policy Optimization (ε\varepsilon-PO), a timing-aware exploration control method that selectively suppresses concentration-inducing token updates when the estimated exploration space falls below a desired margin. An annealing mechanism progressively relaxes this intervention to facilitate the transition from exploration to exploitation. Experiments across four backbone models and five reasoning benchmarks demonstrate improvements in both Pass@1 and Pass@16 over the evaluated baselines, with training dynamics and ablation studies further supporting the effectiveness of the proposed method.

Critical strengths
  1. Novel perspective on exploration control. The paper distinguishes premature policy concentration from beneficial consolidation, motivating a timing-aware approach to balancing exploration and exploitation in RLVR.
  2. Principled optimization mechanism. The proposed ε\varepsilon-PO selectively suppresses concentration-inducing token updates based on rollout confidence, while an annealing strategy progressively shifts optimization toward exploitation.
  3. Comprehensive empirical validation. Experiments across four backbone models and five reasoning benchmarks demonstrate improvements in both Pass@1 and Pass@16. Training dynamics and ablation studies further support the effectiveness of the proposed method.
Critical weaknesses
  • Limited novelty and insufficient comparison with prior work. Several closely related studies have explored entropy collapse and exploration enhancement in RLVR. The authors should provide a systematic comparison with these works to clarify the novelty of the proposed method.
    • The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
    • Rethinking Entropy Regularization in Large Reasoning Models
    • Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
    • Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
    • EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  • Unclear model selection rationale. Why do the authors use Llama-3.2-3B-Instruct rather than a base model, unlike the other experimental settings? The rationale for this choice and its implications for experimental comparability should be clarified.
  • Limited performance gains and insufficient analysis of model scaling. The improvements are modest, with gains of only 1.15 for the 1.7B Base model and 0.39 for the 8B Base model over the baseline. Do these results suggest that the method becomes less effective as model size increases, and can the authors provide further analysis of this trend?
  • Unclear validity of the confidence metric. Why is confidence defined based on token probabilities, and can this metric reliably capture the available exploration space? The authors should provide theoretical or empirical evidence supporting this design choice.
  • Insufficient justification for token-update suppression and annealing. Why does suppressing specific token updates expand the exploration space, and could this intervention undermine effective learning? Is annealing necessary to mitigate such negative effects, and can the authors provide evidence supporting this design choice?
Questions for authors

Explain the concerns in Weaknesses Part.

Reviewer_NGjdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ϵ\epsilon-Policy Optimization (ϵ\epsilon-PO), a modification of GRPO intended to mitigate premature policy concentration during reinforcement learning with verifiable rewards. The method monitors the average probability of sampled tokens within a rollout batch. When this statistic exceeds a scheduled threshold, it masks positive-advantage token losses whose token probabilities also exceed that threshold. An annealed exploration margin progressively relaxes the intervention. The paper motivates this design using a collision-probability measure of concentration and local gradient analysis in logit space. Experiments on four backbone models and five reasoning benchmarks report improvements in average Pass@1 and Pass@16 over the evaluated baselines.

Critical strengths
  • The method uses sampled-token probabilities already available during GRPO training and modifies the objective through a binary mask. It requires neither an additional exploration loss nor full-vocabulary concentration calculations for batch monitoring.
  • The paper connects its concentration statistic to collision probability and Rényi entropy, and derives a sufficient condition under which the masked token losses contribute positively to local concentration drift. The appendix also explicitly acknowledges limitations arising from shared neural parameters and optimizer interactions.
  • The experiments cover four backbones and five benchmarks, reporting both Pass@1 and Pass@16. The margin, trigger, annealing, and token-selection ablations provide useful evidence about individual design choices. The reported average improvements across all four backbones are encouraging.
Critical weaknesses
  • Section 3 repeatedly changes the level of analysis: sampled-token probability, state-level concentration, sampled-coordinate effects, complete logit updates, batch-level drift, and the practical mask. These distinctions are technically important, but the many short paragraphs do not develop their connections sufficiently. In particular, the two mechanisms highlighted after Theorem 1 concern sampled-coordinate contributions, whereas the justification for the implemented mask depends on the complete-update analysis and the majority-token condition discussed later. The current organization requires readers to reconstruct this logical progression themselves.
  • The quantity 1−∑vπθ(v∣s)21-\sum_v\pi_\theta(v\mid s)^2 measures disagreement between two independent next-token samples at a fixed state. Its relationship to useful exploration of alternative reasoning trajectories remains unclear. Appendix A.4 acknowledges this distinction, but the main narrative repeatedly uses “exploration space” in a broader sense. Similarly, exceeding a scheduled concentration threshold does not by itself establish that concentration is premature or harmful. The paper would benefit from a clearer operational interpretation and evidence connecting the trigger to situations where intervention improves learning.
  • Proposition 2 establishes a relative reduction in first-order concentration drift under independently parameterized state logits and a local gradient step. As Appendix B.3 explains, shared neural parameters can change the signs of the relevant contributions, while Adam, gradient clipping, and repeated updates introduce further differences. This limitation is significant because the practical method trains neural policies with Adam. The reported training trajectories are compatible with the proposed explanation, but do not directly establish that the masked losses have the predicted effect under the actual optimizer. This scope should be more prominent in the main text.
  • Improvements over the strongest evaluated baseline are sometimes small. For example, average Pass@16 on Qwen3-4B increases from 74.00 to 74.37. Moreover, the validation accuracy used in Figures 1 and 3 is not sufficiently defined in the experimental description. Its evaluation set, sampling protocol, and relationship to Table 1 should be specified, particularly because the near-zero accuracy without the batch trigger is an important ablation result.
Questions for authors
  • What dataset and evaluation protocol produce the validation accuracy in Figures 1 and 3? How are the checkpoints in Table 1 selected, and how many independent training runs underlie each reported result?
  • Can you compare confidence-based triggering with a training-step-based intervention schedule matched approximately for the amount of masking, or with a relevant adaptive exploration baseline? This would help isolate the value of the proposed timing criterion.
  • Is there direct evidence, evaluated on a fixed diagnostic state set, that the selected losses increase concentration under the actual neural parameterization and optimizer, and that masking them reduces this increase?
  • What motivates using the same threshold for batch activation and token selection? How should the resulting batch statistic be interpreted when high-confidence tokens include predictable formatting or linguistic tokens unrelated to substantive reasoning choices?
Reviewer_R1cPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses premature policy concentration in RLVR, which may limit further exploration, and proposes ε-PO. The method averages sampled-token probabilities over the full rollout batch. When this average exceeds a threshold that increases with training progress, it masks positive-advantage token whose sampled-token probabilities exceed the same threshold, while the remaining tokens still contribute to original GRPO loss. The exploration margin is annealed linearly to relax intervention with training proceeds.

Critical strengths
  1. The method is simple and computationally efficient. It reuses rollout probabilities and retains GRPO’s advantage estimator and sampling procedure. The added operations mainly involve computing statistics and applying masks, with little computational overhead.
  2. The reported results show some effectiveness. Table 1 reports the highest average Pass@1 and Pass@16 for the method across all four models. In addition, the paper includes some ablation studies on design chocies such as the initial exploration margin, the batch trigger, annealing, and alternative token-selection rules.
Critical weaknesses
  1. The incremental contribution over closely related methods needs stronger validation. UloRL [1] already conditionally masks high-confidence positive tokens, while [2] studies scheduled and entropy-guided positive-advantage reweighting. The added value of the full-batch trigger, shared threshold, and linear annealing remains unclear without direct comparisons with these alternatives. Existing ablations support the components in the tested setting, but change both intervention timing and the overall amount of masking. Comparing against a simple intervention schedule with the same token-selection rule and comparable total masking would better isolate the benefit of adaptive timing. The local theory supports selective masking but does not establish the advantages of these specific control choices.

  2. Table 1 contains an inconsistency under the stated evaluation protocol. For Llama-3.2-3B-Instruct / GRPO / AIME25, Pass@1 = 0.23% and Pass@16 = 4.30%. When computed from the same sampled responses with the same per-problem weights, Eq. (28) requires Pass@16 ≤ 16 × Pass@1. However, 4.30% exceeds 3.68%, and rounding cannot explain the discrepancy. Please verify these entries and clarify whether the reported metrics follow Eq. (28).

[1] Du, Dong, et al. “UloRL: An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models’ Reasoning Abilities.” arXiv:2507.19766, 2025.

[2] Jin, Renren, et al. “Revisiting Entropy in Reinforcement Learning for Large Reasoning Models.” Findings of ACL, 2026.

Questions for authors
  1. Could the authors provide additional ablations to isolate the gains from the full-batch trigger, shared threshold, and annealing schedule? Comparisons with closely related alternatives and a simple schedule with a comparable masking budget would help establish their added value.

  2. Could the authors reconcile the inconsistent results in Table 1 with Eq. (28)?

  3. How frequently are masks recomputed during training? If rollout-time masks are reused across multiple policy updates, how does mask staleness affect training stability and performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.