- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reinforcement Learning with Promising Tokens (RLPT), a framework that aligns rollout sampling and policy optimization within a state-dependent subset of high-probability tokens. Specifically, RLPT constructs a Top-K token mask from the behavior policy and applies the same mask during both rollout and optimization, aiming to reduce unnecessary gradient updates on low-probability tokens. The authors provide empirical analyses showing that successful reasoning trajectories are largely concentrated within high-probability token sets, along with a theoretical observation regarding the elimination of masked-token gradients. Experiments across mathematical reasoning, code generation, and domain-specific QA tasks demonstrate improvements over GRPO and DAPO, particularly on mathematical reasoning benchmarks, while ablation studies examine the effects of support alignment and token-set size.
Critical strengths
Clear motivation and systematic empirical evaluation. The paper identifies a practical inconsistency between truncated token sampling during rollout and full-vocabulary policy optimization. The proposed approach is simple and can be integrated into existing RL algorithms without modifying their underlying objectives. The experiments cover multiple reasoning tasks and include controlled ablations and multi-seed evaluations, providing useful empirical evidence for the benefits of aligning rollout and optimization supports.
Critical weaknesses
1. Limited novelty and insufficient comparison with prior work. The core idea of RLPT—preserving the token mask used during truncated rollout and reapplying it during policy optimization—has already been introduced in DeepSeek-V3.2 (2025) as Keep Sampling Mask. Both methods address the same mismatch between rollout and optimization distributions using essentially the same mechanism. However, the paper neither discusses this directly relevant prior work nor provides a comparison that establishes a substantive methodological difference. Furthermore, the theoretical result regarding zero gradients for masked tokens follows directly from standard masked softmax operations. These issues substantially weaken the paper's claimed novelty.
2. Limited empirical gains and insufficient evidence for the broader claims. Although RLPT improves performance on several mathematical reasoning benchmarks, the gains over strong truncated-rollout baselines are relatively modest, and the cross-domain results are inconsistent, with performance degradation in some configurations. Moreover, the analysis of promising tokens focuses on token-level coverage rather than whether complete successful reasoning trajectories remain accessible under restrictive Top-K masking. Consequently, the experiments do not sufficiently establish that the proposed decision-centric formulation provides robust benefits beyond existing support-alignment techniques.
Questions for authors
1. Methodological novelty. How does RLPT fundamentally differ from the Keep Sampling Mask method introduced in DeepSeek-V3.2 (2025)?
2. Missing prior work. Why is DeepSeek-V3.2 not discussed or compared against, despite addressing the same rollout–optimization mismatch?
3. Empirical significance. Are the improvements over Top-K/Top-P rollout baselines statistically significant across multiple random seeds?
4. Exploration limitations. How does the method ensure that restrictive Top-K masking does not exclude essential tokens required for successful reasoning?
2 comments
Author Response (Part 2/2)
Q4: Limited empirical gains and insufficient evidence for the broader claims...
Response: Our primary focus is mathematical reasoning, where the decision-centric formulation is most directly motivated. Table 4 already reports improvements across four mathematical benchmarks over five independent runs, while the code and telecom experiments in Table 5 are intended to examine compatibility rather than establish universal cross-domain superiority.
Furthermore, the controlled ablation in Table 7 shows that RLPT (38.3%) outperforms both Top-k rollout with full-vocabulary optimization (36.9%) and Top-p rollout with full-vocabulary optimization (37.7%) on Math-17k. Thus, the observed benefit cannot be attributed solely to truncating the rollout distribution.
Q5: Empirical significance. Are the improvements over Top-K/Top-P rollout baselines statistically significant across multiple random seeds?
Response: To examine the reliability of the improvements, we conducted five independent training runs for GRPO and the corresponding rollout/optimization ablations on Math-17k using Qwen3-1.7B. We used this smaller backbone due to computational constraints.
| Method | Accuracy (%) |
|---|---|
| GRPO | 28.14 ± 0.94 |
| Top-k rollout + full update | 27.95 ± 1.02 |
| Top-p rollout + full update | 28.13 ± 1.53 |
| Full rollout + masked update | 27.81 ± 0.45 |
| RLPT | 29.26 ± 1.52 |
RLPT achieves the highest mean accuracy across all compared methods. This provides additional multi-seed evidence that the benefit is not limited to a single experimental run. Together with the five-seed results on the larger backbone already reported in Table 4, these additional experiments strengthen the empirical evidence for RLPT's effectiveness.
We hope that our response has addressed your concern and questions satisfactorily. If you had any further concerns, we are glad for discussion.
Author Response (Part 1/2)
Thank you for your comments. We fully understand your concerns about the relation to 'Keep Sampling Mask' proposed in DeepSeek-V3.2 report. We believe this is a misunderstanding and we address your concerns as follow.
Response: We would like to clarify that the primary goal of RLPT is to reformulate LLM RL as decision-making within a compact, state-dependent subspace of promising tokens, rather than resolving the rollout–optimization mismatch. We recognize that DeepSeek-V3.2 employs a similar mask-preservation mechanism. However, its Keep Sampling Mask is primarily introduced to improve training consistency and stability, while our work investigates a different research question: Can RL become more effective by learning to select among plausible reasoning actions, rather than optimizing over the entire vocabulary? Mask preservation is a necessary implementation component of this formulation, but it is not the central scientific contribution we seek to establish.
To further examine this question, we conducted an additional experiment using an adaptive promising-token set, defined at each generation step as the larger of the Top-8 set and the Top-p set (p = 0.95). The selected mask is consistently applied during both rollout and optimization.
The adaptive strategy achieves the best performance across all three benchmarks. Importantly, all three RLPT variants preserve the sampling mask during optimization. Their performance differences therefore highlight the importance of constructing an appropriate decision subspace, beyond the mask-preservation operation itself. In particular, the adaptive results demonstrate that RLPT is not inherently restricted to a fixed Top-k candidate set.
Response: Regarding your concern about theoritical analysis, we clarify that this work does not intend to claim that zero gradients for masked logits constitute a novel mathematical property. Rather, the proposition provides a theoretical interpretation of RLPT: optimization is concentrated on promising token actions instead of distributing gradient components over irrelevant tail tokens. Its practical implications are investigated through the empirical results and gradient analysis in Section 6.
Response: Good point about successful answer coverage from top-k sampling. While Section 3 demonstrates that successful reasoning tokens are highly concentrated among promising candidates, we agree that token-level coverage alone does not guarantee the reachability of a complete successful trajectory.
To directly investigate this concern, we conducted additional experiments comparing Top-4, Top-8, Top-16, and full-vocabulary sampling across two model scales and three mathematical reasoning benchmarks.
The results show that Top-k sampling can achieve end-to-end reasoning accuracy comparable to full-vocabulary sampling. In particular, Top-4 differs from full-vocabulary sampling by at most 0.69 percentage points across all six model–dataset combinations. Unlike token-level coverage, these results directly evaluate whether the restricted action space can produce complete correct answers. They suggest that reducing the sampling space does not necessarily compromise reasoning capability in the evaluated settings.