acceptodds
Under review as a conference paper at ICLR 2027

Decision-Centric Reinforcement Learning over Promising Tokens for Large Language Model Reasoning

Abstract

Reinforcement learning (RL) for large language models (LLMs) is commonly formulated as token-level policy optimization over the full vocabulary. This fixed action space is tens of thousands of tokens wide, although only a small state-dependent subset typically represents plausible next reasoning moves. Existing decoding strategies such as Top-k or nucleus sampling exploit this structure during rollout, but standard RL objectives still compute policy ratios and gradients with respect to the full-vocabulary policy, creating a mismatch between the space where trajectories are collected and the space where the policy is optimized. In this work, we study LLM generation as decision-making over a fixed global action space with a dynamic local decision set. We introduce Reinforcement Learning with Promising Tokens (RLPT), which uses the behavior policy’s own distribution to construct a promising token set at each state and defines a masked policy consistently for both rollout and optimization. This simple policy reparameterization turns token-level RL from full-vocabulary fitting into action selection among plausible reasoning candidates. Across mathematical reasoning, code, and domain QA tasks, RLPT improves GRPO and DAPO, with multi-seed results on Math-17k, MATH500, AIME24, and AIME25 showing improved accuracy and reduced training variability. Controlled ablations further show that the gains come from aligning rollout and optimization supports, rather than from decoding-time truncation alone.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Large language models, Promising set

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VaBCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reinforcement Learning with Promising Tokens (RLPT), a framework that aligns rollout sampling and policy optimization within a state-dependent subset of high-probability tokens. Specifically, RLPT constructs a Top-K token mask from the behavior policy and applies the same mask during both rollout and optimization, aiming to reduce unnecessary gradient updates on low-probability tokens. The authors provide empirical analyses showing that successful reasoning trajectories are largely concentrated within high-probability token sets, along with a theoretical observation regarding the elimination of masked-token gradients. Experiments across mathematical reasoning, code generation, and domain-specific QA tasks demonstrate improvements over GRPO and DAPO, particularly on mathematical reasoning benchmarks, while ablation studies examine the effects of support alignment and token-set size.

Critical strengths

Clear motivation and systematic empirical evaluation. The paper identifies a practical inconsistency between truncated token sampling during rollout and full-vocabulary policy optimization. The proposed approach is simple and can be integrated into existing RL algorithms without modifying their underlying objectives. The experiments cover multiple reasoning tasks and include controlled ablations and multi-seed evaluations, providing useful empirical evidence for the benefits of aligning rollout and optimization supports.

Critical weaknesses

1. Limited novelty and insufficient comparison with prior work. The core idea of RLPT—preserving the token mask used during truncated rollout and reapplying it during policy optimization—has already been introduced in DeepSeek-V3.2 (2025) as Keep Sampling Mask. Both methods address the same mismatch between rollout and optimization distributions using essentially the same mechanism. However, the paper neither discusses this directly relevant prior work nor provides a comparison that establishes a substantive methodological difference. Furthermore, the theoretical result regarding zero gradients for masked tokens follows directly from standard masked softmax operations. These issues substantially weaken the paper's claimed novelty.

2. Limited empirical gains and insufficient evidence for the broader claims. Although RLPT improves performance on several mathematical reasoning benchmarks, the gains over strong truncated-rollout baselines are relatively modest, and the cross-domain results are inconsistent, with performance degradation in some configurations. Moreover, the analysis of promising tokens focuses on token-level coverage rather than whether complete successful reasoning trajectories remain accessible under restrictive Top-K masking. Consequently, the experiments do not sufficiently establish that the proposed decision-centric formulation provides robust benefits beyond existing support-alignment techniques.

Questions for authors

1. Methodological novelty. How does RLPT fundamentally differ from the Keep Sampling Mask method introduced in DeepSeek-V3.2 (2025)?

2. Missing prior work. Why is DeepSeek-V3.2 not discussed or compared against, despite addressing the same rollout–optimization mismatch?

3. Empirical significance. Are the improvements over Top-K/Top-P rollout baselines statistically significant across multiple random seeds?

4. Exploration limitations. How does the method ensure that restrictive Top-K masking does not exclude essential tokens required for successful reasoning?

2 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

Thank you for your comments. We fully understand your concerns about the relation to 'Keep Sampling Mask' proposed in DeepSeek-V3.2 report. We believe this is a misunderstanding and we address your concerns as follow.

Q1: Limited novelty and insufficient comparison with prior work ...

Response: We would like to clarify that the primary goal of RLPT is to reformulate LLM RL as decision-making within a compact, state-dependent subspace of promising tokens, rather than resolving the rollout–optimization mismatch. We recognize that DeepSeek-V3.2 employs a similar mask-preservation mechanism. However, its Keep Sampling Mask is primarily introduced to improve training consistency and stability, while our work investigates a different research question: Can RL become more effective by learning to select among plausible reasoning actions, rather than optimizing over the entire vocabulary? Mask preservation is a necessary implementation component of this formulation, but it is not the central scientific contribution we seek to establish.

To further examine this question, we conducted an additional experiment using an adaptive promising-token set, defined at each generation step as the larger of the Top-8 set and the Top-p set (p = 0.95). The selected mask is consistently applied during both rollout and optimization.

MethodMath-17kAIME24AIME25
No Mask (GRPO)34.720.019.3
RLPT + Top-k38.322.519.5
RLPT + Top-p35.622.416.7
RLPT + Adaptive39.223.221.3

The adaptive strategy achieves the best performance across all three benchmarks. Importantly, all three RLPT variants preserve the sampling mask during optimization. Their performance differences therefore highlight the importance of constructing an appropriate decision subspace, beyond the mask-preservation operation itself. In particular, the adaptive results demonstrate that RLPT is not inherently restricted to a fixed Top-k candidate set.

Q2: the theoretical result regarding zero gradients for masked tokens follows directly from standard masked softmax operations ...

Response: Regarding your concern about theoritical analysis, we clarify that this work does not intend to claim that zero gradients for masked logits constitute a novel mathematical property. Rather, the proposition provides a theoretical interpretation of RLPT: optimization is concentrated on promising token actions instead of distributing gradient components over irrelevant tail tokens. Its practical implications are investigated through the empirical results and gradient analysis in Section 6.

Q3: How does the method ensure that restrictive Top-K masking does not exclude essential tokens required for successful reasoning?

Response: Good point about successful answer coverage from top-k sampling. While Section 3 demonstrates that successful reasoning tokens are highly concentrated among promising candidates, we agree that token-level coverage alone does not guarantee the reachability of a complete successful trajectory.

To directly investigate this concern, we conducted additional experiments comparing Top-4, Top-8, Top-16, and full-vocabulary sampling across two model scales and three mathematical reasoning benchmarks.

ModelDatasetTop-4Top-8Top-16Full Vocabulary
Qwen3-1.7BGSM8K80.4780.7678.5279.79
Qwen3-1.7BMATH50070.8070.0269.8271.39
Qwen3-1.7BMath-17k30.8632.1329.8830.86
Qwen3-4BGSM8K90.3390.7291.6091.02
Qwen3-4BMATH50081.3581.2580.5780.66
Qwen3-4BMath-17k46.9748.0548.0547.07

The results show that Top-k sampling can achieve end-to-end reasoning accuracy comparable to full-vocabulary sampling. In particular, Top-4 differs from full-vocabulary sampling by at most 0.69 percentage points across all six model–dataset combinations. Unlike token-level coverage, these results directly evaluate whether the restricted action space can produce complete correct answers. They suggest that reducing the sampling space does not necessarily compromise reasoning capability in the evaluated settings.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Q4: Limited empirical gains and insufficient evidence for the broader claims...

Response: Our primary focus is mathematical reasoning, where the decision-centric formulation is most directly motivated. Table 4 already reports improvements across four mathematical benchmarks over five independent runs, while the code and telecom experiments in Table 5 are intended to examine compatibility rather than establish universal cross-domain superiority.

Furthermore, the controlled ablation in Table 7 shows that RLPT (38.3%) outperforms both Top-k rollout with full-vocabulary optimization (36.9%) and Top-p rollout with full-vocabulary optimization (37.7%) on Math-17k. Thus, the observed benefit cannot be attributed solely to truncating the rollout distribution.

Q5: Empirical significance. Are the improvements over Top-K/Top-P rollout baselines statistically significant across multiple random seeds?

Response: To examine the reliability of the improvements, we conducted five independent training runs for GRPO and the corresponding rollout/optimization ablations on Math-17k using Qwen3-1.7B. We used this smaller backbone due to computational constraints.

MethodAccuracy (%)
GRPO28.14 ± 0.94
Top-k rollout + full update27.95 ± 1.02
Top-p rollout + full update28.13 ± 1.53
Full rollout + masked update27.81 ± 0.45
RLPT29.26 ± 1.52

RLPT achieves the highest mean accuracy across all compared methods. This provides additional multi-seed evidence that the benefit is not limited to a single experimental run. Together with the five-seed results on the larger backbone already reported in Table 4, these additional experiments strengthen the empirical evidence for RLPT's effectiveness.


We hope that our response has addressed your concern and questions satisfactorily. If you had any further concerns, we are glad for discussion.

Reviewer_YbVKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that token-level RL collects trajectories under a truncated decoding distribution but computes policy ratios and gradients over the full vocabulary, so data collection and optimization use different action supports. It proposes RLPT, which builds a top-K promising-token set from the behavior policy at each state, stores the mask with the trajectory, and evaluates both the behavior and the current policy through that mask during the update. The method is algorithm-agnostic and adds no auxiliary model. Experiments on math, code, and telecom QA report gains over GRPO and DAPO, with five-seed math results and an ablation separating rollout truncation from masked optimization.

Critical strengths
  1. Truncated decoding changes the effective behavior policy while the surrogate still references the untruncated one. That inconsistency is real and rarely discussed, and the fix is cheap and specified precisely enough to reimplement. I also credit holding the mask fixed within an update window, since recomputing it from the evolving policy could place a sampled action outside the new support and leave the ratio undefined.

  2. The main math table reports standard deviations over five runs rather than a single number. Recomputing it, the gains over GRPO on Math-17k, MATH500, and AIME25 hold at roughly p=0.03p = 0.03, 0.0080.008, and 0.020.02, while AIME24 does not separate at p=0.22p = 0.22. The paper also states plainly that there is no wall-clock saving, which avoids a common overclaim.

Critical weaknesses
  1. The paper's second claim, that the gain comes from support alignment rather than from decoding-time truncation, is not supported at the reported noise level. In Table 7 the relevant contrast is RLPT at 38.3 against top-p rollout with full-vocabulary optimization at 37.7, a gap of 0.6, while the seed standard deviations on this benchmark are 1.36 for GRPO and 0.61 for RLPT. The table also mixes a five-seed mean for GRPO with what appear to be single runs elsewhere. Since this ablation is what separates the work from ordinary truncated decoding, it needs the same multi-seed treatment as Table 4. The same configuration is also reported with different numbers across tables. For K=4K = 4, AIME25 appears as 21.93, 18.0, and 19.5 in Tables 4, 5, and 8, and AIME24 as 23.23, 23.3, and 22.5. I could not determine which runs the ablations draw on.

  2. The method restricts the optimized support to four tokens out of a vocabulary above 50k and sets the entropy coefficient to zero, yet reports no entropy curve, no pass@k, and no measurement of how coverage evolves. The coverage study runs on the base model only, so it speaks to initialization rather than to the regime after 200 steps of masked updates, which is where support collapse would appear. Table 3 also reports exactly 100 percent top-8 coverage for every domain and model, which would be partly circular if those samples were drawn with truncated decoding. The decoding configuration is not stated.

Questions for authors
  • Can the Table 7 ablation be rerun with the same five seeds as Table 4, with standard deviations for every row? Does the gap over top-p rollout survive?

  • What decoding configuration produced the model-generated solutions in Table 3, and were those samples drawn without truncation?

  • For the top-k rollout with full-vocabulary optimization baseline, is the ratio computed against the truncated behavior distribution or the untruncated one? That separates correcting the importance ratio from restricting the gradient support.

3 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

Thank you for your careful review. We understand your concerns about the statistical robustness and the distinction between truncated decoding and support-aligned optimization. We clarify the experimental settings and provide additional multi-seed ablations and decoding comparisons below.

Q1: The paper's second claim, that the gain comes from support alignment rather than from decoding-time truncation, is not supported at the reported noise level ...

A1: We apologize for the confusion caused by the inconsistent reporting. The differences arise from distinct experimental runs and reporting protocols, which were not sufficiently distinguished in the manuscript.

Specifically, Table 4 reports the mean and standard deviation over five independent runs, whereas Table 5 reports single-run results for cross-domain comparisons. Tables 7 and 8 contain separately executed single-run ablations. This explains the differences in AIME24 and AIME25 results across tables. In particular, Table 7 uses the multi-seed GRPO mean alongside single-run ablation results, which makes the comparison less consistent than intended. We will make the run provenance and seed settings explicit, and standardize the reporting with Table 4's five-seed protocol as the primary reference.

Regarding the 0.6-point gap over Top-p rollout, we recognize that a single-run comparison is insufficient to establish statistical robustness. To address this concern, we conducted an additional multi-seed ablation on Qwen3-1.7B, reported in Q3. The new results provide further evidence for the benefit of support-aligned optimization, while allowing a more careful interpretation of the observed variance.

Q2: The method restricts the optimized support to four tokens out of a vocabulary above 50k and sets the entropy coefficient to zero ...

Response: We understand the concern that a small promising set might constrain exploration. However, we would like to clarify that RLPT does not restrict the model to a set of promising tokens. As described in Section 4.2, the promising set is state-dependent and is recomputed from the current behavior policy for each new rollout batch. Thus, the available candidate tokens evolve with both the generation context and policy updates.

To further examine whether restricting the sampling support compromises solution quality, we conducted additional experiments comparing Top-4, Top-8, Top-16, and full-vocabulary sampling across two model sizes and three mathematical reasoning benchmarks.

ModelDatasetTop-4Top-8Top-16Full Vocabulary
Qwen3-1.7BGSM8K80.4780.7678.5279.79
Qwen3-1.7BMATH50070.8070.0269.8271.39
Qwen3-1.7BMath-17k30.8632.1329.8830.86
Qwen3-4BGSM8K90.3390.7291.6091.02
Qwen3-4BMATH50081.3581.2580.5780.66
Qwen3-4BMath-17k46.9748.0548.0547.07

The results show that Top-4 sampling achieves accuracy comparable to full-vocabulary sampling across all six settings, with a maximum absolute difference of only 0.69 percentage points. These results suggest that restricting decoding to a compact set of high-probability candidates does not necessarily sacrifice answer quality. This supports the feasibility of using a small promising set for reasoning-oriented tasks. The additional experiments address the quality of truncated sampling, rather than directly measuring post-training entropy or pass@k.

Finally, the model-generated solutions in Table 3 were sampled without vocabulary truncation. Therefore, the reported Top-8 coverage was measured retrospectively from full-vocabulary generations, rather than being enforced by the decoding strategy. We clarify this further in Q4.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Q3: Can the Table 7 ablation be rerun with the same five seeds as Table 4, with standard deviations for every row? Does the gap over top-p rollout survive?

A3: To examine the robustness of the ablation results, we reran all five configurations across multiple seeds using Qwen3-1.7B. Due to computational constraints, this additional experiment uses a smaller backbone than the Qwen3-8B model in Table 4.

MethodAccuracy (%)
GRPO28.14 ± 0.94
Top-k rollout + full-vocabulary optimization27.95 ± 1.02
Top-p rollout + full-vocabulary optimization28.13 ± 1.53
Full rollout + masked optimization27.81 ± 0.45
RLPT (Top-k rollout + masked optimization)29.26 ± 1.52

RLPT achieves the highest mean accuracy among all five configurations. Notably, neither rollout truncation alone nor optimization masking alone matches the mean performance of RLPT. This reproduces the directional pattern observed in the original Qwen3-8B ablation, suggesting that the combination of truncated rollout and support-consistent optimization is beneficial beyond decoding-time truncation alone. We believe these results as additional evidence consistent with the support-alignment hypothesis.

Q4: What decoding configuration produced the model-generated solutions in Table 3, and were those samples drawn without truncation?

A4: The model-generated solutions in Table 3 were obtained using full-vocabulary sampling, without Top-k or Top-p truncation. The Top-K coverage statistics were computed post hoc by measuring the ranks of generated tokens under the model's predicted distribution. Therefore, the 100% Top-8 coverage was not imposed by the sampling procedure, but reflects the observed concentration of tokens in successful trajectories. We will explicitly state the full-vocabulary decoding setting in the revised manuscript to avoid this ambiguity.

Q5: For the top-k rollout with full-vocabulary optimization baseline, is the ratio computed against the truncated behavior distribution or the untruncated one?

A5: In the Top-k rollout + full-vocabulary optimization baseline, the importance ratio is computed using the untruncated policy distributions, even though trajectories are sampled with Top-k decoding. This distinction is central to RLPT. In the decoding-only baseline, the optimization objective still evaluates the original full-vocabulary policy. In contrast, RLPT computes both the behavior and current policy probabilities using the same stored, masked, and renormalized support.


We hope that these responses can address your concerns and questions. If you had any further concerns, please let us know.

AuthorsOpenReview ↗

Request for Follow-up Discussions

Hi Reviewer YbVK,

As the discussion period deadline approaches, we would like to follow up if our response has addressed your concerns? We are glad for discussions if you have any questions.

Submission #51462 Author

Reviewer_KMxmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RLPT restricts each next-token decision to the behavior model’s highest-probability candidates, normally four tokens. During training, it:

  1. Generates responses by sampling from this restricted, renormalized distribution.
  2. Stores the candidate set for every generated token.
  3. Computes rewards and advantages using the underlying GRPO or DAPO procedure.
  4. Reuses the stored candidate sets when computing policy ratios and regularization during optimization.

The candidate sets remain fixed while updating on a collected batch and are refreshed for new rollouts. The intended contribution is consistent treatment of the sampling and training distributions. At inference, the trained model generates answers autoregressively.

This is a simple modification that requires no auxiliary selector model. The strongest positive evidence is the comparison in Table 4.

Critical strengths

RLPT offers a practical modification to existing RL pipelines. The reward design and advantage estimator remain those of the underlying algorithm, which makes the approach compatible with different RL objectives. Its integration with both GRPO and DAPO illustrates this practical flexibility.

The mathematical reasoning results suggest that restricting optimization to very small candidate sets can be useful in practice. With 4 candidates per step, the experiments improve mean accuracy on all 4 math benchmarks. For example, Math-17k increases from 35.78% to 37.78%, and MATH500 from 56.64% to 60.12% (Table 4). The additional improvements on Qwen3-4B and Llama-3.1-8B-Instruct provide preliminary evidence that the approach may also be useful beyond the main Qwen3-8B setting (Table 6).

Critical weaknesses
  1. The central mechanism overlaps directly with uncited prior work. DeepSeek-V3.2 already describes preserving top-k/top-p masks during rollout and applying those masks to the current policy during training so that both policies share the same action space [1]. This is substantially closer than generic invalid-action masking.
  2. The full-rollout/masked-optimization baseline needs an explanation of how sampled actions outside the mask are handled. Its poor performance cannot be interpreted cleanly without that detail.
  3. Token coverage does not establish that complete solutions remain reachable. Section 3 uses high token-level coverage to motivate the sufficiency of the restricted action space. However, a thousand-token solution with one indispensable excluded token has 99.9% token coverage and still cannot be reproduced. Table 2’s 98.15% coverage at the actual default of four candidates therefore does not establish complete-path retention or the availability of an alternative correct solution. The appendix acknowledges rare-token failures; the unresolved issue is whether the coverage analysis supports the stronger empirical interpretation. Table 3 also omits the sampling settings used to generate successful responses, which matter when interpreting its 100% top-8 coverage.
  4. Missing evaluation details affect the interpretation of the gains. The paper does not clearly identify the exact Qwen3 checkpoint and thinking mode, test-time sampling settings, evaluation sample counts, or whether all methods use the same decoder. These details are particularly important because changing the decoder is itself an intervention studied here. Appendix G.1 also describes Math-17k as both training and in-domain evaluation data without specifying the split.
Questions for authors

see above

4 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

Thank you for the thoughtful review and for engaging closely with the mechanism behind RLPT. We would like to clarify the central motivation of our work, address the implementation and evaluation details you raised, and share additional experimental results that help establish the effectiveness of decision-making within a restricted action space.

Q1: The central mechanism overlaps directly with uncited prior work. DeepSeek-V3.2 already describes preserving top-k/top-p masks during rollout ...

A1: We acknowledge that your mentioned mask is relevant to RLPT, as it preserves rollout masks and reapplies them during optimization, an operation shared by RLPT. We will cite this work explicitly.

However, we would like to clarify that the primary goal of this work is to study how RL benefits from making decisions within compact, state-dependent sets of plausible tokens. Support alignment implements this formulation, while top-k is one way to construct the candidate set. Our contribution lies in the systematic investigation of this formulation, including its optimization behavior (Section 4.4), controlled ablations (Section 6.4), and evaluation across RL algorithms and models.

To test whether the benefits extend beyond fixed top-k sets, we additionally evaluate an adaptive construction. At each step, it selects the larger of the top-8 set and the top-p nucleus with p = 0.95 under the behavior policy. This retains at least eight candidates while allowing the set to expand when needed.

MethodMath-17kAIME24AIME25
GRPO (No Mask)34.720.019.3
RLPT + Top-k (k = 4)38.322.519.5
RLPT + Top-p (p = 0.95)35.622.416.7
RLPT + Adaptive39.223.221.3

The adaptive variant improves over unmasked GRPO by 4.5, 3.2, and 2.0 percentage points on Math-17k, AIME24, and AIME25, respectively, and achieves the highest accuracy on all three benchmarks. All three RLPT variants preserve rollout masks during optimization. We believe that these results support the flexibility of the decision-space formulation and show that its benefits are not confined to a fixed top-k construction.

Q2: The full-rollout/masked-optimization baseline needs an explanation of how sampled actions outside the mask are handled ...

A2: In this ablation, a token sampled outside the top-k mask is excluded from the token-level policy-gradient loss. The update for that action is skipped, and the token remains part of the generated trajectory. We will make this handling explicit. This configuration discards learning signals from some sampled actions, so its low score could be interpreted as a diagnostic of this particular mismatched rollout/update configuration. The comparison in Table 8 also supports the benefit of masked optimization. With top-k rollouts, RLPT reaches 38.3%, compared with 36.9% for full-vocabulary optimization. Thus, the empirical case does not depend solely on the poorly performing full-rollout/masked-optimization baseline.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Q3: Token coverage does not establish that complete solutions remain reachable. Section 3 uses high token-level coverage to motivate the sufficiency of the restricted action space. However, a thousand-token solution with one indispensable excluded token has 99.9% token coverage and still cannot be reproduced. Table 2’s 98.15% coverage at the actual default of four candidates therefore does not establish complete-path retention or the availability of an alternative correct solution. The appendix acknowledges rare-token failures; the unresolved issue is whether the coverage analysis supports the stronger empirical interpretation. Table 3 also omits the sampling settings used to generate successful responses, which matter when interpreting its 100% top-8 coverage.

A3: Good point. Token-level coverage measures how concentrated successful trajectories are, but it does not directly establish whether a restricted decoder can generate complete correct solutions. To address this more directly, we conducted additional end-to-end answer accuracy experiments comparing Top-4, Top-8, Top-16, and full-vocabulary sampling across two model sizes and three mathematical reasoning benchmarks.

ModelDatasetTop-4Top-8Top-16Full Vocabulary
Qwen3-1.7BGSM8K80.4780.7678.5279.79
Qwen3-1.7BMATH50070.8070.0269.8271.39
Qwen3-1.7BMath-17k30.8632.1329.8830.86
Qwen3-4BGSM8K90.3390.7291.6091.02
Qwen3-4BMATH50081.3581.2580.5780.66
Qwen3-4BMath-17k46.9748.0548.0547.07

The results show that Top-k sampling can achieve end-to-end reasoning accuracy comparable to full-vocabulary sampling. In particular, Top-4 differs from full-vocabulary sampling by at most 0.69 percentage points across all six model–dataset combinations. Unlike token-level coverage, these results directly evaluate whether the restricted action space can produce complete correct answers. They suggest that reducing the sampling space does not necessarily compromise reasoning capability in the evaluated settings.

We do not claim that every correct trajectory is preserved. As already discussed in Appendix D.1, rare but essential tokens can still fall outside the promising set. Our empirical claim is that sufficiently good complete solutions remain accessible under compact candidate sets for the reasoning tasks studied here.

Q4: Missing evaluation details affect the interpretation of the gains ...

A4: Thank you for pointing out these reproducibility details. Some settings are already provided: Section 6.1 specifies Qwen3-8B as the default backbone, K = 4, and 200 training steps; Appendix G.4 reports the shared training hyperparameters; and Table 4 reports results over five independent runs. Importantly, Table 7 already controls for rollout decoding when isolating the effect of optimization-side masking: Top-k rollout + full-vocabulary optimization and Top-k rollout + masked optimization (RLPT) use the same rollout truncation strategy. Their performance difference therefore cannot be attributed solely to changing the rollout decoder.

However, we acknowledge that the current presentation lacks several details needed for full reproducibility. We will expand the experimental settings to specify the exact checkpoint identifiers and thinking-mode configurations, test-time decoding parameters, evaluation sample counts, and the decoder settings used for each comparison. We will also clarify the precise training/evaluation partition of Math-17k and its data-disjointness status, rather than leaving the split implicit.


Thanks again for the careful review. We are glad to any further discussions.

Thank you for the response and additional experiments. However, some concerns remain:

  1. The authors acknowledge the shared mechanism with DeepSeek-V3.2 [1], but the additional contribution remains insufficiently established. The adaptive results provide encouraging evidence of flexibility, although comparison with unmasked GRPO changes both rollout and optimization distributions and does not isolate the benefit of mask reuse.
  2. The exact checkpoints and thinking modes, test-time decoding settings, evaluation sample counts, and Math-17k training/evaluation split and disjointness are promised but not supplied. Matching rollout decoding does not establish a common test-time protocol or held-out evaluation. The sampling settings behind the original Table 3 also remain unanswered.

No updated manuscript incorporating the promised changes is available. I cannot verify the promised additions to the paper. Given the remaining concerns about the contribution, I maintain my rating.

[1] DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. DeepSeek-AI et al. arXiv, 2025, Section 3.1. https://arxiv.org/html/2512.02556v1

Thank you for your careful follow-up. We appreciate your remaining concerns and have revised the manuscript to make our contributions and experimental protocols clearer and more verifiable. Please check the revised version for the update.

1. Contribution and controlled comparisons. We now explicitly acknowledge the shared mask-preservation mechanism with DeepSeek-V3.2 in Section 5.2. Our contribution focuses on the systematic investigation of compact, state-dependent decision spaces for RL, rather than claiming novelty in mask reuse itself. We agree that the adaptive comparison examines candidate-set construction rather than isolating mask reuse. For the latter, Table 7 compares Top-k rollout with full-vocabulary versus masked optimization (36.9% vs. 38.3%), while the additional five-seed ablation in Table 14 provides further supporting evidence. We have clarified these distinctions and their limitations in Sections 5.2 and 6.4.

2. Reproducibility and evaluation details. The revised manuscript now specifies the default evaluation protocol, including thinking mode, decoding parameters, and sample counts (Appendix G.5); the Math-17k training/evaluation partition and limitations regarding problem-level disjointness (Appendix G.6); and run provenance and baseline implementations (Appendix G.7). We also clarify that Table 3 uses full-vocabulary sampling with temperature 1.0 and a 2048-token limit (Section 3.2 and Appendix H.2). Remaining metadata limitations, including unavailable exact checkpoint revisions, are explicitly acknowledged.


We hope these revisions provide a clearer and more transparent account of our contributions. Thank you again for your thoughtful feedback and consideration. If you have any further concerns, please let us know.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We sincerely thank the reviewers and Area Chair for their constructive feedback. We appreciate the concerns raised regarding novelty, empirical robustness, and the feasibility of restricted action spaces. In our responses, we have clarified the scope of our contributions and provided additional experimental evidence addressing these concerns. Here we provide a summary of the response:

1. Contribution and distinction from prior work. We acknowledge the overlap with DeepSeek-V3.2's Keep Sampling Mask and will explicitly cite it. However, our central contribution is not the mask-preservation operation itself, but the systematic investigation of LLM RL as decision-making within compact, state-dependent sets of promising tokens. Our new adaptive-support experiments further demonstrate the importance of decision-space construction, achieving improvements over both fixed Top-k and Top-p variants across three mathematical reasoning benchmarks.

2. Empirical robustness and support alignment. To address concerns about single-run ablations, we conducted additional five-seed experiments on Qwen3-1.7B. RLPT achieves the highest mean accuracy (29.26 ± 1.52), outperforming GRPO (28.14 ± 0.94) and both Top-k and Top-p rollout baselines with full-vocabulary optimization (27.95 ± 1.02 and 28.13 ± 1.53). While these differences should be interpreted cautiously given the variance, the results provide additional evidence consistent with the benefits of support-aligned optimization.

3. Feasibility of compact decision spaces. To move beyond token-level coverage, we evaluated end-to-end answer accuracy across two model sizes and three reasoning benchmarks. Top-4 sampling achieves accuracy within 0.69 percentage points of full-vocabulary sampling across all six settings. These results support the practical feasibility of compact decision spaces without claiming that every successful reasoning trajectory is preserved.

We have also clarified experimental protocols, baseline implementations, and reporting inconsistencies, and will incorporate these improvements into the revised manuscript.


We believe these additional results and clarifications provide a stronger basis for evaluating the contribution and empirical evidence of our work. We sincerely appreciate the reviewers' and Area Chair's time and consideration.

AuthorsOpenReview ↗

Paper Update

We have updated the manuscript to incorporate the reviewers' valuable suggestions, with changes highlighted in blue\color{blue}{\text{blue}} for your convenience. The main manuscript revisions are:

  1. Clarified the relationship to DeepSeek-V3.2, explicitly acknowledging the shared mask-reuse mechanism and focusing our contribution on the study of compact decision sets (Section 5; R #KMxm and VaBC).
  2. Added adaptive candidate-set results, end-to-end answer accuracy, and five-seed support-alignment ablations (Tables 8, 13, and 14; R #KMxm, YbVK, and VaBC).
  3. Explained baseline policy ratios, out-of-mask action handling, and run provenance (Section 6.4; Appendix G.7; R #KMxm and YbVK).
  4. Expanded training and test-time settings, thinking configurations, evaluation sample counts, Math-17k file partitioning, and the original Table 3 decoding protocol (Appendices G.4–G.6 and H.2; R #KMxm and YbVK).
  5. Qualified token-coverage claims and clarified that tail-gradient removal follows from standard masked softmax (Sections 3 and 4.4; R #KMxm and VaBC).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.