acceptodds
Under review as a conference paper at ICLR 2027

Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning

10 views

Abstract

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token‑level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high‑entropy decision points where the model is uncertain which branch to take. Only 1-4% of token positions are affected, the promoted token almost always lies within the base model's top‑5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL‑trained model, and the entire correction is low‑dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL‑free method that applies contrastive loss only at entropy‑gated decision points, using a few hundred base‑model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single‑GPU training, a reduction in training cost of roughly three orders of magnitude.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, RL, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_stvSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores a thought-provoking question: Does reinforcement learning actually endow large language models with new reasoning capabilities, or does it merely prompt them to select correct paths they were already capable of finding? Through a series of experiments, the authors find that at sparse yet critical branching points, reinforcement learning helps the model select the optimal path from among multiple candidate reasoning paths. Additionally, the paper introduces a method called REASONMAXXER; this approach does not rely on reinforcement learning but instead applies a contrastive loss solely at decision points determined by an entropy-gated mechanism.

Critical strengths
  1. Through a series of experiments, this paper finds that reinforcement learning helps the model select the optimal path from multiple candidate reasoning paths.
  2. This paper proposes REASONMAXXER. It does not rely on reinforcement learning; instead, it improves model performance solely by modifying high-entropy tokens, making it both effective and efficient.
Critical weaknesses
    1. The experiments focus primarily on mathematical reasoning. Although the appendix includes non-mathematical evaluations, these provide only limited evidence of broader applicability. More comprehensive experiments on other reasoning domains, together with competitive baselines, would help establish the method’s generality.
  1. The baseline models used in this paper are outdated.
  2. The method itself retains only "mixed-outcome problems" (i.e., problems that baseline models can sometimes solve correctly). Therefore, a more cautious conclusion would be that, within the context of current mathematical reasoning benchmarks and existing strong baseline models, the primary role of reinforcement learning (RL) manifests as sparse strategy selection. The current experiments are insufficient to support the claim that "reinforcement learning is not capability learning."
  3. How does the ratio of correct to incorrect rollouts affect decision-token selection and training effectiveness? This relationship warrants further analysis.
  4. Why are 20 rollouts used per problem? Sensitivity experiments should assess whether this budget is sufficient across models and tasks.
  5. Regarding the hyperparameter τ=1.2\tau=1.2, experiments are conducted only with Qwen2.5-1.5B. Since entropy distributions may differ across model families, scales, and tasks, it remains unclear whether the observed robustness generalizes. Additional experiments would clarify whether a shared threshold is effective or model-specific calibration is necessary.
Questions for authors

Refer to Weakness

3 comments

We thank the reviewer for the detailed feedback.

W1. We added a direct non-math training experiment with RL baselines, and tested transfer from mathematical training.

We trained one shared adapter on 60 questions from Qwen3-4B-Base across ProofWriter, CommonsenseQA, and OpenBookQA to test if our method can learn from non-math data.

The three RL baselines start from the same Qwen3-4B-Base checkpoint.

Benchmark (questions)BaseReasonMaxxerGeneral-Reasoner [1]SPIRAL [2]Thinking-Space GRPO [3]
ProofWriter (450)70.774.775.674.774.2
CommonsenseQA (1,171)73.975.779.878.577.3
OpenBookQA (450)75.882.084.780.979.8

ReasonMaxxer improves all three tasks. It matches SPIRAL on ProofWriter and exceeds two RL baselines on OpenBookQA, while General-Reasoner remains stronger. Training data and budgets differ across methods.

We also tested math-trained adapters without non-mathematical fine-tuning:

Benchmark (questions)2B Base+ RM4B Base+ RM
ProofWriter (500)54.262.875.087.8
CommonsenseQA (1,221)67.670.478.480.9
OpenBookQA (500)77.678.088.490.0

These use Qwen3.5 backbones (as newer models requested). All scores are accuracy percentages. Transfer is strongest on ProofWriter. These experiments show direct application beyond mathematics and transfer from mathematical training. We will include detailed non math table in the revision.

W2. We extended the evaluation to Qwen3.5-2B-Base and Qwen3.5-4B-Base, using the same ReasonMaxxer training procedure. We also evaluated a public GRPO adapter trained from Qwen3.5-4B-Base on GSM8K [4].

Benchmark2B Base+ RM4B Base+ RM4B GRPO
GSM8K67.483.387.592.788.4
MATH50047.867.869.276.673.6
Minerva12.931.326.838.625.4
OlympiadBench19.440.643.454.745.8
AIME243.310.010.030.010.0
AMC2327.535.055.070.055.0
ProofWriter54.262.875.087.876.6
CommonsenseQA67.670.478.480.979.4
OpenBookQA77.678.088.490.087.8

The non-math rows evaluate transfer from mathematical training, as discussed in W1.

ReasonMaxxer improves over both newer bases on all nine benchmarks and scores above this 4B GRPO reference. These results show that its gains extend to newer backbones.

W3. We retain mixed-outcome training problems because our objective needs choices to encourage and discourage. If every sampled response has the same outcome, there is no within-question reward contrast. This filter applies to ReasonMaxxer's training data; our base–RL analysis and benchmark evaluation are not restricted to mixed-outcome problems.

Independently of this filter, we observe sparse base–RL disagreements, concentrated at uncertain positions and usually favoring another leading base-model candidate.

Yue et al. [5] provide complementary evidence about complete solutions. On MATH500, their Qwen2.5-7B base solves 462 of the 467 questions solved by SimpleRL-Zoo when both receive 128 samples. With 1,024 samples, the base also solves the remaining five (Table 2 and Appendix C.7). The base already demonstrated the capability to solve those questions. This supports interpreting RL's gains in that comparison as making existing capabilities more reliable.

We also tested whether ReasonMaxxer and SimpleRL-Zoo recover the same successes. Among questions the base missed in the original evaluation, we measured how many of each method's one-attempt successes the other could recover within eight attempts.

BackboneBenchmarkRL gains recovered by ReasonMaxxerReasonMaxxer gains recovered by RL
Qwen2.5-1.5BGSM8K95.8% (296/309)98.4% (300/305)
Qwen2.5-1.5BMATH50091.9% (113/123)93.9% (124/132)
Qwen2.5-7BGSM8K99.5% (183/184)97.6% (244/250)
Qwen2.5-7BMATH50096.9% (62/64)95.2% (80/84)

Each denominator is the source model's original one-attempt gains. The overlap shows that ReasonMaxxer recovers many of RL's successes, beyond similar average accuracy.

Together, these results support more reliable use of existing capabilities through sparse policy selection. Our token analysis shows that RL often promotes choices already plausible to the base, mainly at uncertain positions. The learned improvement can therefore lie in knowing which available continuation to favor at these important steps. ReasonMaxxer tests whether this form of learning can reproduce RL's gains by training on correct and incorrect base-model rollouts at uncertain positions. It recovers substantial gains and many of the same successes as RL, showing that learning these choices from the base's own responses can reproduce much of the observed improvement.

W4. At a fixed entropy threshold, changing the correctness ratio does not change an individual trajectory's decision-token mask. It changes which trajectories contribute positive and negative supervision and their per-question advantage weights. The overall number and distribution of selected tokens can therefore change even though the selection rule stays fixed.

We compared 25%-correct and 75%-correct mixtures on Qwen3.5-2B-Base, holding the question pool, entropy threshold, and training budget fixed.

Model / correct-response ratioGSM8KMATH500
Base67.447.8
ReasonMaxxer, 25%82.768.8
ReasonMaxxer, natural 55.5%*83.367.8
ReasonMaxxer, 75%78.867.2

Both tested mixtures improve substantially, including when 75% of responses are incorrect. The 25%-correct mixture scores 3.9 points higher on GSM8K and 1.6 points higher on MATH500 in this run. Thus, effective learning does not require mostly correct responses or an exact balance.

W5. We used 20 rollouts per problem as a common practical budget across the original experiments, not as an established optimum. We added two sensitivity tests on Qwen3.5-2B-Base to separate responses per question from question coverage.

From a pool of 104 questions with eight correct and eight incorrect responses each, we formed nested subsets. All runs use a 50/50 outcome balance, the same learning rate, entropy threshold, seed, and same number of optimizer updates.

Questions × retained rolloutsTrajectoriesGSM8KMATH500AMC23AverageWeighted average
Base—66.648.017.544.060.5
104 × 220878.157.847.561.172.0
104 × 441677.364.240.060.572.9
104 × 883278.262.435.058.573.0
104 × 161,66478.765.650.064.874.6
52 × 420874.353.632.553.567.8
26 × 820871.254.032.552.665.7
13 × 1620873.558.630.054.068.5

With questions fixed, one correct and one incorrect response per question already yield substantial gains. Increasing retained responses from 2 to 16 raises weighted accuracy from 72.0% to 74.6%, with MATH500 improving from 57.8% to 65.6%. Individual benchmark scores are not monotonic.

At 208 retained responses, broader question coverage (104 × 2) gives the highest aggregate accuracy, although 13 × 16 scores slightly higher on MATH500.

Every allocation improves all three benchmarks. These results support flexibility in the training budget on this backbone and seed, without establishing that 20 is sufficient or optimal across models and tasks.

W6. We use a model-family-specific threshold selected through validation (§5.2 and 6), rather than assuming one numerical τ works everywhere. We extended the sweep to Qwen3.5-2B-Base, fixing the training responses, objective, learning rate, seed, and update budget.

τTokens selectedGSM8KMATH500AMC23Average
Base—66.648.017.544.0
1.29.08%80.066.037.561.2
1.65.03%85.170.050.068.4
2.02.10%78.765.650.064.8
2.40.99%82.767.450.066.7

Several thresholds improve all three benchmarks while selecting approximately 1–9% of tokens. A shared τ=1.6 works across these tasks, reaching 68.4% average accuracy versus 44.0% for the base.

These results extend the sensitivity analysis to a newer backbone and support validation-based calibration. They do not establish a universal threshold or show that separate calibration is always necessary.

[1] https://huggingface.co/TIGER-Lab/General-Reasoner-Qwen3-4B

[2] https://huggingface.co/spiral-rl/Spiral-Qwen3-4B

[3] https://huggingface.co/Thinking-Space/Qwen3-4B-Base-GRPO

[4] https://huggingface.co/praysimanjuntak/qwen3.5-4b-grpo-gsm8k

[5] Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? https://arxiv.org/abs/2504.13837

We wanted to follow up and ask whether our response addresses your concerns. We would be happy to provide any further clarifications if needed.

Thank you.

Reviewer_xveDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

research question: RL's real use in LLM reasoning.

Authors think RL's main contribution is not let the model learn new reasoning capabilities, but re-assign the prob on model's existing inference traj.

Core, propose ReasonMaxxer: generate few rollouts in base models, filtered questions by containing both correct and wrong answers. then recognize the decision points by using the token entropy, and only apply advantage-weighted contrastive loss. In addition, use KL regularization to maintain other position's distribution from base models.

Multiple results shows that ReasonMaxxer can reach the same or comparable performance of RL baseline by only using 50 questions with 1k rollouts, reducing much more training budget.

Critical strengths

research questions are important, and observations are valuable.

method is simple, and results are attracitve.

experiments covers mutiple scenarios.

Critical weaknesses
  1. still confused on which cause the main improvements. Entropy gating, contrastive obj, or few high-quality rollout choices.

  2. need more rigorous analysis on budget comparison with others and statistical significance analysis

  3. I think current experiments only support: RL's greedy deconing behavior is more concentrate on few positions, but still need more analysis and theoretical support on RL is policy selection, not capability learning.

Questions for authors

see above

4 comments

Thank you for the positive feedback and the questions.

W1.

Our ablations support contributions from both where we apply feedback and whether we learn from incorrect responses. Applying the same objective to random positions in the same rollouts does not reproduce the gains.

Position selection. On Qwen3.5-2B-Base, we compare entropy-selected tokens with uniformly sampled tokens, matching the exact count per trajectory (about 2% overall). Both use the same 104 questions, eight correct and eight incorrect per question (please refer to stvS for details of this experiment). Random masks remain fixed during training. The objective, KL rule, learning rate, seed and number of updates are held fixed.

Token selectionGSM8KMATH500AMC23
Base66.648.017.5
Entropy78.765.650.0
Random64.742.627.5

Values are single-generation accuracy percentages. Entropy beats random selection by 14.0, 23.0 and 22.5 points. With the same responses and objective, rollout quality alone cannot explain this difference.

Negative feedback. Keeping entropy gating, the positive-only ablation in §6.3 raises Qwen2.5-1.5B's MATH500 accuracy from 29.8% to 39.8%. The full method reaches 50.2%. Learning from incorrect responses therefore adds 10.4 points beyond reinforcing successful responses at uncertain positions.

Rollout allocation. On Qwen3.5-2B-Base, we varied question coverage and retained responses per question, keeping a 50/50 correctness mixture, the same learning rate, entropy threshold, and seed.

Questions × retained rolloutsTrajectoriesGSM8KMATH500AMC23AverageWeighted average
Qwen3.5-2B-Base—66.648.017.544.060.5
104 × 220878.157.847.561.172.0
52 × 420874.353.632.553.567.8
26 × 820871.254.032.552.665.7
13 × 1620873.558.630.054.068.5
104 × 441677.364.240.060.572.9
104 × 883278.262.435.058.573.0
104 × 161,66478.765.650.064.874.6

At 208 responses, covering more questions gives the best pooled result, using just one correct and one incorrect response per question. Adding responses to the fixed question pool also helps. All allocations improve over the base on all three benchmarks.

Correctness balance. Under a separate protocol, the 25%- and 75%-correct runs share the question pool, entropy threshold, and training budget.

Training mixtureGSM8KMATH500
Base67.447.8
25% correct82.768.8
original natural mixture (55.5%)83.367.8
75% correct78.867.2

Both improve substantially, including when most responses are incorrect.

The controls support contributions from both position selection and negative feedback. The data experiments show that gains persist across different allocations and correctness ratios, not specific to certain rollout choices.

W2.

ReasonMaxxer's costs come from measured GPU time covering rollout generation, entropy computation, training and validation. We use the hardware-specific rates in Appendix E: $1.89/hour for our RTX Pro 6000 runs and $2.49/H100-hour for SimpleRL-Zoo.

Baselines have published compute budgets:

Baseline / shared starting modelPublished computeBaseline costReasonMaxxer
SimpleRL-Zoo / Qwen2.5-32B64 H100 × ~36 h = 2,304 GPU-h~$5,737$25
DeepScaleR / DeepSeek-R1-Distill-Qwen-1.5B3,800 A100-h~$4,500$4

SimpleRL-Zoo reports the hardware and durations [1], DeepScaleR reports both GPU-hours and cost [2].

These imply approximately 230× and 1,125× lower post-training cost**, alongside the competitive performance in Table 3. ORZ-32B's schedule implies 9.8M training responses [3], versus ~819K for SimpleRL-Zoo's ~100-step recipe [1] and 3,000 generated for ReasonMaxxer. ORZ's cost is an estimate.

Evaluation uncertainty. We computed 95% problem-level bootstrap confidence intervals for Qwen2.5-1.5B:

BenchmarkBaseSimpleRL-ZooReasonMaxxer
MATH50029.80 [25.80, 33.80]49.60 [45.20, 54.00]50.20 [45.80, 54.60]
GSM8K56.10 [53.37, 58.91]73.31 [70.89, 75.66]71.04 [68.54, 73.46]
AMC 20239.38 [4.69, 14.69]15.62 [9.06, 23.13]16.88 [10.00, 24.69]
AIME 20240.42 [0.00, 1.25]3.75 [1.25, 6.67]2.92 [0.00, 6.67]
Minerva Math4.41 [2.21, 6.99]10.66 [6.99, 14.71]11.40 [7.72, 15.44]
OlympiadBench9.29 [7.06, 11.70]18.93 [15.83, 22.20]16.70 [13.60, 19.79]

ReasonMaxxer's observed gains over the base are 20.40 points on MATH500 and 14.94 on GSM8K. Its differences from SimpleRL-Zoo are much smaller: +0.60 and -2.27 points, respectively. AMC and AIME have wider intervals because they contain few problems.

W3.

Sparse changes in greedy token choices alone do not establish that RL improves reasoning through policy selection rather than learning new capabilities. Our argument also examines whether the base could already generate successful solutions, and whether learning to favor its existing choices can recover RL's improvements.

Yue et al. [5] directly examine the first question. On MATH500, their Qwen2.5-7B base solves 462 of the 467 questions solved by SimpleRL-Zoo when both receive 128 samples: 98.9% coverage of RL's successes. With 1,024 samples, the base also solves the five remaining questions (Table 2 and Appendix C.7). In this comparison, RL's higher single-response accuracy therefore does not require solving problems beyond the base's observed coverage.

Davis and Recht [4] show that several binary-reward RL updates optimize increasing functions of correct-answer probability. Their analysis describes the objective; [5] tests the accessibility of successful solutions.

Our analysis examines how this improvement appears during generation. RL changes relatively few preferred tokens, mainly where the base is uncertain, and usually promotes another leading base-model candidate. This suggests that a few consequential choices can help explain why an already-accessible solution becomes easier to generate.

ReasonMaxxer tests whether we can use this pattern. It learns which choices to favor from correct and incorrect base-model rollouts, with feedback concentrated at uncertain positions. It recovers substantial gains without an RL teacher or repeated online rollout collection.

We also test whether these gains concern the same questions. Similar average accuracy could hide different successes. We identified questions missed by the base but solved by SimpleRL-Zoo in the original one-attempt evaluation, and measured how many ReasonMaxxer recovers:

BackboneBenchmarkIn 1 attemptWithin 8 attempts
Qwen2.5-1.5BGSM8K79.6% (246/309)95.8% (296/309)
Qwen2.5-1.5BMATH50075.6% (93/123)91.9% (113/123)
Qwen2.5-7BGSM8K90.8% (167/184)99.5% (183/184)
Qwen2.5-7BMATH50075.0% (48/64)96.9% (62/64)

These are recovery rates over fixed sets of RL corrections, not full-benchmark pass@8.

ReasonMaxxer recovers 75.0–90.8% of RL's corrections in one attempt and 91.9–99.5% within eight. This tests recovery by ReasonMaxxer; [5] provides the separate evidence about base-model coverage.

The evidence therefore connects base accessibility [5], changes in token choices, and recovery of RL's successes through ReasonMaxxer. This does not exclude learning a better decision rule. It supports our central interpretation that learning which existing continuations to favor can account for substantial reasoning gains, and shows that many of those gains can be obtained from fixed base-model rollouts.

[1] https://github.com/hkust-nlp/simpleRL-reason#training

[2] https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2

[3] https://arxiv.org/abs/2503.24290

[4] Davis and Recht (2025). What Is the Objective of Reasoning with Reinforcement Learning? https://arxiv.org/abs/2510.13651

[5] Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? https://arxiv.org/abs/2504.13837

Reviewer_xveDOpenReview ↗

Thank you

Thank you for your rebuttal, it partially solves my concerns. Therefore, I keep my score. (I slightly improve scores, but I dont think it reach to acceptance score. i.e., from 2.6(initial)->3.3(now), so it is still 3).

We thank the reviewer for their questions, engagement and positive feedback.

Reviewer_ZRa9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studied whether improving LLM reasoning requires an expensive online RL optimization loop. It first finds that disagreements between base and RL-trained models are sparse and concentrated at high-entropy positions, with the RL-preferred token typically among the base model’s top candidates. In addition, the authors propose ReasonMaxxer, which trains a LoRA adapter using advantage-weighted updates at high-entropy positions while anchoring other positions to the base model. Across multiple model families and six math benchmarks, the method achieves performance broadly comparable to publicly available RL-trained checkpoints using only 50 training problems with low cost.

Critical strengths

This paper has the following advantages that I appreciate very much:

  • I think the paper addresses an important and practical question, i.e., how much of RL’s reasoning improvement can be recovered with substantially less data and compute, especially given that models are becomes larger and training becomes more expensive.

  • What's more, the token-level analysis is very informative, and the alternative uncertainty signals in the appendix provide useful evidence that the observed localization is not specific to entropy.

  • Finally, it seems that ReasonMaxxer is simple and achieves competitive results across multiple model families and six math benchmarks, with low reported training cost.

Critical weaknesses

This paper can also benefit from the following clarification and improvement:

  • It seems that the oracle experiment reproduces the RL model’s output mainly by setup, which is not very surprising. If I understand correctly, in section 3.2, the base model’s token is kept when the two models agree and replaced with the RL model’s preferred token when they disagree. If both models are evaluated on the corrected prefix at each step, the emitted token therefore always equals the RL model’s argmax. Matching the RL model’s accuracy is then guaranteed, rather than independent evidence for the proposed causal explanation.

  • In addition, the paper observes that the RL model changes relatively few token choices and usually selects alternatives already ranked highly by the base model. But in my opinion, a newly learned reasoning skill could also being shown as choosing a different token at a small number of critical steps, which does necessarily not mean that no new capability was learned.

  • Finally, for the experiments, I'm curious on the following two questions: first, does SFT also change the base model’s token choices mainly at a small number of high-entropy positions? Also, could we comparing ReasonMaxxer with the same objective applied to randomly selected tokens at the same rate, or to all tokens, to clarify if the actual benefit of ReasonMaxxer comes from selecting high-entropy tokens?

Questions for authors

Please refer to my summary of weaknesses.

4 comments

We thank the reviewer for their careful reading and detailed comments.

W1.

Yes, exact recovery follows by construction. We use the oracle to turn the observations in §3.1 into an explicit decoding rule, then examine which parts of that rule can be approximated.

Section 3.1 shows that disagreements are sparse and often involve alternatives already ranked highly by the base model. Section 3.2 expresses this as teacher-guided decoding: keep the base model's choice unless it differs from the teacher's, and override it at those positions. Matching the teacher is guaranteed; requiring only a few percent of overrides is the empirical property that motivates learning such a correction rule.

The oracle supplies both when to intervene and what token to choose. Figure 2 and §3.3 then replace one component at a time:

  • Teacher-defined positions, random replacements: We retain the disagreement trigger but choose a random alternative from the base model's top-20. This does not recover the RL gains, showing that changing choices at disagreement points is insufficient without useful replacements.
  • Entropy-defined positions, teacher replacements: We replace the disagreement trigger with base-model entropy. Recovery is no longer guaranteed because this gate can miss disagreements. Its performance tests whether uncertainty can identify useful intervention positions without checking teacher disagreement.

The oracle provides the reference for these comparisons. They show why both components matter: uncertainty can help identify where to intervene, while the replacement choices still need to be learned. This motivates the subsequent learning experiments: first testing whether a small adapter can capture the teacher's behavior, then whether outcome feedback from base-model rollouts can teach useful choices without the teacher.

W2.

We agree that a new reasoning rule could affect only a few token choices. Our question is how much of the gain comes from making already-available solutions more likely. For example, on the same question, increasing the probability of a correct answer in a single attempt from 60% to 90% makes success more reliable; the base already demonstrated that it could solve that question.

Yue et al. [1] provide evidence for this distinction: in their evaluated settings, most questions solved by RL models are also solved by the base with additional sampling. This motivates our next question: how could such an improvement arise during generation?

Our analysis finds that the models' preferred tokens differ at relatively few positions, concentrated where the base is uncertain. RL usually chooses an alternative already ranked highly by the base. These findings suggest that choosing better between available continuations can be an important source of improvement. They describe local choices; Yue et al.'s results provide complementary evidence about complete solutions.

ReasonMaxxer tests whether we can use this pattern to learn useful choices directly. It trains on fixed correct and incorrect base-model rollouts, concentrating supervision at high-entropy positions. This recovers substantial gains without an RL teacher or repeated online rollout collection.

We also tested whether ReasonMaxxer and SimpleRL-Zoo recover the same successes. Among questions the base missed in the original evaluation, we measured how many of each method's one-attempt successes the other could recover within eight attempts.

BackboneBenchmarkRL gains recovered by ReasonMaxxerReasonMaxxer gains recovered by RL
Qwen2.5-1.5BGSM8K95.8% (296/309)98.4% (300/305)
Qwen2.5-1.5BMATH50091.9% (113/123)93.9% (124/132)
Qwen2.5-7BGSM8K99.5% (183/184)97.6% (244/250)
Qwen2.5-7BMATH50096.9% (62/64)95.2% (80/84)

Each denominator is the source model's original one-attempt gains.

The overlap shows that ReasonMaxxer recovers many of RL's successes, beyond similar average accuracy. Together, the response-level evidence, local choice patterns, and successful learning method support selection as an explanation for substantial gains. Learning a better selection rule can be part of this process; many of these gains are recoverable from fixed base-model responses and outcome feedback.

[1] Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

W3.

We ran both comparisons.

SFT. We chose OLMo-2-7B for its separate base and SFT checkpoints, which let us examine SFT before preference optimization or RL [2]. Both models were evaluated on identical prefixes from greedy base-model trajectories on MATH500.

Across all 500 questions, 2.47% of token choices change, and 93.6% of changes occur in the highest-entropy 20% of positions. Pooling tokens weights longer trajectories more heavily; 158 reached the length limit. Among the 342 completed trajectories, 13.86% of choices change, with 63.5% of changes in the highest-entropy 20%. We retain all 500 as the main analysis.

Thus, entropy concentration also occurs after SFT in this model pair, while measured sparsity depends on trajectory length. Its value for ReasonMaxxer depends on whether these positions are useful places to apply feedback, which we test next.

Random-token control. On Qwen3.5-2B-Base, both conditions use 104 questions and 1,664 responses, with eight correct and eight incorrect per question (please refer to stvS for details of this experiment). Uniform random masks match the exact selected-token count per trajectory (about 2% overall), are sampled without replacement, and stay fixed during training.

Both runs use the same contrastive objective, KL rule, response order, positive/negative supervision counts per update, learning rate, seed and updates.

Token selectionGSM8KMATH500AMC23
Base66.648.017.5
ReasonMaxxer: entropy78.765.650.0
Random64.742.627.5

Accuracy (%) favors entropy by 14.0, 23.0 and 22.5 points. The same data and objective applied to equally many random tokens do not reproduce the gains.

This supports entropy selection as a contributor to ReasonMaxxer's gains.

[2] https://huggingface.co/allenai/OLMo-2-1124-7B-SFT

I thank the authors for the detailed response. I have carefully read the responses and confirmed that my concerns have been well-addressed. Good luck!

We thank the reviewer again for reviewing our work and for their engagement in rebuttals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.