- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores a thought-provoking question: Does reinforcement learning actually endow large language models with new reasoning capabilities, or does it merely prompt them to select correct paths they were already capable of finding? Through a series of experiments, the authors find that at sparse yet critical branching points, reinforcement learning helps the model select the optimal path from among multiple candidate reasoning paths. Additionally, the paper introduces a method called REASONMAXXER; this approach does not rely on reinforcement learning but instead applies a contrastive loss solely at decision points determined by an entropy-gated mechanism.
Critical strengths
- Through a series of experiments, this paper finds that reinforcement learning helps the model select the optimal path from multiple candidate reasoning paths.
- This paper proposes REASONMAXXER. It does not rely on reinforcement learning; instead, it improves model performance solely by modifying high-entropy tokens, making it both effective and efficient.
Critical weaknesses
-
- The experiments focus primarily on mathematical reasoning. Although the appendix includes non-mathematical evaluations, these provide only limited evidence of broader applicability. More comprehensive experiments on other reasoning domains, together with competitive baselines, would help establish the method’s generality.
- The baseline models used in this paper are outdated.
- The method itself retains only "mixed-outcome problems" (i.e., problems that baseline models can sometimes solve correctly). Therefore, a more cautious conclusion would be that, within the context of current mathematical reasoning benchmarks and existing strong baseline models, the primary role of reinforcement learning (RL) manifests as sparse strategy selection. The current experiments are insufficient to support the claim that "reinforcement learning is not capability learning."
- How does the ratio of correct to incorrect rollouts affect decision-token selection and training effectiveness? This relationship warrants further analysis.
- Why are 20 rollouts used per problem? Sensitivity experiments should assess whether this budget is sufficient across models and tasks.
- Regarding the hyperparameter , experiments are conducted only with Qwen2.5-1.5B. Since entropy distributions may differ across model families, scales, and tasks, it remains unclear whether the observed robustness generalizes. Additional experiments would clarify whether a shared threshold is effective or model-specific calibration is necessary.
Questions for authors
Refer to Weakness
3 comments
W4. At a fixed entropy threshold, changing the correctness ratio does not change an individual trajectory's decision-token mask. It changes which trajectories contribute positive and negative supervision and their per-question advantage weights. The overall number and distribution of selected tokens can therefore change even though the selection rule stays fixed.
We compared 25%-correct and 75%-correct mixtures on Qwen3.5-2B-Base, holding the question pool, entropy threshold, and training budget fixed.
| Model / correct-response ratio | GSM8K | MATH500 |
|---|---|---|
| Base | 67.4 | 47.8 |
| ReasonMaxxer, 25% | 82.7 | 68.8 |
| ReasonMaxxer, natural 55.5%* | 83.3 | 67.8 |
| ReasonMaxxer, 75% | 78.8 | 67.2 |
Both tested mixtures improve substantially, including when 75% of responses are incorrect. The 25%-correct mixture scores 3.9 points higher on GSM8K and 1.6 points higher on MATH500 in this run. Thus, effective learning does not require mostly correct responses or an exact balance.
W5. We used 20 rollouts per problem as a common practical budget across the original experiments, not as an established optimum. We added two sensitivity tests on Qwen3.5-2B-Base to separate responses per question from question coverage.
From a pool of 104 questions with eight correct and eight incorrect responses each, we formed nested subsets. All runs use a 50/50 outcome balance, the same learning rate, entropy threshold, seed, and same number of optimizer updates.
| Questions × retained rollouts | Trajectories | GSM8K | MATH500 | AMC23 | Average | Weighted average |
|---|---|---|---|---|---|---|
| Base | — | 66.6 | 48.0 | 17.5 | 44.0 | 60.5 |
| 104 × 2 | 208 | 78.1 | 57.8 | 47.5 | 61.1 | 72.0 |
| 104 × 4 | 416 | 77.3 | 64.2 | 40.0 | 60.5 | 72.9 |
| 104 × 8 | 832 | 78.2 | 62.4 | 35.0 | 58.5 | 73.0 |
| 104 × 16 | 1,664 | 78.7 | 65.6 | 50.0 | 64.8 | 74.6 |
| 52 × 4 | 208 | 74.3 | 53.6 | 32.5 | 53.5 | 67.8 |
| 26 × 8 | 208 | 71.2 | 54.0 | 32.5 | 52.6 | 65.7 |
| 13 × 16 | 208 | 73.5 | 58.6 | 30.0 | 54.0 | 68.5 |
With questions fixed, one correct and one incorrect response per question already yield substantial gains. Increasing retained responses from 2 to 16 raises weighted accuracy from 72.0% to 74.6%, with MATH500 improving from 57.8% to 65.6%. Individual benchmark scores are not monotonic.
At 208 retained responses, broader question coverage (104 × 2) gives the highest aggregate accuracy, although 13 × 16 scores slightly higher on MATH500.
Every allocation improves all three benchmarks. These results support flexibility in the training budget on this backbone and seed, without establishing that 20 is sufficient or optimal across models and tasks.
W6. We use a model-family-specific threshold selected through validation (§5.2 and 6), rather than assuming one numerical τ works everywhere. We extended the sweep to Qwen3.5-2B-Base, fixing the training responses, objective, learning rate, seed, and update budget.
| τ | Tokens selected | GSM8K | MATH500 | AMC23 | Average |
|---|---|---|---|---|---|
| Base | — | 66.6 | 48.0 | 17.5 | 44.0 |
| 1.2 | 9.08% | 80.0 | 66.0 | 37.5 | 61.2 |
| 1.6 | 5.03% | 85.1 | 70.0 | 50.0 | 68.4 |
| 2.0 | 2.10% | 78.7 | 65.6 | 50.0 | 64.8 |
| 2.4 | 0.99% | 82.7 | 67.4 | 50.0 | 66.7 |
Several thresholds improve all three benchmarks while selecting approximately 1–9% of tokens. A shared τ=1.6 works across these tasks, reaching 68.4% average accuracy versus 44.0% for the base.
These results extend the sensitivity analysis to a newer backbone and support validation-based calibration. They do not establish a universal threshold or show that separate calibration is always necessary.
[1] https://huggingface.co/TIGER-Lab/General-Reasoner-Qwen3-4B
[2] https://huggingface.co/spiral-rl/Spiral-Qwen3-4B
[3] https://huggingface.co/Thinking-Space/Qwen3-4B-Base-GRPO
[4] https://huggingface.co/praysimanjuntak/qwen3.5-4b-grpo-gsm8k
[5] Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? https://arxiv.org/abs/2504.13837
We wanted to follow up and ask whether our response addresses your concerns. We would be happy to provide any further clarifications if needed.
Thank you.
We thank the reviewer for the detailed feedback.
W1. We added a direct non-math training experiment with RL baselines, and tested transfer from mathematical training.
We trained one shared adapter on 60 questions from Qwen3-4B-Base across ProofWriter, CommonsenseQA, and OpenBookQA to test if our method can learn from non-math data.
The three RL baselines start from the same Qwen3-4B-Base checkpoint.
ReasonMaxxer improves all three tasks. It matches SPIRAL on ProofWriter and exceeds two RL baselines on OpenBookQA, while General-Reasoner remains stronger. Training data and budgets differ across methods.
We also tested math-trained adapters without non-mathematical fine-tuning:
These use Qwen3.5 backbones (as newer models requested). All scores are accuracy percentages. Transfer is strongest on ProofWriter. These experiments show direct application beyond mathematics and transfer from mathematical training. We will include detailed non math table in the revision.
W2. We extended the evaluation to Qwen3.5-2B-Base and Qwen3.5-4B-Base, using the same ReasonMaxxer training procedure. We also evaluated a public GRPO adapter trained from Qwen3.5-4B-Base on GSM8K [4].
The non-math rows evaluate transfer from mathematical training, as discussed in W1.
ReasonMaxxer improves over both newer bases on all nine benchmarks and scores above this 4B GRPO reference. These results show that its gains extend to newer backbones.
W3. We retain mixed-outcome training problems because our objective needs choices to encourage and discourage. If every sampled response has the same outcome, there is no within-question reward contrast. This filter applies to ReasonMaxxer's training data; our base–RL analysis and benchmark evaluation are not restricted to mixed-outcome problems.
Independently of this filter, we observe sparse base–RL disagreements, concentrated at uncertain positions and usually favoring another leading base-model candidate.
Yue et al. [5] provide complementary evidence about complete solutions. On MATH500, their Qwen2.5-7B base solves 462 of the 467 questions solved by SimpleRL-Zoo when both receive 128 samples. With 1,024 samples, the base also solves the remaining five (Table 2 and Appendix C.7). The base already demonstrated the capability to solve those questions. This supports interpreting RL's gains in that comparison as making existing capabilities more reliable.
We also tested whether ReasonMaxxer and SimpleRL-Zoo recover the same successes. Among questions the base missed in the original evaluation, we measured how many of each method's one-attempt successes the other could recover within eight attempts.
Each denominator is the source model's original one-attempt gains. The overlap shows that ReasonMaxxer recovers many of RL's successes, beyond similar average accuracy.
Together, these results support more reliable use of existing capabilities through sparse policy selection. Our token analysis shows that RL often promotes choices already plausible to the base, mainly at uncertain positions. The learned improvement can therefore lie in knowing which available continuation to favor at these important steps. ReasonMaxxer tests whether this form of learning can reproduce RL's gains by training on correct and incorrect base-model rollouts at uncertain positions. It recovers substantial gains and many of the same successes as RL, showing that learning these choices from the base's own responses can reproduce much of the observed improvement.