- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EARR, a GRPO reward that combines sampling accuracy and final-answer entropy to modulate penalties on reasoning length, reflection-word count, and continuous reasoning span. It aims to reduce redundant verification and fragmented search while preserving accuracy. The paper reports substantial token reductions and average accuracy gains, together with component, reward-weight, lexicon, and sampling-number analyses.
Critical strengths
-
Substantial reported efficiency gains. Table 1 reports average length reductions of 53.4% and 57.3%, with accuracy gains of 4.0 and 1.4 percentage points over Vanilla. These results suggest practical value beyond a small reduction in output length.
-
Useful component and sensitivity analyses. The paper examines reward components, coefficients, reflection lexicons, and training sample counts. The lexicon analysis also acknowledges that common words can produce false-positive reflection detections.
Critical weaknesses
-
The motivation supports limited correlations. Section 3 analyzes one 1.5B model on MATH-500 without sufficiently documenting the sampling protocol or controlling for length and difficulty. These observations do not establish when further reflection is unproductive, which is the key justification for the proposed modulation rule.
-
Insufficient positioning against entropy-guided methods. While the proposed method integrate entropy as the component, the discussion of other related method that utilize entropy for efficient reasoning is missing, such as PEAR or Step Entropy, which should be included in the Related Work and in the baseline experiments as well. Besides, not all discussed methods in Related Work are compared with the proposed methods.
PEAR: Phase Entropy Aware Reward for Efficient Reasoning (ICLR 2026)
Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026)
Questions for authors
-
Independent value of entropy. Can you compare accuracy-only, entropy-only, and combined modulation at comparable token budgets? A correctness-only GRPO control and a joint length-only reward would also help isolate the new components.
-
Statistical reliability. Can you report training-seed variability and paired uncertainty estimates for the accuracy and length differences? This would clarify which incremental improvements are robust.
-
For the 8B results, why only compare EARR with vanilla and why not include the complete comparision in table 1?
-
Can you provide more details in the experiment settings and more comprehensive results for motivation analysis? For example, what sampling and decoding settings produced Figures 1–2? Do the relationships persist after controlling for difficulty and length, and what evidence shows that entropy predicts reflection utility beyond sampling accuracy? An extended experimental reuslts on more model sizes and more benchmarks will enhance the robustness.
4 comments
Response to the Reviewer
Q4:Experimental Settings and Extended Motivation Analysis.
-
Sampling and Decoding Settings. For the motivation analysis in Figures 1–2, we set the maximum generation length to 16,384 tokens, the temperature to 0.6, and top-p to 0.95. To ensure the reliability of the statistical analysis, we independently sample 20 reasoning trajectories for each problem and compute the relevant statistics based on these samples.
-
Problem Difficulty and Reasoning Length. In Figure 1(a), we group problems according to their difficulty and observe that incorrect responses exhibit more reflections than correct responses even at the same difficulty level. Regarding reasoning length, we adopt a uniform maximum generation length and allow the models to naturally terminate their reasoning within this budget. This is because our study aims to investigate the relationships among reflection count, reasoning length, reasoning continuity, and accuracy during natural reasoning, where reasoning length itself is an important variable potentially influenced by reflection behavior. Forcibly truncating reasoning trajectories to enforce a uniform generation length may disrupt the reasoning process, alter the original reflection behavior and final answer correctness, and introduce additional confounding factors. Therefore, we preserve naturally generated reasoning trajectories to better characterize the relationship between reflection behavior and reasoning performance.
-
Extended Motivation Analysis. In Appendix A, we extend the motivation analysis to two models, DS-1.5B and Qwen3-4B, and two datasets, MATH-500 and MinervaMath. The results show that our original observations remain consistent across different models and datasets, further strengthening the empirical foundation of our motivation.
-
Independent Contribution of Answer Entropy. In Table 2, we add two ablation experiments using only sampling accuracy and only answer entropy, respectively. Compared with these two variants, EARR improves average accuracy by 0.6 and 1.0 percentage points, while increasing the reasoning length reduction ratio by 4.1 and 3.4 percentage points, respectively. These results suggest that answer entropy provides complementary information beyond sampling accuracy and improves the effectiveness of reflection reward modulation.
-
Effectiveness Across Model Scales. In Table 7 of the Appendix, we provide additional comparative experiments on Qwen3-8B and Qwen3-14B. EARR reduces reasoning length by 48.8% and 44.3% while improving average accuracy by 1.5 and 3.1 percentage points, respectively, further demonstrating the effectiveness and scalability of EARR across different model sizes.
Q5:Statistical Reliability.
We would first like to clarify that the evaluation results reported in our paper are averaged over multiple independent runs, as described iin the Implementation Details of Section 5.1 in the revised manuscript.
To further assess the statistical reliability of our results, we report the mean and standard deviation of accuracy and reasoning length over five independent evaluations of EARR on DS-1.5B, as shown below.
| Dataset | Accuracy (%) | Reasoning Length |
|---|---|---|
| AIME24 | 32.00 ± 1.10 | 6638 ± 60.25 |
| AIME25 | 22.50 ± 0.51 | 6498 ± 50.48 |
| AMC23 | 75.33 ± 0.70 | 3451 ± 55.25 |
| MATH500 | 83.16 ± 0.95 | 1932 ± 47.89 |
| MinervaMath | 27.21 ± 0.65 | 2248 ± 37.37 |
| OlympiadBench | 46.50 ± 0.42 | 4190 ± 48.32 |
The results show that the standard deviation of accuracy does not exceed 1.10 percentage points, and that of reasoning length does not exceed 61 tokens across all six benchmarks, demonstrating the stability of EARR across independent stochastic decoding runs.
Looking Forward to Further Discussion
Dear Reviewer,
We have provided detailed responses to your concerns and sincerely hope that our clarifications have addressed your questions.
As the rebuttal period is coming to an end, we would like to kindly follow up on our responses. We greatly value this opportunity to engage in a constructive discussion with you and sincerely appreciate your insights. If you have any remaining concerns or questions, we would be more than happy to provide further clarifications.
We sincerely look forward to hearing your thoughts and appreciate your time and consideration.
Thank you for the response, most of my concerns are resolved, and I will raise my rating.
Response to the Reviewer
We sincerely appreciate your thorough review of our manuscript and constructive feedback. We apologize for the delay in our response, as LLM RL training is computationally intensive. We hope that our responses below adequately address your concerns and clarify the raised issues.
Q1:Independent Contribution of Answer Entropy
In the revised manuscript, we have added two ablation experiments in Table2, using only sampling accuracy and only answer entropy, respectively. Compared with these two variants, EARR improves average accuracy by 0.6 and 1.0 percentage points, while increasing the reasoning length reduction ratio by 4.1 and 3.4 percentage points, respectively. Notably, EARR achieves better reasoning performance with fewer tokens rather than more. These results suggest that answer entropy provides complementary information beyond sampling accuracy, and jointly modeling both signals achieves a better accuracy--efficiency trade-off.
Q2:Evaluation on Larger Models
In the revised manuscript, we have extended the comparison on Qwen3-8B to include SFT, DPO, O1-Pruner, AdaptThink, and ARLCP, and further conducted experiments on Qwen3-14B. Due to time constraints, we select two relatively strong baselines, AdaptThink and ARLCP, for comparison on Qwen3-14B, along with the Vanilla model. The additional results are presented in Table~7 of the Appendix. EARR reduces reasoning length by 48.8% and 44.3% while improving average accuracy by 1.5 and 3.1 percentage points on Qwen3-8B and Qwen3-14B, respectively. These results further demonstrate the effectiveness and scalability of EARR across different model sizes.
Q3: Comparison with Entropy-Guided Methods.
Table 1 of the original PEAR paper demonstrates that PEAR achieves a better accuracy–efficiency trade-off than Step Entropy. Therefore, we select PEAR as a representative entropy-guided baseline and conduct additional comparative experiments on DS-1.5B. The results are presented below.
The results show that, compared with PEAR, EARR improves average accuracy by 1.6 percentage points while increasing the reasoning length reduction ratio by 4.3 percentage points, further demonstrating the effectiveness of EARR in achieving a better trade-off between reasoning accuracy and efficiency.