acceptodds
Under review as a conference paper at ICLR 2027

Reflect Less, Reason Better: Entropy-Aware Reflection Rewards for Efficient Large Reasoning Models

Abstract

Large reasoning models make significant progress in solving complex problems, but their reasoning processes are still often affected by overthinking, which leads to high reasoning latency and computational costs without corresponding improvements in accuracy. Existing methods usually improve reasoning efficiency by directly penalizing reasoning length or the number of reflection words through reinforcement learning. However, these methods often ignore differences between problem difficulty and reflective reasoning structures, which may cause models to frequently switch reasoning paths and thus produce fragmented reflection. To address these issues, this paper proposes a reflection reward method that integrates answer entropy and difficulty awareness, based on analyses of the number of reflection words, continuous reasoning spans, and answer distributions. This method focuses on constraining inefficient reflection behaviors in two types of scenarios: high accuracy with low entropy, and low accuracy with high entropy. It penalizes fragmented and cyclic reflection, and favors reasoning trajectories with low-frequency reflection, long continuous reasoning spans, and continuous stability. Extensive experimental results show that, without reducing model performance, the proposed method effectively reduces token consumption in chain-of-thought reasoning processes, providing a more fine-grained and effective optimization approach for balancing efficiency and accuracy in large language model reasoning.

Primary area:
generative models
Keywords:
Large Reasoning Models, Reasoning Efficiency, Answer Entropy, Reflection Reward

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yPM4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EARR, a GRPO reward that combines sampling accuracy and final-answer entropy to modulate penalties on reasoning length, reflection-word count, and continuous reasoning span. It aims to reduce redundant verification and fragmented search while preserving accuracy. The paper reports substantial token reductions and average accuracy gains, together with component, reward-weight, lexicon, and sampling-number analyses.

Critical strengths
  1. Substantial reported efficiency gains. Table 1 reports average length reductions of 53.4% and 57.3%, with accuracy gains of 4.0 and 1.4 percentage points over Vanilla. These results suggest practical value beyond a small reduction in output length.

  2. Useful component and sensitivity analyses. The paper examines reward components, coefficients, reflection lexicons, and training sample counts. The lexicon analysis also acknowledges that common words can produce false-positive reflection detections.

Critical weaknesses
  1. The motivation supports limited correlations. Section 3 analyzes one 1.5B model on MATH-500 without sufficiently documenting the sampling protocol or controlling for length and difficulty. These observations do not establish when further reflection is unproductive, which is the key justification for the proposed modulation rule.

  2. Insufficient positioning against entropy-guided methods. While the proposed method integrate entropy as the component, the discussion of other related method that utilize entropy for efficient reasoning is missing, such as PEAR or Step Entropy, which should be included in the Related Work and in the baseline experiments as well. Besides, not all discussed methods in Related Work are compared with the proposed methods.


PEAR: Phase Entropy Aware Reward for Efficient Reasoning (ICLR 2026)

Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026)

Questions for authors
  1. Independent value of entropy. Can you compare accuracy-only, entropy-only, and combined modulation at comparable token budgets? A correctness-only GRPO control and a joint length-only reward would also help isolate the new components.

  2. Statistical reliability. Can you report training-seed variability and paired uncertainty estimates for the accuracy and length differences? This would clarify which incremental improvements are robust.

  3. For the 8B results, why only compare EARR with vanilla and why not include the complete comparision in table 1?

  4. Can you provide more details in the experiment settings and more comprehensive results for motivation analysis? For example, what sampling and decoding settings produced Figures 1–2? Do the relationships persist after controlling for difficulty and length, and what evidence shows that entropy predicts reflection utility beyond sampling accuracy? An extended experimental reuslts on more model sizes and more benchmarks will enhance the robustness.

4 comments
AuthorsOpenReview ↗

Response to the Reviewer

We sincerely appreciate your thorough review of our manuscript and constructive feedback. We apologize for the delay in our response, as LLM RL training is computationally intensive. We hope that our responses below adequately address your concerns and clarify the raised issues.

Q1:Independent Contribution of Answer Entropy

In the revised manuscript, we have added two ablation experiments in Table2, using only sampling accuracy and only answer entropy, respectively. Compared with these two variants, EARR improves average accuracy by 0.6 and 1.0 percentage points, while increasing the reasoning length reduction ratio by 4.1 and 3.4 percentage points, respectively. Notably, EARR achieves better reasoning performance with fewer tokens rather than more. These results suggest that answer entropy provides complementary information beyond sampling accuracy, and jointly modeling both signals achieves a better accuracy--efficiency trade-off.

Q2:Evaluation on Larger Models

In the revised manuscript, we have extended the comparison on Qwen3-8B to include SFT, DPO, O1-Pruner, AdaptThink, and ARLCP, and further conducted experiments on Qwen3-14B. Due to time constraints, we select two relatively strong baselines, AdaptThink and ARLCP, for comparison on Qwen3-14B, along with the Vanilla model. The additional results are presented in Table~7 of the Appendix. EARR reduces reasoning length by 48.8% and 44.3% while improving average accuracy by 1.5 and 3.1 percentage points on Qwen3-8B and Qwen3-14B, respectively. These results further demonstrate the effectiveness and scalability of EARR across different model sizes.

Q3: Comparison with Entropy-Guided Methods.

Table 1 of the original PEAR paper demonstrates that PEAR achieves a better accuracy–efficiency trade-off than Step Entropy. Therefore, we select PEAR as a representative entropy-guided baseline and conduct additional comparative experiments on DS-1.5B. The results are presented below.

MethodAIME24AIME25AMC23MATH500MinervaMathOlympiadΔAccΔLength
Vanilla26.3 / 1234021.3 / 1186165.6 / 772481.1 / 467525.1 / 621242.1 / 8686--
PEAR29.0 / 779523.3 / 752569.1 / 370283.4 / 181825.1 / 234145.8 / 4701+2.4-49.1%
EARR31.7 / 665222.5 / 650575.2 / 346083.1 / 193126.6 / 224746.7 / 4206+4.0-53.4%

The results show that, compared with PEAR, EARR improves average accuracy by 1.6 percentage points while increasing the reasoning length reduction ratio by 4.3 percentage points, further demonstrating the effectiveness of EARR in achieving a better trade-off between reasoning accuracy and efficiency.

AuthorsOpenReview ↗

Response to the Reviewer

Q4:Experimental Settings and Extended Motivation Analysis.

  1. Sampling and Decoding Settings. For the motivation analysis in Figures 1–2, we set the maximum generation length to 16,384 tokens, the temperature to 0.6, and top-p to 0.95. To ensure the reliability of the statistical analysis, we independently sample 20 reasoning trajectories for each problem and compute the relevant statistics based on these samples.

  2. Problem Difficulty and Reasoning Length. In Figure 1(a), we group problems according to their difficulty and observe that incorrect responses exhibit more reflections than correct responses even at the same difficulty level. Regarding reasoning length, we adopt a uniform maximum generation length and allow the models to naturally terminate their reasoning within this budget. This is because our study aims to investigate the relationships among reflection count, reasoning length, reasoning continuity, and accuracy during natural reasoning, where reasoning length itself is an important variable potentially influenced by reflection behavior. Forcibly truncating reasoning trajectories to enforce a uniform generation length may disrupt the reasoning process, alter the original reflection behavior and final answer correctness, and introduce additional confounding factors. Therefore, we preserve naturally generated reasoning trajectories to better characterize the relationship between reflection behavior and reasoning performance.

  3. Extended Motivation Analysis. In Appendix A, we extend the motivation analysis to two models, DS-1.5B and Qwen3-4B, and two datasets, MATH-500 and MinervaMath. The results show that our original observations remain consistent across different models and datasets, further strengthening the empirical foundation of our motivation.

  4. Independent Contribution of Answer Entropy. In Table 2, we add two ablation experiments using only sampling accuracy and only answer entropy, respectively. Compared with these two variants, EARR improves average accuracy by 0.6 and 1.0 percentage points, while increasing the reasoning length reduction ratio by 4.1 and 3.4 percentage points, respectively. These results suggest that answer entropy provides complementary information beyond sampling accuracy and improves the effectiveness of reflection reward modulation.

  5. Effectiveness Across Model Scales. In Table 7 of the Appendix, we provide additional comparative experiments on Qwen3-8B and Qwen3-14B. EARR reduces reasoning length by 48.8% and 44.3% while improving average accuracy by 1.5 and 3.1 percentage points, respectively, further demonstrating the effectiveness and scalability of EARR across different model sizes.

Q5:Statistical Reliability.

We would first like to clarify that the evaluation results reported in our paper are averaged over multiple independent runs, as described iin the Implementation Details of Section 5.1 in the revised manuscript.

To further assess the statistical reliability of our results, we report the mean and standard deviation of accuracy and reasoning length over five independent evaluations of EARR on DS-1.5B, as shown below.

DatasetAccuracy (%)Reasoning Length
AIME2432.00 ± 1.106638 ± 60.25
AIME2522.50 ± 0.516498 ± 50.48
AMC2375.33 ± 0.703451 ± 55.25
MATH50083.16 ± 0.951932 ± 47.89
MinervaMath27.21 ± 0.652248 ± 37.37
OlympiadBench46.50 ± 0.424190 ± 48.32

The results show that the standard deviation of accuracy does not exceed 1.10 percentage points, and that of reasoning length does not exceed 61 tokens across all six benchmarks, demonstrating the stability of EARR across independent stochastic decoding runs.

AuthorsOpenReview ↗

Looking Forward to Further Discussion

Dear Reviewer,

We have provided detailed responses to your concerns and sincerely hope that our clarifications have addressed your questions.

As the rebuttal period is coming to an end, we would like to kindly follow up on our responses. We greatly value this opportunity to engage in a constructive discussion with you and sincerely appreciate your insights. If you have any remaining concerns or questions, we would be more than happy to provide further clarifications.

We sincerely look forward to hearing your thoughts and appreciate your time and consideration.

Thank you for the response, most of my concerns are resolved, and I will raise my rating.

Reviewer_5k7gOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets overthinking in large reasoning models and proposes EARR, an outcome reward for GRPO. For each training question, N sampled trajectories are profiled by reasoning length, reflection count, and continuous reasoning span. Each statistic is z-scored within the group and passed through a sigmoid to form a penalty, and the three penalties are mixed with weights 0.15, 0.05 and 0.05. The penalty is then scaled by a difficulty factor equal to acc⋅(1−H)acc \cdot (1-H) for correct trajectories and (1−acc)⋅H(1-acc) \cdot H for incorrect ones, where accacc is the group accuracy and HH is the normalized entropy of the sampled answers. Models are trained on the 7,500 MATH training problems with N = 10. On DeepSeek-R1-Distill-Qwen-1.5B, average length over six math benchmarks drops by 53.4% with a 4.0 point accuracy gain, and on Qwen3-4B it drops by 57.3% with a 1.4 point gain.

Critical strengths

1. Consistent efficiency gains over strong baselines. In Table 1, EARR gives the largest average length reduction on both backbones while keeping or improving average accuracy, and it beats ARLCP, the closest method, on both averages. The trend holds on Qwen3-8B and on MMLU, where the 1.5B model's average length falls from 2,183 to 772 tokens.

2. The method does not rely on a carefully built lexicon. Table 9 shows that using only "wait" as the reflection marker already gives a 5.1 point accuracy gain and a 43.5% length reduction. Larger word lists mostly trade a little accuracy for shorter outputs. This makes the approach cheap to apply to other models, and the authors openly discuss false positives from words like "but" and "however".

Critical weaknesses

1. The value of answer entropy is not shown. No variant uses accuracy alone in gg, so the main claim is never tested. Removing gg gives higher accuracy (+4.7 vs +4.0) with less compression (34.7% vs 53.4%), so gg seems to trade accuracy for length rather than improve both.

2. The reflection analysis measures the training target. ARN and CRS are counted with the same word list the reward penalizes, including common words like "but" and "check". A lower ARN may only mean the model stopped using these words, which does not show "fewer but more effective reflections".

Questions for authors
  1. How is HH normalized to [0, 1]? Could you report a variant where gg uses accuracy only? If "w/o gg" means g=1g = 1, its penalty is never weaker than in EARR, so why does the length reduction drop from 53.4% to 34.7%?

  2. Do ARN and CRS still improve when measured by a detector independent of the training lexicon, for example an LLM judge?

2 comments
AuthorsOpenReview ↗

Response to the Reviewer

We apologize for the delayed response due to the computational demands of LLM RL training. Thank you very much for carefully reviewing our manuscript and providing valuable comments. We hope that the following responses satisfactorily address the points you have raised.

Q1:Independent Contribution of Answer Entropy.

We acknowledge that, compared with the variant without the difficulty-aware factor gg, the full EARR achieves 0.7 percentage points lower average accuracy but increases the length reduction ratio by 18.7 percentage points. Given that EARR aims to reduce reasoning costs while maintaining reasoning performance, we consider this trade-off reasonable. Notably, compared with the original model, EARR still achieves an accuracy improvement of 4.0 percentage points and a length reduction of 53.4%.

Furthermore, to investigate the independent contribution of answer entropy, we have added two ablation experiments to Table 2 of the revised manuscript, using only sampling accuracy (gaccg_{\mathrm{acc}}) and only answer entropy (gentg_{\mathrm{ent}}), respectively. Compared with these two variants, the full EARR improves average accuracy by 0.6 and 1.0 percentage points and increases the length reduction ratio by 4.1 and 3.4 percentage points, respectively. These results indicate that although introducing the difficulty-aware factor involves a certain accuracy–length trade-off, jointly modeling answer entropy and sampling accuracy can improve both metrics simultaneously, rather than merely achieving greater compression at the expense of accuracy.

Q2:Entropy Normalization and Difficulty-Aware Modulation.

First, regarding answer entropy normalization, we have explicitly provided its formulation in Line 249 of the revised manuscript. Specifically, we use the maximum possible entropy corresponding to the number of distinct answer classes as the normalization factor, ensuring that answer entropy lies within [0,1][0,1].

Second, regarding the reduced compression effectiveness after removing the difficulty-aware factor, although setting g=1g=1 results in absolute behavior penalties no smaller than those in the full EARR, GRPO optimizes the policy based on the relative rewards of sampled trajectories within each group rather than the absolute magnitude of the penalties. Therefore, stronger absolute penalties do not necessarily lead to more effective reasoning compression. Unlike uniformly applying behavior penalties, EARR adaptively adjusts the penalty strength according to sampling accuracy, answer entropy, and trajectory correctness, thereby reshaping the relative reward relationships among different reasoning trajectories and potentially enabling more targeted suppression of redundant reflection and ineffective exploration.

Q3:Lexicon-Independent Reflection Behavior Analysis.

To further validate the effectiveness of EARR beyond the training reflection lexicon, we have conducted additional experiments using an independent LLM judge, with the results presented in Figure 7 of Appendix C and the evaluation prompt provided in Figure 8.

The advantages of EARR in both ARN and CRS persist when reflection events are identified semantically rather than lexically. Specifically, EARR achieves the lowest ARN across all six benchmarks and the best or tied-best CRS on every benchmark. These results provide further evidence that EARR improves the underlying reasoning behavior rather than merely suppressing specific reflection words used during training.

Thank you for your effort in this rebuttal. I think most of my concerns have been addressed, and I will be raising my rating accordingly.

Reviewer_sgQoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses overthinking in large reasoning models. A sampling study on MATH-500 with R1-Distill-Qwen-1.5B suggests that more reflection words go with lower accuracy and shorter "continuous reasoning spans" (the average text length between two reflections), and that the entropy of the sampled answers tracks problem difficulty.

EARR is a GRPO reward: a correctness reward minus a penalty built from group-normalized response length, reflection count, and reasoning span. The penalty is scaled by a difficulty factor that is large for correct rollouts on easy, low-entropy problems and for incorrect rollouts on hard, high-entropy problems. The authors train on MATH with R1-Distill-1.5B and Qwen3-4B (plus Qwen3-8B in the appendix) and evaluate on six math benchmarks and MMLU. They report 53–57% shorter responses with +1.4 to +4.0 average accuracy, ahead of ARLCP, AdaptThink, O1-Pruner, DPO, SFT and CoD. Ablations, coefficient sweeps, and reflection-lexicon variants are also included.

Critical strengths

The empirical results are strong and fairly broad. On both backbones EARR gives the largest length reduction in Table 1 while keeping or improving accuracy, against a reasonable set of recent baselines. The reward is simple and cheap to add to any GRPO pipeline.

Critical weaknesses
  1. The novelty is limitted. Joint length and reflection penalties are basically ARLCP, and difficulty-aware length penalties already exist (e.g., DAST). The ablation in Table 2 shows that removing the entropy factor or the length term gives higher accuracy with less compression, so the components seem to move along an accuracy–length curve rather than improve on it.

  2. The motivation in Sec. 3 comes from one small model (R1-Distill-1.5B) on MATH-500. The experiments also stop at Qwen3-4B/8B, with no recent thinking models such as Qwen3.8.

  3. All results are single runs,. Decoding settings are not given, and it is unclear whether baselines were retrained under the same setup.

Questions for authors
  1. Do the observations in Sec. 3 still hold for Qwen3-4B/8B, and within problems of the same difficulty? Can you add results on a recent or larger thinking model such as Qwen3.8 or a 14B/32B model?

  2. What are the evaluation decoding settings (temperature, max tokens, samples per problem), what is the variance across seeds, and were all baselines retrained under the same setup?

5 comments
AuthorsOpenReview ↗

Response to the Reviewer

We apologize for the delayed response due to the computational demands of LLM RL training. We sincerely appreciate your careful review and positive feedback, and hope our responses address your questions.

Q1: Novelty

First, unlike ARLCP, which primarily focuses on reflection count and reasoning length, EARR introduces the CRS to explicitly characterize reasoning continuity between reflection events, thereby suppressing fragmented reasoning. Second, unlike DAST, which adjusts reasoning length according to problem difficulty, EARR jointly incorporates sampling accuracy and answer entropy to characterize reasoning uncertainty and adaptively modulates behavior penalties according to trajectory correctness, targeting redundant reflection in simple problems and ineffective exploration in highly uncertain problems.

Regarding the accuracy–length trade-off, we acknowledge that removing the length penalty or the difficulty-aware factor yields higher accuracy at the expense of compression effectiveness. However, to further investigate the independent contribution of answer entropy, we have added two ablation variants to the revised Table 2, using only sampling accuracy and only answer entropy, respectively. Compared with these two variants, EARR improves average accuracy by 0.6 and 1.0 percentage points while increasing the length reduction ratio by 4.1 and 3.4 percentage points, respectively. Moreover, removing the CRS penalty leads to decreases in both accuracy and compression effectiveness. These results demonstrate that jointly modeling sampling accuracy and answer entropy, together with reasoning continuity constraints, can simultaneously improve accuracy and reasoning efficiency, rather than merely shifting the operating point along the accuracy–length trade-off curve.

Furthermore, as shown in Table 1, EARR achieves both higher average accuracy and greater length reduction than ARLCP on DS-1.5B and Qwen3-4B under the same evaluation benchmarks, further supporting the effectiveness of our proposed mechanisms.

Q2:Generality of Motivating Observations

To address this concern, we have extended our motivation analysis to two reasoning models (DS-1.5B and Qwen3-4B) across two mathematical reasoning benchmarks (MATH-500 and MinervaMath). As shown in Figures 5 and 6 in Appendix A, the three observations in Section 3 remain consistent across different models and benchmarks, further supporting the generality of our motivation.

Q3:Scalability to Larger and More Recent Reasoning Models

To address this concern, we have added experimental results on Qwen3-14B to Table 7 of the revised manuscript. Due to time constraints, we select two relatively strong baselines, AdaptThink and ARLCP, for comparison, while also reporting the performance of the Vanilla model. The results further validate the effectiveness of EARR. With this addition, our method has been evaluated on four models of different sizes: DS-1.5B, Qwen3-4B, Qwen3-8B, and Qwen3-14B, providing further empirical evidence for the scalability of EARR. Regarding Qwen3.8 suggested by the reviewer, its smallest model has 27B parameters. Due to our current computational resource constraints, we are unable to conduct training experiments at this scale and leave the exploration of larger reasoning models to future work.

Q4:Experimental Settings and Reproducibility

First, we would like to clarify that all experimental results reported in our paper are averaged over multiple runs rather than obtained from a single run. We have explicitly clarified this in the Implementation Details of Section 5.1 in the revised manuscript, where we also provide the inference settings, including temperature, maximum generation length, and the number of samples per problem. Furthermore, to ensure a fair comparison, all baselines requiring training were retrained under the same experimental settings.

To further assess the statistical reliability of our results, we report the mean and standard deviation of accuracy and reasoning length over five independent evaluations of EARR on DS-1.5B, as shown below.

DatasetAccuracy (%)Reasoning Length
AIME2432.00 ± 1.106638 ± 60.25
AIME2522.50 ± 0.516498 ± 50.48
AMC2375.33 ± 0.703451 ± 55.25
MATH50083.16 ± 0.951932 ± 47.89
MinervaMath27.21 ± 0.652248 ± 37.37
OlympiadBench46.50 ± 0.424190 ± 48.32

The results show that the standard deviation of accuracy does not exceed 1.10 percentage points, and that of reasoning length does not exceed 61 tokens across all six benchmarks, demonstrating the stability of EARR across independent stochastic decoding runs.

Thanks for your reponse, however, without the experiments on larger models, my concern remains, I will keep my original score.

AuthorsOpenReview ↗

Response to the Reviewer

Thank you for your response. We would like to emphasize that, to address your concerns regarding scalability, we have conducted additional experiments on Qwen3-14B, with the results reported in Table 7 of the revised manuscript. On Qwen3-14B, EARR reduces reasoning length by 44.3% while improving average accuracy by 3.1 percentage points, outperforming both AdaptThink and ARLCP in terms of overall accuracy and reasoning efficiency.

Our method has now been evaluated on models of four different sizes (1.5B, 4B, 8B, and 14B), consistently achieving significant reductions in reasoning length while improving average accuracy. We believe that these consistent experimental results across a broad range of model sizes provide sufficient empirical evidence for the scalability of EARR within the evaluated range.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.