- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes High-Entropy Sum (HES), a training-free metric for selecting high-quality reasoning data for large language models. The method calculates the sum of entropy values from the highest-entropy tokens in a reasoning trajectory to estimate its quality. The authors apply HES to three training paradigms: Supervised Fine-Tuning (SFT), Rejection Fine-Tuning (RFT), and Reinforcement Learning (RL). Experimental results across mathematical reasoning, STEM, and coding tasks demonstrate that HES can improve data selection efficiency and model performance. In particular, selecting only 20% of the training data achieves performance comparable to using the full dataset in SFT, while HES-based selection also shows improvements in RFT and RL.
Critical strengths
-
The proposed HES method is simple and practical. Instead of evaluating the entire reasoning trajectory equally, the method focuses on high-entropy tokens that may represent important decision points during reasoning.
-
The same metric can be applied across SFT, RFT, and RL without requiring an additional reward model. The experimental results are promising, and the comparisons with random selection, length-based selection, and other entropy-based metrics provide useful evidence for the effectiveness of HES.
Critical weaknesses
- The relationship between high-entropy tokens and reasoning quality could be explained more clearly. The paper assumes that high-entropy tokens represent important decision points in the reasoning process. Although the experimental results support the usefulness of HES, high entropy may also indicate uncertainty or confusion rather than meaningful reasoning. It would be helpful to provide more qualitative examples or analysis to explain why high-entropy tokens are reliable indicators of reasoning quality.
- The experimental improvements are not always substantial. For example, Table 7 shows that the proposed RL selection strategy achieves an average accuracy of 19.26%, compared with 18.34% for the full-batch baseline. Although this is an improvement, the difference is relatively small. It would be useful to include additional experimental runs or statistical analysis to demonstrate the reliability of these improvements.
Questions for authors
please refer to "Weaknesses"
2 comments
Dear Reviewer,
As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.
If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.
Thank you again for your time and valuable feedback.
Dear Reviewer,
Thank you for recognizing the practicality of HES. Our experiments support two central advantages: effective selection of successful reasoning trajectories and improved accuracy–cost trade-offs.
W1: High-entropy decisions and training value
HES combines correctness verification with cumulative uncertainty. Appendix O specifies verified SFT demonstrations, correctness-filtered RFT candidates, and high-HES positive RL rollouts with random negatives. Verification identifies successful trajectories, and HES prioritizes their uncertain decision points.
Figure 1 characterizes uncertainty across correct and incorrect paths; the training experiments evaluate the benefit of HES within the corresponding correctness-conditioned pools. This gives a concrete interpretation: successful trajectories containing uncertain decisions can provide informative demonstrations under a limited training budget.
We provide both qualitative and quantitative evidence. Figures 4–5 and Appendix H visualize the sparse high-entropy tail. The token-colored example on pp. 26–27 illustrates strategy choice and verification: the trajectory reformulates the signed sum from 0 to 2000 as the sum of all integers minus twice the sum of powers of two, then checks the formula on smaller cases.
Downstream ablations measure the utility of the selection rule directly:
The RFT comparison holds query identity fixed, and the RL comparison holds negative sampling fixed (Tables 5–7). These controls support the training value of prioritizing high-HES successful trajectories. The qualitative analysis illustrates the reasoning interpretation, while the downstream gains establish practical selection utility.
Length-based selection provides another direct comparison. HES exceeds Length by 2.29 points in Math SFT, 1.14 on GPQA, 0.59 on LiveCodeBench, 1.40 in RFT, and 1.18 in RL (Table 16). At equal 20% example retention, Table 13 reports 139.10M versus 180.48M selected tokens, alongside scores of 31.34 versus 29.05. This 22.9% token saving with higher accuracy demonstrates the practical advantage of using tail uncertainty to prioritize reasoning demonstrations.
W2: Performance gains, evaluation evidence, and efficiency
HES improves average performance across SFT, RFT, and RL. Appendix L reports the following comparisons with 95% interval half-widths:
Math evaluation uses Average@16, averaging 16 sampled generations. The intervals quantify uncertainty around the reported scores, and the positive point-estimate gains across training paradigms support a recurring selection benefit.
Appendix G extends the evidence across datasets and model families. With Qwen3-8B on OpenR1-Math-220k, HES-80% achieves 31.13 versus 29.38 for Fullset (Table 8). With Qwen3-4B on Open-Math-Reasoning, it achieves 29.14 versus 26.63 (Table 9). With DeepSeek-R1-Distill-Qwen-7B on OpenR1-Math-220k, HES-20% achieves 33.41 versus 29.08 for Fullset (Table 10). On the Llama-based RFT model, HES exceeds Random in both per-query selection (13.23 versus 12.42) and global selection (13.54 versus 11.85) at k=4 (Table 11). These additional comparisons broaden the empirical support for HES across data sources, model sizes, and architectures. Each comparison evaluates selected subsets using the corresponding common training and evaluation setup.
In RL, HES improves over Full-Batch on six of seven benchmarks and exceeds equally sized Random and Length selections by 1.75 and 1.18 points, respectively (Table 7). The asymmetric design is also supported by the ablation: high-HES positives with random negatives achieve 19.26 versus 17.28 with low-HES negatives.
The performance benefit comes with measured training savings. Both HES and Full-Batch generate 32 rollouts per prompt. HES selects 50% for policy updates, reducing end-to-end time from approximately 15 to 13 hours on 8×A100 GPUs, including generation, scoring, and updates (Appendix N). This 13.3% wall-clock saving, together with the higher average score, demonstrates the practical accuracy–cost advantage of HES in the evaluated setting.