acceptodds
Under review as a conference paper at ICLR 2027

Unified Data Selection for LLM Reasoning

Abstract

Effectively training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data. Existing methods are either computationally expensive or fail to reliably distinguish high- from low-quality reasoning samples. To address this, we propose High-Entropy Sum (HES), a training-free metric that quantifies reasoning quality by summing only the entropy of the top (e.g., 0.5%) highest-entropy tokens in each reasoning sample. We validate HES across three mainstream training paradigms: Supervised Fine- tuning (SFT), Rejection Fine-tuning (RFT), and Reinforcement Learning (RL), with extensive results demonstrating its consistent effectiveness and significantly reduced computational overhead. In SFT, training on the top 20% HES-ranked data matches full-dataset performance, while using the lowest-HES data degrades it. In RFT, our HES-based training approach significantly outperforms baseline methods. In RL, HES-selected successful trajectories enable the model to learn strong reasoning patterns, significantly surpassing other compared methods. Our findings establish HES as a robust, training-free metric that enables a unified, effective, and efficient method for developing advanced reasoning in LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning, data selection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_SjwhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes High-Entropy Sum (HES), a training-free metric for selecting high-quality reasoning data for large language models. The method calculates the sum of entropy values from the highest-entropy tokens in a reasoning trajectory to estimate its quality. The authors apply HES to three training paradigms: Supervised Fine-Tuning (SFT), Rejection Fine-Tuning (RFT), and Reinforcement Learning (RL). Experimental results across mathematical reasoning, STEM, and coding tasks demonstrate that HES can improve data selection efficiency and model performance. In particular, selecting only 20% of the training data achieves performance comparable to using the full dataset in SFT, while HES-based selection also shows improvements in RFT and RL.

Critical strengths
  • The proposed HES method is simple and practical. Instead of evaluating the entire reasoning trajectory equally, the method focuses on high-entropy tokens that may represent important decision points during reasoning.

  • The same metric can be applied across SFT, RFT, and RL without requiring an additional reward model. The experimental results are promising, and the comparisons with random selection, length-based selection, and other entropy-based metrics provide useful evidence for the effectiveness of HES.

Critical weaknesses
  • The relationship between high-entropy tokens and reasoning quality could be explained more clearly. The paper assumes that high-entropy tokens represent important decision points in the reasoning process. Although the experimental results support the usefulness of HES, high entropy may also indicate uncertainty or confusion rather than meaningful reasoning. It would be helpful to provide more qualitative examples or analysis to explain why high-entropy tokens are reliable indicators of reasoning quality.
  • The experimental improvements are not always substantial. For example, Table 7 shows that the proposed RL selection strategy achieves an average accuracy of 19.26%, compared with 18.34% for the full-batch baseline. Although this is an improvement, the difference is relatively small. It would be useful to include additional experimental runs or statistical analysis to demonstrate the reliability of these improvements.
Questions for authors

please refer to "Weaknesses"

2 comments

Dear Reviewer,

Thank you for recognizing the practicality of HES. Our experiments support two central advantages: effective selection of successful reasoning trajectories and improved accuracy–cost trade-offs.

W1: High-entropy decisions and training value

HES combines correctness verification with cumulative uncertainty. Appendix O specifies verified SFT demonstrations, correctness-filtered RFT candidates, and high-HES positive RL rollouts with random negatives. Verification identifies successful trajectories, and HES prioritizes their uncertain decision points.

Figure 1 characterizes uncertainty across correct and incorrect paths; the training experiments evaluate the benefit of HES within the corresponding correctness-conditioned pools. This gives a concrete interpretation: successful trajectories containing uncertain decisions can provide informative demonstrations under a limited training budget.

We provide both qualitative and quantitative evidence. Figures 4–5 and Appendix H visualize the sparse high-entropy tail. The token-colored example on pp. 26–27 illustrates strategy choice and verification: the trajectory reformulates the signed sum from 0 to 2000 as the sum of all integers minus twice the sum of powers of two, then checks the formula on smaller cases.

Downstream ablations measure the utility of the selection rule directly:

ComparisonHESComparatorGain
SFT: HES vs. all-token ES, 20% retention31.3429.23+2.11
RFT: Highest-HES vs. Lowest-HES, per-query k=4k=430.2027.61+2.59
RL: high-HES vs. low-HES positives, random negatives19.2617.18+2.08

The RFT comparison holds query identity fixed, and the RL comparison holds negative sampling fixed (Tables 5–7). These controls support the training value of prioritizing high-HES successful trajectories. The qualitative analysis illustrates the reasoning interpretation, while the downstream gains establish practical selection utility.

Length-based selection provides another direct comparison. HES exceeds Length by 2.29 points in Math SFT, 1.14 on GPQA, 0.59 on LiveCodeBench, 1.40 in RFT, and 1.18 in RL (Table 16). At equal 20% example retention, Table 13 reports 139.10M versus 180.48M selected tokens, alongside scores of 31.34 versus 29.05. This 22.9% token saving with higher accuracy demonstrates the practical advantage of using tail uncertainty to prioritize reasoning demonstrations.

W2: Performance gains, evaluation evidence, and efficiency

HES improves average performance across SFT, RFT, and RL. Appendix L reports the following comparisons with 95% interval half-widths:

ComparisonHESBaselineGain
SFT, HES-20% vs. Random-20%31.34 ± 1.3623.86 ± 1.32+7.48
SFT, HES-80% vs. Fullset34.61 ± 1.4331.83 ± 1.41+2.78
RFT, per-query k=4k=4, vs. Length30.20 ± 1.3928.80 ± 1.37+1.40
RL, vs. Full-Batch19.26 ± 1.2018.34 ± 1.18+0.92

Math evaluation uses Average@16, averaging 16 sampled generations. The intervals quantify uncertainty around the reported scores, and the positive point-estimate gains across training paradigms support a recurring selection benefit.

Appendix G extends the evidence across datasets and model families. With Qwen3-8B on OpenR1-Math-220k, HES-80% achieves 31.13 versus 29.38 for Fullset (Table 8). With Qwen3-4B on Open-Math-Reasoning, it achieves 29.14 versus 26.63 (Table 9). With DeepSeek-R1-Distill-Qwen-7B on OpenR1-Math-220k, HES-20% achieves 33.41 versus 29.08 for Fullset (Table 10). On the Llama-based RFT model, HES exceeds Random in both per-query selection (13.23 versus 12.42) and global selection (13.54 versus 11.85) at k=4k=4 (Table 11). These additional comparisons broaden the empirical support for HES across data sources, model sizes, and architectures. Each comparison evaluates selected subsets using the corresponding common training and evaluation setup.

In RL, HES improves over Full-Batch on six of seven benchmarks and exceeds equally sized Random and Length selections by 1.75 and 1.18 points, respectively (Table 7). The asymmetric design is also supported by the ablation: high-HES positives with random negatives achieve 19.26 versus 17.28 with low-HES negatives.

The performance benefit comes with measured training savings. Both HES and Full-Batch generate 32 rollouts per prompt. HES selects 50% for policy updates, reducing end-to-end time from approximately 15 to 13 hours on 8×A100 GPUs, including generation, scoring, and updates (Appendix N). This 13.3% wall-clock saving, together with the higher average score, demonstrates the practical accuracy–cost advantage of HES in the evaluated setting.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Reviewer_idJrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I find the paper clear and generally well written. I was able to follow the motivation, the proposed method, and the experimental results without much difficulty. This paper proposes High-Entropy Sum (HES), a simple metric that uses high-entropy tokens to select reasoning data for SFT, RFT, and RL. The results show that HES improves training efficiency and often outperforms the tested baselines. I find the work useful and the results encouraging.

Critical strengths
  1. The method is simple and efficient. HES is easy to understand and implement. It does not require an additional trained selector. It can reuse logits during RFT and RL generation, and use a smaller model for offline SFT data selection. This makes the method practical and potentially cost-effective.

  2. The experiments cover SFT, RFT, and RL, with SFT results on Math, STEM, and Code. HES generally outperforms the tested baselines. In Math SFT, using only 20% of the data achieves performance close to full-data training. The paper also compares different entropy metrics and studies the effect of selection ratios. These results provide useful evidence for the effectiveness of HES.

Critical weaknesses
  1. I found that Zhu et al. (2025) [1] already proposed EAS, which uses cumulative token-level entropy to select reasoning data for SFT. This idea seems closely related to HES. I would like the authors to clarify the key differences between HES and EAS. A direct comparison under the same settings would be better. Additionally, it is necessary to include comparisons with other relevant data sampling baselines (such as diversity-based sampling).

[1] Zhu et al. "Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLM." 2025. https://arxiv.org/abs/2508.20384

  1. Since HES sums the entropy of the top 0.5% of tokens, longer responses with more reasoning steps may receive higher HES scores. HES may favor longer responses. Moreover, incorrect trajectories often have higher HES scores than correct ones. Although the method selects from verified correct trajectories, it remains unclear whether HES captures reasoning quality beyond length and uncertainty.

  2. The title seem too broad. The paper focuses on a specific entropy-based method for selecting reasoning trajectories. A more precise title would be better.

Questions for authors
  1. How does HES differ from EAS? Could the authors clarify the key differences and explain the advantage of HES over EAS?
  2. The improvement in RL is relatively small (19.26 vs. 18.34 for Full-Batch). Were the experiments repeated with different training seeds? Could the observed improvement be due to evaluation variance?
2 comments

Dear Reviewer,

Thank you for recognizing HES's simplicity and practical usefulness. We highlight its native-logit implementation, consistent gains over length selection, and measured accuracy–cost benefit.

W1 / Q1: EAS positioning and coverage

HES and EAS both accumulate uncertainty. HES's distinctive design combines ordinary trajectory probabilities with focused tail aggregation:

AspectEASHES
Scoring contextAnswer-oriented prefix probes with \boxed{ and an answer prefixOrdinary next-token distributions
AggregationSum across probed positionsSum of the highest-entropy 0.5%
Online integrationConstructed probing contextsReuse of generation logits

HES integrates directly into rollout generation, and its common scoring construction supports Math, STEM, and Code SFT as well as RFT and RL (Tables 2–7; Appendix M). We validate the aggregation component with a fixed probability source: HES scores 31.34 versus 29.23 for native-logit ES (Table 5). This 2.11-point gain supports focused tail selection within our scoring setup. HES exceeds ES on all seven Math benchmarks in Table 5, demonstrating gains across the evaluation suite.

We also incorporate coverage into selection. Per-query HES achieves 30.20 versus 28.59 for global HES at k=4k=4 (Table 6). In RL, high-HES positives with random negatives achieve 19.26 versus 17.28 with low-HES negatives (Table 7). These experiments support combining informative-example ranking with broad coverage of questions and negative trajectories.

W2: Training utility beyond length ranking

Verification and HES work together: SFT uses verified demonstrations, RFT verifies candidate answers, and RL ranks positive trajectories while sampling negatives randomly (Appendix O). This protocol prioritizes successful demonstrations with substantial cumulative uncertainty.

Table 16 directly compares HES with length selection:

SettingLengthHESGain
Math SFT, 20%29.0531.34+2.29
GPQA, 20%50.8251.96+1.14
LiveCodeBench, 20%53.8454.43+0.59
RFT, per-query k=4k=428.8030.20+1.40
RL18.0819.26+1.18

HES improves all five comparisons. Appendix J reports a length correlation of 0.8703 together with 49.69% top-20% subset overlap, showing substantially different selections. Table 13 further reports 139.10M versus 180.48M selected tokens: HES gains 2.29 points with 22.9% fewer tokens at equal example retention. These results demonstrate the practical value of combining length and tail uncertainty.

Table 5 further isolates the cumulative component. Since HES/N≈p AvgHE\mathrm{HES}/N\approx p\,\mathrm{AvgHE}, AvgHE provides the corresponding length-normalized ranking up to rounding. HES reaches 31.34 versus 25.96 for AvgHE, a 5.38-point gain, supporting the value of retaining cumulative uncertainty. The deployment benefit extends to offline scoring: a 0.6B proxy selects data for the 8B target with 85.89% subset overlap with self-selection and a downstream score of 30.20 (Table 4; Appendix M). Together, these results support both the scoring design and its inexpensive implementation.

W3: Scope of the shared selection framework

“Unified” describes the same HES score across three training paradigms. Section 3.2 specifies the SFT, RFT, and RL selection protocols; Tables 2–7 validate their applications. The contribution is precisely captured by “High-Entropy Sum for Reasoning Data Selection across SFT, RFT, and RL.”

Q2: Evaluation evidence and RL efficiency

Our evaluation evidence combines Average@16 sampling, reported 95% confidence intervals, and controlled selection ablations. Average@16 averages 16 sampled generations for Math evaluation. Appendix L reports:

RL strategyScore and 95% interval half-width
Full-Batch18.34 ± 1.18
Random positives and negatives17.51 ± 1.17
Length-selected positives, random negatives18.08 ± 1.18
High-HES positives, random negatives19.26 ± 1.20

HES achieves the highest average point estimate: +0.92 over Full-Batch, +1.75 over Random, and +1.18 over Length, with higher scores than Full-Batch on six of seven benchmarks (Table 7). Controlled ablations also favor the proposed combination: 19.26 versus 17.28 with low-HES negatives, and versus 17.18 with low-HES positives.

Both methods generate 32 rollouts per prompt; HES selects 50% for policy updates. Including generation, scoring, and updates, training takes 13 versus 15 hours on 8×A100 GPUs, a 13.3% wall-clock saving (Appendix N). The experiments therefore demonstrate an observed improvement in average accuracy alongside lower end-to-end cost.

Thank you for the response and clarification. This addresses some of my concerns. However, I'm still not fully convinced that HES can serve as a unified approach to data selection. I agree with Reviewer cfyx that experimental comparisons with EAS and other relevant data selection baselines under consistent settings would further strengthen the paper.

Reviewer_cfyxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Since training an LLM on large volumes of data is inefficient and time-consuming, the paper proposes a method to select a subset of the data. The method builds on prior work [1], which shows that a small number of high-entropy “forking” tokens drive performance gains in RL. The authors use this idea for data selection and propose High-Entropy Sum (HES). Specifically, HES computes token-level entropy from the model’s predicted distribution and sums the entropies of the top 0.5% highest-entropy tokens. The authors apply HES to three training paradigms: (1) SFT keeps the top-ranked samples, (2) RFT selects the highest-HES correct responses, and (3) GRPO keeps the highest-HES correct rollouts together with randomly sampled incorrect rollouts. The authors report that training on the top 20% of HES-ranked SFT data matches full-dataset performance. In RFT and RL, HES selection outperforms random and length-based baselines.

[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning NeurIPS 2025

Critical strengths
  1. The paper is clearly written and organized, making the metric and its use in each training paradigm easy to follow.
  2. The proposed method HES is a simple, training-free metric works for data selection across SFT, RFT, and RL.
  3. The paper includes a lot of experiments.
Critical weaknesses
  1. Many methods exist for selecting subsets of data for LLM training, and a thorough comparison with them would be valuable. The paper compares HES only with basic criteria such as random, length, difficulty, and other entropy statistics, which is not sufficient. Several relevant methods are missing: (1) DEITA [2], which selects data using learned complexity and quality scores. (2) IFD [3], which scores samples using the model’s own loss. The paper cites it but does not compare against it. Superfiltering [4] computes this score with a small proxy model, similar to the proxy setting in the paper. (3) LIMR [5], which selects training samples for RL.
  2. The paper appears to assume that data on which the model is likely to produce an incorrect response is high-quality data. The reviewer finds this assumption difficult to accept, for two reasons: (1) Data that the model would answer correctly may also contain important knowledge. (2) Training on data biased in this way could degrade the model, and a balanced set of data can sometimes be more important. It would be helpful if the authors could address these concerns about what they define as high-quality data.
  3. The motivation in Figure 1 seems largely expected in this field: summing only high-entropy tokens would naturally widen the distance between the correct and incorrect means. Let ES denote the entropy sum over all tokens and HES the sum over high-entropy tokens. ES also separates correct and incorrect responses well, and the separation does not appear to differ much from that of HES. In Figure 3, applying ES (ratio 1.0) yields performance close to that of HES. This may reduce the value of the paper’s main claim that scoring should focus on forking tokens.
  4. The core idea of focusing on high-entropy forking tokens comes from [1]. Since data selection and RL both belong to LLM training, the reviewer considers the application of this idea to data selection a limited contribution. The paper presents many experiments, but the results do not seem to offer surprising findings or new insights to the community.

[1] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning, NeurIPS 2025
[2] What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, ICLR 2024
[3] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning, NAACL 2024
[4] Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning, ACL 2024
[5] LIMR: Less is More for RL Scaling, arXiv 2025

Questions for authors
  1. The title is misleading. What does "unified" mean in "Unified Data Selection"?
  2. The formulation does not make the importance of high-entropy tokens clear. By the paper's definition, HES = |T_high| × AvgHE = pN × AvgHE, where N is the response length. Since AvgHE is nearly constant across samples (Figure 1(b)), HES appears to scale linearly with response length. Length-based selection is already effective as a baseline in the paper. Could the authors explain the relationship between HES and length?
3 comments

Dear Reviewer,

Thank you for the detailed feedback. Our experiments demonstrate strong sample efficiency, gains from focused entropy aggregation, and practical reuse across training paradigms.

W1: Positioning and experimental advantages

The methods use complementary signals:

Related approachHES contribution
DEITA: complexity, quality, diversityCumulative high-entropy information within verified reasoning demonstrations.
IFD: loss-derived difficultyDistributional uncertainty along reasoning trajectories.
Superfiltering: weak-to-strong filteringHES proxy selection from 0.6B to 8B.
LIMR: RL question selectionWithin-pool rollout selection with asymmetric positive/negative sampling.

Our controlled comparisons quantify HES's advantages: at 20% SFT retention, it scores 31.34 versus 29.05 for Length, 23.30 for intermediate Difficulty, and 23.86 for Random (Table 2). The 0.6B proxy achieves 30.20 for the 8B target, with 85.89% selection overlap with self-selection (Table 4; Appendix M). Online scoring reuses generation logits, and end-to-end RL takes 13 versus 15 hours for Full-Batch (Appendix N).

W2: Correctness, training value, and coverage

Correctness verification precedes positive selection: SFT ranks verified demonstrations, RFT verifies generated answers, and RL applies HES to positive rollouts while sampling negatives randomly (Appendix O). Figure 1 describes uncertainty across all generations; our training protocols use that signal within correctness-conditioned pools.

Math SFT selectionRetentionScore
Fullset100%31.83
Random20%23.86
Highest-HES20%31.34
Highest-HES80%34.61

HES approaches full-data performance with one-fifth of the examples and exceeds it by 2.78 points at 80% retention (Table 2). Coverage is explicitly incorporated: per-query HES scores 30.20 versus 28.59 for global selection at k=4k=4 (Table 6).

W3: Advantage over all-token entropy

Table 5 isolates aggregation under the same SFT setup and 20% retention:

AvgEAvgHEESHES absoluteHES
25.4825.9629.2328.4731.34

HES improves over ES by 2.11 points, with gains on all seven benchmarks, and over AvgHE by 5.38 points. These results validate both cumulative aggregation and high-entropy focus. Figure 3 complements this comparison with sensitivity sweeps supporting the 0.5% operating point.

The relative-tail design also improves over a fixed entropy threshold by 2.87 points (31.34 versus 28.47). Using each trajectory's entropy ranking adapts the selected tail to its distribution, while retaining a common scoring rule across examples. The five-way ablation therefore supports the specific combination of summation, high-entropy focus, and a relative threshold.

W4 / Q1: Contribution and cross-paradigm unification

Building on Wang et al.'s token-level RL insight, HES introduces a trajectory-level ranking rule used in SFT, RFT, and RL. Section 3.2 defines the shared score and its selection protocols; Tables 2–7 evaluate Math/STEM/Code SFT and Math RFT/RL. This shared selection mechanism is the meaning of “unified.”

The RL ablation adds a concrete design insight: high-HES positives with random negatives achieve 19.26 versus 17.28 with low-HES negatives (Table 7), supporting informative positive selection combined with broad negative coverage.

Q2: Advantage over length selection

The identity HES≈pN AvgHE\mathrm{HES}\approx pN\,\mathrm{AvgHE} shows that HES combines length with tail uncertainty. We evaluate this combination directly. Appendix J reports a length correlation of 0.8703 and 49.69% overlap between the top-20% HES and length subsets. Table 13 shows:

SelectionTokens (millions)Score
Length180.4829.05
HES139.1031.34

Thus HES delivers +2.29 points using 22.9% fewer selected tokens at equal example retention. The normalization ablation further supports cumulative scoring: HES/N≈p AvgHE\mathrm{HES}/N\approx p\,\mathrm{AvgHE}, and HES achieves 31.34 versus 25.96 for AvgHE (Table 5). These results demonstrate a better accuracy–token trade-off.

The comparison extends across domains and training settings: HES exceeds Length by 1.14 points on GPQA, 0.59 on LiveCodeBench, 1.40 in per-query RFT, and 1.18 in RL (Table 16). Output lengths also show efficient learned behavior. HES produces shorter outputs than Random in five of six evaluated settings, including 407.18 fewer tokens in RL and 679.28 fewer tokens in Code SFT (Table 15). These measurements connect effective data ranking with both training-time savings and concise model outputs.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

Thank you for the response and the additional results. Some of my concerns have not been fully addressed, and I summarize them below.

  1. The reviewer still believe the paper lacks discussion of, and experimental comparison with, prior work on data filtering. The baselines are still limited to random, length, difficulty, and other entropy statistics.
  2. Thank you for providing the experiments for W2 and W3. However, the method selects data based on the belief that good-quality samples are those the model does not know well. With longer training or more data, the reviewer expects that training on data selected this way may lead to different results. Using balanced data is a common strategy, and it should be noted that curriculum learning, regularization, and periodic re-injection of previously learned knowledge are widely used in existing data filtering methods.
  3. As written in the summary, the reviewer understood that "unified" means HES can be used in SFT, RFT, and RL. The concern is that the expression itself is still confusing. The current wording gives the impression of a method that uniformly selects from all data in the world, rather than a data filtering method. A title such as "High-Entropy Sum for Data Filtering in Diverse LLM Training" could be considered.
  4. The reviewer agrees with the novelty concern raised by Reviewer Znnv. The forking tokens have already been discovered, and the proposed method comes down to summing their entropy.

The reviewer intends to maintain the original score, and hopes the reviewers' valuable feedback helps improve the work.

Reviewer_ZnnvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes High-Entropy Sum (HES), a training-free reasoning quality metric that sums the entropy of only the top 0.5 percent highest-entropy tokens per sample, since the value of long chains of thought concentrates in a few high-entropy decision forks that averaging dilutes. A unified selection framework applies HES across SFT ranking, RFT correctness-then-HES filtering, and RL asymmetric sampling with top-HES positives and random negatives. The metric accumulates from generation logits at no extra cost and supports small-model proxies.

Critical strengths

HES piggybacks on generation logits with no extra forward pass or training, and the 0.6B proxy transfers to 8B training, making the method immediately deployable in large-scale data pipelines at negligible cost.

Critical weaknesses
  1. Novelty is limited. The role of high-entropy critical tokens comes directly from Wang et al. 2025, and HES merely reformulates it as a sum for ranking. The unified framework is essentially one ranking rule reused in three settings, with no theoretical analysis of the metric itself.
  2. Figure 1 shows incorrect paths have substantially higher HES than correct ones, yet all selection experiments retain the highest HES, a superficially contradictory direction. The paper does not state whether the SFT data was pre-filtered for correctness, nor explain why high HES indicates quality rather than difficulty within correct-only pools.
Questions for authors
  1. Was Mixture-of-Thoughts pre-filtered for correctness in SFT? If it contains incorrect traces, how do you reconcile selecting highest HES with Figure 1 showing higher HES for incorrect paths?
  2. Within correct-only pools, how does HES correlate with problem difficulty or solution diversity? Could high HES simply indicate harder problems?
2 comments

Dear Reviewer,

Thank you for recognizing HES's practical efficiency and proxy transferability. Our contribution combines effective trajectory selection, a shared score across SFT/RFT/RL, and lightweight deployment.

W1: Contribution and metric design

HES turns the high-entropy token insight into a data selection rule. Wang et al. study token-level RL updates; HES aggregates uncertainty to select demonstrations and rollouts, then trains on complete selected trajectories. Our experiments establish three advantages:

AdvantageEvidence
Effective aggregationHES scores 31.34 versus 29.23 for ES and 25.96 for AvgHE at 20% SFT retention (Table 5).
Effective asymmetric RL selectionHigh-HES positives with random negatives score 19.26 versus 17.28 with low-HES negatives (Table 7).
Small-model proxy transferA 0.6B selector achieves 30.20 for an 8B target, versus 31.34 for self-selection, with 85.89% subset overlap (Table 4; Appendix M).

The score also has a direct analytical interpretation. For descending token entropies and k=∣Thigh∣≈pNk=|T_{\mathrm{high}}|\approx pN,

HES=∑i=1kH(i)=k AvgHE.\mathrm{HES}=\sum_{i=1}^{k}H_{(i)}=k\,\mathrm{AvgHE}.

HES retains both the number of selected positions and their uncertainty. The controlled ablation validates this design: +5.38 points over high-entropy averaging and +2.11 over all-token summation.

The benefit extends across data budgets and domains. Math SFT with the top 20% reaches 31.34 versus 31.83 for full-data training; retaining 80% raises the score to 34.61 (Table 2). On GPQA, the top 20% achieves 51.96 versus 47.66 for Fullset; on LiveCodeBench, retaining 80% achieves 61.65 versus 58.76 (Table 3). These results demonstrate both sample efficiency and stronger downstream performance through selective training.

W2 / Q1: Correctness-conditioned selection

Yes—the Mixture-of-Thoughts SFT protocol ranks verified demonstrations, as specified in Appendix O and the correct-demonstration formulation in Section 2.1. Verification and HES provide complementary signals:

ParadigmSelection protocol
SFTRank verified demonstrations by HES.
RFTVerify generated answers, then rank correct candidates.
RLSelect high-HES positive rollouts and randomly sample negatives.

Figure 1 measures uncertainty across correct and incorrect generations. Its higher HES for incorrect paths motivates conditioning selection on correctness. Within the verified pool, HES prioritizes successful trajectories containing uncertain decisions. Table 6 directly validates their training value: at per-query k=4k=4, Highest-HES achieves 30.20 versus 27.61 for Lowest-HES, with both selecting correct solutions to the same questions.

Q2: Trajectory value and coverage

Per-query RFT controls query identity and coverage, allowing us to evaluate selection among successful solutions to each problem:

Responses retained per queryRandomLengthHES
k=2k=228.9228.8030.10
k=4k=428.6428.8030.20
k=8k=829.3329.5430.13

HES improves over both alternatives at every tested kk (Table 6). These gains demonstrate useful trajectory selection with the problem mix held fixed. Coverage contributes further: per-query HES scores 30.20 versus 28.59 for global HES at k=4k=4, consistent with the coverage benefit discussed in Section 4.3. Together, these experiments support informative trajectory selection combined with broad query coverage.

We further compare HES with the intermediate-Difficulty baseline: HES scores 31.34 versus 23.30 in SFT and 19.26 versus 18.19 in RL (Tables 2 and 7). These comparisons complement the fixed-query RFT results by demonstrating an advantage over difficulty-based prioritization in two additional training settings.

Length and uncertainty are examined jointly in Appendix J. HES and length have a Spearman correlation of 0.8703, while their top-20% subsets overlap by 49.69%. Table 13 reports 139.10M selected tokens and a 31.34 score for HES, versus 180.48M and 29.05 for length selection. Thus, at equal example retention, HES improves accuracy by 2.29 points with 22.9% fewer tokens, supporting its value as a trajectory-level ranking criterion.

This selection benefit also translates into measured training efficiency: both RL strategies generate 32 rollouts per prompt, and HES selects 50% for policy updates. Appendix N reports 13 versus 15 hours on 8×A100 GPUs, including generation, scoring, and updates. The 13.3% wall-clock saving accompanies a higher average score, demonstrating a practical accuracy–cost benefit. HES uses a fixed entropy statistic for scoring, making the same selection rule easy to deploy across stages.

Dear Reviewer,

As the author–reviewer discussion deadline approaches, I would like to remind you to review the authors’ rebuttal and indicate whether your concerns have been adequately addressed.

If you have any remaining concerns or questions, please share them with the authors before the discussion period ends.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.