acceptodds
Under review as a conference paper at ICLR 2027

CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection

Abstract

Evaluating Large Language Models (LLMs) incurs substantial computational overhead during continuous development, since a fixed benchmark is rerun for many checkpoints. Coreset selection reduces this workload, but existing methods either require historical response logs that new benchmarks lack, as in Item Response Theory, or inherit the lexical bias of question-only embeddings. We propose CoT-Core, a training-free core question selection framework. Because lexically disparate questions can require related reasoning operations, CoT-Core prompts an LLM to unroll a zero-shot Chain-of-Thought (CoT) trajectory for each question and clusters the resulting question-trajectory representations to choose representative items. The subset is constructed once, without reference answers or historical target-model responses, and reused across evaluations. On GSM8K, MMLU, MMLU-Pro, and GPQA, CoT-Core attains the strongest history-free score estimation in most dataset-budget settings. Rerunning the full evaluation pipeline for eleven target models on MMLU-Pro further shows that a 5% subset reduces output tokens by 94.77%, while its estimated scores deviate from full-benchmark scores by only 0.0065 on average. CoT-Core therefore provides a practical option for repeated evaluation when historical responses are unavailable.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, LLM Evaluation, Efficiency, Data Selection

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_p8c6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoT-Core, a method for selecting subsets of evaluation sets without needing historical model responses. The method first generates CoT reasoning paths for questions, encodes them together with the original questions, and then selects representative samples through K-Means clustering to reduce redundant evaluation costs. The authors validated the method on GSM8K, MMLU, MMLU-Pro, and GPQA, and assessed its effectiveness through ablation studies, control experiments, and model ranking analysis.

Critical strengths
  • The method is simple and practically useful. Buildingd reusable evaluation subsets without needing historical evaluation records, making it suitable for new benchmarks or scenarios with no historical data.

  • The experiments are fairly comprehensive. They cover multiple evaluation benchmarks, sampling ratios, and models, and also analyze the role of reasoning trajectories through length-matched controls and ablation studies.

Critical weaknesses
  • The unique contribution of CoT features is not fully clear. Existing baselines mostly use unrelated text and lack more competitive representations like question summaries, key reasoning steps, or semantic rewrites, making it hard to prove the necessity of a full CoT trajectory.

  • Performance gains and practical efficiency benefits are still unclear. In some settings, improvements over random sampling are small, and there’s no statistical significance testing for method differences.

  • Additionally, token savings don’t account for the full preprocessing cost, nor is end-to-end runtime reported.

Questions for authors
  1. Is it possible to add low-cost semantic representation baselines like question summaries or key reasoning steps to check if the full CoT trajectory is really necessary?

  2. Could you also include a statistical significance analysis compared to random sampling, as well as end-to-end efficiency and cost amortization results that account for the cost of generating CoT?

Reviewer_XRBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoT-Core, a training-free framework for selecting reusable subsets of LLM evaluation benchmarks without reference answers or historical target-model responses at selection time. It generates a zero-shot reasoning trajectory for each question, embeds the question–trajectory pair, and selects the nearest-to-centroid item from each K-Means cluster. The selected items support either standard aggregation or a historically calibrated GP-IRT estimator. Experiments on GSM8K, MMLU, MMLU-Pro, and GPQA Diamond show the lowest MAE among history-free selectors in 12 of 19 settings under standard aggregation. Additional studies examine generators, encoders, clustering algorithms, and length-matched auxiliary text. A separate eleven-model study reports substantial downstream output-token savings, although its score-fidelity statistic averages subset scores across trials before computing absolute deviations.

Critical strengths
  1. A practical formulation for history-free benchmark compression. The method addresses a useful setting where historical response matrices are unavailable. Selection is cleanly separated from performance estimation, and the generated trajectories are not supplied to target models. Table 1 provides encouraging evidence across four benchmarks, particularly on GPQA at low budgets: at 5%, MAE decreases from 0.1252 for Random to 0.1044, while ranking agreement also improves. These results support the practical value of augmenting question representations with solution-related information.

  2. Useful ablations and explicit reporting of limitations. The paper goes beyond a single configuration: Table 14 reports lower MAE with question–CoT features than question-only features across three encoders at the listed budgets, and the length-matched controls investigate whether simply appending text explains the gains. The authors also report counterexamples and acknowledge that incorrect trajectories can still supply useful concepts. This makes the empirical contribution more informative and appropriately limits what can be concluded about the mechanism.

Critical weaknesses
  1. The evidence does not yet establish the reliability of a single reusable coreset. Several improvements over Random are small—for example, MMLU-Pro at 5% improves from 0.0132 to 0.0124 MAE in Table 1—but round-level variability or uncertainty in paired method differences is not reported. The pooled individual-error standard deviations are not confidence intervals. Moreover, the headline 0.65 pp deviation in Table 16 is computed after averaging subset scores across trials, allowing positive and negative errors to cancel. Appendix G explicitly acknowledges this distinction, so this is not a reporting error; however, the statistic does not demonstrate the reliability of the single fixed subset envisaged for repeated deployment. Per-subset error distributions would materially strengthen the practical claim.

  2. The default aggregation choice is insufficiently validated against the intended full-benchmark score. Equation (1) defines an item-weighted mean, whereas the default estimator assigns equal weight to each cluster representative regardless of cluster size. Appendix E correctly includes the resulting weight-discrepancy term, but the experiments do not establish how much it contributes or compare uniform and cluster-mass weighting on the same selected subsets. This matters for interpreting the advantage over question-only clustering: the results establish an advantage under the chosen estimator, but not whether that advantage survives a natural alternative aggregation rule. A controlled comparison using identical subject quotas and both weighting schemes would clarify the strength of the selection contribution.

Questions for authors

No question.

Reviewer_QWpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CoT-Core is a training-free coreset selector for LLM benchmarks. It clusters question-plus-generated-reasoning-trajectory embeddings instead of question-only embeddings, so it can pick items by the solution procedure they require. It is evaluated on GSM8K, MMLU, MMLU-Pro, and GPQA, with reconstruction error (MAE) and ranking correlation (Sim) as the metrics, plus a token-cost study, placebo controls, and a generator-scale ablation.

Critical strengths

1.Solves a real cold-start problem. The method needs no historical model responses, so it applies to new or private benchmarks where IRT-style methods cannot be calibrated. 2.Good controls. The length-matched placebo study (Faker text and unrelated Wikipedia text at matched token counts) tests whether the gain comes from content or from extra tokens. Most coreset papers skip this check.

Critical weaknesses

1.The headline efficiency number leaves out the construction cost. The 94.77% token reduction covers only downstream generation. The one-time trajectory generation over the full benchmark is never quantified, and there is no break-even analysis for how many reuses are needed. 2.Statistical support is thin. Many "best" margins are smaller than the pooled error standard deviations. Trajectory-generation randomness is never varied, since all five rounds reuse one cached set of trajectories. Random gets 20 trials while the other methods get 5, and the GPQA 5% result rests on roughly ten items. 3.The central mechanism is asserted, not measured. The claim that clusters group items by shared solution procedure is supported only by two hand-picked case studies, which contain arithmetic errors. The "semantic equalizer" and "semantic repeller" explanations are post hoc. The placebo advantage also disappears at 10% and reverses at 1% and 20% in some settings.

Questions for authors

As in weakness

1 comment

I'd maintain my score, as there is no rebuttal.

Reviewer_9Y7BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This contribution proposes an coreset selection strategy for LLM benchmarks, that also work in cold start (without past repeated evaluations from other models, as often required in IRT f.e.). The approach generates a CoT, attach it to the question and process the concatenated sequence through an embedding model, before K-mean clustering the representations. The item closest to the centroid of each cluster is selected as part of the core set. The paper also provides a derivation for a bound of the SA error.

This relies on the intuition that questions may look semantically similar but the reasoning required to answer may be very different, and vice versa.

Critical strengths
  1. The paper describes a real problem, that of cold start subset selection for new or private benchmarks. The technique is useful to reduce the computational overhead of benchmarks that often contain redundant tasks. The paper is well written and forward with the results and the analysis.

  2. The approach is simple to interpret, natural and quite cheap (albeit incremental in nature), as the authors also show that a model of moderate size capable of generating CoTs is sufficient for the technique to select a relevant core set over popular standard such as MMLU and GPQA.

Critical weaknesses
  1. A major concern is the magnitude of the effects, which seem to be mostly within noise. The sample size is quite small, and this is compounded by the fact that models tend to make mistakes on the same examples (which is what is reported as error bar if I understand correctly), and CoTs generation randomness is also ignored. Benchmarks where I would expect for CoT to bring a sizable improvements, such as MMLU(-pro), show rather small gains, with weak baselines (such as random) winning outright at 15%.

  2. It is unclear to me if building the coreset is worth compared to just sampling a bit more at random? The authors did not factor in the cost of running the reasoning model to collect the CoT. Considering the random baseline is quite close to their technique, there should be a comparison between the two methods at compute parity for fairness. I expect (given random wins at 15%) that it may only marginally pay off?

Questions for authors

The paper was very clear and I have no clarification

  1. Did the authors consider testing on less mature benchmarks? That would provide a more convincing test bed for cold-start. All the benchmarks used in this paper are pretty famous.
  2. Considering the metric results are all very close one to another, the authors should strengthen significantly the statistical analysis: indicate sample sizes for each coreset, report single subset error, add confidence intervals, explain how the random ties policy they adopted impact the results in ranking 100 models on as few as a few dozens questions (for GPQA).
  3. Add competing approaches that solve a similar problem with history-free selection from item-level descriptions (AI pulled https://arxiv.org/abs/2510.26384 and https://arxiv.org/abs/2503.13335 which seems pretty relevant).
  4. Bring a wider variety of benchmarks. The ones presented here are all multiple-choice, or short answer. Code and long-form generation are missing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.