- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CoT-Core, a method for selecting subsets of evaluation sets without needing historical model responses. The method first generates CoT reasoning paths for questions, encodes them together with the original questions, and then selects representative samples through K-Means clustering to reduce redundant evaluation costs. The authors validated the method on GSM8K, MMLU, MMLU-Pro, and GPQA, and assessed its effectiveness through ablation studies, control experiments, and model ranking analysis.
Critical strengths
-
The method is simple and practically useful. Buildingd reusable evaluation subsets without needing historical evaluation records, making it suitable for new benchmarks or scenarios with no historical data.
-
The experiments are fairly comprehensive. They cover multiple evaluation benchmarks, sampling ratios, and models, and also analyze the role of reasoning trajectories through length-matched controls and ablation studies.
Critical weaknesses
-
The unique contribution of CoT features is not fully clear. Existing baselines mostly use unrelated text and lack more competitive representations like question summaries, key reasoning steps, or semantic rewrites, making it hard to prove the necessity of a full CoT trajectory.
-
Performance gains and practical efficiency benefits are still unclear. In some settings, improvements over random sampling are small, and there’s no statistical significance testing for method differences.
-
Additionally, token savings don’t account for the full preprocessing cost, nor is end-to-end runtime reported.
Questions for authors
-
Is it possible to add low-cost semantic representation baselines like question summaries or key reasoning steps to check if the full CoT trajectory is really necessary?
-
Could you also include a statistical significance analysis compared to random sampling, as well as end-to-end efficiency and cost amortization results that account for the cost of generating CoT?
I'd maintain my score, as there is no rebuttal.