- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LoadCalib, a framework that adjusts the verbosity of teacher-generated chain-of-thought (CoT) reasoning based on problem difficulty and student model capacity. The method first constructs a capacity matrix by evaluating different CoT verbosity levels, then selects an appropriate level for each difficulty category. The authors further introduce a closed-loop calibration process to refine the selection policy using student performance feedback. Experiments with Qwen2.5 models of different sizes show that LoadCalib can reduce generated tokens while maintaining comparable accuracy in some settings.
Critical strengths
-
The paper investigates an interesting problem in CoT distillation, particularly how to adapt reasoning length to different problem difficulties and student capacities.
-
The proposed framework considers both problem difficulty and student capacity when selecting CoT verbosity, providing an intuitive approach to improving distillation efficiency.
-
The paper presents experiments across multiple student model sizes, including ablation studies and comparisons with different CoT verbosity settings.
Critical weaknesses
-
The contribution of token reduction is not sufficiently convincing. Shorter CoT traces naturally reduce supervision length, and the paper does not clearly establish the advantages of its approach over existing efficient reasoning methods, particularly fast/slow thinking strategies. Since token efficiency is a central motivation of this work, comparisons with relevant approaches would help better demonstrate its contribution.
-
The experiments are limited to the relatively outdated Qwen2.5 model family. More recent models, such as Qwen3 and Qwen3.5, support explicit thinking modes and can generate substantially longer reasoning traces, making them particularly relevant for studying reasoning efficiency. Evaluations on modern thinking-enabled models would strengthen the practical relevance and generalizability of the proposed approach.
-
The proposed framework introduces considerable computational overhead, including training multiple student models to construct the capacity matrix and performing additional training during closed-loop calibration. However, the reported token savings are relatively modest. It remains unclear whether these savings justify the additional computational cost. A more comprehensive analysis of the overall training and inference costs would be valuable.
-
The empirical improvements are limited. In most settings, LoadCalib reduces token usage at the expense of some accuracy. Even for the 3B model, the accuracy improvement over Uniform-medium is not statistically significant. Furthermore, the closed-loop calibration provides only marginal additional gains. These results raise concerns about whether the proposed framework offers a meaningful advantage over simpler alternatives.
-
The difficulty prediction component is not sufficiently convincing. The learned predictor achieves only 56.5% classification accuracy and relies on heuristic difficulty labels. Its reliability on more complex problems or different domains remains unclear. It would be helpful to investigate how the quality of difficulty estimation affects the overall performance of LoadCalib.
Questions for authors
See weaknesses!