- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TokenDistill, an iterative self-distillation framework for shortening CoT reasoning while preserving answer accuracy. Each iteration generates concise reasoning via a concise-output instruction, filters by answer correctness, fine-tunes on retained responses without the instruction, and decides whether to continue. Its main advantage is avoiding an external teacher, preference comparisons, and RL. Experiments on several Qwen2.5, LLaMA-3.1, and DeepSeek-R1-distilled models over GSM8K, MATH-500, and CommonsenseQA show substantial CoT-length reductions with competitive accuracy. However, the evidence does not establish that iterative data reconstruction is necessary, especially versus simpler repeated SFT under matched compute. And the novelty of the proposed method is limited. Using the test set for stopping and an underspecified checkpoint-selection procedure also raises validity and reproducibility concerns about the reported accuracy-compression trade-offs.
Critical strengths
S1. A simple and practical self-distillation framework without external teacher models The paper proposes an intuitive and computationally accessible approach to CoT compression. Unlike methods requiring external teachers, auxiliary models, or reinforcement learning with specialized reward functions, TokenDistill relies on the model's own generated responses and SFT. By transferring the concise-output requirement from explicit prompting into the model's generation behavior, the proposed framework avoids introducing additional model components or specialized optimization objectives. This design makes the method relatively easy to implement and integrate into existing LLM fine-tuning pipelines. The simplicity and self-contained nature of the framework constitute a meaningful practical contribution, as they reduce implementation complexity and dependence on external models.
S2. Promising empirical results across models and useful analysis of iterative compression The empirical results demonstrate substantial reductions in CoT length across multiple model families and reasoning benchmarks. Particularly encouraging results are reported for DeepSeek-R1-Distill-Qwen-7B. On MATH-500, TokenDistill improves accuracy from 75.6% to 83.0% while reducing average CoT length from 2519.03 to 1221.44 tokens. Similar improvements are observed on GSM8K. Beyond the main comparisons, the authors provide several useful analyses. Appendix A.1 demonstrates that multi-round distillation can achieve stronger compression than single-round training. Section 4.5 further compares two training strategies, showing that the compression improvements do not solely depend on accumulating parameter updates across rounds. The cross-dataset experiments and random-seed analysis also provide additional evidence of transferability and stability. These results support the practical effectiveness of the proposed framework and demonstrate its potential applicability to different reasoning models and tasks.
Critical weaknesses
W1. Insufficient evidence for the additional benefits of iterative data reconstruction My primary concern is that the paper does not sufficiently establish the additional benefits of iterative data reconstruction, which constitutes a central design choice of the proposed framework. I acknowledge that Appendix A.1 demonstrates the benefits of multi-round distillation over single-round training. Furthermore, Section 4.5 provides useful evidence that compression improvements do not solely depend on accumulating parameter updates across rounds. However, these experiments do not directly isolate the contribution of progressively reconstructing the training data. An essential missing baseline is repeated SFT using the same correctness-filtered concise dataset generated in the first round, under a matched training budget. Such a comparison would determine whether the additional compression achieved in later rounds results from progressively reconstructed training data or can be similarly obtained through repeated optimization on existing concise supervision. This distinction is particularly important because TokenDistill requires additional model inference and correctness filtering to reconstruct the dataset at each iteration. Without demonstrating a meaningful advantage over fixed-data repeated SFT, it remains unclear whether this additional computational overhead is justified. While the current experiments support the effectiveness of the overall framework, they do not sufficiently establish the independent benefit of its central iterative data reconstruction mechanism. This limits the strength of the paper's methodological contribution.
W2. Unclear checkpoint selection and potential evaluation bias across distillation rounds The second major concern is the lack of a clearly defined checkpoint-selection strategy across distillation rounds. Comparing Table 1 with Appendix A.1 (Table 4) reveals that the main results use checkpoints from different rounds. For example, on GSM8K, Qwen2.5-3B uses round 2, while Qwen2.5-7B uses round 5. However, the paper does not explain how these checkpoints are selected. This is particularly concerning because the stopping criterion in Section 3.4 (Eq. 12) depends only on average CoT length. While an accuracy constraint is not necessarily required, this criterion does not sufficiently explain why earlier checkpoints are reported even when subsequent rounds achieve further compression. It remains unclear whether the reported checkpoints were selected according to a predefined stopping rule or retrospectively chosen based on their observed accuracy-compression trade-offs. If the latter, selecting checkpoints after inspecting benchmark test accuracy could introduce selection bias and overestimate the effectiveness of the proposed method. Furthermore, Eq. (12) uses the benchmark test set for stopping decisions. Although only response lengths are considered, this still introduces test-set dependence. The authors should clarify the checkpoint-selection protocol and demonstrate that the reported results can be reproduced using predefined criteria and an independent validation set.
Questions for authors
Q1. Necessity of iterative data reconstruction Appendix A.1 demonstrates that multi-round distillation achieves stronger compression than single-round training, while Section 4.5 compares two strategies that both rely on progressively reconstructed datasets. However, have the authors considered a baseline that repeatedly performs SFT on the same correctness-filtered concise dataset generated in the first round, without reconstructing the training data? Could the authors compare this baseline with TokenDistill under matched optimization steps and training-token budgets, reporting the accuracy-compression trade-offs across rounds? This comparison would directly help determine whether iterative data reconstruction provides additional benefits beyond repeated optimization on fixed concise supervision.
Q2. Stopping criteria and checkpoint selection The proposed stopping criterion considers only average CoT length, whereas Appendix A.1 shows that further compression can sometimes cause substantial accuracy degradation. How are the final checkpoints reported in the main experiments selected from different training rounds? In particular, are these selections determined solely by the stopping conditions, or is answer accuracy also considered? Could the authors report the selected round and corresponding stopping-condition values for each experimental setting, and clarify how the proposed procedure ensures a favorable accuracy-compression trade-off?
Q3. Independence of the evaluation protocol Can the authors reproduce the main results using an independent validation set for stopping and checkpoint selection, reserving the test set exclusively for final evaluation? Please also report the selected round and corresponding stopping decisions for each setting.
Thank you for the detailed and constructive review. We appreciate your recognition of the framework’s simplicity, self-contained design, and empirical compression results. Your concerns identify two important issues: isolating the contribution of data reconstruction and making checkpoint selection independent of the test set. We address them separately below.
W1 / Q1: Additional benefits of iterative data reconstruction
Thank you for suggesting this comparison. Accordingly, we evaluated fixed-data repeated SFT on Qwen2.5-7B/MATH: all five rounds reuse the same 3,845 correctness-filtered concise examples from Round 1, with each round continuing from the preceding model.
We compare this baseline with the two training strategies described in Sections 3.3 and 4.5 and evaluate all three on MATH-500.
Table R1. Comparison of fixed-data SFT and iterative data reconstruction on MATH-500
Mk-based denotes the training in each round based on the previous-round model, while M0-based denotes the alternative strategy that always uses the initial model as the base model. Both strategies reconstruct the training dataset each round. Fixed-data repeated SFT follows the Mk-based strategy but reuses the same correctness-filtered concise dataset constructed in the first round.
As shown in Table R1, fixed-data SFT yielded little additional compression after the initial round. Both TokenDistill variants continued to shorten reasoning traces across rounds. These results suggest that iterative data reconstruction helps sustain compression across rounds.
W2 / Q2 / Q3: Stopping criterion and checkpoint selection
Thank you for highlighting this issue. The length-based stopping criterion was intended to guide the exploration of further compression across iterations, rather than to select the checkpoint with the best accuracy–length trade-off.
Our original aim was to examine the best observed trade-offs across training rounds. Accordingly, the checkpoints reported in Table 1 of the submitted paper were selected retrospectively based on test-set accuracy and ATL. For TokenSkip, we also reported results across all evaluated compression ratios to illustrate its accuracy–compression trade-offs.
We used no separate validation set for stopping or checkpoint selection and acknowledge the risk of selection bias. We plan to re-evaluate with predefined criteria on an independent validation set, reserving the test set for final evaluation. Results are not yet available. In the revision, we will document the stopping and checkpoint-selection procedures and report the selected round for each setting.
As supplementary evidence, we compare TokenDistill’s cross-dataset transfer with TokenSkip’s in-domain performance. TokenDistill is trained and its checkpoint selected only on GSM8K; the same fixed checkpoint is then evaluated on MATH-500 and CSQA. In contrast, the TokenSkip baselines are trained separately on the corresponding MATH and CSQA training sets and evaluated on MATH-500 and CSQA, respectively. Their results and selected compression ratios are taken from Table 1 of the submitted paper.
Table R2. Comparison of GSM8K-trained TokenDistill with in-domain TokenSkip
Compared with in-domain TokenSkip, the fixed GSM8K-trained TokenDistill checkpoints produced 16.2–20.4% shorter outputs in all four settings and achieved higher accuracy in three. The only exception was Qwen2.5-7B on CSQA, where accuracy was 0.29 percentage points lower but outputs were 20.4% shorter. These results suggest that a TokenDistill model trained on a single source dataset can achieve competitive accuracy–compression trade-offs on target tasks using the checkpoint selected on that same source dataset, without further training or checkpoint selection.