- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduced Student-Error-Guided Distillation (SEGD), extending SDFT by incorporating student-generated incorrect reasoning into the teacher's context. It uses an error-evidence gate based on the diversity of incorrect attempts and a compact error summary. Results show improvements over SDFT on mathematical reasoning and some other reasoning benchmarks.
Critical strengths
- The results are promising: SEGD improves MATH accuracy from 62.67% (SDFT) to 69.00%, and MATH-500 from 63.80% to 70.33%.
- There is a reasonable efficiency overhead. The reported computational overhead is only approximately 6% compared with SDFT.
Critical weaknesses
- The key innovation of asking students to learn from mistakes during distillation is not new. Furthermore, more related work should be cited in the "Learning from Errors." part in section 2, including but not limited to: [1, 2, 3].
- For Table 1, it should explicitly add words in the table or caption to denote that all 4 baselines are based on Qwen2.5-7B-Instruct. Moreover, the baselines ERRDPO and STEPKL are not cited. If they are 2 variables based on your own method SEGD, please explain the full meaning of these 2 abbreviations.
- Although Table 2 includes ablations on gates and representations, there are no error summaries without gating. So it's still not clear how many improvements come from the gate or from the error summaries themselves.
- Equation 10 needs more clarification and explanation of the structure. For example, the current one simply combines the number of unique incorrect traces and distinct incorrect answers. However, is this 2 terms enough to represent reasoning diversity, or are other types of equations, like a dynamic gate, better? More analysis is needed.
- Threshold γ is fixed at 70% for all experiments. Since gating is a key component of SEGD, more ablation studies should be conducted on different values.
[1] Capture the Key in Reasoning to Enhance CoT Distillation Generalization
[2] Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review
[3] Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks
Questions for authors
- For Table 1, does the first section show the inference results of different original LLMs (without being trained) on various benchmarks? If so, why are the math tasks called "In-domain" and the other tasks called "Transfer"?
- I suggest changing the title layout, now it's in 3 rows. Also, "Primary Area: other topics in machine learning (i.e., none of the above)" is also inaccurate for this submission.
W1. How does SEGD differ from EDIT, FAIR and LEMA? [1] uses minimum edit distance between paired correct and incorrect teacher CoTs to identify key tokens for training. [2] fine-tunes on CoT examples and GPT-4-generated mistake corrections. [3] trains on student-error feedback and teacher rationales filtered through multi-teacher peer review. In SEGD, selected mistakes from the current student's rollouts go into the EMA teacher's prompt alongside the gold solution. The student sees only the problem. On prefixes the student generated, it learns from the teacher's token distribution by reverse KL. The distinction is how mistakes enter distillation; learning from mistakes itself is established work.
W2 / Q1. Which models and losses does Table 1 use, and why is GSM8K called transfer? The top rows are single-run zero-shot evaluations with no additional training in this study. The intended backbone for the four trained baselines below is Qwen2.5-7B-Instruct. We use “in-domain” relative to the MATH training set. We will clarify that GSM8K measures cross-dataset transfer within mathematical reasoning. Thanks for noting that.
ERRDPO and STEPKL are baseline variants implemented for this study, rather than names of separately published methods. ERRDPO is designed to use DPO with the reference solution preferred over an incorrect student rollout, while STEPKL is designed to apply distillation after the first localized divergence. We will clarify their definitions and cite DPO directly alongside ERRDPO.
W3. Do summaries help without the gate? Table 2 compares summary-based methods under different gating rules. Our error-evidence gate performs best among the tested gates on MATH and MATH-500.
This result demonstrates that summary-only can improve around 2.17 compared baseline.
W4 / W5. Why use these two counts and γ=0.7? Would a learned gate work better? Thank you for suggesting a learned gate as an alternative to our heuristic. One count distinguishes wrong reasoning traces even when they reach the same answer. The other counts different wrong final answers. Equation 10 divides each by the number of attempts and averages the two. Equal weighting is a heuristic: paraphrases can inflate the trace count, equivalent expressions can inflate the answer count, and more mistakes can raise the score. We count strings, not semantic error types.
The cutoff already changes with the current batch, while the quantile γ stays fixed. At γ=0.7, one of four or three of eight eligible problems passes when scores are distinct; if all scores tie, all pass. We also tested γ values from 0.7 to 0.9. The results were broadly comparable, although the best value differed between benchmarks. We have not compared a learned gate.
Q2. Could the title and primary area be clearer? Thank you for pointing out the three-line title. A line break after “Error-Guided Self-Distillation:” would separate the method name from the subtitle without changing the wording. We also agree that “other topics in machine learning” is too broad: an area covering language-model training would better reflect our focus on self-distillation.
[1] Dai, C., Li, K., Zhou, W., & Hu, S. (2025). Capture the key in reasoning to enhance CoT distillation generalization. ACL 2025.
[2] An, S., Ma, Z., Cai, S., Lin, Z., Zheng, N., Lou, J. G., & Chen, W. (2024). Can LLMs learn from mistakes? An empirical study on reasoning tasks. Findings of EMNLP 2024.
[3] Li, Z., Ji, Y., Meng, R., & He, D. (2025). Learning from committee: Reasoning distillation from a mixture of teachers with peer-review. Findings of ACL 2025.