acceptodds
Under review as a conference paper at ICLR 2027

Error-Guided Self-Distillation: Turning Student Failures into Distillation Signals

Abstract

Incorrect student attempts contain evidence of how a model fails, yet demonstration-conditioned self-distillation does not explicitly use these failures across attempts as teacher-side context. Simply adding wrong traces is not necessarily helpful: they can mix informative mistakes with irrelevant or misleading reasoning. We introduce Student-Error-Guided Distillation (SEGD), which reuses on-policy student failures as additional privileged information for the teacher. An error-evidence gate selects training examples based on counts of distinct wrong traces and final answers, and an error representation supplies problem-aligned evidence alongside the gold solution. Experiments show improvements on the evaluated mathematical reasoning benchmarks and selected non-math tasks, with mixed broad-transfer results. These findings support investigating student failures as a resource for teacher construction, with selection and representation as central design choices.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ECTdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduced Student-Error-Guided Distillation (SEGD), extending SDFT by incorporating student-generated incorrect reasoning into the teacher's context. It uses an error-evidence gate based on the diversity of incorrect attempts and a compact error summary. Results show improvements over SDFT on mathematical reasoning and some other reasoning benchmarks.

Critical strengths
  1. The results are promising: SEGD improves MATH accuracy from 62.67% (SDFT) to 69.00%, and MATH-500 from 63.80% to 70.33%.
  2. There is a reasonable efficiency overhead. The reported computational overhead is only approximately 6% compared with SDFT.
Critical weaknesses
  1. The key innovation of asking students to learn from mistakes during distillation is not new. Furthermore, more related work should be cited in the "Learning from Errors." part in section 2, including but not limited to: [1, 2, 3].
  2. For Table 1, it should explicitly add words in the table or caption to denote that all 4 baselines are based on Qwen2.5-7B-Instruct. Moreover, the baselines ERRDPO and STEPKL are not cited. If they are 2 variables based on your own method SEGD, please explain the full meaning of these 2 abbreviations.
  3. Although Table 2 includes ablations on gates and representations, there are no error summaries without gating. So it's still not clear how many improvements come from the gate or from the error summaries themselves.
  4. Equation 10 needs more clarification and explanation of the structure. For example, the current one simply combines the number of unique incorrect traces and distinct incorrect answers. However, is this 2 terms enough to represent reasoning diversity, or are other types of equations, like a dynamic gate, better? More analysis is needed.
  5. Threshold γ is fixed at 70% for all experiments. Since gating is a key component of SEGD, more ablation studies should be conducted on different values.

[1] Capture the Key in Reasoning to Enhance CoT Distillation Generalization

[2] Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review

[3] Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks

Questions for authors
  1. For Table 1, does the first section show the inference results of different original LLMs (without being trained) on various benchmarks? If so, why are the math tasks called "In-domain" and the other tasks called "Transfer"?
  2. I suggest changing the title layout, now it's in 3 rows. Also, "Primary Area: other topics in machine learning (i.e., none of the above)" is also inaccurate for this submission.
1 comment

W1. How does SEGD differ from EDIT, FAIR and LEMA? [1] uses minimum edit distance between paired correct and incorrect teacher CoTs to identify key tokens for training. [2] fine-tunes on CoT examples and GPT-4-generated mistake corrections. [3] trains on student-error feedback and teacher rationales filtered through multi-teacher peer review. In SEGD, selected mistakes from the current student's rollouts go into the EMA teacher's prompt alongside the gold solution. The student sees only the problem. On prefixes the student generated, it learns from the teacher's token distribution by reverse KL. The distinction is how mistakes enter distillation; learning from mistakes itself is established work.

W2 / Q1. Which models and losses does Table 1 use, and why is GSM8K called transfer? The top rows are single-run zero-shot evaluations with no additional training in this study. The intended backbone for the four trained baselines below is Qwen2.5-7B-Instruct. We use “in-domain” relative to the MATH training set. We will clarify that GSM8K measures cross-dataset transfer within mathematical reasoning. Thanks for noting that.

ERRDPO and STEPKL are baseline variants implemented for this study, rather than names of separately published methods. ERRDPO is designed to use DPO with the reference solution preferred over an incorrect student rollout, while STEPKL is designed to apply distillation after the first localized divergence. We will clarify their definitions and cite DPO directly alongside ERRDPO.

W3. Do summaries help without the gate? Table 2 compares summary-based methods under different gating rules. Our error-evidence gate performs best among the tested gates on MATH and MATH-500.

MethodMATH improvement over SDFT
SDFT (baseline)—
SDFT + summary, without gate+2.17
SDFT + summary + gate (SEGD)+6.33

This result demonstrates that summary-only can improve around 2.17 compared baseline.

W4 / W5. Why use these two counts and γ=0.7? Would a learned gate work better? Thank you for suggesting a learned gate as an alternative to our heuristic. One count distinguishes wrong reasoning traces even when they reach the same answer. The other counts different wrong final answers. Equation 10 divides each by the number of attempts and averages the two. Equal weighting is a heuristic: paraphrases can inflate the trace count, equivalent expressions can inflate the answer count, and more mistakes can raise the score. We count strings, not semantic error types.

The cutoff already changes with the current batch, while the quantile γ stays fixed. At γ=0.7, one of four or three of eight eligible problems passes when scores are distinct; if all scores tie, all pass. We also tested γ values from 0.7 to 0.9. The results were broadly comparable, although the best value differed between benchmarks. We have not compared a learned gate.

γMATHMATH-500
0.762.66%67.00%
0.863.54%66.60%
0.963.40%65.00%

Q2. Could the title and primary area be clearer? Thank you for pointing out the three-line title. A line break after “Error-Guided Self-Distillation:” would separate the method name from the subtitle without changing the wording. We also agree that “other topics in machine learning” is too broad: an area covering language-model training would better reflect our focus on self-distillation.

[1] Dai, C., Li, K., Zhou, W., & Hu, S. (2025). Capture the key in reasoning to enhance CoT distillation generalization. ACL 2025.

[2] An, S., Ma, Z., Cai, S., Lin, Z., Zheng, N., Lou, J. G., & Chen, W. (2024). Can LLMs learn from mistakes? An empirical study on reasoning tasks. Findings of EMNLP 2024.

[3] Li, Z., Ji, Y., Meng, R., & He, D. (2025). Learning from committee: Reasoning distillation from a mixture of teachers with peer-review. Findings of ACL 2025.

Reviewer_62eQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work uses verified student failures as teacher-side diagnostic context rather than as targets to imitate. For each problem, the student samples multiple attempts; incorrect attempts are summarized, and a batch-relative evidence gate determines whether the summary is appended to the gold-conditioned EMA teacher prompt. Training otherwise retains the SDFT reverse-KL objective. On Qwen2.5-7B, SEGD reaches 69.00 on MATH and 70.33 on MATH-500, improving over the strongest listed trainable baseline by 5.75 and 4.93 points. The paper also evaluates alternative gates, raw traces, shuffled summaries, transfer tasks, and runtime overhead.

Critical strengths
  1. The distinction between failures as diagnostic context and failures as imitation/preference targets is clear and useful.
  2. The mathematics gains are large and consistent over three seeds, with relatively small standard deviations.
  3. The paper includes meaningful controls, including alternative gates, raw traces, and shuffled summaries, and reports only about 1.06× overhead at 7B.
Critical weaknesses
  1. Existing controls do not fully isolate the claimed mechanism. The paper still lacks a random gate with matched activation, always-on summaries, cross-problem summaries, and equal-length irrelevant context.
  2. The gate relies on string-level failure diversity and a batch-relative percentile; distinct traces need not represent distinct reasoning errors, and activation depends on batch composition.
  3. The pooled one-sided Wilcoxon tests reuse the same benchmark items across seeds. The paper acknowledges this, but the very small p-values should remain descriptive rather than inferential.
  4. Generality is mixed: GSM8K gains are negligible, SocialIQA is not best, and the ToolUse pilot is not direct SEGD evidence.
Questions for authors
  1. Can the authors add random-gate, always-on, cross-problem-summary, and equal-length irrelevant-context controls?
  2. How sensitive are results to gate percentile, number of attempts, and batch composition? What is the realized gate-on rate over training?
  3. Do the main gains remain significant under hierarchical bootstrap or mixed-effects analysis?
  4. Were all baselines matched in rollout, teacher-scoring, optimizer, masking, and hyperparameter-search budgets?
1 comment

W1 / Q1. Do the gains come from gate selection, relevant errors or simply extra context? Tables 1–2 report the following 7B MATH results. All configurations include the gold solution.

ConfigurationAdditional teacher context (as configured)MATH (%)
SDFT (baseline)None62.67
SEGD (ours)Selected error summary69.00
SEGD w/o summarizationOne representative wrong trace68.19
SEGD with shuffled summariesShuffled error summary65.85

The original summary configuration had the highest reported MATH accuracy, followed by raw traces and shuffled summaries.

W2 / Q2. How often does the gate fire, and how do γ, attempt count and batch composition affect it? Each cell averages 13 logged gate-on ratios during the phase when gating was enabled.

Run historyFirst (steps 703–775)Middle (781–853)Last (859–931)
139%34%37%
233%37%47%
344%42%40%

The overall means were 36.87%, 39.11% and 41.91%. The gate remained active in the last third of all three recorded histories (37–47%), with no consistent upward or downward trend across them. With γ=0.7, the gate selects scores at or above the batch’s 70th percentile: if all scores differ, one of four or three of eight eligible problems pass. Ties can admit more. The number of attempts changes how diversity is estimated; batch composition changes the cutoff. We also tested γ values from 0.7 to 0.9. The results were broadly comparable, although the best value differed between benchmarks.

W3 / Q3. Are the math gains significant after accounting for shared items and seeds? Table 1 reports gains over SDFT of 6.33 points on MATH and 6.53 on MATH-500. The same questions recur across seeds, and MATH-500 is a subset of MATH. We therefore treat the pooled Wilcoxon tests as descriptive.

Q4. Were training and search budgets equal? The 7B on-policy MATH runs used 12 student rollouts per problem (Section 5). SFT used gold solutions without student rollouts. The ERRDPO and STEPKL configurations specify the same rollout count and the AdamW optimizer. SDFT and SEGD share the reverse-KL loss and masking rules (Appendix C). SEGD took about 7.4 minutes longer than the two-hour 7B SDFT run (1.06×), mainly to generate summaries; at 3B, the time ratio was 1.02× (Section 6). The added work is to summarize errors from those same rollouts and select which summaries to include in the teacher’s prompt.

W4. How broad are the gains? After fine-tuning on MATH alone, 7B SEGD scored higher than SDFT on other transfer benchmarks (other than math) in Table 1: +8.52 points on CommonsenseQA, +2.71 on MMLU-Pro, and +2.02 on SocialIQA.

Reviewer_iaXSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Student-Error-Guided Distillation (SEGD) on top of demo-conditioned self-distillation (SDFT / OPD-style reverse KL). The student only sees the problem. The EMA teacher still gets the gold solution, and on some examples also a short summary of the student's own wrong attempts. An error-evidence gate picks those examples from counts of distinct wrong traces and wrong answers. Wrong attempts are teacher context, not imitation targets. Most experiments are Qwen2.5-7B LoRA on MATH / MATH-500 / GSM8K, with gate and representation ablations, a small MATH audit, and a few 3B MC-QA / ToolUse pilots. They report clear gains on math vs SDFT and ERRDPO. Transfer outside math looks mixed.

Critical strengths
  • The intervention relative to SDFT is clean and easy to understand. Same loss, same gold demo; only teacher context changes when the gate fires. I do not need a new training objective to get the idea.

  • Math gains look real, not noise. Ablations in Table 2 are useful: alt gates and shuffled summary land clearly below SEGD, so it is not just "add more text". The raw-error control in the appendix also helps - dumping raw wrongs without gating/summarization is worse than gold-only, which motivates their design a bit.

Critical weaknesses
  • The mechanism story looks thin to me. The paper wants an "error-aware correction" reading, but the audit does not really nail that. On fixed wrong traces, SEGD can make wrong boxed answers more likely, and the authors themselves note possible self-conditioning of the scored trajectory into the evidence pool. Without isolating that, I doubt whether the MATH gains come from useful failure diagnostics or just other prompt/training side effects.

  • Outside verifier-rich math, the case is not convincing. Same-domain CSQA favors SFT over SEGD. SocialIQA transfer loses to ERRDPO. Broad transfer vs the base instruct model is mixed. The 3B ARC-Challenge jump looks suspiciously large compared to very weak SFT/SDFT numbers there; without a clearer failure account, I cant put much weight on it as non-math support. ToolUse is basically an EMA self-distill pilot, not SEGD.

  • Novelty feels incremental. Using privileged context for the teacher is already in the SDFT / OPD line. The main add-on is gating + summarizing student failures. That is a reasonable engineering idea, but with the mechanism still open and transfer weak, I do not think it clears the bar yet.

Questions for authors
  • Please rebuild the error set / summary while excluding the trajectory being scored (and near-duplicates of it). Do MATH gains and the teacher-grading deltas still hold? That would test self-conditioning directly.

  • For the 3B ARC-Challenge run: why are SFT/SDFT so low, and what does a typical SEGD error summary look like there? A short failure breakdown would change how much weight I put on that table.

  • How does gate-on rate and summary quality change over training? Does the evidence stay informative late when wrongs are rarer, or does most of the help land early?

2 comments

W1. Do the MATH gains reflect useful failure information or other prompt and training effects? We address this question in Table 2 by comparing the original failure summaries with shuffled summaries and raw failure trajectories. On MATH, the original summary setting outperforms both raw trajectories and shuffled summaries. This comparison is consistent with the idea that matched failure information is useful, rather than the gain coming only from adding more context.

Q1. Does the teacher’s score change when the scored attempt is excluded from the summary? We regenerated summaries after excluding the scored attempt and compared them with a control that retained it but removed the same number of other attempts. Excluding the scored trajectory from the summary had little effect on the teacher’s score. One possible reason for the small average change is that some remaining attempts gave the same wrong answer, so the summary could still capture related information.

W2 / Q2. Why were the ARC baseline scores low? We mistakenly used the default generation limit of 64 tokens for ARC. However, our prompt asked the model to provide a brief explanation before the final answer, so some responses were cut off before the answer appeared. This affected the baselines more because they tended to produce longer explanations. We apologize for this evaluation error. Without an explicit answer, the scorer used the last standalone option letter in the final 120 characters, or marked the response incorrect if it found none.

We re-evaluated one frozen checkpoint per method using identical prompts, greedy decoding and the same scorer. We changed only the generation limit:

ARC accuracy64 tokens256 tokens
SFT15.05%75.25%
SDFT35.45%82.27%
SEGD79.26%84.28%

The longer limit substantially narrowed the gap in this paired comparison. Thank you again for raising this important issue.

Q3. Does the gate stay active, and do summaries stay useful late in training? The table shows gate use across thirds of the period when the gate was enabled. Each cell averages 13 logged ratios.

Run historyFirst (steps 703–775)Middle (781–853)Last (859–931)
139%34%37%
233%37%47%
344%42%40%

The gate remained active in the last third of all three histories. From the first to the last third, the mean gate-on rate rose from 33% to 47% in run 2 but fell slightly in runs 1 (39% to 37%) and 3 (44% to 40%).

W2 / W3. How well does SEGD transfer, and what is new relative to SDFT? After fine-tuning on MATH alone, the 7B SEGD model scored higher than SDFT on all four transfer benchmarks in Table 1. The largest gain was on CommonsenseQA; GSM8K changed little.

BenchmarkSDFT (%)SEGD (%)Gain (pp)
GSM8K83.8384.00+0.17
MMLU-Pro53.3256.03+2.71
SocialIQA63.3965.41+2.02
CommonsenseQA63.9672.48+8.52

The transfer gains are relative to SDFT, not to every baseline: ERRDPO scored higher on SocialIQA, and SFT did better in the separate in-domain CommonsenseQA experiment. The ToolUse pilot tested EMA distillation, not SEGD.

Errors in the student’s own rollouts can provide an additional source of supervision that SDFT does not use. SEGD gives selected error summaries to the teacher without changing the student’s loss. Our ablation results support the benefit of incorporating error summaries.

Thank you for providing these new results and the new evaluation, and I am glad to involve in fix the evaluation protocol bug of the previous experiments :).

However, seems like the reviewer didn't address the mechanism concern, so I retain my score for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.