Error-Guided Self-Distillation: Turning Student Failures into Distillation Signals

Incorrect student attempts contain evidence of how a model fails, yet demonstration-conditioned self-distillation does not explicitly use these failures across attempts as teacher-side context. Simply adding wrong traces is not necessarily helpful: they can mix informative mistakes with irrelevant or misleading reasoning. We introduce Student-Error-Guided Distillation (SEGD), which reuses on-policy student failures as additional privileged information for the teacher. An error-evidence gate selects training examples based on counts of distinct wrong traces and final answers, and an error representation supplies problem-aligned evidence alongside the gold solution. Experiments show improvements on the evaluated mathematical reasoning benchmarks and selected non-math tasks, with mixed broad-transfer results. These findings support investigating student failures as a resource for teacher construction, with selection and representation as central design choices.