SURGE: Sparse Update Routing via Gated Entropy for Efficient Active Distillation

On-policy distillation (OPD) is widely used to transfer capabilities from large teacher language models to smaller student models. However, supervising every token incurs substantial computational cost without distinguishing where guidance is truly needed and can induce unnecessary updates that disrupt existing student capabilities. To address these limitations, we introduce $\text{SURGE}$, an active distillation framework that provides selective control over student adaptation and knowledge transfer by using two separate gating mechanisms for updates. A token gate applies teacher supervision only at high-entropy positions, while a step gate forces updates to remain inside low-scoring generation steps. Additionally, a decoupled KL objective is used to apply updates selectively on such active tokens while retaining regularization toward a frozen reference student over the entire sequence. Across the evaluated reasoning, scientific knowledge, and hate speech tasks, SURGE achieves the highest reported accuracy among the compared baselines. When distilling Gemma4-31B into Gemma4-E4B, SURGE improves over the base student by a relative $\text{8.61\%}$ on average and achieves a mean recovery of $\text{40.93\%}$ of the initial teacher–student accuracy gap. Relative to OPD, estimated distillation FLOPs fall by $\text{49.89\%}$ on average, and the share of that gap SURGE closes is $\text{66.60\%}$ larger. Average response length falls by $\text{45.21\%}$, while the mean format-failure rate falls by $\text{24.78\%}$ relative to OPD. Finally, we assess the generalization of our method by distilling Qwen3.5-122B-A10B into Qwen3.5-2B on these benchmarks, also noting improvements across accuracy and training FLOPs.