The Joint-Distillation Gap: Which Teacher Corrections Fit Together Under Shared Student Updates?

Why can a shared student learn less from several specialized teachers than from each teacher separately, even when routing is correct and domain exposure is matched? We study this joint-distillation gap through the updates needed to learn teacher corrections. The cost of adding a correction depends on what the student is already learning, so a correction favored in isolation may be a poor addition to the selected supervision. This observation motivates Jointly Feasible Mode Distillation (JFMD). It decomposes teacher–student residuals into modes and selects corrections by their reliability-adjusted value and the projected cost of adding them to the current set. Across three model families and eight benchmarks, JFMD improves aggregate transfer over the evaluated student baselines. Controlled training comparisons across candidate pairs and matched-count ablations provide empirical support for conditional cost, while held-out analyses connect calibration reliability to stability on new examples. The gains persist across the tested student sizes and under teacher-routing errors. Together, the results suggest that learning requirements shared across corrections provide a useful explanation for the gap and a guide for selecting supervision. Our code is available at https://anonymous.4open.science/r/JFMD-392B.