Learning to Generalize from the Vicinity: Knowledge Expansion from a Single Teacher

When empirical data distributions poorly represent the true data manifold, models exposed to local continuities around data tend to generalize better than those that tightly fit individual data. Similarly, in Knowledge Distillation (KD), when teacher representations poorly reflect the true representation manifold, can exposing a student to local neighborhoods provide greater benefits than fitting to individual teacher targets? In other words, *Can vicinal distributions around teacher targets help students learn better?* We investigate this by introducing a novel distillation objective where we admit a generative vicinal distribution for each individual teacher target; **Generative Vicinity Distillation (GVD)**. GVD learns a generative prior over the teacher representation space to enable sampling from vicinities and derives guidance functions to impose desired constraints on the generated samples (class consistency, locality). Departing from conventional KD thinking that depends on multi-teacher supervision for robustness, diversity, and fidelity, we instead exploit the ambiguity of a generative model’s solution space to derive such properties from a single teacher. Extensive experiments across datasets and architectures demonstrate the effectiveness of GVD in improving student generalization and robustness.