- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Generative Vicinity Distillation (GVD) to address the pointwise nature of conventional knowledge distillation. GVD learns a score-based generative prior in the teacher's representation space and uses class-consistency and locality guidance to sample multiple teacher-like representations around each target, providing diverse and progressive supervision to the student. The evaluation covers CIFAR-10/100, Tiny ImageNet, STL-10, and UCR time-series tasks, including predictive performance, teacher-student fidelity, calibration, teacher-error subsets, cross-dataset transfer, and limited-data settings.
Critical strengths
-
Extending vicinal risk minimization to the teacher's representation space is novel, and the paper clearly distinguishes learned teacher-manifold neighborhoods from simple noise injection and multi-teacher supervision.
-
The experiments cover homogeneous and heterogeneous teacher-student architectures, image and time-series tasks, and metrics involving accuracy, F1, calibration, fidelity, transfer, and teacher-misclassified examples.
-
GVD improves over KD, FitNet, DIST, DiffKD, and TeKAP in many configurations. The analysis on teacher-error subsets and low-data regimes is particularly valuable because it suggests benefits beyond merely reproducing the teacher's original output.
Critical weaknesses
-
GVD requires training an additional diffusion prior and performing reverse-diffusion steps during student training. Although deployment inference is unchanged, the training pipeline is substantially more complex than standard KD, and detailed cost analysis focuses mainly on one CIFAR-100 setting.
-
The paper does not directly establish that generated feature representations correspond to semantically valid input-space neighborhoods. Class guidance preserves a predicted label but does not guarantee semantic or causal validity, so part of the benefit may be generic stochastic regularization.
-
The method includes a generative prior, class and locality guidance, diffusion steps, the number of vicinity samples, and a progressive schedule. Existing ablations do not fully isolate these components, and comparisons with simpler feature interpolation, covariance matching, or manifold regularization are limited.
Questions for authors
-
Do the authors have existing results for simple local baselines on the same teacher representations, such as within-class interpolation, local-covariance Gaussian sampling, or Mixup? If these methods approach GVD, the necessity of the diffusion prior would be weakened; a stable gap would strengthen the contribution.
-
Were all table entries obtained from the same number of independent training repetitions? Please report means and standard deviations from existing runs for key configurations, especially those with gains below one percentage point. Improvements within training variance would lower my confidence in the claimed consistency.
-
On the teacher-misclassified subset, is GVD's improvement over the teacher and KD primarily caused by guidance that uses ground-truth labels, or by generation from teacher representations alone? Please clarify which training components use labels; if teacher-error correction relies on strong label supervision, the interpretation of knowledge expansion from a single teacher should be narrowed.
3 comments
W3, Q1 (isolating components; simple local baselines). We thank the reviewer. The revision adds controlled ablations that isolate each component, and compares GVD against simple local baselines on the same teacher representations, both in distillation accuracy (Sec. 3.1) and in sample quality (Sec. 5.3.2, Tables 21–23).
1. Each component is isolated.
| Component | Ablation | Top-1 |
|---|---|---|
| Full GVD | – | 75.74 |
| Progressive schedule | Fixed , | 75.48 |
| Generative prior | Prior with partial noising, no guidance | 74.93 (KD: 74.40) |
| Prior-free: classifier-guided Gaussian, matched displacement | 74.22 | |
| Prior-free: direct classifier-gradient refinement, matched displacement | 73.01 | |
| Locality guidance | Target guidance only, from pure noise | 65.61 |
| Class guidance | Anchor guidance only, from pure noise | 75.17 |
| Both guidance terms | Neither, from pure noise | 64.54 |
| Denoising steps | 1 / 5 / 10 steps | 73.68 / 75.74 / 75.82 |
| Number of samples | 1 / 2 / 4 | 75.74 / 75.76 / 75.92 |
The prior alone improves over KD, removing it under matched displacement costs 1.3–2.5 points even with classifier guidance, locality guidance is essential when sampling from noise, and the schedule, steps and number of samples act as secondary refinements with saturating returns.
2. A stable gap over simple local baselines (Q1). Under the same fixed configuration and with perturbation magnitude matched to GVD, simple local baselines on the same teacher representations remain clearly below GVD:
| Vicinal targets (matched displacement) | Top-1 |
|---|---|
| Fixed teacher targets | 74.45 |
| Isotropic Gaussian | 74.11 |
| Mixup | 73.43 |
| Classifier-guided Gaussian | 74.22 |
| Classifier-guided direct feature refinement | 73.01 |
| GVD (fixed configuration, matched displacement) | 75.48 |
| GVD (with progressive schedule) | 75.74 |
3. The gap is reflected in the targets themselves. Comparing generated features against 50,000 teacher representations of real training images at matched displacement (Table 23), GVD lies closest to same-class real features (5-NN distance 8.04 vs. 8.37 for same-class interpolation and 9.07 for Gaussian), moves closer to them than the original feature in 98.71% of cases (vs. 89.73% and 37.99%), and best preserves local intrinsic dimensionality (LID error 0.91 vs. 3.17 and 0.99). Same-class interpolation stays near the class but distorts local structure and drops class purity (74.52% vs. 90.03%), consistent with manifold intrusion.
These results indicate that simple local constructions do not approach GVD, and that the learned prior is the main source of the gain.
Q2 (repetitions and variance). We thank the reviewer. All student results are obtained from five independent training runs with different random seeds, under the same protocol for every method and table entry (Sec. 5.2). We now report means and standard deviations for all CIFAR-100 results (Table 29) and assess significance in the new Sec. 5.3.4 (Experiments for Statistical Significance).
Image classification. Using Welch's -test, GVD significantly outperforms direct feature distillation (FitNets), DIST, diffusion-based DiffKD and single-teacher augmentation TeKAP in all six CIFAR-100 settings. Against logit distillation (KD), GVD's gains are significant in four of six settings.
Time-series classification. Across all 60 experiments (20 UCR datasets × 3 teacher–student settings), a paired two-sided Wilcoxon signed-rank test shows that GVD significantly outperforms Base (), KD (), FitNets () and TeKAP ().
Q3 (role of labels in teacher-error correction). We thank the reviewer for this precise question. Only two components use ground-truth labels: class guidance (Eq. 6, via the frozen teacher classifier) and the student task loss, which is identical to KD and all baselines. The diffusion prior (trained unconditionally, Eq. 2) and anchor guidance (pulling samples toward the teacher's own representation) are label-free.
Label-free generation alone already improves over the teacher's point targets. The revised ablations (Sec. 3.1) isolate this: sampling from the prior with no guidance improves over KD (74.93 vs. 74.40) and over fixed teacher targets (74.45), and adding label-free anchor guidance raises this to 75.17, against 75.74 for full GVD. Removing class guidance also barely changes target quality: 5-NN label purity against real training features drops only from 90.03% to 89.79% (Table 23), so class consistency comes mainly from the prior.
Q3. Labels alone do not produce the gain. Applying the same class guidance without the prior, under matched displacement, gives 74.22 (classifier-guided Gaussian) and 73.01 (direct classifier-gradient refinement), against 75.48 for GVD in the same configuration. Label information therefore helps only when combined with the learned teacher distribution.
On the teacher-misclassified subset, we evaluate GVD with and without class guidance (Top-1 gain over each reference model):
| Teacher | Base | KD | FitNets | TeKAP | |
|---|---|---|---|---|---|
| GVD without class guidance (label-free generation) | +29.24 | +2.32 | +1.27 | +1.05 | +2.09 |
| GVD (full) | +30.15 | +3.23 | +2.18 | +1.96 | +3.00 |
Even without any label-guided generation, GVD students correct teacher errors more often than all baselines, including KD (+1.27) and TeKAP (+2.09). Label-free generation thus accounts for the majority of the improvement over KD (1.27 of 2.18 points), while class guidance contributes a consistent additional +0.91 points.
Interpretation. Knowledge expansion arises primarily from label-free generation: sampling teacher-consistent targets beyond the teacher's single deterministic output already corrects a substantial share of teacher errors. Class guidance is a complementary, label-guided refinement that combines these targets with ground-truth supervision, which standard KD also uses through its task loss.
-- Thank you again for your valuable feedback, which has substantially strengthened the paper. We have addressed each of your concerns and would be grateful if you could consider the revised manuscript in your final evaluation.
We sincerely thank the reviewer for their thoughtful review. We appreciate the recognition of our novel teacher-space vicinal formulation, our broad evaluation across architectures, tasks and metrics, and the value of our teacher-error and low-data analyses. We address each concern below.
W1 (training complexity and scope of the cost analysis). We thank the reviewer. The revised Sec. 5.3.3 (Tables 25–28) now reports full cost analyses for three settings across two datasets, against logit-based KD, direct feature matching (FitNets) and diffusion-based DiffKD.
KD and FitNets memory differ by at most 0.1 GB in each setting.
Overhead shrinks with scale. The prior operates on the teacher's final feature map, not on images, so its relative cost falls as models grow: GVD's step time is at most 2.0× that of feature distillation and only 1.4× KD on TinyImageNet. It matches or beats DiffKD in step time and uses less memory in all settings.
The prior is a one-time cost, cheaper than the teacher. It is trained once per teacher, reused across students, and is cheaper to train than the teacher in every setting (e.g., 0.28 vs. 0.98 s/step and 3.56 vs. 24.92 GB on TinyImageNet).
Deployment is unaffected. The student is unchanged, so inference cost equals that of a non-distilled student.
Limited added complexity. GVD adds one stage, a feature-space prior trained with the standard diffusion loss, and uses the same hyperparameters for all image experiments (Table 9). Fewer denoising steps reduce cost further (Table 27).
The overhead buys real gains. GVD outperforms feature matching by +1.43 to +3.77 points at comparable cost, while the cheaper augmentation TeKAP falls below KD; GVD improves over TeKAP by +4.47 (ResNet-50 → MobileNetV2) and +2.86 points (TinyImageNet).
W2 (semantic validity of generated features). We thank the reviewer for this point. We agree that class guidance alone does not guarantee semantic or causal validity, and we do not claim that generated features correspond to specific input-space images. Our claim is narrower: GVD targets are consistent with the teacher's representation of real data, and their benefit is not explained by generic stochastic regularization or by class guidance. The revised paper tests this directly (Sec. 3.1 and Sec. 5.3.2, Tables 21–23).
1. Generated features are grounded in neighbourhoods of real images. We evaluate generated features against a bank of 50,000 teacher representations of real CIFAR-100 training images, under matched perturbation magnitude (Table 23).
GVD features lie closest to real same-class images in the teacher space, move closer to them than the original teacher feature in 98.71% of cases, and follow the local directions of variation of real data. 90% of their nearest real neighbours share the ground-truth class. The prior also generalises to unseen data: it reconstructs held-out test-set features with cosine similarity 0.998 (Table 21).
2. The benefit is not generic stochastic regularization. With perturbation magnitude matched to GVD and progressive scheduling disabled, Gaussian and Mixup vicinals reach 74.11 and 73.43 Top-1, against 75.48 for GVD. Unmatched Gaussian vicinals (74.43) give no gain over fixed teacher targets (74.45).
3. The benefit is not explained by class guidance. Prior-free, classifier-guided alternatives at matched displacement reach 73.01 (direct classifier-gradient refinement) and 74.22 (classifier-guided Gaussian), against 75.48 for GVD. Conversely, the prior without any guidance still improves over KD (74.93 vs. 74.40) and retains 88.81% purity, against 69.75% for Gaussian (Table 23). Most of the semantic consistency therefore comes from the learned prior, with guidance adding locality and class correction.
GVD models vicinities in the teacher's representation space, where our neighbourhood-consistency analyses support the plausibility of generated targets; input-space causal validity are difficult to assess directly and are left for future work.