acceptodds
Under review as a conference paper at ICLR 2027

Learning to Generalize from the Vicinity: Knowledge Expansion from a Single Teacher

Abstract

When empirical data distributions poorly represent the true data manifold, models exposed to local continuities around data tend to generalize better than those that tightly fit individual data. Similarly, in Knowledge Distillation (KD), when teacher representations poorly reflect the true representation manifold, can exposing a student to local neighborhoods provide greater benefits than fitting to individual teacher targets? In other words, *Can vicinal distributions around teacher targets help students learn better?* We investigate this by introducing a novel distillation objective where we admit a generative vicinal distribution for each individual teacher target; **Generative Vicinity Distillation (GVD)**. GVD learns a generative prior over the teacher representation space to enable sampling from vicinities and derives guidance functions to impose desired constraints on the generated samples (class consistency, locality). Departing from conventional KD thinking that depends on multi-teacher supervision for robustness, diversity, and fidelity, we instead exploit the ambiguity of a generative model’s solution space to derive such properties from a single teacher. Extensive experiments across datasets and architectures demonstrate the effectiveness of GVD in improving student generalization and robustness.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Distillation, Diffusion Models, Efficient Deep Learning, Generative Models, Vicinal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_o8MmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Generative Vicinity Distillation (GVD) to address the pointwise nature of conventional knowledge distillation. GVD learns a score-based generative prior in the teacher's representation space and uses class-consistency and locality guidance to sample multiple teacher-like representations around each target, providing diverse and progressive supervision to the student. The evaluation covers CIFAR-10/100, Tiny ImageNet, STL-10, and UCR time-series tasks, including predictive performance, teacher-student fidelity, calibration, teacher-error subsets, cross-dataset transfer, and limited-data settings.

Critical strengths
  1. Extending vicinal risk minimization to the teacher's representation space is novel, and the paper clearly distinguishes learned teacher-manifold neighborhoods from simple noise injection and multi-teacher supervision.

  2. The experiments cover homogeneous and heterogeneous teacher-student architectures, image and time-series tasks, and metrics involving accuracy, F1, calibration, fidelity, transfer, and teacher-misclassified examples.

  3. GVD improves over KD, FitNet, DIST, DiffKD, and TeKAP in many configurations. The analysis on teacher-error subsets and low-data regimes is particularly valuable because it suggests benefits beyond merely reproducing the teacher's original output.

Critical weaknesses
  1. GVD requires training an additional diffusion prior and performing reverse-diffusion steps during student training. Although deployment inference is unchanged, the training pipeline is substantially more complex than standard KD, and detailed cost analysis focuses mainly on one CIFAR-100 setting.

  2. The paper does not directly establish that generated feature representations correspond to semantically valid input-space neighborhoods. Class guidance preserves a predicted label but does not guarantee semantic or causal validity, so part of the benefit may be generic stochastic regularization.

  3. The method includes a generative prior, class and locality guidance, diffusion steps, the number of vicinity samples, and a progressive schedule. Existing ablations do not fully isolate these components, and comparisons with simpler feature interpolation, covariance matching, or manifold regularization are limited.

Questions for authors
  1. Do the authors have existing results for simple local baselines on the same teacher representations, such as within-class interpolation, local-covariance Gaussian sampling, or Mixup? If these methods approach GVD, the necessity of the diffusion prior would be weakened; a stable gap would strengthen the contribution.

  2. Were all table entries obtained from the same number of independent training repetitions? Please report means and standard deviations from existing runs for key configurations, especially those with gains below one percentage point. Improvements within training variance would lower my confidence in the claimed consistency.

  3. On the teacher-misclassified subset, is GVD's improvement over the teacher and KD primarily caused by guidance that uses ground-truth labels, or by generation from teacher representations alone? Please clarify which training components use labels; if teacher-error correction relies on strong label supervision, the interpretation of knowledge expansion from a single teacher should be narrowed.

3 comments

We sincerely thank the reviewer for their thoughtful review. We appreciate the recognition of our novel teacher-space vicinal formulation, our broad evaluation across architectures, tasks and metrics, and the value of our teacher-error and low-data analyses. We address each concern below.

W1 (training complexity and scope of the cost analysis). We thank the reviewer. The revised Sec. 5.3.3 (Tables 25–28) now reports full cost analyses for three settings across two datasets, against logit-based KD, direct feature matching (FitNets) and diffusion-based DiffKD.

SettingGVD / KD, FitNets step timeGVD − KD, FitNets memoryGVD vs. DiffKD (step, memory)Top-1 gain over KDTop-1 gain over FitNets
CIFAR-100, WRN-40-2 → WRN-40-10.12 / 0.03, 0.07 s+1.62 GB0.12 vs. 0.15 s; 2.07 vs. 2.46 GB+1.34+1.43
CIFAR-100, ResNet-50 → MobileNetV20.10 / 0.05 s (2.0×)+0.07 GB0.10 vs. 0.10 s; 0.75 vs. 0.80 GB+1.98+3.77
TinyImageNet, ResNet-50 → MobileNetV10.43 / 0.31 s (1.4×)+0.89 GB0.43 vs. 0.45 s; 4.36 vs. 5.93 GB+0.45+2.74

KD and FitNets memory differ by at most 0.1 GB in each setting.

Overhead shrinks with scale. The prior operates on the teacher's final feature map, not on images, so its relative cost falls as models grow: GVD's step time is at most 2.0× that of feature distillation and only 1.4× KD on TinyImageNet. It matches or beats DiffKD in step time and uses less memory in all settings.

The prior is a one-time cost, cheaper than the teacher. It is trained once per teacher, reused across students, and is cheaper to train than the teacher in every setting (e.g., 0.28 vs. 0.98 s/step and 3.56 vs. 24.92 GB on TinyImageNet).

Deployment is unaffected. The student is unchanged, so inference cost equals that of a non-distilled student.

Limited added complexity. GVD adds one stage, a feature-space prior trained with the standard diffusion loss, and uses the same hyperparameters for all image experiments (Table 9). Fewer denoising steps reduce cost further (Table 27).

The overhead buys real gains. GVD outperforms feature matching by +1.43 to +3.77 points at comparable cost, while the cheaper augmentation TeKAP falls below KD; GVD improves over TeKAP by +4.47 (ResNet-50 → MobileNetV2) and +2.86 points (TinyImageNet).


W2 (semantic validity of generated features). We thank the reviewer for this point. We agree that class guidance alone does not guarantee semantic or causal validity, and we do not claim that generated features correspond to specific input-space images. Our claim is narrower: GVD targets are consistent with the teacher's representation of real data, and their benefit is not explained by generic stochastic regularization or by class guidance. The revised paper tests this directly (Sec. 3.1 and Sec. 5.3.2, Tables 21–23).

1. Generated features are grounded in neighbourhoods of real images. We evaluate generated features against a bank of 50,000 teacher representations of real CIFAR-100 training images, under matched perturbation magnitude (Table 23).

Method5-NN distance ↓Closer than original (%) ↑Off-tangent residual ↓5-NN purity (%) ↑
Gaussian9.0737.990.92269.75
Gaussian + classifier guidance9.0470.650.75671.34
Same-class Mixup8.3789.730.75274.52
GVD8.0498.710.74590.03

GVD features lie closest to real same-class images in the teacher space, move closer to them than the original teacher feature in 98.71% of cases, and follow the local directions of variation of real data. 90% of their nearest real neighbours share the ground-truth class. The prior also generalises to unseen data: it reconstructs held-out test-set features with cosine similarity 0.998 (Table 21).

2. The benefit is not generic stochastic regularization. With perturbation magnitude matched to GVD and progressive scheduling disabled, Gaussian and Mixup vicinals reach 74.11 and 73.43 Top-1, against 75.48 for GVD. Unmatched Gaussian vicinals (74.43) give no gain over fixed teacher targets (74.45).

3. The benefit is not explained by class guidance. Prior-free, classifier-guided alternatives at matched displacement reach 73.01 (direct classifier-gradient refinement) and 74.22 (classifier-guided Gaussian), against 75.48 for GVD. Conversely, the prior without any guidance still improves over KD (74.93 vs. 74.40) and retains 88.81% purity, against 69.75% for Gaussian (Table 23). Most of the semantic consistency therefore comes from the learned prior, with guidance adding locality and class correction.

GVD models vicinities in the teacher's representation space, where our neighbourhood-consistency analyses support the plausibility of generated targets; input-space causal validity are difficult to assess directly and are left for future work.

W3, Q1 (isolating components; simple local baselines). We thank the reviewer. The revision adds controlled ablations that isolate each component, and compares GVD against simple local baselines on the same teacher representations, both in distillation accuracy (Sec. 3.1) and in sample quality (Sec. 5.3.2, Tables 21–23).

1. Each component is isolated.

ComponentAblationTop-1
Full GVD–75.74
Progressive scheduleFixed tinit=30t_{\text{init}}=30, λ=0.3\lambda=0.375.48
Generative priorPrior with partial noising, no guidance74.93 (KD: 74.40)
Prior-free: classifier-guided Gaussian, matched displacement74.22
Prior-free: direct classifier-gradient refinement, matched displacement73.01
Locality guidanceTarget guidance only, from pure noise65.61
Class guidanceAnchor guidance only, from pure noise75.17
Both guidance termsNeither, from pure noise64.54
Denoising steps1 / 5 / 10 steps73.68 / 75.74 / 75.82
Number of samples MM1 / 2 / 475.74 / 75.76 / 75.92

The prior alone improves over KD, removing it under matched displacement costs 1.3–2.5 points even with classifier guidance, locality guidance is essential when sampling from noise, and the schedule, steps and number of samples act as secondary refinements with saturating returns.

2. A stable gap over simple local baselines (Q1). Under the same fixed configuration and with perturbation magnitude matched to GVD, simple local baselines on the same teacher representations remain clearly below GVD:

Vicinal targets (matched displacement)Top-1
Fixed teacher targets74.45
Isotropic Gaussian74.11
Mixup73.43
Classifier-guided Gaussian74.22
Classifier-guided direct feature refinement73.01
GVD (fixed configuration, matched displacement)75.48
GVD (with progressive schedule)75.74

3. The gap is reflected in the targets themselves. Comparing generated features against 50,000 teacher representations of real training images at matched displacement (Table 23), GVD lies closest to same-class real features (5-NN distance 8.04 vs. 8.37 for same-class interpolation and 9.07 for Gaussian), moves closer to them than the original feature in 98.71% of cases (vs. 89.73% and 37.99%), and best preserves local intrinsic dimensionality (LID error 0.91 vs. 3.17 and 0.99). Same-class interpolation stays near the class but distorts local structure and drops class purity (74.52% vs. 90.03%), consistent with manifold intrusion.

These results indicate that simple local constructions do not approach GVD, and that the learned prior is the main source of the gain.


Q2 (repetitions and variance). We thank the reviewer. All student results are obtained from five independent training runs with different random seeds, under the same protocol for every method and table entry (Sec. 5.2). We now report means and standard deviations for all CIFAR-100 results (Table 29) and assess significance in the new Sec. 5.3.4 (Experiments for Statistical Significance).

Image classification. Using Welch's tt-test, GVD significantly outperforms direct feature distillation (FitNets), DIST, diffusion-based DiffKD and single-teacher augmentation TeKAP in all six CIFAR-100 settings. Against logit distillation (KD), GVD's gains are significant in four of six settings.

Time-series classification. Across all 60 experiments (20 UCR datasets × 3 teacher–student settings), a paired two-sided Wilcoxon signed-rank test shows that GVD significantly outperforms Base (p=1.19×10−9p=1.19\times10^{-9}), KD (p=4.13×10−6p=4.13\times10^{-6}), FitNets (p=2.36×10−5p=2.36\times10^{-5}) and TeKAP (p=7.25×10−6p=7.25\times10^{-6}).


Q3 (role of labels in teacher-error correction). We thank the reviewer for this precise question. Only two components use ground-truth labels: class guidance (Eq. 6, via the frozen teacher classifier) and the student task loss, which is identical to KD and all baselines. The diffusion prior (trained unconditionally, Eq. 2) and anchor guidance (pulling samples toward the teacher's own representation) are label-free.

Label-free generation alone already improves over the teacher's point targets. The revised ablations (Sec. 3.1) isolate this: sampling from the prior with no guidance improves over KD (74.93 vs. 74.40) and over fixed teacher targets (74.45), and adding label-free anchor guidance raises this to 75.17, against 75.74 for full GVD. Removing class guidance also barely changes target quality: 5-NN label purity against real training features drops only from 90.03% to 89.79% (Table 23), so class consistency comes mainly from the prior.

Q3. Labels alone do not produce the gain. Applying the same class guidance without the prior, under matched displacement, gives 74.22 (classifier-guided Gaussian) and 73.01 (direct classifier-gradient refinement), against 75.48 for GVD in the same configuration. Label information therefore helps only when combined with the learned teacher distribution.

On the teacher-misclassified subset, we evaluate GVD with and without class guidance (Top-1 gain over each reference model):

TeacherBaseKDFitNetsTeKAP
GVD without class guidance (label-free generation)+29.24+2.32+1.27+1.05+2.09
GVD (full)+30.15+3.23+2.18+1.96+3.00

Even without any label-guided generation, GVD students correct teacher errors more often than all baselines, including KD (+1.27) and TeKAP (+2.09). Label-free generation thus accounts for the majority of the improvement over KD (1.27 of 2.18 points), while class guidance contributes a consistent additional +0.91 points.

Interpretation. Knowledge expansion arises primarily from label-free generation: sampling teacher-consistent targets beyond the teacher's single deterministic output already corrects a substantial share of teacher errors. Class guidance is a complementary, label-guided refinement that combines these targets with ground-truth supervision, which standard KD also uses through its task loss.

-- Thank you again for your valuable feedback, which has substantially strengthened the paper. We have addressed each of your concerns and would be grateful if you could consider the revised manuscript in your final evaluation.

Reviewer_gbceOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generative Vicinity Distillation (GVD), which replaces conventional point-wise teacher supervision with a local distribution of plausible teacher representations. GVD learns a diffusion-based prior over the teacher feature space and uses locality and class-consistency guidance to generate vicinal targets for student training. Experiments across image and time-series benchmarks show improved accuracy, teacher–student fidelity, calibration, robustness, and transferability over standard KD baselines.

Critical strengths
  1. Clear and well-motivated formulation: The paper provides an intuitive motivation for extending conventional point-wise teacher supervision to a vicinal distribution around each teacher representation. This offers a clear perspective on addressing imperfect and overly rigid teacher targets.

  2. Extensive analysis beyond predictive accuracy: The paper evaluates not only classification accuracy but also teacher–student fidelity, calibration, robustness to imperfect teachers and limited data, and transferability. These analyses provide a relatively comprehensive view of the benefits of vicinal supervision.

Critical weaknesses
  1. Unclear novelty over Angular-KD: The core idea of expanding supervision from a single teacher into multiple nearby or diverse teacher views appears closely related to Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (Angular-KD) [A]. Although GVD constructs vicinal targets using a diffusion-based generative prior, the conceptual distinction from Angular-KD is not sufficiently clear. The paper would benefit from a more explicit discussion of what new knowledge GVD captures beyond existing single-teacher view augmentation.

[A] Yu et al., Single-teacher view augmentation: Boosting Knowledge Distillation via Angular Diversity, NeurIPS 2026.

  1. Missing direct comparison with Angular-KD: A direct empirical comparison with Angular-KD seems important. Beyond predictive accuracy, it would be useful to compare the generated teacher views themselves—for example, how much of the local neighborhood around each teacher representation is covered, how diverse the generated targets are, and whether GVD better preserves locality and class consistency. Such analysis would make the claimed advantage of learning a vicinal distribution more convincing.

  2. Unclear cost-performance trade-off of the diffusion prior: 1. GVD requires training and sampling from an additional diffusion model, resulting in noticeably higher training cost and memory usage than simpler KD approaches. For example, the reported training step time is 0.12 s for GVD versus 0.03 s for TeKAP, with memory usage of 2.07 GB versus 0.47 GB. While GVD generally outperforms TeKAP, the improvement is relatively modest in several teacher-student settings. Therefore, stronger evidence is needed to show that the additional diffusion-based machinery provides a favorable accuracy-efficiency trade-off over simpler teacher-view augmentation methods.

Questions for authors

Could the authors clarify the distinction between GVD and prior single-teacher view augmentation methods such as Angular-KD? In particular, it would be helpful to provide a direct comparison in both performance and the properties of the generated teacher views, such as diversity, locality, and coverage around the original teacher representation. Given that GVD introduces an additional diffusion prior and higher training cost, could the authors also discuss whether the performance gains justify this added complexity compared with simpler teacher-view augmentation approaches?

2 comments

We sincerely thank the reviewer for their thoughtful review. We appreciate the recognition of our well-motivated vicinal formulation for addressing imperfect, rigid teacher targets, and of our analysis beyond accuracy, including fidelity, calibration, robustness and transferability. We address each concern below.


W1, Q1. (Novelty and Separation from Angular-KD) We thank the reviewer for highlighting Angular-KD. Both methods enrich supervision from a single teacher, but they differ in how targets are constructed, what they can express, and their theoretical motivation.

Learned density versus predefined angular geometry. Angular-KD diversifies the outputs of NN linear heads with angular objectives defined on cosine similarity, which do not model the density of teacher features. GVD instead learns a score-based prior over teacher representations and samples along its score field, so perturbations concentrate in high-density directions. This matters empirically: a density-agnostic isotropic vicinity gives no gain over fixed targets (74.43 vs. 74.45), whereas GVD reaches 75.74. Our new view-quality analysis confirms the difference directly (see W2).

Different principles. Angular-KD is motivated by ensemble diversity, where more diverse views tighten a bound on the ensemble loss; GVD is motivated by Vicinal Risk Minimisation, where supervision from a local neighbourhood of each teacher target is intended to promote locally stable student predictions.

A shifted, controllable target. Angular-KD averages its views with the teacher output, so supervision remains centred on the original prediction. In GVD, the score prior together with anchor guidance (locality) and target guidance (class consistency) shifts targets toward high-density, class-consistent regions at sampling time, without retraining the prior. This enables counterfactual targets that correct teacher errors (Tables 5–6), for which Angular-KD offers no comparable control or analysis.

A single generative prior versus one head per view. Angular-KD needs a separate feature and logit head per view, so NN is fixed by the architecture and parameters grow with NN. GVD performs all generation with a single prior, trained once and reused: the number of views is a sampling-time choice that adds no parameters, and fresh targets are drawn every epoch. Each GVD view does cost several denoising steps against one linear map, so the advantage is flexibility and parameter scaling, not per-view compute.

Progressive supervision. Because dispersion is controlled by reverse-step initialisation and guidance strength, GVD explicitly schedules targets from broad to narrow over training. Angular-KD's view dispersion is governed by a learned margin, which may change during training but follows no designed coarse-to-fine schedule.

Following the reviewer’s suggestion, we have revised the related work section to clarify GVD’s novelty over Angular-KD and other single-teacher augmentation methods.


W2, Q1. (Direct comparison with Angular-KD) We thank the reviewer for this suggestion. We have added both an accuracy comparison and a direct analysis of the generated teacher views (Table 13, Table 24 ).

1. Predictive accuracy. We compare GVD with Angular-KD and TeKAP on CIFAR-100 under our training protocol, using the number of views from their original papers (five heads for Angular-KD, three views for TeKAP).

MethodWRN-40-2 → WRN-40-1ResNet-50 → MobileNetV2
TeKAP74.5262.75
Angular-KD74.9565.42
GVD75.7467.22

GVD improves over Angular-KD by +0.79 and +1.80 points, with the larger margin in the heterogeneous setting, where teacher and student representations differ most.

2. Quality of the generated views. Following the reviewer's suggestion, we compare the views themselves on CIFAR-100 (1,000 anchors × 5 views, pooled teacher features). Besides same-class 5-NN distance and 5-NN label purity, we introduce two metrics: pairwise diversity, the mean angle between views of the same anchor, and local coverage, the percentage of an anchor's 50 nearest same-class teacher representations that have a generated view within their own 5-NN radius.

Method5-NN distance ↓5-NN purity (%) ↑Local coverage (%) ↑Diversity, feature (°)Diversity, logit (°)
Angular-KD17.5429.710.0090.8510.41
GVD8.0490.142.9310.312.34

(continued in the next comment)

W2, Q1 (Direct comparison with Angular-KD) These results address each of the reviewer's questions: Diversity: Angular-KD produces substantially more diverse views, as expected from its explicit angular-diversity objective. Locality: GVD views lie much closer to same-class teacher representations (5-NN distance 8.04 vs. 17.54). Class consistency: 90.14% of the nearest teacher neighbours of GVD views share the ground-truth class, compared with 29.71% for Angular-KD. Neighbourhood coverage: GVD views cover part of the local same-class neighbourhood, whereas Angular-KD views cover none of it.

The two methods therefore occupy different points of the diversity–consistency trade-off. Angular-KD obtains diversity by moving views away from the teacher feature distribution; we note that this is compatible with its CRD-based objective, which does not require views to lie on that distribution. GVD instead concentrates its variation within the region occupied by same-class teacher representations, which is the property its vicinal formulation is designed to provide. Together with the accuracy gains above, this supports that learning a vicinal distribution yields targets that are local, class-consistent and more effective for distillation.

3. Scalability of views. Angular-KD requires a separate pair of trainable heads per view, so the number of views is fixed by the architecture. GVD uses a single prior trained once per teacher, so the number of views is a sampling-time choice that adds no parameters.


W3, Q1 (cost–performance trade-off). We agree that GVD costs more to train than lightweight augmentation such as TeKAP, and we address why we believe the trade-off is favourable. The revised Sec. 5.3.3 (Tables 25–28) now analyses this trade-off in three settings across two datasets, against baseline meethods.

SettingStep time, GVD / TeKAPMemory, GVD / TeKAPTop-1 gain over TeKAP
CIFAR-100, WRN-40-2 → WRN-40-10.12 / 0.03 s (4.0×)2.07 / 0.47 GB+1.22
CIFAR-100, ResNet-50 → MobileNetV20.10 / 0.05 s (2.0×)0.75 / 0.69 GB+4.47
TinyImageNet, ResNet-50 → MobileNetV10.43 / 0.32 s (1.3×)4.36 / 3.69 GB+2.86

Overhead shrinks as models grow. The prior operates on the teacher's final feature map, so its relative cost falls with model size: from 4.0× TeKAP's step time in the smallest setting to 1.3× on TinyImageNet, with only +0.06 GB extra memory for ResNet-50 → MobileNetV2. GVD also has equal or lower step time and lower memory than diffusion-based DiffKD in all settings ((Table 26).

Gains over simpler view-augmentation methods are largest where distillation is hardest. GVD significantly outperforms TeKAP in all six CIFAR-100 pairs (p<0.001p<0.001; Sec. 5.3.4), by +3.34 to +4.69 points in the three heterogeneous pairs and +0.45 to +1.22 in the homogeneous ones, and by +1.43 and +2.86 on TinyImageNet. Against Angular-KD under the same protocol (Tab. 13), GVD improves by +0.79 (WRN-40-2 → WRN-40-1) and +1.80 (ResNet-50 → MobileNetV2).

The prior is a one-time cost, cheaper than the teacher. It is trained once per teacher and reused across all its students, and its training step is cheaper than the teacher's in every setting (e.g., 0.28 vs. 0.98 s/step and 3.56 vs. 24.92 GB on TinyImageNet). Inference cost is identical for all methods.

The gain is attributable to the prior. At matched displacement, simple view augmentations give no gain over fixed targets (74.45): Gaussian perturbations reach 74.11 and Mixup 73.43, whereas GVD reaches 75.48. Our new sample-quality analysis (Sec. 5.3.2) further shows that GVD views are closer to the teacher feature neighbourhood and more class-consistent than cheaper alternatives at matched displacement.

Benefits beyond accuracy. Compared with TeKAP, GVD is markedly stronger with limited data (50.02 vs. 42.02 at 10% of the data), on teacher-misclassified samples (+3.00), and under overfitted teachers (drop of 1.79 vs. 2.04), and it improves fidelity and calibration.

Controllable cost. We use one sample and 5 denoising steps. Unlike multi-teacher distillation (training several teachers), teacher-assistant methods (training intermediate models), or Angular-KD (trainable heads per view), GVD uses a single prior trained once per teacher and reused across students, so its cost does not grow with the number of views.


Thank you again for your valuable feedback, which has substantially strengthened the paper. We have addressed each of your concerns and would be grateful if you could consider the revised manuscript in your final evaluation.

Reviewer_kteAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generative Vicinity Distillation (GVD), a knowledge distillation framework that extends fixed teacher targets to learned vicinal target distributions. A diffusion prior is trained over teacher representations, while anchor and target guidance encourage locality and class consistency. Epoch-dependent schedules adjust guidance strength and reverse-process initialization to provide progressive supervision. Experiments on CIFAR-10/100, TinyImageNet, and 20 UCR time-series datasets report improvements over several KD baselines in classification performance, teacher–student fidelity, and calibration. Additional analyses examine performance under imperfect teacher supervision and on teacher-misclassified examples.

Critical strengths

• The paper explores an interesting way to enrich supervision from a single teacher by generating targets around its representations. The diffusion prior, combined with anchor and class guidance, provides a well-motivated alternative to simply adding random noise to teacher features.

• The experiments cover both image and time-series classification across different teacher–student architectures, assessing calibration and fidelity alongside accuracy. The ablations and analyses of teacher-misclassified samples also help explain how the method’s components contribute to student learning.

Critical weaknesses

• The contribution of the generative prior is not sufficiently isolated. In Table 5, improvements in ground-truth class probability and hard-label KL closely follow the objective used for target guidance in Eq. (6). Although Table 6 shows downstream student gains, it evaluates the complete pipeline. A comparison with directly refining teacher features using the same classification objective, without diffusion and with comparable displacement from the anchor, would help determine whether the learned prior contributes beyond supervised target correction. This distinction is important for establishing the value of the paper’s main generative component.

• The claimed manifold consistency of generated targets lacks direct validation. This property is central to the motivation for using a learned diffusion prior rather than Gaussian perturbations or Mixup. However, improved student accuracy and movement toward class prototypes do not establish that guided samples remain near plausible local teacher features. Quantitative diagnostics of locality and proximity to real teacher-feature neighborhoods, before and after guidance and at comparable perturbation magnitudes, would help substantiate the proposed mechanism.

Questions for authors
  1. How does GVD compare with directly refining teacher features using the same ground-truth classification objective without diffusion, while controlling displacement from the original features? This would help isolate the contribution of the learned prior from label-guided target correction.

  2. What quantitative evidence supports the quality of the learned prior and the manifold consistency of its generated targets? In particular, are held-out denoising results available, and do guided samples remain near plausible teacher-feature neighborhoods compared with Gaussian and Mixup targets at similar perturbation magnitudes?

  3. How does GVD compare with relational or representation-alignment approaches, such as RKD or a CKA-based alignment baseline, under the same teacher–student settings?

  4. Could you clarify the sign of the anchor guidance in Eq. (5)? As written, the positive sign appears to move the predicted clean representation away from the anchor rather than toward it. Should the sign be reversed, or is the guidance direction defined differently?

  5. Could you verify the accuracy drops in Table 18? KD decreases from 74.40% to 72.89%, a drop of 1.51 rather than 2.51 percentage points, which is smaller than GVD’s drop. While GVD retains the highest absolute accuracy, how does this correction affect the claim that it is the least affected by overfitted teacher supervision?

2 comments

We sincerely thank the reviewer for their thoughtful review. We appreciate the recognition of GVD's guided diffusion prior as an interesting and well-motivated alternative to random noise, and of our evaluation across image and time-series tasks, including calibration, fidelity, ablations and teacher-misclassified analyses. We address each concern below.

W1 and Q1. We thank the reviewer for this valuable suggestion. To isolate the contribution of the learned diffusion prior from classifier-guided correction, we conducted the requested additional experiments.

First, we evaluate GVD without either guidance term. Partial noising without any guidance (tinit:50→30, λ=0t_{\mathrm{init}}:50\rightarrow30,\ \lambda=0) achieves 74.93% accuracy, outperforming KD (74.40%), FitNets (74.31%), and their combined fixed-target supervision (74.45%). This demonstrates that the learned diffusion prior provides useful supervision even without explicit guidance.

Second, we compare GVD against two prior-free alternatives under matched feature displacement: (i) direct classifier-gradient refinement of teacher features (η=3\eta=3) and (ii) classifier-guided Gaussian perturbations (η=0.03\eta=0.03). For a controlled comparison, we fix GVD's sampling configuration (tinit=30, λ=0.3t_{\mathrm{init}}=30,\ \lambda=0.3), eliminating progressive scheduling. All methods use five generation/refinement steps, approximately matched mean feature displacement (ℓ2≈12.0\ell_2\approx12.0), and identical student training settings. Under these conditions, GVD achieves 75.48% accuracy, compared with 73.01% for direct refinement and 74.22% for classifier-guided Gaussian perturbations. These results show that GVD's improvements cannot be explained only by classifier-guided feature correction, supporting the value of diffusion-based target generation.

We have incorporated these experiments and the corresponding discussion into the ablation section of the revised manuscript.


W2 and Q2. We thank the reviewer for this valuable suggestion. We have added a new subsection, Section 5.3.2: Quantitative Evaluation of GVD Sample Quality, providing direct quantitative evidence of held-out denoising quality and the local geometric consistency of generated teacher representations.

First, we evaluate the learned prior on held-out teacher features. Unguided diffusion reconstruction achieves cosine similarity of 0.9980 (MSE: 0.0188) at t=30t=30, remaining above 0.96 at t=200t=200. This demonstrates high reconstruction fidelity on unseen teacher representations without relying on classifier guidance.

Second, we assess manifold-consistency indicators under matched feature displacement. Using 1,000 held-out CIFAR-100 images, we generate 10 vicinal targets per image and compare them against a reference bank of 50,000 real teacher representations. We evaluate five complementary metrics: same-class 5-NN distance, closer-than-original rate, off-tangent residual, paired local intrinsic dimensionality (LID) error, and 5-NN label purity. Gaussian and Mixup perturbations are matched to GVD's per-sample displacement, while other baselines use approximately matched magnitudes.

GVD achieves the lowest same-class 5-NN distance (8.0425), highest closer-than-original rate (98.71%), lowest off-tangent residual (0.7453), and lowest paired LID error (0.9116). Compared with classifier-guided Gaussian perturbations, GVD improves neighborhood proximity (8.0425 vs. 9.0351) and label purity (90.03% vs. 71.34%). Direct classifier-gradient refinement also yields weaker neighborhood proximity (8.7923) and substantially higher LID error (4.9875), while same-class Mixup exhibits higher LID error (3.1666). Furthermore, GVD without either guidance term retains favorable neighborhood metrics, supporting the contribution of the learned prior beyond explicit guidance.

Together, these results provide direct quantitative evidence that GVD-generated targets better preserve the observed local geometry and semantic structure of teacher representations than the evaluated alternatives. We have incorporated the experimental protocol, metric definitions, ablations, and results into Section 5.3.2 of the revised manuscript.

Q3. We thank the reviewer for this suggestion. We have added comparisons with distance-wise relational distillation (RKD-D) and combined distance- and angle-wise distillation (RKD-DA) under the same TinyImageNet teacher–student settings. Following the original RKD formulation, we use λA=2λD\lambda_A=2\lambda_D.

For ResNet-34 →\rightarrow ResNet-18, GVD achieves 64.07% Top-1 accuracy, outperforming RKD-DA (63.52%) and RKD-D (63.09%). Similarly, for ResNet-50 →\rightarrow MobileNetV1, GVD achieves 61.24%, compared with 60.11% and 58.72%, respectively. GVD also achieves the highest Top-5 accuracy in both settings.

These results demonstrate that GVD consistently outperforms relational distillation baselines, supporting the effectiveness of generative vicinal supervision beyond conventional relational alignment. We have incorporated these comparisons into the revised manuscript.


Q4 (sign of Eq. 5). We thank the reviewer for spotting this. The error was in the presentation, not the implementation (see our anonymised code repository): anchor guidance is applied directly to the predicted clean representation, ẑ₀ᵃ = ẑ₀ + λᵛᵃ·√(1 − αₜ)·(a − ẑ₀), equivalently ε̃ = ε − √αₜ·λᵛᵃ·(a − ẑ₀) in noise space, which moves the estimate toward the anchor. We have corrected the sign and scaling of Eq. (5) in the revised manuscript.


Q5: We thank the reviewer for identifying this inconsistency. Verification of the original experimental records confirmed a transcription error: KD's accuracy should be 71.89%, not 72.89%. Thus, the reported 2.51-point drop is correct, and GVD retains the smallest accuracy drop. We have corrected Table 18 (Table 20 in Revision) accordingly.


Thank you again for your valuable feedback, which has substantially strengthened the paper. We have addressed each of your concerns and would be grateful if you could consider the revised manuscript in your final evaluation.

Reviewer_zatxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that knowledge distillation supervises each input with a single fixed teacher target and proposes Generative Vicinity Distillation (GVD), which instead samples targets from a neighborhood of that target. A DDIM prior is trained on teacher features (the input to the teacher's classification tail). During student training, each teacher feature is noised to a small timestep (t_init from 50 to 30, T = 1000) and denoised in five steps with two guidance terms: an anchor term meant to keep the sample near the original feature (Eq. 5), and a class term that applies the gradient of the teacher tail's cross-entropy to the predicted clean feature (Eq. 6). Over training, t_init decreases and the guidance scale increases on cosine schedules, which the authors describe as coarse-to-fine supervision. When the teacher misclassifies a sample, the class term is supposed to turn its feature into a "counterfactual" target for the true class. The student is trained with cross-entropy, an MSE loss to one sampled feature, and a KL loss to the teacher-tail logits of that feature, and the paper interprets this as a teacher-induced form of vicinal risk minimization (VRM). GVD has the best Top-1 accuracy in all six CIFAR-100 teacher–student pairs, 0.09 to 1.98 points above the strongest baseline, and is also best on TinyImageNet, CIFAR-10, and the UCR average. Further experiments cover calibration, teacher–student agreement, limited data, an overfitted teacher, transfer to STL-10, ablations, and training cost.

Critical strengths
  1. The idea of replacing a fixed teacher target with generated, controllable targets is a sensible response to known problems of single-teacher KD, such as teacher errors and overconfidence, and it avoids training several teachers. The guidance can in principle encode other target criteria (Sec. 5.4). The class guidance in Eq. 6 is done well: the gradient is taken on the predicted clean feature, where the teacher tail was trained, and then mapped back to the noise prediction. I checked the derivation and it is correct. The prior is only used during training, so inference cost does not change.

  2. The experiments are broad: four image datasets, 20 UCR datasets, and six homogeneous and heterogeneous CIFAR-100 pairs. GVD has the best Top-1 in every CIFAR-100 pair (Table 1) and the most UCR wins in all three settings (Table 4). The calibration results are the most consistent finding, with the lowest ECE and Brier score in all six CIFAR-100 pairs (Table 15). Costs are reported in detail (Tables 19–22), including per-stage time and memory and the effect of the number of sampling steps and samples, so the training overhead (0.12 s vs. 0.03 s per step for KD) is clear.

Critical weaknesses
  1. The learned vicinity is the core of the method, but the paper does not show that it matters, and the objective does not behave as described. This is my main concern.

(a) In expectation the loss is still point-wise. For the feature term, E‖z̃ − z_s‖² = ‖μ(x) − z_s‖² + E‖z̃ − μ(x)‖², where μ(x) is the mean vicinal sample and the second term does not involve the student. If the logit term is KL(teacher‖student), its expectation is KL(p̄(x)‖p_s) plus a constant. The student is therefore still fitted to one (epoch-dependent) target per input, plus sampling noise (M = 1). It cannot learn a local distribution, and unlike in VRM the input is never perturbed. The claims in lines 165–170 and 310–341 (and Fig. 3) do not follow from Eqs. 9–11.

(b) The ablations do not isolate the prior. "No guidance" (64.54) and "class guidance only" (65.61) start from pure noise, so their samples do not depend on x, while "anchor guidance only" (75.17) presumably starts from the noised anchor (lines 453–456). There is no run with partial noising alone. Eq. 5 also looks wrong: plugging it into Eq. 1 gives ẑ₀′ − a = (ẑ₀ − a)(1 + (1 − α_t)λ/√α_t), which moves the estimate away from the anchor, and under a standard linear schedule (1 − α_t)/√α_t is at most about 0.03 for t ≤ 50. Locality seems to come from the small starting noise, not from guidance or a learned geometry.

(c) "Manifold consistency" is never measured. No statistics of the generated samples are reported, and the Gaussian and Mixup controls use a single setting each (σ = 0.1, α = 0.4).

The results are equally consistent with a simpler explanation, namely softened, label-corrected teacher targets plus target noise, which is not the contribution claimed in lines 102–106.

  1. The novelty over existing work is limited.

(a) Generating several targets from one teacher by perturbing its outputs is the idea of TeKAP (lines 52–53). GVD replaces the Gaussian noise with the feature-space DDIM of DiffKD (line 784), trained on teacher instead of student features.

(b) The sampling step combines known techniques without citing them: partial noising and denoising to obtain nearby variants (SDEdit, Meng et al., 2022), and a loss gradient on the predicted clean sample mapped back to the noise prediction (Eq. 6; backward guidance in Bansal et al., 2023).

(c) The "counterfactual supervision" is the class guidance pushing a feature toward its label, so the gains in Table 5 are expected by construction. The 30.15-point gain over the teacher in Table 6 is built in, since the teacher is wrong on every sample in that subset. The progressive schedule adds 0.26 points (lines 457–460).

Given TeKAP, the claim to be the first to move from individual teacher targets to vicinal distributions (lines 694–697) is too strong. The paper needs to show what the learned prior adds over a tuned Gaussian vicinity.

  1. The size of the gains is hard to judge.

(a) Several baselines look under-tuned. DIST is below vanilla KD in all six CIFAR-100 pairs, TeKAP in five of six (by 3.60 points on ShuffleNetV2), and DiffKD is only 0.15–0.50 points above the non-distilled student (Table 1). TeKAP is the most direct comparison for a learned vicinity, so this weakens the main evidence.

(b) No standard deviations are reported, although five runs were done (line 348). Some margins (+0.09 on ResNet-20, +0.12 on ShuffleNetV1) are smaller than the changes between neighboring hyperparameter values (Tables 21–22), and those hyperparameters are selected on the test set (lines 1017–1018, 1036–1037).

(c) GVD uses both feature and logit losses, while KD and FitNets use one each. The fixed-target combination is reported for only one pair (74.45 vs. 75.74, lines 443–448).

Minor: the KL direction and temperature, the noise schedule, the feature projector for heterogeneous pairs, and the meaning of c^(ep) in Eq. 10 are not given. Table 11 lists GVD with 0 losses in every setting, although it has only 14, 14, and 11 Top-1 wins.

Questions for authors
  1. Main concern: Which KL direction and temperature are used in Eq. 11? Could you report distillation to Monte Carlo estimates of μ(x) and p̄(x) (e.g., 10 samples per input) on WRN-40-2→WRN-40-1 with a few seeds? If it comes close to 75.74, how would you revise the vicinal/VRM interpretation?

  2. Role of the prior: What sign, scaling, and noise schedule does the code use for Eq. 5? Please add (a) t_init 50→30 with no guidance, and (b) the same class guidance applied to a Gaussian perturbation of matched size, without the prior. Simple statistics of the generated samples, such as distance and cosine similarity to the anchor, would also help.

  3. Baselines and variance: How were the baseline hyperparameters set, given that DIST and TeKAP fall below KD in most pairs? Please give mean ± std for Table 1, teacher accuracies, and KD + FitNets for all six pairs.

  4. Data use: Which split is the teacher-misclassified subset in Table 6 drawn from, and how does a non-distilled student do on it? In Table 17, was the prior trained on the full training set?

3 comments

We sincerely thank the reviewer for their careful review and time. We appreciate the recognition of GVD as a sensible response to single-teacher limitations without training multiple teachers, its unchanged inference cost, the breadth of our evaluation, the consistency of the calibration results, and the detailed cost analysis. We address each concern below.


W1(a), Q1. Eq. (11) uses forward KL (teacher ‖ student) with T=4T=4. Distilling to Monte Carlo estimates of μ(x)\mu(x) and pˉ(x)\bar p(x) (M=5M=5, same progressive schedule) achieves 75.81 ± 0.13, compared with 75.74 ± 0.07 for GVD (M=1M=1). The reviewer's derivation, reducing the objective to mean-target matching, holds for any MSE-based stochastic supervision, and does not render the learned distribution irrelevant: its quality matters through the mean it induces.

Why the VRM motivation still holds. The student is trained to produce, for each input, the prediction that best agrees on average with all plausible teacher targets in its vicinity. These induced targets differ from the teacher's own output in ways determined by the vicinity: First-order local structure: For a denoising prior, the conditional mean shifts each target approximately along the learned score, toward higher-density regions of the teacher representation distribution, so the student inherits local structure of the teacher manifold. Class uncertainty: The averaged prediction reflects class uncertainty across the vicinity, consistent with GVD's improved calibration (Table 17). Progressive targets: Under the guidance schedule, the induced target evolves from broad to narrow over training, providing dynamically changing, neighbourhood-informed supervision rather than a single fixed target.

Empirical support. The vicinity's geometry, not stochasticity, drives the gain: An isotropic vicinity has its mean at the original teacher feature and yields no gain: Gaussian vicinals reach 74.43 (74.11 at matched displacement), on par with fixed teacher targets (74.45).A learned vicinity shifts the mean toward high-density, class-consistent regions: the prior alone reaches 74.93, and full GVD 75.74. At matched displacement, prior samples move closer to same-class real teacher features than the anchor in 92% of cases, against 38% for Gaussian samples (Table 23).

Relation to VRM (lines 165–170, 310–341). Unlike conventional VRM, GVD does not perturb inputs directly. Instead, it operates in the expert model's (teacher) learned representation space, where semantic similarities are better captured than in raw input space (the same motivation underlying manifold mixup over raw-input mixup). By generating vicinal perturbations along the learned score field, with class-consistency guidance to mitigate manifold intrusion (a key challenge in VRM), we argue that GVD's learned vicinities can serve as more faithful proxies for input-space neighborhoods/heuristically constructed vicinities. GVD shares VRM's motivation of leveraging local neighborhoods for improved generalization, using learned vicinities as proxies for local representation-space neighborhoods.

Figure 3 remains consistent with the reviewer's conditional-mean view: (left) point-wise supervision places student features at the original teacher feature; (centre) a heuristic vicinity guides toward a mean that poorly represents the local feature geometry; (right) a learned, geometry-aware vicinity guides toward a more representative mean. Mean-target equivalence thus does not negate the role of the learned vicinity; it explains how the quality of the generated neighbourhood determines student feature placement.


W1(b, c), Q2 (Eq. 5, isolating the prior, manifold consistency). We thank the reviewer for the careful analysis. We have added all the requested ablations (Sec. 3.1) and a quantitative analysis of the manifold consistency of generated samples (Sec. 5.3.2).

Eq. 5. The error was in the mathematical presentation, not the implementation. Anchor guidance is applied directly to the predicted clean representation, moving it toward the anchor, and we have corrected the sign and scaling in the manuscript. We use a 1,000-step linear schedule (βt\beta_t from 10−410^{-4} to 0.020.02, αt\alpha_t cumulative), with λva\lambda^{\mathrm{va}} increasing from 0.3 to 0.5 and tinitt_{\text{init}} decreasing from 50 to 30 over training. The per-step coefficient λva1−αt\lambda^{\mathrm{va}}\sqrt{1-\alpha_t} ranges from about 0.03 to 0.09; although modest per step, it acts directly on the clean prediction at every DDIM step, and the steps compound to remove roughly 13–14% of the distance to the anchor.

W1(b, c), Q2 (Locality.) Partial noising and anchor guidance both keep samples close to the anchor. We agree that partial noising contributes substantially: it sets the radius of the vicinity, while anchor guidance explicitly refines proximity to the original representation. We have added ablations that separate the contributions of partial noising and guidance.

Requested ablations (Q2).

  • (a) Partial noising, no guidance (tinit:50→30t_{\text{init}}: 50\to30, λ=0\lambda=0): 74.93, above KD (74.40) and fixed teacher targets (74.45). Prior-based sampling is thus effective on its own, and guidance (75.74%) adds a further +0.81.
  • (b) Class guidance without the prior, at matched displacement: 74.22 (classifier-guided Gaussian) and 73.01 (direct classifier-gradient refinement), against 75.48 for GVD in the same configuration.
  • Gaussian and Mixup at matched displacement: instead of a single σ or α, both are now scaled to GVD's per-sample displacement, reaching 74.11 and 73.43 (vs. 74.43 and 73.22 at the original settings). GVD's advantage therefore holds both at the originally reported settings and at matched perturbation size.

Manifold consistency (W1c). We evaluate generated features against a reference bank of 50,000 teacher representations of real CIFAR-100 training images at matched displacement (Tables 22–23). The prior provides learned geometry beyond small noise: At the same perturbation size, prior samples move toward real same-class features, follow their local directions of variation and preserve local dimensionality, even without guidance; Gaussian noise drifts off the manifold, while Mixup distorts local structure. We also verify that the prior generalises to unseen data: it reconstructs held-out test-set features with cosine similarity 0.998 (Table 21).

Method5-NN distance ↓Closer than anchor (%) ↑Off-tangent ↓LID error ↓Purity (%) ↑
Gaussian9.0737.990.9220.9969.75
Same-class Mixup8.3789.730.7523.1774.52
Prior, no guidance8.1592.230.7460.9388.81
Full GVD8.0498.710.7450.9190.03

The distillation results further seperate candidate cause. Target noise alone (matched Gaussian, 74.11) and label correction without the prior (74.22 and 73.01) do not reach GVD (75.48); softening is already present in KD's temperature-scaled targets (74.40). The prior alone, using neither labels nor guidance, outperforms all three (74.93), so GVD's gain is not explained by softened, label-corrected targets plus noise.


W2. (a) Novelty and separation from TeKAP and DiffKD: GVD's contribution is to replace each teacher target with a learned vicinity of teacher representations, following VRM's motivation that supervision should reflect the local neighbourhood of each example rather than a single observation. TeKAP perturbs teacher outputs with isotropic random noise centred on the teacher target, so its views do not depend on the teacher's representation distribution. DiffKD uses diffusion on the student side, denoising student features toward a single fixed teacher target. GVD learns a prior over teacher representations and samples locally constrained, class-consistent targets from it, so supervision is shaped by where plausible teacher representations lie.

This separation is reflected empirically. At matched displacement, isotropic perturbations give no gain over fixed targets (74.11 vs. 74.45) and Mixup is lower (73.43), whereas GVD reaches 75.48, and its samples move toward real same-class teacher features in 98.7% of cases against 38.0% for Gaussian ones (Table 23). GVD also significantly outperforms TeKAP and DiffKD in all six CIFAR-100 pairs.

(b) Sampling components. We have added the citations to SDEdit (Meng et al., 2021) and Bansal et al. (2024). Our contribution is not these primitives but their use to construct controllable, class-consistent vicinal targets for distillation, together with guidance functions designed for this purpose (anchor locality and class consistency).

(c) Counterfactual supervision. Table 5 is intended to verify that class-guided targets on the teacher-misclassified set are realised close to the anchor, not as evidence of gains. On teacher-misclassified test samples: GVD reaches 30.15%, against 26.92% (Base), 27.97% (KD), 28.19% (FitNets) and 27.15% (TeKAP). Without class guidance, GVD still reaches 29.24%, above all baselines, so most of the correction does not stem from label guidance. We agree the progressive schedule is a refinement (+0.26), not a core contribution.

W3 / Q3 (baselines, variance and gain size). We thank the reviewer.

a) Baseline hyperparameters. All baselines use their original papers' recommended hyperparameters under the same protocol as GVD (Table 8): KD (τ=4\tau=4); DIST (τ=4\tau=4, α=1\alpha=1, β=1\beta=1); TeKAP's defaults (N(0,1)\mathcal{N}(0,1) noise, mixing coefficient α=0.1\alpha=0.1, 3 augmented views). GVD's defaults (Table 9) are fixed once and used unchanged across all datasets and pairs. They are chosen for efficiency, not test accuracy: more denoising steps or samples give higher test accuracy (Tables 27–28), yet we report 5 steps and M=1M=1. The reported results are therefore not inflated by test-set selection. That DIST and TeKAP can fall below KD under a shared protocol is consistent with prior reports of small, setting-dependent gains (e.g., TeKAP falls below its host method in several settings in Angular-KD, Yu et al., 2025, Table 2). Moreover, the comparison with learned single-teacher vicinities does not rest on TeKAP alone: GVD also outperforms Angular-KD (74.95 → 75.74; 65.42 → 67.22; Table 13) and outperforms TeKAP in all six pairs (p<0.001p<0.001) (Sec 5.3.4 in revised paper).

b) Variance and significance. Mean ± std for all Table 1 results are now reported in Table 29, and the new Sec. 5.3.4 tests significance with Welch's tt-test. GVD significantly outperforms FitNets, DIST, DiffKD and TeKAP in all six CIFAR-100 pairs. Against KD, its gains are significant in four of six pairs (p≤0.001p \leq 0.001) and within training variance for other two pairs. On time series, a paired Wilcoxon test over 60 experiments confirms significant gains over all baselines (p≤2.36×10−5p \leq 2.36\times10^{-5}).

c) Combined feature and logit baseline. We now report KD + FitNets (same two losses as GVD, fixed teacher targets) and teacher accuracies for all six pairs (Table 29, Sec. 5.3.4). KD + FitNets never significantly improves over KD, whereas GVD significantly outperforms it in five of six pairs (p<0.05p<0.05), showing that the gains stem from vicinal supervision rather than from combining the two losses.


Q4. Teacher-misclassified subset (Table 6) contains all CIFAR-100 test images misclassified by the teacher, which we have clarified in the revised paper. A non-distilled student (Base) reaches 26.92% on this subset:

TeacherBase (no distillation)TeKAPKDFitNetsGVD
Top-1 on misclassified test-set (%)0.0026.9227.1527.9728.1930.15

Table 17(Table 19 in Revision). All components use the subsampled training set, except the teacher.


Minor points. (i) For heterogeneous pairs, student features are mapped to the teacher feature dimension with [a 1×1 convolutional projector followed by BatchNorm], trained jointly with the student, as in FitNets. (ii) In Eq. (10), c^(ep) denotes the epoch-dependent sampling configuration, i.e. the guidance strengths λ^(ep) (Eq. 7) and initial timestep t_init^(ep) (Eq. 8). (iii) Table 11 (wins and losses). "Wins/Draws" and "Losses" are pairwise: each baseline column counts the datasets where GVD is equal or better than, or worse than, that baseline, whereas "Top-1 Wins" counts datasets where a method is best among all methods (definitions in Appendix 5.2). The GVD column's "20" and "0" were placeholders and have been replaced with "–", as in Table 4.


Thank you again for your valuable feedback, which has substantially strengthened the paper. We have addressed each of your concerns and would be grateful if you could consider the revised manuscript in your final evaluation.

Comments on the paper

1 comment

General Response to All Reviewers and the Area Chair

We sincerely thank all reviewers and the Area Chair for their careful and constructive feedback. The reviews have substantially strengthened the paper. We have uploaded a revised manuscript and summarise the main changes below; detailed responses are provided in each reviewer's thread.

1. Isolating the role of the learned prior (zatx, kteA, o8Mm)

  • Prior without any guidance (partial noising only): 74.93, above KD (74.40) and fixed teacher targets (74.45).
  • Prior-free, classifier-guided alternatives at matched displacement: classifier-guided Gaussian (74.22) and direct classifier-gradient refinement (73.01), against 75.48 for GVD in the same configuration.
  • Simple local baselines at matched displacement: Gaussian (74.11) and Mixup (73.43) remain clearly below GVD.
  • Monte Carlo mean-target distillation (75.81 ± 0.13 vs. 75.74 ± 0.07): we clarify that GVD's benefit lies in where the learned vicinity places this target.

2. Direct evaluation of generated targets (zatx, kteA, o8Mm, gbce) — new Sec. 5.3.2

  • Held-out denoising quality of the prior (cosine similarity 0.998).
  • Five manifold-consistency metrics against 50,000 real teacher representations at matched displacement: GVD achieves the lowest same-class 5-NN distance, the highest closer-than-anchor rate (98.7% vs. 38.0% for Gaussian), the lowest off-tangent residual and LID error, and 90% label purity.
  • View-level comparison with Angular-KD: diversity, locality, class consistency and local coverage.

3. New baselines and comparisons (gbce, kteA, zatx)

  • Angular-KD: GVD improves by +0.79 and +1.80 points (homogeneous and heterogeneous pairs).
  • Relational KD (RKD-D, RKD-DA) on TinyImageNet.
  • KD + FitNets (same two losses as GVD with fixed targets) for all six CIFAR-100 pairs.

4. Statistical significance and variance (zatx, o8Mm) — new Sec. 5.3.4

  • Mean ± std over five runs and teacher accuracies for all CIFAR-100 pairs (Table 29).
  • Welch's t-test: GVD significantly outperforms FitNets, DIST, DiffKD, TeKAP and KD + FitNets across settings, and KD in four of six pairs, while being on par in the remaining two.
  • Wilcoxon signed-rank test over 60 time-series experiments: significant gains over all baselines.

5. Computational cost (gbce, o8Mm) — extended Sec. 5.3.3

  • Full cost analysis in three settings across two datasets. GVD's relative overhead decreases with model size (1.3–1.4× step time on TinyImageNet), the prior is a one-time cost cheaper to train than the teacher, and inference cost is unchanged.

6. Role of labels in teacher-error correction (o8Mm, zatx)

  • Without class guidance, GVD still outperforms all baselines on teacher-misclassified test samples (29.24% vs. 27.97% for KD), so most of the correction arises from label-free generation.

7. Corrections and clarifications

  • Corrected the sign and scaling of Eq. (5) (presentation only; the implementation was correct), and added citations to SDEdit and Universal Guidance.
  • Clarified the KL direction and temperature, noise schedule, data splits, pairwise win/loss definitions, and corrected a transcription error in the overfitted-teacher table.

We believe these additions address the reviewers' all main concerns, in particular the contribution of the learned prior, supported by an extensive analysis of manifold consistency, and the robustness of the reported gains. We would be grateful if the reviewers could consider the revised manuscript and responses in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.