- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CondKD, which reformulates knowledge distillation as conditional generation of teacher features given student representations. It introduces SplitTok to address high-dimensional optimization and Distribution Contraction to incorporate label supervision. Experiments show competitive results across several benchmarks.
Critical strengths
- Reformulating KD as conditional generation offers an interesting perspective. The proposed components are reasonably motivated and fit well within the overall framework.
- The experiments cover diverse datasets and teacher-student architectures, providing a relatively broad evaluation of the proposed method.
Critical weaknesses
- The paper lacks a direct comparison with deterministic feature regression under the same settings. It remains unclear whether the gains come from generative modeling or simply from transferring teacher features.
- Theorem 1 establishes a gradient relationship for a single-step feature estimate, but this does not necessarily imply more stable network optimization. The connection between the theorem and the actual training objective also needs clarification.
- While SplitTok alleviates high-dimensional optimization difficulties, its token-wise reconstruction objective does not explicitly model dependencies across feature dimensions. It remains unclear whether this affects feature reconstruction and how SplitTok compares with simpler dimensionality-reduction strategies.
Questions for authors
- How does CondKD compare with simple MSE-based feature regression, particularly when using the same Distribution Contraction targets?
- Can the authors clarify how Theorem 1 supports the claimed optimization stability beyond the single-step gradient relationship?
- Could the authors clarify how SplitTok handles dependencies across feature dimensions, and whether it offers advantages over simpler dimensionality-reduction strategies?
5 comments
Q2: Clarification on Theorem 1.
We thank the reviewer for the question and make the following clarification on Theorem 1. Theorem 1 serves two purposes:
(1) Explaining how Distribution Contraction incorporates label supervision. By rewriting the actual noise-prediction objective using the single-step feature estimate, the theorem decomposes its feature gradient into teacher-feature reconstruction and a label-dependent correction anchored to the ground-truth class prototype. This explains how label supervision enters the unified reconstruction objective without an explicit classification loss.
(2) Establishing the connection between CondKD and multi-task learning at the feature-gradient level. The theorem identifies how CondKD replaces the probability-dependent classification component with a teacher-anchored correction, quantifies their gradient discrepancy, and specifies the condition under which they are close.
These results provide a mechanistic justification for Distribution Contraction and its interpretation as a teacher-guided surrogate of multi-task feature gradients. We agree that they do not alone establish more stable network optimization. We will clarify this scope and revise the corresponding stability claims.
Q3: Dependency modeling across feature dimensions.
(I) Dependencies across feature dimensions. CondKD reconstructs teacher-feature tokens separately, while jointly reconstructing the dimensions within each token. Every token is conditioned on the complete student representation, which is learned jointly with the diffusion head. Reconstruction gradients from all tokens encourage this global condition to encode information useful across multiple teacher-feature groups. Consequently, dependencies across tokens can be captured implicitly through information encoded in the shared student representation.
Although CondKD does not explicitly model residual dependencies beyond this learned condition, our reconstruction-error analysis and overlapping-token ablation will assess the practical consequences of token-wise reconstruction. We leave more advanced cross-token dependency modeling to future work.
(II-a) Reconstruction fidelity. We compare SplitTok with deterministic whole-feature regression on ImageNet validation data. Boundary regions include dimensions on each side of internal token boundaries; interiors exclude dimensions from each token end.
| Method | Overall MSE | Boundary (r=4) | Interior (r=4) | Boundary (r=8) | Interior (r=8) |
|---|---|---|---|---|---|
| CondKD | 3.25284 | 3.24044 | 3.25222 | 3.25067 | 3.25334 |
| Deterministic regression | 3.87007 | 3.85790 | 3.86919 | 3.86908 | 3.87042 |
Table R14. Reconstruction errors. All values are reported in units of 10⁻³.
CondKD reduces overall MSE by 15.95%, with comparable boundary and interior errors. This suggests no evident boundary-specific reconstruction penalty, although it does not establish explicit complete cross-token dependency modeling.
(II-b) Ablation on overlapping tokens. We compare the original nonoverlapping partition with 50% overlapping tokens. Overlap allows dimensions spanning the original token boundaries to be reconstructed jointly within a token. Repeated dimensions receive inverse-coverage loss weights, and their predictions are averaged when assembling the complete feature. As shown in the Table, accuracy remains nearly unchanged, suggesting limited benefit from overlap in this setting.
| SplitTok configuration | Top-1 (%) |
|---|---|
| Without overlap | 72.40 |
| With 50% overlap | 72.39 |
Table R15. Ablation on overlapping tokens for SplitTok.
Q4: Comparison with simpler dimensionality-reduction strategies.
We compare SplitTok with fixed random projections that compress teacher features to 64 or 128 dimensions. SplitTok instead partitions the full teacher representation into 64-dimensional tokens, retaining all feature dimensions. We evaluate downstream distillation accuracy on ImageNet using R34 as the teacher and R18 as the student. As shown in the table, random projection yields much lower accuracy, possibly because compression discards useful teacher information.
| Method | Top-1 accuracy (%) |
|---|---|
| SplitTok (64-dimensional tokens) | 72.40 |
| Random projection (64 dimensions) | 70.94 |
| Random projection (128 dimensions) | 71.20 |
Table R16. Comparison with fixed random projections on ImageNet using R34 as the teacher and R18 as the student.
Thank you for the additional experiments and clarifications. The new ImageNet comparisons and SplitTok ablations address several of my initial concerns. Three limitations still inform my assessment.
- CC3M comparison. The main CC3M result compares CondKD with logit-based KL distillation. A matched deterministic feature-regression baseline is still missing, leaving the specific contribution of diffusion unclear in this unlabeled setting.
- Theoretical scope of Distribution Contraction. For a single-step feature estimate and a frozen teacher classifier, Theorem 1 quantifies the difference between the CondKD and multi-task feature gradients and specifies when they are close. As the authors acknowledge, the claim that this surrogate leads to more stable updates to the student network during training goes beyond what the theorem establishes.
- SplitTok. All tokens are conditioned on the full student representation. Once that representation is fixed, each token is reconstructed separately, leaving any remaining dependencies between tokens unmodeled. The overlap ablation shows that jointly reconstructing some dimensions across token boundaries has little effect on accuracy in the tested setting.
Given these limitations, I am retaining my original rating of 2.
Q1: Comparison with deterministic feature prediction and the benefits of uncertainty modeling.
(I) Comparison with deterministic feature prediction. We replace CondKD’s diffusion head with a deterministic MLP predictor, keeping all other training settings unchanged. As shown in the Table, CondKD consistently outperforms the deterministic prediction, supporting the effectiveness of diffusion modeling.
(II) Performance on uncertainty groups. We assess local conditional ambiguity on ImageNet using frozen ResNet-34 and ResNet-18 reference models from PyTorch. We extract features immediately before their linear classifiers and align training features by image identity. For each validation image xi, we retrieve its k=50 nearest training examples in the ResNet-18 feature space, using Euclidean distance after per-dimension standardization with training-set statistics. We measure the dispersion of the corresponding ResNet-34 features:
\|t_j-\bar{t}_i\|_2^2}{\frac{1}{N} \sum^{N}_{j=1} \| t_j - \mu_t \|_2^2}$$ Here, $t_j$ denotes the teacher feature, $\bar{t}_i$ is the mean teacher feature within the retrieved neighborhood, $\mu_t$ is the training-set teacher-feature mean, and $N$ is the training-set size. A larger score indicates greater teacher-feature variation among nearby student representations, providing a **local proxy for conditional uncertainty**. We partition validation images into five uncertainty quintiles. Both methods use identical group assignments derived from the frozen reference models, independently of their predictions. CondKD achieves larger accuracy gains in the higher-uncertainty groups, supporting its benefit on examples with greater local conditional ambiguity. | Group | Mean $U$ | PyTorch | Deterministic | CondKD | Gain (pp) | |:--|--:|--:|--:|--:|--:| | Bin 1: lowest uncertainty | 0.237 | 84.51 | 84.98 | **85.07** | +0.09 | | Bin 2 | 0.334 | 78.70 | 79.72 | **79.78** | +0.06 | | Bin 3 | 0.418 | 70.18 | 72.68 | **73.28** | +0.60 | | Bin 4 | 0.511 | 61.66 | 63.68 | **65.28** | +1.60 | | Bin 5: highest uncertainty | 0.647 | 53.74 | 56.50 | **58.57** | +2.07 | | Overall | 0.429 | 69.76 | 71.51 | **72.40** | +0.89 | *Table R12. ImageNet accuracy (%) of R18 across uncertainty quintiles, each containing 10,000 images. PyTorch denotes the supervised pretrained ResNet-18 reference model. Gain denotes CondKD’s improvement over deterministic MSE* **(III) Teacher-feature predictability and local residual ambiguity.** We freeze both trained students and fit matched linear and MLP probes to predict the same raw teacher features. To measure local residual dispersion, we compute $r_j=t_j-g(z_j)$. Here, $g$ averages the predictions of three independently initialized MLP probes. For each validation image, we retrieve its $k$ nearest examples in the corresponding student-feature space and calculate: $$V_i = \frac{1}{D(k-1)} \sum_{j \in \mathcal{N}_{k(i)}} \|r_j - \bar{r}_i \|_2^2, \qquad \bar{r}_i = \frac{1}{k} \sum_{j \in \mathcal{N}_{k(i)}} r_j,$$ Here, $D$ is the teacher-feature dimension. We report the average over all 50,000 ImageNet validation images. | Evaluation metric | Deterministic | CondKD | Reduction | |:--|--:|--:|--:| | Linear-probe MSE | 0.28997 | **0.23880** | **17.65%** | | MLP-probe MSE | 0.26104 | **0.20882** | **20.00%** | | Local residual dispersion, k=25 | 0.21256 | **0.16233** | **23.63%** | | Local residual dispersion, k=50 | 0.21791 | **0.16540** | **24.10%** | | Local residual dispersion, k=100 | 0.22602 | **0.16915** | **25.16%** | *Table R13. Teacher-feature predictability and local residual ambiguity. Lower is better. * CondKD improves teacher-feature predictability and reduces local residual dispersion across neighborhood sizes. These results indicate that its student representation preserves more readily recoverable teacher information and is consistent with reduced effective conditional ambiguity.