- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Dyna-DINO distills a large DINOv2 teacher into a smaller ViT using a shallow-to-deep sequence of teacher targets. A CKA-based rule controls when the student moves to the next teacher block, with a patience fallback in the algorithm. Using shallower targets also allows the teacher forward pass to stop early. The paper evaluates classification, instance retrieval, segmentation, and training efficiency on ImageNet-100 and ImageNet-1K.
Critical strengths
Using intermediate teacher targets may be a practical way to reduce teacher computation during early training. The paper reports improvements over fixed-anchor distillation and evaluates the resulting representations on classification, retrieval, and segmentation. This broader task coverage is useful, although the baseline set remains too limited.
Critical weaknesses
-
The novelty and new insight are not clear enough. Intermediate teacher supervision and progressive distillation are established ideas. TinyMIM already reported that an intermediate teacher layer can be a better target when teacher and student depths differ. The potential contribution here is the CKA-controlled progression. However, the paper does not compare it with a simple non-adaptive shallow-to-deep schedule. Appendix A.1 compares against individual fixed layers, which does not answer this question. Without this control, it is hard to tell what the adaptive rule adds or what new lesson the paper establishes.
-
The comparisons are seriously insufficient for VFM distillation. The main tables rely on fixed-anchor variants and ViTKD. A closely related method such as Proteus, which distills DINOv2 using ImageNet-1K, is missing. The paper needs stronger VFM distillation baselines under matched teachers, students, data, and training budgets. The ViTKD result also needs more checking: Table 6 reports only 12.09% CLS linear-probe accuracy, compared with 77.63% for the fixed-anchor ViT-S baseline. Such a large failure calls for an implementation and tuning check, beyond attributing it to the capacity gap. Beating this result does not establish competitiveness with existing VFM distillation methods.
-
The method description is difficult to follow and internally inconsistent. Section 3.2 says the target advances if and only if CKA reaches the threshold. Algorithm 1 also advances when the patience limit is reached. These are different rules. The main algorithm supervises the final student feature, while the ImageNet-1K results use a configuration called Dyna with three anchors without clearly explaining how its anchors and progression are coordinated. Please give one complete training recipe, including the CKA calculation, transition rule, patience setting, and multi-anchor configuration. These details are central to the method.
-
The figures do not explain the method clearly enough. Figure 1 has small labels and repeated block diagrams, but does not clearly show how the actual transition rule and three-anchor setup work. Figure 3 also uses small text, making the convergence curves and curriculum progression harder to inspect at normal page size. Please enlarge the labels and simplify the diagrams. The figures should make the method easier to understand, especially given the ambiguities in the text. My main concerns remain the limited contribution and weak comparisons.
Questions for authors
-
How does CKA-triggered progression compare with a fixed shallow-to-deep schedule under the same training budget? This is needed to assess the contribution of the adaptive rule.
-
Can you compare with recent VFM distillation methods such as Proteus under a matched protocol? Please also verify the implementation and tuning behind the very low ImageNet-1K ViTKD result.
-
What exact transition rule and multi-anchor setup produced the main results? Please reconcile Section 3.2 with Algorithm 1 and define the three-anchor configuration explicitly.
4 comments
We would like to provide a brief follow-up regarding the ViTKD baseline and additional comparisons on ImageNet-1K. After further tuning the training hyperparameters for ViTKD, we eventually identified a configuration that yields substantially stronger performance. Finding these settings required significant compute and time because the default hyperparameters from the official codebase do not transfer well to our challenging large teacher–student capacity gap (40-layer ViT-G to 12-layer ViT-S) at ImageNet-1K scale. We report the updated comparison below:
| Method | Linear Probing (CLS) (%) | Linear Probing (Patch) (%) |
|---|---|---|
| Dyna-DINO | 78.10 | 77.03 |
| Proteus | 76.38 | 74.47 |
| ViTKD (tuned) | 74.74 | 76.59 |
These updated results and the additional VFM distillation baseline further demonstrate the effectiveness of Dyna-DINO: under a large teacher–student capacity gap, an adaptive curriculum achieves stronger convergence than fixed-anchor and feature-generation baselines. We will update Table 6 and the corresponding discussion in the revised manuscript accordingly.
I thank the authors for their rebuttal and additional experiments. However, my main concerns remain unresolved. The novelty of the proposed method is still limited, and the additional fixed-schedule comparison does not convincingly establish the necessity of CKA-based adaptive progression, especially given the unusually weak baseline performance. The Proteus comparison under a restricted training budget and the unexplained poor ViTKD results also raise concerns about experimental fairness and reliability. Furthermore, the manuscript still suffers from substantial issues in writing, presentation, and technical clarity. Overall, I do not believe the rebuttal sufficiently addresses the fundamental weaknesses of the paper.
I therefore maintain my original rating of 1 (Clear Rejection).
We thank Reviewer M7XW for the follow-up. We would be glad to provide further clarification if the reviewer could specify the remaining concerns in more detail:
1. Fixed-Schedule Comparison and "Unusually Weak Baseline Performance" We conducted the exact fixed-schedule ablation requested by the reviewer and faithfully reported the resulting performance. The poor performance of the fixed schedule directly demonstrates why a static progression is suboptimal: without adaptive pacing, shifting targets too early or too late destabilizes representation alignment, whereas CKA-based progression adjusts automatically to the student's learning pace. Could the reviewer clarify what is meant by "unusually weak baseline performance" (e.g., whether this refers to the fixed schedule or another baseline) and what specific experimental setup would resolve this concern?
2. Proteus Training Budget and Updated ViTKD Results For Proteus, we used the official implementation on ImageNet-1K under the exact same compute and epoch budget as Dyna-DINO. Because both methods optimize similar feature-distillation objectives, evaluating them under a matched compute budget provides a fair comparison of convergence efficiency rather than penalizing one method with fewer updates.
Regarding ViTKD, please note that in our follow-up comment above, we reported updated ImageNet-1K results after additional hyperparameter tuning (74.74% CLS / 76.59% Patch linear probing, compared to 78.10% / 77.03% for Dyna-DINO), along with the explanation that the default hyperparameters in the official ViTKD codebase were designed for smaller teacher–student capacity gaps and required substantial compute to retune at ImageNet-1K scale. Could the reviewer clarify whether the remark on "unexplained poor ViTKD results" refers to the original Table 6 numbers prior to our follow-up tuning, or if further clarification on the tuned ViTKD setup is needed?
We thank the reviewer again for their time and feedback, and we remain available throughout the rest of the discussion period to address any specific questions.
We thank Reviewer M7XW for recognizing the practicality of our method, as well as for providing constructive feedback. We address each weakness and question below:
1. Novelty & Fixed-Schedule Comparison
While prior work explores intermediate teacher supervision and progressive training, our core contribution is demonstrating that progressively deeper teacher feature maps form an effective training curriculum—non-trivially motivated by the shifting geometric similarity between student and teacher representations throughout training. Furthermore, the CKA-based progression rule is essential: a central challenge in curriculum learning is controlling the curriculum pace. If the curriculum advances too quickly, the subsequent target becomes too difficult and the curriculum loses its purpose; conversely, an overly slow schedule stagnates learning. Because the optimal pace varies across datasets and model architectures, online CKA similarity enables the student to adaptively learn at its own pace by directly measuring feature-map geometric alignment, avoiding manual schedule cross-validation.
To verify this directly, we conducted an additional experiment using a ViT-G teacher and a ViT-S student under a fixed shallow-to-deep schedule, where the student trains for exactly 3 epochs at each teacher layer sequentially from layer 1 to layer 40 (120 epochs total). We compare the results below:
We will include this ablation study in the updated manuscript.
2. Proteus Baseline Comparison & ViTKD Validation
We evaluated Proteus using its official codebase under same protocol: ViT-G teacher with ViT-S student for 100 epochs on ImageNet-1K. Result is as below:
Under matched compute and training duration, Proteus underperforms Dyna-DINO across both classification and retrieval. We attribute Dyna-DINO's faster convergence to its layer-skipping curriculum: by sequentially matching target layers of increasing difficulty, the curriculum smooths the optimization landscape and accelerates convergence, whereas Proteus requires 300 epochs in its original setting to reach peak performance.
Regarding ViTKD, we tuned the hyperparameters and faithfully followed the official codebase, yet observed low performance on ImageNet-1K. We attribute this to the severe capacity and architectural gap between teacher and student in our benchmark. In the original ViTKD study, experiments primarily pair teachers and students of matching depth (e.g., 12-layer ViT-Base with 768-dim to 12-layer ViT-Small with 384-dim), with at most a 24-layer ViT-Large (1024-dim) to 12-layer ViT-Base setup. In contrast, our ViT-G teacher has 40 layers and a 1536-dimensional embedding space, creating a substantially wider capacity gap relative to ViT-S (12 layers, 384-dim) that ViTKD's direct feature-generation/mimicking mechanism struggles to bridge. This underscores a key advantage of Dyna-DINO: even under extreme teacher–student capacity gaps, Dyna-DINO dynamically adapts its target layer so the student always distills from an appropriate intermediate representation.
3. Complete Training Recipe & Multi-Anchor Clarification
Section 3.2 provides a high-level overview of the curriculum, whereas Algorithm 1 presents the exact implementation procedure. We will reconcile and provide the complete unified training recipe in the revision:
4. Figure Clarity
In the revised manuscript, we will enlarge all text labels and simplify the block diagrams.