acceptodds
Under review as a conference paper at ICLR 2027

Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring

Abstract

Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment. Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitigate this bottleneck, we propose Dyna-DINO: Efficient ViT Distillation Via Adaptive Representation Anchoring, a training curriculum for ViT feature-based knowledge distillation. By utilizing the teacher’s intermediate feature maps as a sequence of progressively more difficult targets, our curriculum allows the student to build a foundational representation before tackling higher-level abstractions. Our results demonstrate that this paradigm significantly accelerates convergence through adaptive difficulty selection across various student model sizes and dataset scales. With our curriculum, the Dyna-DINO distilled ViT-S achieves accuracy on ImageNet-100, a improvement compared with baseline. On ImageNet-1K, Dyna-DINO achieves and improvement for the instance retrieval task on the Oxford and Paris datasets, improvements on semantic segmentation task, as well as meaningful performance gain on classification task. Furthermore, the curriculum enables savings in training FLOPs and savings in training time on ImageNet-100 by implementing early-stopping for teacher inference during the initial stages of training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Vision Foundation Model, Knowledge Distillation, Efficient Training, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M7XWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Dyna-DINO distills a large DINOv2 teacher into a smaller ViT using a shallow-to-deep sequence of teacher targets. A CKA-based rule controls when the student moves to the next teacher block, with a patience fallback in the algorithm. Using shallower targets also allows the teacher forward pass to stop early. The paper evaluates classification, instance retrieval, segmentation, and training efficiency on ImageNet-100 and ImageNet-1K.

Critical strengths

Using intermediate teacher targets may be a practical way to reduce teacher computation during early training. The paper reports improvements over fixed-anchor distillation and evaluates the resulting representations on classification, retrieval, and segmentation. This broader task coverage is useful, although the baseline set remains too limited.

Critical weaknesses
  1. The novelty and new insight are not clear enough. Intermediate teacher supervision and progressive distillation are established ideas. TinyMIM already reported that an intermediate teacher layer can be a better target when teacher and student depths differ. The potential contribution here is the CKA-controlled progression. However, the paper does not compare it with a simple non-adaptive shallow-to-deep schedule. Appendix A.1 compares against individual fixed layers, which does not answer this question. Without this control, it is hard to tell what the adaptive rule adds or what new lesson the paper establishes.

  2. The comparisons are seriously insufficient for VFM distillation. The main tables rely on fixed-anchor variants and ViTKD. A closely related method such as Proteus, which distills DINOv2 using ImageNet-1K, is missing. The paper needs stronger VFM distillation baselines under matched teachers, students, data, and training budgets. The ViTKD result also needs more checking: Table 6 reports only 12.09% CLS linear-probe accuracy, compared with 77.63% for the fixed-anchor ViT-S baseline. Such a large failure calls for an implementation and tuning check, beyond attributing it to the capacity gap. Beating this result does not establish competitiveness with existing VFM distillation methods.

  3. The method description is difficult to follow and internally inconsistent. Section 3.2 says the target advances if and only if CKA reaches the threshold. Algorithm 1 also advances when the patience limit is reached. These are different rules. The main algorithm supervises the final student feature, while the ImageNet-1K results use a configuration called Dyna with three anchors without clearly explaining how its anchors and progression are coordinated. Please give one complete training recipe, including the CKA calculation, transition rule, patience setting, and multi-anchor configuration. These details are central to the method.

  4. The figures do not explain the method clearly enough. Figure 1 has small labels and repeated block diagrams, but does not clearly show how the actual transition rule and three-anchor setup work. Figure 3 also uses small text, making the convergence curves and curriculum progression harder to inspect at normal page size. Please enlarge the labels and simplify the diagrams. The figures should make the method easier to understand, especially given the ambiguities in the text. My main concerns remain the limited contribution and weak comparisons.

Questions for authors
  1. How does CKA-triggered progression compare with a fixed shallow-to-deep schedule under the same training budget? This is needed to assess the contribution of the adaptive rule.

  2. Can you compare with recent VFM distillation methods such as Proteus under a matched protocol? Please also verify the implementation and tuning behind the very low ImageNet-1K ViTKD result.

  3. What exact transition rule and multi-anchor setup produced the main results? Please reconcile Section 3.2 with Algorithm 1 and define the three-anchor configuration explicitly.

4 comments

We thank Reviewer M7XW for recognizing the practicality of our method, as well as for providing constructive feedback. We address each weakness and question below:


1. Novelty & Fixed-Schedule Comparison

While prior work explores intermediate teacher supervision and progressive training, our core contribution is demonstrating that progressively deeper teacher feature maps form an effective training curriculum—non-trivially motivated by the shifting geometric similarity between student and teacher representations throughout training. Furthermore, the CKA-based progression rule is essential: a central challenge in curriculum learning is controlling the curriculum pace. If the curriculum advances too quickly, the subsequent target becomes too difficult and the curriculum loses its purpose; conversely, an overly slow schedule stagnates learning. Because the optimal pace varies across datasets and model architectures, online CKA similarity enables the student to adaptively learn at its own pace by directly measuring feature-map geometric alignment, avoiding manual schedule cross-validation.

To verify this directly, we conducted an additional experiment using a ViT-G teacher and a ViT-S student under a fixed shallow-to-deep schedule, where the student trains for exactly 3 epochs at each teacher layer sequentially from layer 1 to layer 40 (120 epochs total). We compare the results below:

MethodLinear Probing (%)Oxford5k (E / M / H)Paris6k (E / M / H)
Fixed Schedule60.406.52 / 5.78 / 1.5825.31 / 19.36 / 7.73
Fixed Anchor77.8610.28 / 8.80 / 2.1524.83 / 21.29 / 6.63
Dyna-DINO90.1022.56 / 17.30 / 4.8153.84 / 40.99 / 15.97

We will include this ablation study in the updated manuscript.


2. Proteus Baseline Comparison & ViTKD Validation

We evaluated Proteus using its official codebase under same protocol: ViT-G teacher with ViT-S student for 100 epochs on ImageNet-1K. Result is as below:

MethodLinear Probing (Patch / CLS %)Oxford5k (E / M / H)Paris6k (E / M / H)
Proteus74.47 / 76.3817.04 / 14.52 / 3.5151.92 / 41.79 / 20.76
Dyna-DINO77.03 / 78.1034.12 / 24.31 / 4.8668.04 / 55.78 / 33.54

Under matched compute and training duration, Proteus underperforms Dyna-DINO across both classification and retrieval. We attribute Dyna-DINO's faster convergence to its layer-skipping curriculum: by sequentially matching target layers of increasing difficulty, the curriculum smooths the optimization landscape and accelerates convergence, whereas Proteus requires 300 epochs in its original setting to reach peak performance.

Regarding ViTKD, we tuned the hyperparameters and faithfully followed the official codebase, yet observed low performance on ImageNet-1K. We attribute this to the severe capacity and architectural gap between teacher and student in our benchmark. In the original ViTKD study, experiments primarily pair teachers and students of matching depth (e.g., 12-layer ViT-Base with 768-dim to 12-layer ViT-Small with 384-dim), with at most a 24-layer ViT-Large (1024-dim) to 12-layer ViT-Base setup. In contrast, our ViT-G teacher has 40 layers and a 1536-dimensional embedding space, creating a substantially wider capacity gap relative to ViT-S (12 layers, 384-dim) that ViTKD's direct feature-generation/mimicking mechanism struggles to bridge. This underscores a key advantage of Dyna-DINO: even under extreme teacher–student capacity gaps, Dyna-DINO dynamically adapts its target layer so the student always distills from an appropriate intermediate representation.


3. Complete Training Recipe & Multi-Anchor Clarification

Section 3.2 provides a high-level overview of the curriculum, whereas Algorithm 1 presents the exact implementation procedure. We will reconcile and provide the complete unified training recipe in the revision:

  • CKA evaluation frequency: CKA is computed at every training batch using a small batch of feature maps already generated during the forward pass, adding negligible overhead (<0.5%< 0.5\% of total training GFLOPs).
  • Transition rule & patience limit: A student anchor advances to the next teacher layer whenever its online CKA similarity reaches the CKA threshold, or if it reaches a maximum limit (patience) of 5 epochs on the current target layer (preventing stagnation on complex deep layers).
  • Three-anchor configuration: In the three-anchor setup, the student attaches two intermediate projectors (at layers 1 and 6) alongside the final layer projector (layer 12). All three anchors follow the same progressive rule independently: each begins by teacher layer 1 and advances to deeper teacher layers whenever its respective online CKA score reaches the shared threshold or hits the patience limit.

4. Figure Clarity

In the revised manuscript, we will enlarge all text labels and simplify the block diagrams.

We would like to provide a brief follow-up regarding the ViTKD baseline and additional comparisons on ImageNet-1K. After further tuning the training hyperparameters for ViTKD, we eventually identified a configuration that yields substantially stronger performance. Finding these settings required significant compute and time because the default hyperparameters from the official codebase do not transfer well to our challenging large teacher–student capacity gap (40-layer ViT-G to 12-layer ViT-S) at ImageNet-1K scale. We report the updated comparison below:

MethodLinear Probing (CLS) (%)Linear Probing (Patch) (%)
Dyna-DINO78.1077.03
Proteus76.3874.47
ViTKD (tuned)74.7476.59

These updated results and the additional VFM distillation baseline further demonstrate the effectiveness of Dyna-DINO: under a large teacher–student capacity gap, an adaptive curriculum achieves stronger convergence than fixed-anchor and feature-generation baselines. We will update Table 6 and the corresponding discussion in the revised manuscript accordingly.

I thank the authors for their rebuttal and additional experiments. However, my main concerns remain unresolved. The novelty of the proposed method is still limited, and the additional fixed-schedule comparison does not convincingly establish the necessity of CKA-based adaptive progression, especially given the unusually weak baseline performance. The Proteus comparison under a restricted training budget and the unexplained poor ViTKD results also raise concerns about experimental fairness and reliability. Furthermore, the manuscript still suffers from substantial issues in writing, presentation, and technical clarity. Overall, I do not believe the rebuttal sufficiently addresses the fundamental weaknesses of the paper.

I therefore maintain my original rating of 1 (Clear Rejection).

We thank Reviewer M7XW for the follow-up. We would be glad to provide further clarification if the reviewer could specify the remaining concerns in more detail:


1. Fixed-Schedule Comparison and "Unusually Weak Baseline Performance" We conducted the exact fixed-schedule ablation requested by the reviewer and faithfully reported the resulting performance. The poor performance of the fixed schedule directly demonstrates why a static progression is suboptimal: without adaptive pacing, shifting targets too early or too late destabilizes representation alignment, whereas CKA-based progression adjusts automatically to the student's learning pace. Could the reviewer clarify what is meant by "unusually weak baseline performance" (e.g., whether this refers to the fixed schedule or another baseline) and what specific experimental setup would resolve this concern?


2. Proteus Training Budget and Updated ViTKD Results For Proteus, we used the official implementation on ImageNet-1K under the exact same compute and epoch budget as Dyna-DINO. Because both methods optimize similar feature-distillation objectives, evaluating them under a matched compute budget provides a fair comparison of convergence efficiency rather than penalizing one method with fewer updates.

Regarding ViTKD, please note that in our follow-up comment above, we reported updated ImageNet-1K results after additional hyperparameter tuning (74.74% CLS / 76.59% Patch linear probing, compared to 78.10% / 77.03% for Dyna-DINO), along with the explanation that the default hyperparameters in the official ViTKD codebase were designed for smaller teacher–student capacity gaps and required substantial compute to retune at ImageNet-1K scale. Could the reviewer clarify whether the remark on "unexplained poor ViTKD results" refers to the original Table 6 numbers prior to our follow-up tuning, or if further clarification on the tuned ViTKD setup is needed?


We thank the reviewer again for their time and feedback, and we remain available throughout the rest of the discussion period to address any specific questions.

Reviewer_he4COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Dyna-DINO introduces a progressive feature-distillation curriculum in which a student initially matches shallow teacher representations and advances toward deeper representations based on CKA similarity. Early termination of teacher computation reduces training overhead. The method is evaluated on classification, retrieval, and semantic segmentation.

Critical strengths
  1. The method is conceptually simple and has a clear mechanism for reducing teacher-side computation.
  2. Evaluation across several downstream tasks provides evidence that the learned representations are not exclusively optimized for classification.
  3. The fixed-intermediate-layer experiments and dense layer-alignment comparison are useful controls.
  4. The reported ImageNet-100 improvements are substantial and accompanied by reduced training time.
Critical weaknesses
  1. The central benefit of adaptive progression is not sufficiently isolated. Comparing against fixed teacher layers establishes the usefulness of changing supervision targets, but does not establish the necessity of CKA-based adaptation. A predetermined shallow-to-deep schedule, particularly one matched to the observed layer dwell times, would be a critical baseline.
  2. The algorithm description is inconsistent. Section 3.2 states that progression occurs if and only if the CKA threshold is reached, whereas Algorithm 1 also advances after a patience limit. The practical values and implementation of this limit, the frequency of CKA evaluation, and the progression mechanism for the three-anchor configuration require clarification.
  3. The large-scale results are more modest than the headline results suggest. On ImageNet-1K, Table 6 shows CLS-token classification improvements of 0.47 percentage points for ViT-S and 0.22 points for ViT-Tiny over the single-anchor baseline. Reported time savings are 4.83% and 7.5%, respectively. These are useful results, but their reliability and practical significance would be clearer with repeated runs and consistent end-to-end efficiency accounting.
  4. The ViTKD baseline requires stronger validation. Its extremely low ImageNet-1K classification and segmentation results raise questions about optimization and adaptation to the present setting. The explanation based on the teacher–student capacity gap is plausible, but does not replace a systematic implementation and hyperparameter check.
  5. Some numerical claims are inconsistent with the tables. In Table 4, the ViT-S Paris mean improvement over the single-anchor baseline is 5.46 points, rather than the reported 6.09. The three Paris scores for Dyna-T average to approximately 38.79, whereas the reported mean is 35.94.
Questions for authors
  1. Responses to weaknesses.

  2. And also,

  • Does CKA-based adaptation outperform a fixed progressive curriculum under matched compute?
  • What are the patience values, CKA evaluation frequency, and exact three-anchor update rules?
  • Can the authors validate the ViTKD implementation and correct the inconsistencies in Table 4?
4 comments

We thank Reviewer he4C for recognizing the motivation and simplicity of our approach, as well as the effectiveness of our curriculum training method across classification, segmentation, and image retrieval tasks and the usefulness of the dense-layer matching ablation experiments. We address each weakness and question below:


1. The effectiveness of CKA adaptation: We thank the reviewer for raising this practical question. To evaluate this directly, we conducted an additional experiment using a ViT-G teacher and a ViT-S student under a fixed shallow-to-deep schedule. The student stays for exactly 3 epochs at each teacher layer, progressing sequentially from layer 1 to layer 40 with 120 epochs in total. We report the result below:

MethodLinear Probing (%)Oxford5k (E / M / H)Paris6k (E / M / H)
Fixed Schedule60.406.52 / 5.78 / 1.5825.31 / 19.36 / 7.73
Dyna90.1022.56 / 17.30 / 4.8153.84 / 40.99 / 15.97
Fixed Anchor77.8610.28 / 8.80 / 2.1524.83 / 21.29 / 6.63

A central challenge in designing a curriculum learning strategy is controlling the curriculum pace. If the curriculum progresses too quickly, the subsequent target becomes too difficult and the curriculum loses its purpose; conversely, an overly slow schedule stagnates the learning process. Furthermore, the optimal pace varies across different datasets and model architectures. This motivates our use of online CKA similarity for pace control: without explicitly cross-validating the schedule, the student adaptively learns at its own pace simply by measuring alignment in feature map geometry.

We thank reviewer again for the comment and will add this ablation result to the updated manuscript.


2. Algorithm Clarification: We thank the reviewer for pointing out this discrepancy. Section 3.2 provides a high-level overview of the curriculum, while Algorithm 1 presents the implementation-level procedure. We will align the wording in the revision. More specifically:

  • Patience limit: In some cases, the student model may stall at a deep, complex teacher layer. To prevent stagnation, we enforce a maximum epoch limit per layer. We use a patience value of 5 epochs across all experiments; our cross-validation over patience values showed that 3–7 epochs works reliably for a 100-epoch training budget.
  • CKA frequency: CKA is computed at every training batch. Because it simply reuses a small batch of feature maps already generated during the training forward pass, it adds negligible overhead (<0.5%< 0.5\% of total training GFLOPs).
  • Three-anchor update: The three-anchor setup evaluates Dyna-DINO when the student attaches two additional projectors (at layers 1 and 6) alongside the final layer projector. All three student anchors follow the same update rule as the single-anchor setup: each starts by targeting the first teacher layer and independently advances to deeper teacher layers whenever its respective online CKA score reaches the shared CKA threshold.

3. Variability on ImageNet-1K: We thank the reviewer for highlighting this point. We conducted 2 additional runs on ImageNet-1K for both ViT-Tiny and ViT-S across distinct seeds, reporting CLS linear probing accuracy and end-to-end training time savings:

ModelRunLinear Probing (%)Time Saving (%)
ViT-TinySeed 164.707.53
ViT-TinySeed 264.557.51
ViT-SSeed 178.124.90
ViT-SSeed 278.094.85

These results demonstrate that while minor run-to-run variability exists, Dyna-DINO consistently outperforms the baseline while maintaining stable end-to-end training time reductions (which include all student updates and CKA calculations).


4. ViTKD Baseline Validation and Table 4 Corrections: We appreciate the reviewer catching the numerical discrepancies in Table 4. We will correct the reported mean values in Table 4 in the updated manuscript.

Regarding ViTKD, we tuned the hyperparameters and faithfully followed the official codebase, yet observed suboptimal performance on ImageNet-1K. We attribute this to the severe capacity and architectural gap between the teacher and student in our benchmark. In the original ViTKD study, experiments primarily pair teachers and students with matching depth (e.g., 12-layer ViT-Base with 768-dim to 12-layer ViT-Small with 384-dim), with at most a 24-layer ViT-Large (1024-dim) to 12-layer ViT-Base setup. In contrast, our ViT-G teacher has 40 layers and a 1536-dimensional embedding space, which create a substantially capacity gap relative to ViT-S (12 layers, 384-dim) that ViTKD's direct feature-generation/mimicking mechanism struggles to bridge. This highlights a core advantage of Dyna-DINO: even under extreme teacher–student capacity gaps, Dyna-DINO dynamically adapts its target layer so the student always distills from an appropriate intermediate representation.

We would like to provide a brief follow-up regarding the ViTKD baseline and additional comparisons on ImageNet-1K. After further tuning the training hyperparameters for ViTKD, we identified a configuration that yields substantially stronger performance. Finding these settings required significant compute and time because the default hyperparameters from the official codebase do not transfer well to our challenging large teacher–student capacity gap (40-layer ViT-G to 12-layer ViT-S) at ImageNet-1K scale.

In addition, we include Proteus (Accessing Vision Foundation Models via ImageNet-1K) as an additional baseline trained under the exact same protocol (ViT-G teacher, ViT-S student, 100 epochs on ImageNet-1K). We report the updated comparison below:

MethodLinear Probing (CLS) (%)Linear Probing (Patch) (%)
Dyna-DINO78.1077.03
Proteus76.3874.47
ViTKD (tuned)74.7476.59

These updated results and the additional VFM distillation baseline further demonstrate the effectiveness of Dyna-DINO: under a large teacher–student capacity gap, an adaptive curriculum achieves stronger convergence than fixed-anchor and feature-generation baselines. We will update Table 6 and the corresponding discussion in the revised manuscript accordingly.

Reviewer_he4COpenReview ↗

Responses to rebuttal

There has been substantial progress in rebuttal, especially in the re-tuning of ViTKD, the supplementation of Proteus, and multi-seed experiments; however, the core issue of "whether CKA adaptation is necessary" has not been fully isolated.

The determining factor should be whether the missing table provides evidence:

  • CKA has benefits from relatively reasonable and budget-matched fixed courses;
  • Large-scale improvements are reliable under multiple seeds;
  • The updated strong baseline has not eliminated the main advantage.

We thank the reviewer for follow up. We believe that our response has already included the determining factor listed by the reviewer:

1. CKA has benefits from relatively reasonable and budget-matched fixed courses: we allow the fixed courses cka free baseline to train with more budget (120 epochs vs 100 epochs for Dyna) for the best performance, and it underperforms our method. We believe the CKA is benefiting our Dyna-DINO method demonstrated by the performance gain.

2. Large-scale improvements are reliable under multiple seeds: we have reported the IN-1K experiments across multiple seeds and the performance improvement is consistent. We consider this as an evidence that the improvements are reliable.

3. The updated strong baseline has not eliminated the main advantage: we refine the updated baseline with the best-performing set of parameter, while it's performance still below our method, so the main advantage stays with our method.

We would like the reviewer to clarify however, the core issue of "whether CKA adaptation is necessary" has not been fully isolated. and what additional experiments would be helpful to isolated the effect of CKA, and we will stay active for the rest of rebuttal.

Reviewer_UQBbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Dyna-DINO progressively shifts student supervision from shallower to deeper teacher features during distillation, using online CKA similarity to determine transitions. Stopping teacher computation at the current target can reduce teacher cost. Evaluation covers image classification, instance retrieval, and semantic segmentation.

Critical strengths
  1. I like the simplicity of the progressive supervision strategy. In Table 7, ViT-S to ViT-Tiny distillation reaches 83.36% versus 83.38% for dense alignment, using one projector rather than twelve. This gives a concrete reason to consider the curriculum: effective supervision need not match every layer simultaneously.

  2. The segmentation gains in Table 5 and Paris retrieval gains in Table 4 make the transfer results worth taking seriously. I also find Section A.1 useful: outperforming every static teacher layer in that setting helps rule out the explanation that the method merely picked a particularly favorable intermediate target.

Critical weaknesses
  1. I am not yet convinced that online CKA adaptation adds to progressive supervision itself. The static-layer study in Section A.1 and dense-alignment comparison in Table 7 do not answer whether a fixed shallow-to-deep schedule would work equally well. A comparison under the same optimization settings and a comparable teacher budget would directly test the contribution claimed in Section 3.2.

  2. In Table 6, ViT-Tiny gains only 0.22 percentage points on CLS probing and 0.02 on patch probing. The Oxford mean gain in Table 4 is 0.50 mAP points against the stronger three-anchor baseline. I would like to see variability across runs before treating these small differences as reliable gains at ImageNet-1K scale.

  3. I find the efficiency claim difficult to assess because Tables 1 and 6 label the savings as teacher overhead, whereas Section 4.2 refers to total training time and training FLOPs. Full training cost, including student updates and CKA scheduling, is needed to judge how much efficiency is actually gained at a given accuracy.

Questions for authors
  1. How does the online rule compare with a fixed shallow-to-deep schedule using the same supervision, optimizer, training duration, and a comparable teacher budget?

  2. Are the small ImageNet-1K differences, especially ViT-Tiny probing and Oxford retrieval, consistent across seeds?

  3. What is the full-training accuracy-cost tradeoff under matched measurement conditions, including student updates and CKA scheduling, and which reported savings reflect teacher computation only?

4 comments

We thank Reviewer UQBb for their constructive comments, as well as their appreciation for the simplicity of our progressive training strategy and the effectiveness of our downstream transfer results and training curriculum. We address each weakness and question below:


1. The effectiveness of CKA adaptation: We thank the reviewer for raising this practical question. To evaluate this directly, we conducted an additional experiment using a ViT-G teacher and a ViT-S student under a fixed shallow-to-deep schedule. Specifically, the student stays for exactly 3 epochs at each teacher layer, progressing sequentially from layer 1 to layer 40 with 120 epochs in total. We report the linear probing accuracy and image retrieval performance on ROxford5k and RParis6k below:

Schedule MethodLinear Probing (%)ROxford5k (E / M / H)RParis6k (E / M / H)
Fixed Schedule (3 epochs/layer)60.406.52 / 5.78 / 1.5825.31 / 19.36 / 7.73
Dyna90.1022.56 / 17.30 / 4.8153.84 / 40.99 / 15.97
Fixed Anchor77.8610.28 / 8.80 / 2.1524.83 / 21.29 / 6.63

A central challenge in designing a curriculum learning strategy is controlling the curriculum pace. If the curriculum progresses too quickly, the subsequent target becomes too difficult and the curriculum loses its purpose; conversely, an overly slow schedule stagnates the learning process. Furthermore, the optimal pace varies across different datasets and model architectures. This motivates our use of online CKA similarity for pace control: without explicitly cross-validating the schedule, the student adaptively learns at its own pace simply by measuring alignment in feature map geometry.

We thank reviewer again for the comment and will add this ablation result to the updated manuscript.


2. Result Consistency Across Seeds: We thank the reviewer for pointing this out. Due to compute constraints, we conducted 2 additional runs of Dyna-DINO using a ViT-G teacher with ViT-S and ViT-Tiny students across 2 distinct random seeds. We report the ROxford5k retrieval performance and linear probing accuracy below:

RunROxford5k mAP (E / M / H)Linear Probing (Patch / CLS %)
Fixed Anchor33.85 / 23.31 / 4.6463.35 / 64.4
Seed 134.50 / 24.43 / 4.9063.39 / 64.70
Seed 234.22 / 24.25 / 4.8263.36 / 64.55

These results demonstrate that while minor run-to-run variability exists, our method consistently outperforms the baseline across both linear probing classification and instance retrieval.


3. Clarification for the efficiency: We apologize for the confusion caused by the table labels. We would like to clarify that all savings reported in Tables 1 and 6 refer to total training compute savings (both total training time and total training FLOPs), which fully include student updates and CKA scheduling overhead. Because the online CKA calculation simply reuses a small batch of feature maps already generated during the training forward pass, it accounts for less than 0.5% of the total training FLOPs. We will revise the table headers and captions in the updated manuscript to make this explicit.

We would like to provide a brief follow-up regarding the ViTKD baseline and additional comparisons on ImageNet-1K. After further tuning the training hyperparameters for ViTKD, we identified a configuration that yields substantially stronger performance. Finding these settings required significant compute and time because the default hyperparameters from the official codebase do not transfer well to our challenging large teacher–student capacity gap (40-layer ViT-G to 12-layer ViT-S) at ImageNet-1K scale.

In addition, we include Proteus (Accessing Vision Foundation Models via ImageNet-1K) as an additional baseline trained under the exact same protocol (ViT-G teacher, ViT-S student, 100 epochs on ImageNet-1K). We report the updated comparison below:

MethodLinear Probing (CLS) (%)Linear Probing (Patch) (%)
Dyna-DINO78.1077.03
Proteus76.3874.47
ViTKD (tuned)74.7476.59

These updated results and the additional VFM distillation baseline further demonstrate the effectiveness of Dyna-DINO: under a large teacher–student capacity gap, an adaptive curriculum achieves stronger convergence than fixed-anchor and feature-generation baselines. We will update Table 6 and the corresponding discussion in the revised manuscript accordingly.

Thanks for running the additional experiments and rebuttal. The new experiments are useful.

Could you clarify whether the fixed schedule used comparable teacher FLOPs and the same optimization settings? I’m keeping my original rating for now because the benefit of CKA at a matched budget is still unclear.

We thank the reviewer for the follow up!

We would like to clarify that the fixed schedule baseline uses the exact same optimization setting, and we even give more training epochs and compute budget to the fixed schedule method (120 epochs, while 100 epochs for Dyna, due to the fact that ViT-G teacher has 40 layers and we allocated 3 epochs for each layer) to allow for best performing baseline. We would like to point out that CKA threshold control is one of the major contributions of this project: fixed schedule baseline might be able to achieve good performance, but it would be extremely costly (and sometimes impossible to tune) since it requires significant amount of tuning on the curriculum schedule, which will not generalize to a different model architecture / training dataset. With CKA scheduling, it would be much more elegant and one CKA threshold parameter is able to produce performing curriculum, and it generalize across model size / dataset scales.

We would be happy to conduct any additional experiments for any further concerns and questions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remark

Dear Reviewers and AC,

We appreciate the active engagement of the reviewers, as well as the effort from the AC in overseeing the rebuttal process. We would like to provide a brief summary of our clarifications and supplemental experiments conducted during the rebuttal:

  1. Effectiveness of CKA scheduling: In response to all three reviewers, we conducted a fixed-schedule progressive curriculum ablation to isolate the contribution of CKA-based adaptation. Using the exact same optimization settings and training configuration—while allocating a larger compute budget to the fixed schedule (120 epochs vs. 100 epochs for Dyna-DINO)—Dyna-DINO outperforms the fixed schedule by nearly 30% in linear probing (90.10% vs. 60.40%), demonstrating the necessity and effectiveness of adaptive curriculum pacing via online CKA similarity.

  2. Performance consistency across multiple seeds: To address the concern raised by Reviewers UQBb and he4C, we ran additional ImageNet-1K experiments across multiple random seeds for both ViT-Tiny and ViT-S. The results confirm that Dyna-DINO consistently outperforms the baseline across seeds while maintaining stable end-to-end training time savings.

  3. ViTKD baseline retuning: Reviewers M7XW and he4C noted the low initial performance of the ViTKD baseline. After extensive hyperparameter tuning tailored to our large teacher–student capacity gap (40-layer ViT-G to 12-layer ViT-S) at ImageNet-1K scale, we identified a strong configuration for ViTKD (74.74% CLS / 76.59% Patch linear probing); however, it still underperforms Dyna-DINO (78.10% / 77.03%).

  4. Proteus baseline comparison: As requested by Reviewer M7XW, we trained Proteus using its official implementation on ImageNet-1K under a matched compute budget (100 epochs). Dyna-DINO outperforms Proteus across both classification (78.10% vs. 76.38% CLS; 77.03% vs. 74.47% Patch) and instance retrieval benchmarks.

We have uploaded a revised version of the manuscript incorporating the updated ViTKD results. We thank the reviewers and AC again for their time and constructive feedback.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.