acceptodds
Under review as a conference paper at ICLR 2027

Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers

Abstract

Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9–24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized attention in MAE and earlier local-to-global attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, vision transformers, progressive depth growth, efficient training, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1SvAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies existing progressive depth-growth strategies (Stacking, MIDAS, and LIDAS) for MAE and DINOv3 pretraining. At matched training steps and final architectures, selected configurations improve frozen-feature classification, segmentation, and depth estimation while reducing FLOPs by 13.9–24.8%. Layer-wise probes, attention statistics, and skipping interventions reveal recurring attention patterns in MAE and earlier local-to-global progression in DINOv3. Additional experiments examine longer training, frozen-teacher distillation, robustness, and fine-tuning.

Critical strengths
  1. The study addresses an important efficiency problem across two distinct SSL objectives and both global and dense tasks. For example, DINOv3 LIDAS improves linear accuracy from 83.40 to 83.73 and ADE20K mIoU from 44.70 to 45.77 using 86.1% of baseline FLOPs (Table 1), with unchanged final architecture.
  2. The extensive ablations make the empirical study informative: stronger baselines, long training, frozen-teacher distillation, and intermediate-layer analyses expose meaningful dependencies and failure cases. Persistent MAE attention cycles after extended training are a particularly interesting observation.
Critical weaknesses
  1. The empirical contribution lacks comparisons with published growth and efficient-training methods under a common SSL recipe: LiGO (ICLR 2023), AutoProg/MoGrow (CVPR 2022), and EfficientTrain++ (TPAMI 2024). All are cited but not experimentally compared; EfficientTrain++ already reports MAE results, while LiGO and AutoProg provide relevant ViT growth baselines to adapt. Without comparisons using matched objectives, final architectures, and compute budgets, the relative efficiency of the studied recipes remains unclear.
  2. The evidence does not fully establish the stability and source of the gains. Probe-seed error bars cannot establish the reliability of DINOv3's 0.33-point classification gain across pretraining runs. Its growth transitions also reset the teacher backbone from the student and recompute layer-wise learning-rate multipliers, without corresponding controls. These omissions limit attribution to depth growth; attention statistics and residual-stream perturbations characterize differences but do not establish why downstream performance improves.
Questions for authors
  • How does the proposed recipe compare with representative existing growth methods and an efficient MAE curriculum under the same objective, final architecture, and compute budget? Please include elapsed training time accounting for growth transitions.
  • Do the main DINOv3 gains persist across independent pretraining seeds and controls that separate depth growth from teacher-backbone resets and changes in layer-wise learning-rate multipliers?
3 comments

We thank the reviewer for the careful reading and the positive assessment of our work. We are particularly glad that the reviewer recognizes the value of studying pretraining efficiency across two self-supervised objectives and both global and dense tasks, and finds the ablations and the persistence of the MAE attention patterns informative. We address your questions and concerns below.

W1/Q1: Comparison with prior efficiency methods

Comparison with MoGrow and LiGO. We thank the reviewer for this suggestion. We now compare Stacking with MoGrow and depth-only LiGO under the same MAE ViT-L/16 protocol: identical data, pixel-reconstruction objective, Prop-1 growing schedule, final architecture, and 50k-step cosine training schedule. MoGrow uses momentum-averaged weights and the official interleaving map. LiGO learns depth-mixing coefficients for 100 steps at each growth transition, initialized from our Stacking map using LiGO’s noisy initialization. These comparisons evaluate the growth operators under a common training protocol. LiGO’s additional 500 operator-fitting steps are included in its reported FLOPs.

MethodPretraining FLOPs (%) ↓Linear top-1 ↑k-NN top-1 ↑ADE20K mIoU ↑NYUv2 RMSE ↓
Fixed depth100.060.2831.4026.310.6400
Stacking (ours)79.663.7033.8627.790.5842
LiGO80.463.8533.6828.070.5863
MoGrow79.661.3231.5826.990.6068

Each row uses one pretraining run and one downstream seed, while k-NN is deterministic.

Stacking outperforms MoGrow on all four readouts at the same FLOPs, including gains of 2.38 percentage points in linear probing and 0.80 points in segmentation. LiGO and Stacking achieve closely comparable results, with neither method leading across all tasks: LiGO scores slightly higher on linear probing and segmentation, while Stacking scores higher on k-NN and depth estimation. Thus, the learned operator provides no consistent advantage across tasks despite its additional fitting phase and compute. All three growing methods outperform the fixed-depth baseline on every readout while using approximately 80% of its FLOPs. These results strengthen the evidence for depth growth in this setting and show that simple Stacking remains competitive with both learned and momentum-based alternatives.

EfficientTrain++. EfficientTrain++ saves compute through an input curriculum, whereas our study grows encoder depth. Its published MAE experiment also uses HOG reconstruction targets rather than our pixel targets. A controlled comparison would therefore require adapting its curriculum to our objective and matching the total compute budget. We focus here on comparing depth-growth operators while keeping the inputs, objective, and depth schedule fixed. Combining depth growth with an input curriculum is an interesting direction for future work.

Elapsed training time. Please see G4 of the General Response for the elapsed-time estimates for the main MAE and DINOv3 settings, including checkpoint I/O and job restarts at growth transitions.

W2/Q2: Stability and source of the gains

Pretraining seed variability. Please see G1 of the General Response for the additional pretraining seeds and downstream evaluations for fixed-depth DINOv3 and LIDAS Prop-3.

Teacher and learning-rate handling. Please see G3 of the General Response for the controls on the teacher-backbone reset and layer-wise learning-rate multipliers. The main comparison evaluates the complete growing recipe. The controls test how sensitive its results are to these choices.

Computational patterns and performance. Please see G2 of the General Response for our interpretation of the attention patterns and layer-skipping results. We hypothesize that the shift toward earlier computation contributes to downstream gains by giving later layers richer representations to refine.

We thank the reviewer again for the insightful feedback and additional suggestions. In our response, we have addressed all concerns raised by adding controlled comparisons with prior growing variants (LiGO and MoGrow) and running new DINOv3 multi-seed, teacher-update, and learning-rate controls. We are happy to address any remaining questions and hope the reviewer will consider these updates when revisiting the evaluation.

Thank you for the detailed response. After reading it and the other reviews, I maintain my original score . The additional comparisons and experiments show consistent improvements for MAE and in DINOv3 linear probing. However, the mechanism remains a hypothesis, and the gains become smaller across pretraining seeds and under stronger baselines. I encourage the authors to revise the claims in the final version accordingly.

Reviewer_NbeuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies progressive depth growth for self-supervised vision transformers using existing Stacking, MIDAS, and LIDAS strategies in MAE and DINOv3. It evaluates downstream quality and pretraining compute across schedules, then analyzes intermediate representations, attention patterns, and layer-skipping effects. Its contribution is systematic empirical knowledge about growth-based training.

Critical strengths
  • The MAE compute–quality benefit survives important controls. Under the stronger WSD recipe, Stacking improves linear accuracy from 64.02% to 64.54%, segmentation mIoU from 27.07 to 28.20, and depth RMSE from 0.6214 to 0.5803 at 79.6% of baseline FLOPs (p. 25, Table 13). Full-horizon growth over 500k WSD steps retains the 20.4% saving and improves all four reported metrics (p. 29, Table 15). These results support useful efficiency gains within the tested recipes.

  • Complementary analyses provide value beyond final scores. At the main training budgets, intermediate readouts improve substantially, and layer-skipping interventions show redistributed layer influence (p. 7, Figure 3; p. 35, Figure 17). The study also reports unfavorable configurations and adaptation boundaries, including nearly tied long-run fine-tuning results (pp. 43–44, Table 20). This gives the empirical contribution useful breadth.

Critical weaknesses
  • The DINO results do not isolate structural growth from associated optimization changes. Growth recomputes layer-wise learning-rate multipliers; ordinary self-distillation also resets the teacher backbone while retaining its EMA heads (p. 17, Appendix A.2). These operations may contribute to the effects attributed to growing. Reported uncertainty covers probe seeds, not independently repeated pretraining, leaving small gains and rankings less established (pp. 4–5 and 18). Mechanistic explanation is also limited: skipping measures residual-stream changes rather than task effects, and the authors acknowledge that periodicity does not explain strategy differences (p. 21, Eq. 12; p. 33).

  • Practical efficiency and transfer claims need tighter boundaries. Table 16 gives WSD extension deltas relative to grown references, so direct matched-compute comparison requires matching absolute scores; the cosine values in Table 12 cannot supply those references (pp. 23–24 and 29). Hardware is specified, but I found no wall-clock results, so FLOP savings do not establish time savings (p. 13). Long-run fine-tuning is nearly tied only for the tested early-growth configuration; Table 20 does not evaluate full-horizon 500k growth (pp. 43–44).

Questions for authors
  • What existing controls or independent pretraining repeats help separate DINO growth effects from learning-rate and teacher-state changes, and establish strategy-ranking stability? Please distinguish ordinary self-distillation from frozen-teacher distillation and specify which claims concern the complete training recipe rather than isolated structural growth (pp. 15–18).

  • Could you provide the existing absolute scores and complete reference configurations behind Table 16, alongside the corresponding fixed-depth comparison, to make matched-FLOP performance directly assessable? Please also provide any available runtime measurements or explicitly restrict the efficiency claim to FLOPs (p. 13; p. 29, Table 16).

2 comments

We thank the reviewer for the careful reading and the positive assessment of our work. We are particularly glad that the reviewer recognizes that the MAE compute–performance gains hold under the stronger WSD recipe and in the full-horizon 500k-step experiment, and finds the intermediate-layer analyses informative. We address your questions and concerns below.

W1/Q1: DINOv3 growth and interpretation

Teacher and learning-rate handling. Please see G3 of the General Response for the distinction between evaluating the complete growing recipe and testing the contribution of its components. There, we report controls for the layer-wise learning-rate rule and the teacher-backbone reset in self-distillation. In frozen-teacher distillation, the target teacher remains fixed throughout training. The reset at each growth transition applies to the student’s evaluation EMA.

Pretraining seed variability. Please see G1 of the General Response for the additional pretraining seeds and downstream evaluations. These runs assess the repeatability of the comparison between fixed-depth DINOv3 and LIDAS Prop-3.

Computational patterns and performance. Please see G2 of the General Response for our interpretation of the attention patterns and layer-skipping results. We hypothesize that the shift toward earlier computation contributes to downstream gains by giving later layers richer representations to refine.

W2/Q2: Practical efficiency and transfer

Absolute scores at matched FLOPs. We agree that absolute scores make the comparison easier to assess. We have uploaded a revised manuscript with an updated Table 16 that reports the absolute scores for the 125k-step grown WSD references, their 160k-step extensions, and the 125k-step fixed-depth comparison at matched FLOPs.

Elapsed training time. Please see G4 of the General Response for the elapsed-time estimates from training logs, including checkpoint I/O and job restarts at growth transitions. We will clarify that our primary efficiency claims refer to FLOPs and report the elapsed-time estimates alongside them.

Scope of the fine-tuning comparison. We agree that this distinction should be explicit. The long-run models in Table 20 complete growth during the first 50k of their 500k pretraining steps. Their fine-tuned performance is comparable to the fixed-depth baseline. The models grown throughout all 500k steps in Table 15 were evaluated only with frozen features, so our fine-tuning conclusion applies to the early-growth setting. We will clarify this in the camera-ready version.

Thank you to the author for addressing my questions and concerns; I will maintain my original positive rating.

Reviewer_worSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies progressive depth growth as a pretraining strategy for self-supervised vision transformers. Starting from a shallow network, the authors progressively insert copied Transformer blocks until reaching a target architecture. They compare three block-copying strategies, i.e., Stacking, MIDAS, and LIDAS under different growth schedules, and evaluate MAE and DINOv3 models trained from scratch on ImageNet-1k. Grown models are compared with fixed-depth baselines using matched training steps, data exposure, final architectures, and inference costs. The main experiments report that suitably configured growth reduces pretraining FLOPs by 13.9-24.8% while improving downstream performance on ImageNet-1k classification, ADE20K semantic segmentation, and NYUv2 depth estimation. The paper further evaluates layer-wise representations and finds that task-relevant information often becomes accessible at earlier depths in grown models. Attention measurements and layer-skipping interventions are used to characterize internal computation: MAE exhibits recurring four-layer attention patterns, whereas DINOv3 shows an earlier local-to-global attention transition. Additional experiments consider extended training, frozen-teacher distillation, training hyperparameters, robustness to missing visual evidence, nonlinear probes, and full fine-tuning.

Critical strengths
  1. The empirical study is systematic and carefully controlled within the chosen settings. The comparisons match training steps, data exposure, final architecture, and inference cost, while accounting for the actual pretraining FLOPs. Evaluating both global and dense tasks, multiple growth strategies and schedules, two MAE model sizes, and extended training makes the study substantially more convincing than a single-model demonstration. The appendix also examines alternative training recipes rather than relying exclusively on the original baseline configuration.

  2. The central empirical phenomenon is practically relevant. For selected configurations, particularly MAE with Stacking or LIDAS, growth improves frozen representations while reducing pretraining computation. The layer-wise results further suggest that useful representations can emerge at earlier depths. These findings could be valuable for efficient pretraining and motivate further work on intermediate-layer inference or feature extraction.

Critical weaknesses
  1. The reliability and generality of the main empirical claims remain insufficiently established. Each pretraining configuration appears to be evaluated using a single pretraining run. The reported standard deviations measure variation across downstream probe seeds, but do not capture run-to-run variation in pretraining itself. This is particularly important for DINOv3, where some final-layer gains are small; for example, the main LIDAS linear-probe improvement is 0.33 percentage points, and where the conclusions depend strongly on the growth schedule. The same concern applies to the internal analyses, which are also based on individual trained models. Moreover, all models are pretrained on ImageNet-1k, and the DINOv3 experiments disable register tokens and Gram anchoring. Together with the mixed results across strategies and schedules, these choices make it difficult to determine whether the findings reflect a general property of self-supervised vision pretraining or are specific to the selected datasets, recipes, and runs. Since the paper’s primary contribution is empirical, this limitation materially affects my assessment.

  2. The computational-pattern analysis is descriptive and does not yet explain the performance benefits. The attention and layer-skipping results show that growth leaves recognizable structural signatures, but they do not establish a connection between these signatures and improved downstream representations. In particular, all three MAE growth strategies exhibit similar four-layer attention cycles, including configurations that do not improve final-layer classification or segmentation. Thus, the presence of periodicity alone cannot explain the performance differences. Likewise, layer skipping measures changes in residual-stream magnitude rather than changes in task performance, so stronger early-layer influence should not be equated directly with more useful computation. As a result, the evidence supports the observation that growth changes attention and representation organization, but provides limited mechanistic understanding of why growth helps. Clarifying this distinction, or providing evidence linking the observed patterns to performance, would be important for the paper’s second main contribution.

Questions for authors
  1. Can the authors provide results across multiple pretraining seeds for the key comparisons, especially the main DINOv3 Prop-3 results and the strongest MAE configurations? How large is pretraining run-to-run variation relative to the reported gains?
  2. What evidence distinguishes the recurring MAE attention patterns from persistent signatures of block copying? Given that similar cycles occur across strategies with substantially different downstream performance, what mechanistic conclusion should be drawn about their role in improving representations?
2 comments

We thank the reviewer for the careful reading and the positive assessment of our work. We are particularly glad that the reviewer finds the study systematic and carefully controlled, and recognizes the practical value of improving both final and intermediate representations while reducing pretraining compute. We address your questions and concerns below.

W1/Q1: Reliability and generality of the results

Pretraining seed variability. Please see G1 of the General Response for the additional DINOv3 pretraining seeds, the corresponding downstream results, and the attention and layer-skipping analyses across seeds.

We prioritize the additional pretraining runs for DINOv3 because its main linear-probe gain is smaller. For MAE, the existing evidence additionally covers two model sizes and multiple training recipes. For example, ViT-L Stacking improves all four readouts under both cosine and WSD at 79.6% of baseline FLOPs.

Pretraining dataset. We would like to clarify that ImageNet-1k is an established benchmark for training image-only vision transformers from scratch. It is the primary pretraining dataset in the original MAE study, and DINOv3 provides a public reference recipe for this setting. This gives us a common dataset for evaluating depth growth across masked reconstruction and self-distillation under established recipes. We agree that evaluating growth on other pretraining datasets and at larger data scales would help establish how broadly the findings generalize, and we leave this to future work.

DINOv3 reference recipe. We follow the authors’ public ViT-L/16 ImageNet-1k recipe, which uses zero register tokens and leaves Gram anchoring disabled. Both the fixed-depth and grown models share these settings, so the comparison measures the effect of the growing procedure within this reference recipe.

Strategy and schedule dependence. The reviewer correctly observes that the outcomes depend on the strategy and schedule. This dependence is indeed one of our findings. In DINOv3, Prop-1 offers larger compute savings but lower classification and segmentation performance than the fixed-depth baseline across all three tested strategies, whereas LIDAS Prop-3 improves linear probing and both dense readouts at 86.1% of baseline FLOPs. Our conclusion is that suitable growth configurations can improve the compute–performance trade-off in both families, with the preferred configuration depending on the model and task.

W2/Q2: Computational patterns and representation quality

Please see G2 of the General Response for our interpretation of the observed computational patterns. We interpret the recurring MAE attention pattern as a persistent signature of growth, while the intermediate downstream readouts provide the direct evidence that useful representations become accessible earlier. We hypothesize that this shift toward earlier computation contributes to downstream gains by giving later layers richer representations to refine.

We thank the reviewer again for the insightful feedback and constructive questions. In our response, we have addressed all concerns raised by running multi-seed DINOv3 pretraining experiments confirming repeatable downstream gains and computational patterns, and clarifying how stronger intermediate-layer readouts connect earlier computation to downstream performance. We are happy to address any remaining questions and hope the reviewer will consider these updates when revisiting the evaluation.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their helpful feedback. We are happy to see that all reviewers assessed our work positively, highlighting the clarity of the paper (Nbeu, 1SvA), the systematic and carefully controlled comparisons (worS), and the breadth of the experiments and ablations (Nbeu, 1SvA).

Common concerns were raised regarding 1) variability across pretraining seeds, 2) the relationship between computational patterns and downstream performance, 3) the role of the teacher and learning-rate updates in DINOv3, and 4) elapsed training time. We address these points below.

G1. Pretraining seed variability. We agree on the importance of measuring variability across pretraining runs. The standard deviations in the submission quantify variation across downstream probe seeds for each pretrained encoder. Given the cost of repeating the full pretraining sweep, we focus the additional analysis on fixed-depth DINOv3 and LIDAS Prop-3, where the original linear-probe gain is 0.33 percentage points. We repeat each configuration with two additional pretraining seeds and evaluate the same downstream tasks, attention patterns, and layer-skipping effects.

The table below reports the mean ± standard error over three pretraining seeds. The linear-probe gain of LIDAS is robust to the pretraining seed: LIDAS improves over the fixed-depth baseline by 0.28 pp, a margin well beyond the seed-to-seed variation of either model, and every LIDAS run outperforms every baseline run. ADE20k and NYUv2 also improve on average, but with larger seed-to-seed variation for LIDAS, particularly for depth estimation, while k-NN accuracy is unchanged. The computational patterns reported recur in all three seeds. In every LIDAS run, attention is most localized and query-dependent (lowest attention distance and entropy, highest query–key NMI) at layers 3–5, compared with layers 10–11 in every baseline run. The middle layers of LIDAS likewise show broader, less query-dependent attention and less patch-to-[CLS] attention. Layer skipping reproduces the shift in layer influence: skipping early LIDAS layers changes middle-layer representations more than in the baseline, whereas skipping late layers has a smaller effect on nearby readouts. Overall, the linear-probe gain and the computational signatures of growth are robust to the pretraining seed, whereas the dense-prediction gains are smaller relative to the variation between seeds.

ModelIN1k kNN ↑IN1k Linear ↑ADE20k mIoU ↑NYU RMSE ↓
DINOv3, cosine, baseline82.18 ± 0.0283.49 ± 0.0344.76 ± 0.150.4634 ± 0.0008
LIDAS, cosine, Prop-382.18 ± 0.0483.77 ± 0.00345.44 ± 0.240.4593 ± 0.0047

G2. Computational patterns and downstream performance. We agree that the connection between the observed patterns and performance is an important question. In MAE, the four-layer attention pattern persists at 500k steps even as alignment between copied weights weakens. This is consistent with growth establishing an organization that persists as the copied blocks continue to learn. Its recurrence across strategies with different downstream outcomes also shows that periodicity alone does not determine representation quality.

The clearest connection to performance comes from the intermediate readouts: useful features become accessible at shallower depths. MAE ViT-L Stacking approaches its final classification and segmentation scores by layers 16–18, while the fixed-depth model continues to improve at later layers. DINOv3 also transitions to broader attention at shallower layers. The layer-skipping analysis complements these observations by measuring how layer influence propagates through subsequent residual streams; it does not directly measure task performance after skipping. Together, these findings motivate our hypothesis that the shallow training stages encourage early blocks to extract useful features, giving later blocks richer representations to refine. This shift toward earlier computation offers a plausible explanation for the downstream gains in the stronger growing configurations.

G3. DINOv3 growth recipe. Our main comparison evaluates the full growing recipe at matched data, training steps, and final architecture. When the student grows, we need to expand the EMA teacher and set the layer-wise learning-rate multipliers for the new depth. We initialize the expanded teacher backbone from the student, keep the teacher’s EMA heads, and recompute the multipliers at each transition. These simple choices are part of our recipe and partly a practical strength of our setup: we improve the compute–performance trade-off without changing the self-distillation objective or adding an extra training phase.

We agree that testing the sensitivity of the results to these choices is informative. We therefore evaluate three controls: a grown model with a fixed layer-wise learning-rate rule same as the baseline (layer-wise LR fixed), a grown model whose teacher's EMA heads are also reset (teacher reset_full) and a grown model whose teacher is expanded from its own EMA weights (teacher grow).

The table below reports the three controls for LIDAS Prop-3 (mean ± standard error over three downstream seeds). Resetting the teacher's projection heads together with its backbone at each growth step (teacher reset_full) performs similarly to the default, which resets only the backbone, except for lower ADE20k segmentation. Fixing the layer-wise learning-rate multipliers to those of the full 24-layer model throughout training (layer-wise LR fixed) is similarly comparable but degrades segmentation further. Growing the teacher separately from its own EMA (teacher grow), without resetting it from the student, matches the default in linear probing and segmentation but is worse in k-NN accuracy and depth estimation. All variants retain the improvements over the fixed-depth baseline in linear probing, segmentation, and depth estimation. This indicates that the gains are not an artifact of the teacher reset or the recomputed learning-rate multipliers, although the size of the dense-prediction gains depends on these choices.

ModelIN1k kNN ↑IN1k Linear ↑ADE20k mIoU ↑NYU RMSE ↓
DINOv3, cosine, baseline82.1783.40 ± 0.0244.70 ± 0.060.4650 ± 0.00005
LIDAS, cosine, Prop-382.1683.73 ± 0.0245.77 ± 0.050.4523 ± 0.0003
+ layer-wise LR fixed82.2083.69 ± 0.0444.95 ± 0.070.4525 ± 0.0002
+ teacher reset_full82.2483.74 ± 0.0245.22 ± 0.030.4519 ± 0.0004
+ teacher grow81.9883.75 ± 0.0345.52 ± 0.020.4579 ± 0.0003

G4. Elapsed training time. We report FLOPs as our primary measure of computational cost because they quantify the arithmetic work independently of the hardware. Elapsed time also depends on node throughput, data loading, communication, and filesystem load, which can vary across runs. We agree that elapsed time complements the FLOP comparison. We therefore estimate training time on 32 GH200 GPUs using logged median step times at each depth, pooled across grown and baseline runs within each recipe.

SettingFixed depth (h)Grown (h)Time savedFLOPs saved
MAE ViT-L/16, 50k1.711.597.0%20.4%
MAE ViT-H/14, 50k3.032.6014.2%24.8%
MAE ViT-L/16, 500k, growth in first 50k17.1117.020.5%2.0%
DINOv3, cosine, Prop-117.4112.9625.6%27.5%
DINOv3, cosine, Prop-217.4114.1418.8%19.0%
DINOv3, cosine, Prop-317.4114.9414.2%13.9%
DINOv3, WSD, Prop-318.8416.6611.5%13.9%
DINOv3, frozen-teacher distillation, Prop-118.9315.6917.1%23.0%

The estimated time savings broadly follow the FLOP savings, most closely for DINOv3 with cosine decay. For the shorter MAE runs, costs such as data loading and communication decrease less than FLOPs at shallow depths, and stage restarts account for a larger share of the total time. As a result, the elapsed-time savings are smaller.

We address the remaining concerns and questions in the responses to each review, which include the scope of the tested settings, the absolute scores for the matched-FLOP comparison in Table 16, and comparisons with prior growth methods.

AuthorsOpenReview ↗

Final Summary

We thank all three reviewers and the Area Chair for the time and care they invested in evaluating our work. We are glad that the reviewers recognized the systematic comparisons, the breadth of the experiments, and the practical value of improving visual representations while reducing pretraining compute.

This study examines how progressive depth growth affects the quality of self-supervised visual representations and where useful features emerge in the network. Across MAE and DINOv3, we identify growth configurations that improve downstream performance while reducing pretraining FLOPs by up to 25%. During the rebuttal, we addressed all concerns raised by the reviewers through additional experiments and clarifications, with the main updates summarized below.

  • Reliability across pretraining runs. We evaluated the main DINOv3 comparison across three independent pretraining seeds. LIDAS Prop-3 improves mean linear-probe accuracy by around 0.3 percentage points, with every LIDAS run outperforming every fixed-depth run. Both dense tasks also improve on average, with greater variation across seeds. The earlier transition from local to broader attention and the greater influence of early layers in the layer-skipping analysis recur across all three seeds.
  • Computational patterns and downstream performance. The recurring MAE attention pattern persists across strategies with different downstream outcomes. The intermediate probes provide a more direct connection to performance. For example, MAE Stacking approaches its final classification and segmentation scores by middle layers, while the fixed-depth model continues to improve at later layers. These observations motivate our hypothesis that shifting useful computation toward earlier layers contributes to the downstream gains.
  • DINOv3 growth recipe. We added three controls that vary how the teacher is updated or how layer-wise learning rates are assigned during growth. All three retain gains over the fixed-depth baseline in linear probing, segmentation, and depth estimation, although the size of the dense-task gains depends on these choices.
  • Comparison with prior methods. We added MoGrow and LiGO as two growing baselines. Under a common MAE protocol, Stacking outperforms MoGrow on all four readouts at the same FLOPs. It is also competitive with LiGO, with each performing better on two downstream tasks, while LiGO requires additional training stages, resulting in more FLOPs.

We appreciate the reviewers’ constructive feedback and are encouraged that all three recommended acceptance. We hope the additional experiments and clarifications are helpful to the Area Chair in reaching a final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.