- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies existing progressive depth-growth strategies (Stacking, MIDAS, and LIDAS) for MAE and DINOv3 pretraining. At matched training steps and final architectures, selected configurations improve frozen-feature classification, segmentation, and depth estimation while reducing FLOPs by 13.9–24.8%. Layer-wise probes, attention statistics, and skipping interventions reveal recurring attention patterns in MAE and earlier local-to-global progression in DINOv3. Additional experiments examine longer training, frozen-teacher distillation, robustness, and fine-tuning.
Critical strengths
- The study addresses an important efficiency problem across two distinct SSL objectives and both global and dense tasks. For example, DINOv3 LIDAS improves linear accuracy from 83.40 to 83.73 and ADE20K mIoU from 44.70 to 45.77 using 86.1% of baseline FLOPs (Table 1), with unchanged final architecture.
- The extensive ablations make the empirical study informative: stronger baselines, long training, frozen-teacher distillation, and intermediate-layer analyses expose meaningful dependencies and failure cases. Persistent MAE attention cycles after extended training are a particularly interesting observation.
Critical weaknesses
- The empirical contribution lacks comparisons with published growth and efficient-training methods under a common SSL recipe: LiGO (ICLR 2023), AutoProg/MoGrow (CVPR 2022), and EfficientTrain++ (TPAMI 2024). All are cited but not experimentally compared; EfficientTrain++ already reports MAE results, while LiGO and AutoProg provide relevant ViT growth baselines to adapt. Without comparisons using matched objectives, final architectures, and compute budgets, the relative efficiency of the studied recipes remains unclear.
- The evidence does not fully establish the stability and source of the gains. Probe-seed error bars cannot establish the reliability of DINOv3's 0.33-point classification gain across pretraining runs. Its growth transitions also reset the teacher backbone from the student and recompute layer-wise learning-rate multipliers, without corresponding controls. These omissions limit attribution to depth growth; attention statistics and residual-stream perturbations characterize differences but do not establish why downstream performance improves.
Questions for authors
- How does the proposed recipe compare with representative existing growth methods and an efficient MAE curriculum under the same objective, final architecture, and compute budget? Please include elapsed training time accounting for growth transitions.
- Do the main DINOv3 gains persist across independent pretraining seeds and controls that separate depth growth from teacher-backbone resets and changes in layer-wise learning-rate multipliers?
3 comments
We thank the reviewer again for the insightful feedback and additional suggestions. In our response, we have addressed all concerns raised by adding controlled comparisons with prior growing variants (LiGO and MoGrow) and running new DINOv3 multi-seed, teacher-update, and learning-rate controls. We are happy to address any remaining questions and hope the reviewer will consider these updates when revisiting the evaluation.
Thank you for the detailed response. After reading it and the other reviews, I maintain my original score . The additional comparisons and experiments show consistent improvements for MAE and in DINOv3 linear probing. However, the mechanism remains a hypothesis, and the gains become smaller across pretraining seeds and under stronger baselines. I encourage the authors to revise the claims in the final version accordingly.
We thank the reviewer for the careful reading and the positive assessment of our work. We are particularly glad that the reviewer recognizes the value of studying pretraining efficiency across two self-supervised objectives and both global and dense tasks, and finds the ablations and the persistence of the MAE attention patterns informative. We address your questions and concerns below.
W1/Q1: Comparison with prior efficiency methods
Comparison with MoGrow and LiGO. We thank the reviewer for this suggestion. We now compare Stacking with MoGrow and depth-only LiGO under the same MAE ViT-L/16 protocol: identical data, pixel-reconstruction objective, Prop-1 growing schedule, final architecture, and 50k-step cosine training schedule. MoGrow uses momentum-averaged weights and the official interleaving map. LiGO learns depth-mixing coefficients for 100 steps at each growth transition, initialized from our Stacking map using LiGO’s noisy initialization. These comparisons evaluate the growth operators under a common training protocol. LiGO’s additional 500 operator-fitting steps are included in its reported FLOPs.
Each row uses one pretraining run and one downstream seed, while k-NN is deterministic.
Stacking outperforms MoGrow on all four readouts at the same FLOPs, including gains of 2.38 percentage points in linear probing and 0.80 points in segmentation. LiGO and Stacking achieve closely comparable results, with neither method leading across all tasks: LiGO scores slightly higher on linear probing and segmentation, while Stacking scores higher on k-NN and depth estimation. Thus, the learned operator provides no consistent advantage across tasks despite its additional fitting phase and compute. All three growing methods outperform the fixed-depth baseline on every readout while using approximately 80% of its FLOPs. These results strengthen the evidence for depth growth in this setting and show that simple Stacking remains competitive with both learned and momentum-based alternatives.
EfficientTrain++. EfficientTrain++ saves compute through an input curriculum, whereas our study grows encoder depth. Its published MAE experiment also uses HOG reconstruction targets rather than our pixel targets. A controlled comparison would therefore require adapting its curriculum to our objective and matching the total compute budget. We focus here on comparing depth-growth operators while keeping the inputs, objective, and depth schedule fixed. Combining depth growth with an input curriculum is an interesting direction for future work.
Elapsed training time. Please see G4 of the General Response for the elapsed-time estimates for the main MAE and DINOv3 settings, including checkpoint I/O and job restarts at growth transitions.
W2/Q2: Stability and source of the gains
Pretraining seed variability. Please see G1 of the General Response for the additional pretraining seeds and downstream evaluations for fixed-depth DINOv3 and LIDAS Prop-3.
Teacher and learning-rate handling. Please see G3 of the General Response for the controls on the teacher-backbone reset and layer-wise learning-rate multipliers. The main comparison evaluates the complete growing recipe. The controls test how sensitive its results are to these choices.
Computational patterns and performance. Please see G2 of the General Response for our interpretation of the attention patterns and layer-skipping results. We hypothesize that the shift toward earlier computation contributes to downstream gains by giving later layers richer representations to refine.