- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Prior work showed that training a ViT on non-visual data (e.g., strings of balanced parentheses) before training it on real images makes it learn better representations. But it is unclear why, because such a full warmed-up training schedule mixes attention, MLPs, normalization and embeddings together, so the resulting weights do not reveal which specific part makes the core contribution. The paper therefore asks which part of this warmed-up training actually carries the benefit, including the following 3 questions: (1) how much of the gain survives once most warmed-up weights are discarded; (2) whether the surviving part can be expressed as attention QK/OV products and moved into freshly initialized weights; (3) whether such an initializer remains effective at ImageNet-1K scale.
With an elimination study on a ViT with such warmed-up training on DYCK, the paper concludes that: (1) essentially all of the gain survives when only the late-block attention weights are kept, while early layers and MLPs contribute little or even hurt; (2) these weights can be reduced to the per-head QK/OV products and refactorized by SVD into fresh weights, which matches the full warm-up on CIFAR-100 and Tiny-ImageNet; (3) this initializer still works with ViT-B on ImageNet-1K once it covers more than the last 4 blocks. It also finds that this effect is specific to procedural warm-up, as the same intervention recovers only ~15% of the gain from supervised DeiT pre-trained weights.
Critical strengths
A well-posed follow-up question studied with controlled analysis: the paper explores one natural follow-up question of recent work on procedural warm-up of ViTs and answers it with well-designed controlled ablations.
Critical weaknesses
(1) The experiments do not support the central claim.
(a) The analysis cannot yet separate the learned procedural structure from the generic attention prior: mimetic initialization [1] gives these same two products a simple identity shape with no warm-up and already improves ViTs trained from scratch, and its MLP version [2] finds that MLP initialization helps much less than attention initialization. The paper's explanation of what transfers is incomplete without the comparison with this initialization.
(b) The ImageNet baseline is lower than the official reported score: the from-scratch ViT-B in Table 2 is 77.72%, which is far below the standard DeiT-B result of 81.8% in [3], and even the best initializer (80.54%) stays below it. The authors should explain this gap or use the widely adopted baseline performance for proper comparison, which is essential to support the effectiveness claim.
(c) The statistical support is insufficient: the ImageNet references and the elimination study are single runs, and the margins on CIFAR-100 in Table 1 are quite small, which cannot fully support the claim.
(d) The claim that the effect is specific to procedural warm-up rests on comparing a 15k-step DYCK model with a fully trained ImageNet DeiT on a single target dataset, which mixes up procedural vs. visual data & short vs. mature training. A short visual warm-up of the same training duration should be included as a control.
(e) The evidence is too narrow for the paper's general framing: only plain ViTs with classification tasks on 3 datasets are tested, and model size is tied to the dataset (Tiny for 2 small datasets, Base for ImageNet-1K). More classification datasets should be included, such as CIFAR-10, STL-10, Flowers-102, Oxford Pets and Food-101. A wide sweep on model sizes including Tiny, Small, Base and Large; ViT variants such as Swin and PVT; and downstream tasks like dense prediction (COCO detection & ADE20K segmentation) are needed to show the findings hold beyond the current narrow setting.
(2) The paper is hard to read.
Some terms such as "coverage screen", "gauge" and "loader-defined late subset" need to be explained in plain language, (e.g., in a footnote), and the main findings of this work are hard to extract from the paper which should be highlighted more clearly in the introduction and conclusion.
Reference
[1] Mimetic Initialization of Self-Attention Layers.
[2] Mimetic Initialization of MLPs.
[3] Training data-efficient image transformers & distillation through attention.
Questions for authors
Apart from the main concerns in Weaknesses, here are the minor concerns:
-
Under your recipe, how do (a) mimetic initialization and (b) late DYCK QK+OV with matched spectrum but random singular directions compare with your initializer on downstream tasks? How much of each late DYCK QK and OV product is captured by a scaled identity? If DYCK clearly beats both, the learned-structure claim holds; if not, the paper should present the warm-up as mainly installing a generic attention prior.
-
A discussion of linear-attention ViTs and DiTs would further strengthen the robustness of the findings.