acceptodds
Under review as a conference paper at ICLR 2027

What Transfers from Non-Visual Procedural Warm-Up? Attention Operators as Structured Initializers for Vision Transformers

Abstract

Non-visual procedural warm-up can improve the subsequent training of Vision Transformers (ViTs), despite exposing the model to sequences with no visual or semantic overlap with images. What initialization structure does this warm-up leave behind? We study this question in a controlled setting: a ViT is warmed up for 15,000 steps on DYCK-64 and then trained for image classification. Through elimination interventions, we progressively discard warm-up parameters and isolate the bias-free query-key (QK) and output-value (OV) weight operators of selected attention blocks. We then refactor these operators into fresh projection matrices while randomly initializing the rest of the model, separating operator transfer from raw weight copying. Across three matched source/downstream seeds, late-block QK+OV initialization reaches 75.17 ± 0.30% on CIFAR-100 and 60.41 ± 0.64% on Tiny-ImageNet, compared with 74.74 ± 0.51% and 60.30 ± 0.70% for full warm-up transfer, retaining 121% and 103% of its gain over scratch. On ViT-Base/ImageNet-1K, expanding QK+OV coverage from the last four to the last 6/8/10/12 blocks yields 79.86/80.40/80.27/80.54% Top-1, versus 77.72% from scratch and 79.50% from full warm-up; with the source fixed, the last-eight setting reaches 80.32 ± 0.12% across three downstream seeds. Exact alternative factorizations further show that optimization can depend on factor coordinates even when the transferred operators are fixed. This behavior is source-dependent: under supervised ImageNet pretraining, the same QK+OV intervention recovers only 15% of the full transfer gain. Our results show that late-block attention-operator initialization can retain the benefit of a short procedural warm-up, while establishing a clear empirical boundary between procedural initialization and conventional visual feature transfer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers,Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9VbROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Prior work showed that training a ViT on non-visual data (e.g., strings of balanced parentheses) before training it on real images makes it learn better representations. But it is unclear why, because such a full warmed-up training schedule mixes attention, MLPs, normalization and embeddings together, so the resulting weights do not reveal which specific part makes the core contribution. The paper therefore asks which part of this warmed-up training actually carries the benefit, including the following 3 questions: (1) how much of the gain survives once most warmed-up weights are discarded; (2) whether the surviving part can be expressed as attention QK/OV products and moved into freshly initialized weights; (3) whether such an initializer remains effective at ImageNet-1K scale.

With an elimination study on a ViT with such warmed-up training on DYCK, the paper concludes that: (1) essentially all of the gain survives when only the late-block attention weights are kept, while early layers and MLPs contribute little or even hurt; (2) these weights can be reduced to the per-head QK/OV products and refactorized by SVD into fresh weights, which matches the full warm-up on CIFAR-100 and Tiny-ImageNet; (3) this initializer still works with ViT-B on ImageNet-1K once it covers more than the last 4 blocks. It also finds that this effect is specific to procedural warm-up, as the same intervention recovers only ~15% of the gain from supervised DeiT pre-trained weights.

Critical strengths

A well-posed follow-up question studied with controlled analysis: the paper explores one natural follow-up question of recent work on procedural warm-up of ViTs and answers it with well-designed controlled ablations.

Critical weaknesses

(1) The experiments do not support the central claim.

(a) The analysis cannot yet separate the learned procedural structure from the generic attention prior: mimetic initialization [1] gives these same two products a simple identity shape with no warm-up and already improves ViTs trained from scratch, and its MLP version [2] finds that MLP initialization helps much less than attention initialization. The paper's explanation of what transfers is incomplete without the comparison with this initialization.

(b) The ImageNet baseline is lower than the official reported score: the from-scratch ViT-B in Table 2 is 77.72%, which is far below the standard DeiT-B result of 81.8% in [3], and even the best initializer (80.54%) stays below it. The authors should explain this gap or use the widely adopted baseline performance for proper comparison, which is essential to support the effectiveness claim.

(c) The statistical support is insufficient: the ImageNet references and the elimination study are single runs, and the margins on CIFAR-100 in Table 1 are quite small, which cannot fully support the claim.

(d) The claim that the effect is specific to procedural warm-up rests on comparing a 15k-step DYCK model with a fully trained ImageNet DeiT on a single target dataset, which mixes up procedural vs. visual data & short vs. mature training. A short visual warm-up of the same training duration should be included as a control.

(e) The evidence is too narrow for the paper's general framing: only plain ViTs with classification tasks on 3 datasets are tested, and model size is tied to the dataset (Tiny for 2 small datasets, Base for ImageNet-1K). More classification datasets should be included, such as CIFAR-10, STL-10, Flowers-102, Oxford Pets and Food-101. A wide sweep on model sizes including Tiny, Small, Base and Large; ViT variants such as Swin and PVT; and downstream tasks like dense prediction (COCO detection & ADE20K segmentation) are needed to show the findings hold beyond the current narrow setting.

(2) The paper is hard to read.

Some terms such as "coverage screen", "gauge" and "loader-defined late subset" need to be explained in plain language, (e.g., in a footnote), and the main findings of this work are hard to extract from the paper which should be highlighted more clearly in the introduction and conclusion.

Reference

[1] Mimetic Initialization of Self-Attention Layers.

[2] Mimetic Initialization of MLPs.

[3] Training data-efficient image transformers & distillation through attention.

Questions for authors

Apart from the main concerns in Weaknesses, here are the minor concerns:

  • Under your recipe, how do (a) mimetic initialization and (b) late DYCK QK+OV with matched spectrum but random singular directions compare with your initializer on downstream tasks? How much of each late DYCK QK and OV product is captured by a scaled identity? If DYCK clearly beats both, the learned-structure claim holds; if not, the paper should present the warm-up as mainly installing a generic attention prior.

  • A discussion of linear-attention ViTs and DiTs would further strengthen the robustness of the findings.

Reviewer_ckXfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores where the benefits come from a procedural warm-up on non visual data for ViTs. The paper finds the attention QK and OV weights particularity for the later blocks to carry most of the benefits.

Critical strengths
  • The paper looks into an interesting question, about what is driving the transfer from a non visual procedural warmup which helps ViTs subsequently train on images.

  • The experiments have been carefully designed and run on multiple datasets, model sizes and over multiple seeds (in most parts of the paper).

Critical weaknesses
  • The authors do not establish their contribution sufficiently beyond prior work. Shinnick et al. already concluded that majority of the benefits were concentrated in later parts of the network and explored the individual contribution of the attention and mlp blocks in isolation. This paper does go further by looking specifically at the QK and OV operators, but the benefit of its SVD refactorisation over direct transfer of the attention block remains unclear. In table 3 that transferring the raw attention weights (as Shinnick et al. do) is the same performance as the best QK+OV setting the paper introduces (both using blocks 8 - 11)

  • The explanation to the titles question remains limited. The paper does not explain which properties of the procedural operators are responsible for the benefits. It has been covered in prior work where transferable structure resides Shinnick et al. (2025), Lee et al. (2025) Jiang et al. (2026) that conclude the attention weights can be responsible for majority of the performance of procedural warmup (aka. pre-pre training). Figures 2 and 3 provide observations about matrix ranks and training dynamics but do not explicitly connect them to why the attention operators are beneficial. The DeiT experiments also do not provide any evidence to support the title of the paper. Applying the extra controls used here to the procedural warmup model would be more beneficial.

  • Shinnick et al. (2025), Can You Learn to See Without Images? Procedural Warm-up for Vision Transformers

  • Lee et al. (2026), Training Language Models via Neural Cellular Automata

  • Jiang et al. (2026), Procedural Pretraining: Warming Up Language Models with Abstract Data

Questions for authors
  • Can you clarify what the advantage of SVD refactorization over raw attention transfer is, given the similar performance in table 3?

  • Do the operator-direction and product preserving factorisations from DeIT provide any insights into DYCK? This could be useful in strengthening the papers positioning and answering the titles question.

Reviewer_sM8gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which components of a Vision Transformer retain the benefits of non-visual procedural warm-up. The authors first warm up a ViT on DYCK-64 sequences and then progressively remove pretrained parameters, ultimately retaining selected query–key (QK) and output–value (OV) weight products from late attention blocks. These products are refactorized into new projection matrices, while the remaining target parameters are randomly initialized.

Critical strengths

Rather than relying only on correlations between pretrained weights and downstream performance, the paper progressively removes parameters and tests which retained components are sufficient to preserve the warm-up benefit.

Separating QK and OV products provides an interpretable way to investigate attention initialization while distinguishing the learned operator from its original projection-matrix coordinates.

The paper reports matched-seed experiments on CIFAR-100 and Tiny-ImageNet, followed by a ViT-Base/ImageNet-1K coverage study and independent three-seed confirmation of the selected last-eight-block setting.

Critical weaknesses
  1. The evidence is concentrated on a narrow model and task setting. The main experiments use plain ViT architectures and image classification on three datasets. Although the paper discusses broader implications, it does not establish that the same operator-transfer mechanism applies to dense prediction, video models, or generative Transformers.

  2. The causal mechanism remains only partially identified. The intervention results support the sufficiency of late QK+OV initialization under the evaluated protocol, but they do not establish why these operators carry the benefit. The observed OV/QK asymmetry and weight statistics are informative, yet alternative explanations involving activation scales, optimization geometry, or interactions with randomly initialized parameters remain plausible.

  3. The selection of late-layer coverage is target-dependent. The ImageNet coverage curve is non-monotonic, and the preferred suffix differs from the four-block suffix selected in the smaller-dataset intervention study. This makes it difficult to infer a general rule for choosing which layers to transfer without additional validation.

  4. The method preserves only bias-free operators, not the complete attention function. Source biases and other components are discarded. As the paper's numerical audit indicates, equality of the QK/OV products does not imply equality of the complete biased module. The extent to which this limitation affects the interpretation of the transferred structure deserves further clarification.

  5. The parameterization results complicate the claim that the operators themselves are the transferable unit. Different factorizations of the same initial products produce different outcomes, suggesting that the benefit depends not only on the operators but also on the coordinates in which they are represented and optimized. This is an interesting finding, but it weakens any interpretation that treats the products as a complete characterization of the transferable initialization.

  6. The generality of the source-dependent finding is uncertain. The conventional visual-pretraining comparison uses a supervised DeiT checkpoint. The paper does not perform corresponding transfer interventions from MAE, DINO, or DINOv2, so the evidence is insufficient to characterize how operator-level transfer behaves across different types of mature visual pretraining.

  7. The computational advantage is not fully characterized. The proposed method requires an additional procedural warm-up stage. A more explicit accounting of source-training cost, target-training cost, and reuse across multiple target models would help establish when the approach is practically beneficial.

Questions for authors
  1. Generalization beyond image classification. Have the authors tested the proposed QK+OV initializer on dense prediction tasks, video Transformers, or diffusion Transformers? If not, what aspects of the method are expected to transfer, and which may depend on the specific ViT architecture and classification objective?

  2. Mechanism behind the OV/QK asymmetry. OV-only transfer is more consistently beneficial than QK-only transfer in the reported experiments. Can the authors provide further evidence that this difference reflects the information encoded in the operators, rather than differences in parameter scales, optimization conditioning, or interactions with the randomly initialized network?

  3. Choosing the transferred layer range. The suffix coverage results are non-monotonic, and the best-performing range varies with the downstream setting. Can the authors identify a selection criterion that predicts which blocks should be transferred without relying on extensive downstream sweeps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.