acceptodds
Under review as a conference paper at ICLR 2027

A Lasting Bias of Initialization: How Initialization Shapes Generalization in Vision Transformers

Abstract

Initialization is typically designed to stabilize optimization, yet its effects can persist throughout training. Recent work for example showed that warming up Vision Transformers (ViTs) on procedural text, such as Dyck languages, surprisingly improves downstream image classification despite seeing no images during warm-up. We investigate which changes to initialization led to this. We find that much of the improvement can be recovered by matching a small set of layerwise parameter and activation statistics, including effective weight scales, LayerNorm statistics, low attention entropy and negative MLP pre-activations in early-to-middle layers, and large residual write ratios in late layers. We extract a total of 108 of these scalar statistics across the layers of procedurally warmed-up reference models and show that imposing them on a randomly initialized ViT is sufficient to match or exceed the improvements from procedural warm-up. Models trained from procedural and our modified initializations also show persistently lower intermediate-layer prediction accuracy and memorize fewer corrupted labels. These observations are consistent with an implicit regularizing effect. Together, our results show that a few initialization statistics can reproduce much of the benefit of procedural warm-up and provide evidence that initialization can impose a lasting inductive bias on learning and generalization.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
initialization, procedural, generalization, regularization, vision transformer, ViT

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sGaTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to explain why procedural warm-up (training a ViT on Dyck-language sequences before image training) improves image classification. First the authors show that warmed-up models make classes decodable only in the last layers and memorize fewer corrupted labels, and they read this as a regularizing effect. Then they compare the warmed-up weights with a random initialization and find a few differences: late layers write more to the residual stream, while early and middle layers have sharper attention, suppressed MLPs and different weight and LayerNorm scales. To test whether these differences explain the gain, they impose them on a randomly initialized ViT with two interventions. The late-layer intervention rescales some weights of the last layers so that they write as much to the residual stream as in the warmed-up model. The early-layer intervention modifies the weights and LayerNorm parameters of the early and middle layers to match their attention sharpness, MLP activations and scales. Overall, imposing these 108 scalar statistics recovers most or all of the gain of procedural warm-up on ImageNet and on several distribution shifts. Combining the two interventions gives a model that is on par with or slightly better than procedural warm-up, and that also memorizes fewer corrupted labels.

Critical strengths
  1. Clear question and experimental design. The paper asks a concrete question, what procedural warm-up actually changes in the network, and tries to answer it with interventions and not only with correlations. I appreciated several choices: the hybrid models (random early layers, warmed-up late layers) isolate which part of the network carries the gain. The residual write ratios are tested in both directions: scaling a random model up to the procedural ratios and scaling a hybrid model down to the random ratios. The rescaling only touches WVW_V, WOW_O and Wfc2W_{fc2}, so that the output scale changes without changing the attention sharpness, and the effect of the larger writes is separated from the smaller relative update size by adjusting the learning rate. The early-layer components are ablated one by one , and the late-layer results are repeated on ViT-S over eight procedural variants (Figure 11). The modified models are also checked to reproduce the logit-lens and write-ratio profiles of procedural warm-up, and they are evaluated on six distribution shifts and not only on ImageNet. The procedures are described precisely, with pseudocode, which makes the work easy to follow and to reproduce.

  2. Interesting message on initialization. Initialization is usually designed only to make optimization stable. This paper shows that a few scalar statistics set at initialization can change where learning happens across the depth of the network, and that this effect is still visible after 300 epochs, on distribution shifts and on label memorization. It also has a practical side: 108 scalars are enough to obtain most of the benefit of a full procedural warm-up, which needs all the parameters of the model, and the preliminary result in Appendix L suggests that some of these targets could even be set by hand. This could lead to a simple new initialization scheme, and it is interesting for the community also outside procedural warm-up.

Critical weaknesses
  1. Copying the properties of procedural warm-up and getting similar behaviour is somewhat expected. The main contribution measures statistics of the warmed-up model and imposes the same statistics on a random model, so in my opinion it is not very surprising that the modified model behaves like the warmed-up one. More importantly, the experiments do not separate what is specific to procedural warm-up from a generic re-tuning of the initialization. Setting all late-layer write ratios to an arbitrary 1.4 gives almost the same accuracy as the procedural ratios (79.69 vs 79.99, Table 8). The effective weight scales and LayerNorm statistics alone give +1.8 of the +2.3 pp (Table 9). These results show that changing the initialization helps, but not that the specific properties of procedural warm-up are needed: a simpler change might work as well. To check this, the authors could train randomly initialized models with smaller or larger initial weights (e.g. a different standard deviation), without copying anything from the warmed-up model. If this also gives a similar gain, the improvement comes from better initial weight scales in general and not from properties specific to procedural warm-up.

  2. The combination of interventions is underexplored. For me the most interesting result is Section 3.4, where combining the early-layer and late-layer interventions gives a model on par with or better than procedural warm-up. This suggests the two interventions may capture different properties, and with some tuning (of the target statistics, of the split layer, or by mixing targets from different procedural variants) the combined initialization could outperform both. But this part is a single short paragraph, tested with one procedural variant and one seed, and the gap over procedural warm-up (80.3 vs 80) is too small to conclude anything. Most of the paper is spent on matching single properties, which is the less surprising part.

  3. Possible different explanation of the low decodability of early layers. At initialization the early and middle layers of the warmed-up model write very little to the residual stream (Fig. 2a). The fc1 pre-activations are strongly negative (Fig. 2c), so GELU outputs and gradients are close to zero, as the authors also note in Section 3.3. At the start of training these layers therefore do almost nothing, and the network behaves like a shallower one. This could explain the low logit-lens accuracy of early layers more simply than "delayed class decodability", and it connects the paper to near-identity residual initializations, which are not discussed.

Questions for authors
  1. Could the low logit-lens accuracy of early layers be explained by these layers being close to the identity at initialization (low residual writes, suppressed MLPs), instead of by delayed class information?

  2. In Section 2.2 you train the warmed-up models with an auxiliary classification loss, which applies the final LayerNorm and classification head to the cls token of layer 7 and adds it to the standard loss with weight 0.3 during the first 50 epochs. Why layer 7, weight 0.3 and 50 epochs? Does the conclusion that earlier decodability hurts final accuracy still hold for other layers (e.g. layer 10), other weights, and over several seeds?

  3. Did you try to tune or vary the combined initialization? I would find the following experiments useful:

  • different split points between the early-layer and late-layer interventions;
  • early and late targets taken from different procedural variants (e.g. early from k-Dyck D4, late from k-Dyck Shuffled D98);
4 comments

We thank the reviewer for the thorough review and the constructive criticisms. Below, we address the remaining points.

Copying the properties of procedural warm-up and getting similar behaviour is somewhat expected (W1)

We think that it is far from obvious which of the many possible weight- and activation-level statistics and properties are responsible for the performance improvements. Indeed, for brevity and clarity of the paper, we omitted numerous unsuccessful properties that correlated with the improvements but were non-causal. Further, for example, Shinnick et al. (https://arxiv.org/abs/2511.13945) hypothesized that pre-pretraining instills an “abstract computational prior” and further claimed that useful information is “encoded in precise weight structure of both attention and MLP components” (Section 4.4 in Shinnick et al.). Our results challenge these hypotheses. Instead, we find that few weight- and activation-level statistics are sufficient to reproduce the performance improvements.

The experiments do not separate what is specific to procedural warm-up from a generic re-tuning of the initialization [...] a simpler change might work as well (W1)

We believe this is precisely one of the core takeaways of our paper! Our results suggest that re-tuning the initialization, as demonstrated by our interventions in Section 3, can capture the benefits of procedural warm-up. We hope these findings inspire future work to revisit how models are initialized. As the reviewer points out, initialization is typically viewed only as a means of stabilizing optimization. However, we find that it can have lasting effects. The properties we identify offer a promising starting point for designing better and simpler initializations (inferences from Figure 2): large residual writes in the first and final layers, low attention entropy throughout the network, negative FC1 pre-activations, and effective weight scales approximately 1.5× larger than those typically used. However, we leave further investigation for future work.

train randomly initialized models with smaller or larger initial weights (e.g. a different standard deviation), without copying anything from the warmed-up model (W1)

In our reply to reviewer Hw2f, we did a closely related experiment in which we extracted per-parameter mean and standard deviation statistics from warmed-up models and used them to randomly initialize a new model. However, this recovered only 56-61% of the performance improvement (Figure 18). In contrast, our interventions recovered the full gain and even surpassed the performance of procedural warm-up (Figure 3). While this still requires the warmed-up model, we hope this experiment addresses the spirit of the reviewer’s suggested experiment.

The combination of interventions is underexplored (W2, Q3)

We now report the combined intervention across three seeds (Figure 3). It achieves 80.43 ± 0.05% ImageNet top-1 accuracy, compared with 80.09 ± 0.12% for k-Dyck D4 warm-up and 80.35 ± 0.09% for the early intervention alone. The gains over the early intervention alone are larger on ImageNet-R and ObjectNet. We also varied which blocks receive each intervention (Q3). All tested allocations improved over random initialization (Appendix N, Table 11). These experiments use k-Dyck D4 targets throughout. Whether early and late targets from different warm-up variants work well together remains open.

Possible different explanation of the low decodability of early layers (W3, Q1)

We agree that near-identity behavior in the early blocks is a plausible explanation for their low logit-lens accuracy at initialization. However, our observation extends into training: early and middle layer readout accuracy remains low after procedural warm-up (Figure 1a), and several early blocks still contribute less to the residual stream after training (Figure 12). Across the early-block ablations (Table 9), block 7 logit-lens accuracy was strongly negatively correlated with final accuracy throughout training (Figure 17). Regarding near-identity initialization (Q1): We trained models with LayerScale with scales set to 1e-4 (as in DeiT-III). However, this yields lower ImageNet top-1 accuracy than random initialization (Figure 18) and shows a weaker effect on delayed class decodability (Figure 19).

Choice of auxiliary-loss settings and robustness (Q2)

We chose block 7 because it shows a pronounced early gap in class readout between random initialization and procedural warm-up (Figure 1a). Applying the auxiliary loss during the first 50 epochs covers this early divergence while letting both conditions finish training under the same objective. The exact cutoff and the weight of 0.3 were pragmatic choices and not optimized. We suspect that we could achieve even sharper effects with larger weights or later cutoff, but find that this wouldn’t add much additional insight. We used two complementary interventions to test whether the depth of class readout matters for final accuracy. Encouraging earlier readout reduced final accuracy for both warm-up variants (Section 2.2). Restricting direct readout to the last three blocks improved final accuracy by 1.3 pp in a randomly initialized ViT (App. D).

I thank the authors for the detailed response. The results on the combined intervention with three seeds and the analysis of the block split points address my concerns.

On W1, the baseline I suggested (changing the standard deviation of the random initialization without using the warmed-up model) is still missing, since the Mean+SD initialization still uses statistics from the warm-up. I recommend expanding the discussion of the combined intervention in the paper, since in my opinion it is the most interesting contribution.

I keep my score.

Thanks for your reply! We are glad that we were able to address most of your concerns.

Regarding W1: Following your suggestion, we ran additional experiments in which we varied the standard deviation of random initialization, without using the warmed-up model. Due to time and computational constraints, we were able to evaluate standard deviations of 0.01 (0.5x of the standard deviation used in all previous experiments) and 0.04 (2x) with one seed each. We also tested 0.08, but training diverged.

The table below shows that neither decreasing or increasing the standard deviation improves performance over the standard initialization. This suggests that simply adjusting the scale of randomly initialized weights is insufficient to reproduce the performance gains. Instead, it supports the importance of the properties of the warmed-up models that we identified.

Standard deviationImageNet top-1 accuracy
0.01 (0.5x)77.77
0.02 (1x, standard)78.28/78.04/77.91
0.04 (2x)78.07
0.08 (4x)diverged
Reviewer_xqLMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds off recent work of Shinnick et al that applies procedural warm up for vision transformers, using procedurally generated data with no vision contents, to obtain better performance on vision tasks. The authors of this paper seek to understand which changes to initialization leads to such results. They empirically show that they are able to match and sometimes surpass results of Shinnick et al., with an initialization based on statistics of the weights and activations. In this way they show that such an inductive bias can be brought about for ViTs directly through initialization.

Critical strengths

The paper is well written and the authors do a good job explaining the motivation of their work. The experimental analysis is also presented well.

Critical weaknesses
  1. The major issue I had with the paper is that it didn't really give strong insight for why either using procedurally generated warmup or an initialization built to mimic that does so well. Shinnick et al., have already shown that procedural warm up offers benefit even though it has no vision contents. The authors of this paper then seek to build an initialization that mimics that ability but there is a real lack of explanation as to why this type of approach works so well. Furthermore, the building of this initialization requires an already pretrained model so that they can look at the weight and activation statistics.

  2. While Shinnick et al., worked purely on vision transformers it would have been good if you pushed the exploration a bit and considered other modalities.

Questions for authors
  1. Can you provide some insight into why having such a vision content free procedural warm up phase or one built into the network at initialization offers better generalization performance?

  2. Can you offer more insight into how exactly your initialization affects the optimization trajectory?

3 comments

We thank the reviewer for taking the time to review and the constructive feedback. We address the remaining concerns and questions below.

Why the initialization helps and how it changes training (W1, Q1, Q2)

We first emphasize that our primary goal is to distill the procedural pre-pretraining approach of Shinnick et al. into its core initialization properties. By doing so, we challenge their hypothesis that pre-pretraining instills an “abstract computational prior” and instead demonstrate that simple initialization properties (Sections 3.1 & 3.2) can account for the performance gains. We complement this main result with further analysis in Section 2.2 that links these performance improvements to a lasting regularization effect. For example, Figure 1c shows that procedurally warmed-up models and our reconstructions memorize fewer randomly corrupted labels, following the standard protocol of Zhang et al. (https://arxiv.org/abs/1611.03530). Moreover, we consistently observed delayed class decodability in warmed-up models and our reconstructions (Figures 1a, 1b, 14a, 15a, 17a): class readout accuracy remains low in early and middle blocks throughout training, whereas randomly initialized models have higher accuracy already in intermediate layers. To investigate this class decodability observation further, we explicitly encouraged early class decodability (at block 7) by applying an auxiliary intermediate classification loss during the first 50 epochs in procedurally warmed-up models. While this improves performance at ⅓ of training (+1.49 pp for k-Dyck), it ultimately ends up deteriorating performance by 1.58 pp. Conversely, restricting the classifier’s direct readout to the contributions of the final 3 blocks improves accuracy by 1.3 pp in a randomly initialized model (Appendix D). Together, these findings support our hypothesis that the procedural initialization induces a lasting regularization effect, where the delayed class decodability is downstream to the identified initialization properties (Section 3).

building of this initialization requires an already pretrained model so that they can look at the weight and activation statistics (W1)

As mentioned above, our primary goal is to identify and validate the core initialization properties carried by procedurally pre-pretrained weights that account for the performance improvements. However, we have also conducted preliminary experiments showing that these properties can be implemented without relying on these weights (see Appendix L for late-layer residual writes scaling). We believe the remaining properties can similarly be implemented, as they seem to have remarkable consistent patterns:

  • Residual writes: large in the first and final layers (Figure 2a and Appendix L),
  • Attention entropy: low throughout most layers (Figure 2b),
  • fc1 pre-activations: negative with a mean of ca. 2-3 (Figure 2c), and
  • Effective weight scales: ca. 1.5x larger than originally initialized (Figure 2d).

Results beyond vision (W2)

As suggested, we extended our analysis beyond vision to language modeling (Appendix P). Concretely, following a concurrent submission (https://openreview.net/forum?id=P8cMuhbbfz), we first pre-pretrained a 500M-parameter transformer on shuffled k-Dyck sequences and subsequently trained it on natural language texts from the C4 dataset (https://arxiv.org/abs/1910.10683) using the popular next-token prediction objective. We then extracted the residual write ratios from all attention and MLP layers and transferred them to a randomly initialized model, following the procedure described in our Section 3.1.

The table below shows that transferring the residual write ratios recovers ca. 65% of the performance improvement achieved by pre-pretraining. These preliminary results suggest that the benefits of pre-pretraining may not come from learning computational inductive biases (e.g., grammatical priors), but rather from setting favorable initialization properties, consistent with our findings in vision. This result may also complement the explanation of the concurrent submission’s observation that pre-pretraining strengthens long-range dependency modeling that might be downstream of the initialization-related mechanisms we identified.

initializationloss
Random init3.029
PW - k-Dyck shuffled2.950
Random init w/ PW residual write ratios2.978

Thank you very much for your rebuttal and the responses for my queries. I have read the paper a few more times in light of the rebuttal and still don't feel the paper is at the standard of ICLR. While the results and analysis are interesting the paper doesn't give much insight into the authors' initialization. Furthermore, given the work of Shinnick et al., I believe this paper makes an incremental advance. Therefore, I unfortunately cannot recommend this paper for publication and will vote for rejection.

Thank you for your continued engagement. We respect the reviewer’s assessment, but would like to clarify two points, as we believe some of our key contributions may not have come across clearly.

  • Regarding lack of insight: We believe that identifying the specific initialization properties responsible for the performance gains from procedural pre-pretraining is itself an important step toward understanding the underlying mechanisms. Our findings challenge the hypothesis of Shinnick et al. that these improvements come from learned computational priors. Instead, we demonstrate that a small set of relatively simple initialization properties is sufficient to reproduce, and sometimes surpass, the gains from procedural pre-pretraining. We believe that identifying these properties and challenging the proposed explanation of prior work offers an important insight for the pre-pretraining field, and possibly also other communities. Furthermore, our experiments on label memorization, class decodability, and intermediate supervision provide evidence linking these initialization properties to persistent regularization effects. We believe that this provides a new perspective on the origins of the benefits of pre-pretraining, adding to the field’s current explanation (i.e., learned task-specific computational priors) with this initialization-induced regularization. We’d be happy to hear from the reviewer’s perspective on these (and in particular the latter) findings, which haven’t been explicitly discussed in their comments.
  • Regarding incremental nature: We respectfully disagree that our work is merely an incremental extension of Shinnick et al. While their work establishes that procedural pre-pretraining benefits vision transformers, ours investigates what drives these improvements. By identifying and reconstructing the initialization properties that account for the performance gains, we go beyond demonstrating the phenomenon to isolating its underlying factors. This not only challenges the interpretation of the original work (and most of the pre-pretraining field) but also raises the more fundamental question of whether procedural pre-pretraining is necessary to obtain these benefits.
Reviewer_Hw2fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses image classification with Vision Transformers (ViTs) trained from scratch, and asks why "procedural warm-up" (Shinnick et al., 2026), i.e. first training the transformer on bracket sequences from formal languages, improves ImageNet accuracy by about 2 points. The authors show that warm-up has lasting effects on training: when the final classifier is applied to each layer (the "logit lens"), warmed-up models predict the class accurately only in the last few layers, and they memorize fewer randomly flipped labels. They then measure layerwise statistics that warm-up changes at initialization, such as weight scales, attention entropy and how much late layers write to the residual stream. Imposing these 108 statistics on a random initialization recovers, and sometimes slightly exceeds, the gain of warm-up for ViT-B on ImageNet-1k, and reproduces the same training behavior.

Critical strengths

S1. A clear and interesting question. Why training on brackets helps image classification is a puzzling result, and the paper turns it into a testable hypothesis: the gain comes from a few initialization statistics rather than from something learned on the bracket task that carries over to images.

S2. Simple interventions recover the gain. Imposing a few statistics on a random initialization, without any warm-up, recovers the warm-up gain (Fig. 3). Rescaling the last three layers alone helps consistently, for all eight warm-up variants with ViT-S (Fig. 11).

S3. Interesting analysis of training dynamics. Several experiments give a consistent picture: warm-up delays the layer at which the class becomes predictable and reduces the memorization of wrong labels, and forcing earlier prediction seems to lower final accuracy (Sec. 2.2).

Critical weaknesses

W1. The main finding overlaps with prior work, and simple baselines are missing. The central claim that the gain from synthetic pretraining comes from a few weight statistics has been shown before:

  • (a) Wu et al. (NeurIPS 2022, "Insights into Pre-training via Simpler Synthetic Tasks") reach the same conclusion for language models: initializing each weight tensor with the mean and standard deviation of a synthetically pretrained model recovers most of the gain (Sec. 5.1, Table 2). They even single out the same statistic, lower LayerNorm gains (Sec. 5.3; cf. l. 292–293 here).
  • (b) The paper's own ablation points the same way: weight scales and LayerNorm statistics alone give +1.82 of the +2.00 points of warm-up, and the attention-entropy and MLP terms highlighted in the abstract add nothing when added separately (Table 9).
  • (c) Mimetic initialization (Trockman & Kolter, 2023) follows the same idea for ViTs: it replaces pretraining with an initialization that mimics pretrained attention weights, and already gives +1.19 of the +1.72 points of warm-up (ViT-B on ImageNet-1k, in Shinnick et al.'s setup, Table 2).

Wu et al. are not cited, although Shinnick et al. cite them, and mimetic initialization is mentioned (l. 448) but not compared. Please discuss these works and add both as baselines.

W2. The main property the paper builds on is hard to interpret. Warmed-up models predict the class only in their last layers, and the paper presents this as part of its explanation of the gain (Sec. 2.2). Yet the early- and late-layer interventions both reproduce this (App. J), while the late-layer one is worse on all seven benchmarks (Fig. 3), so this property does not reliably predict performance. As a result, it is difficult to understand what this observation teaches us, which limits the reach of the paper (see Q1).

W3. Training from scratch limits practical applicability. The paper shows that initialization can have a lasting effect when ViTs are trained from scratch. However, this is a fairly specific setting: in current practice, supervised training is often performed by fine-tuning a pretrained ViT rather than starting from random initialization. This limits how broadly the reported benefits can be interpreted, and the paper should discuss more clearly why the from-scratch setting remains relevant.

Questions for authors

Q1. What does delayed class decodability teach us (W2)? Is it a cause of the better generalization, or a side effect of the changed weight scales? For instance, does initializing with the warmed-up weight scales and LayerNorm statistics alone (Table 9) already reproduce the delayed profile?

Q2. The weight-scale and LayerNorm component of the early-layer intervention (Table 9) is close to Wu et al.'s per-tensor mean/std initialization (W1), but covers only the query, key and first MLP weights of layers 0–7. How does Wu et al.'s initialization, applied to all weights of the warmed-up blocks, compare with your combined intervention (Fig. 3)? This would show what the activation-level targets add beyond per-tensor statistics.

Q3. The attention-entropy intervention creates an attention sink: all queries attend to a few common keys (Fig. 16). Does the attention of warmed-up models look like this, or does the intervention only match the average entropy with a different pattern?

Q4. Minor: the paper needs proofreading (e.g. l. 58–59, 89, 250, 268, 310, 404, 424, 432).

3 comments

We thank the reviewer for their thorough and constructive feedback. Below, we address remaining concerns and questions.

Overlap with prior work and missing baselines (W1, Q2)

Thanks for pointing us to Wu et al. We agree that their work is related. As suggested in W1a and Q2, we evaluated Wu et al.’s all-tensor (“Per_Param_Mean_SD”) and LayerNorm-only (“Pre-attn_LN_Per_Param_Scale”) variants. Since Wu et al. also apply parameter-statistics matching to the final normalization layer, whereas our interventions do not, we evaluated two versions of their all-tensor approach: one applied only to the transformer blocks and another including the final normalization layer. The newly added Figure 18 and the table below show that these approaches recover only 56-61% of the performance gain achieved by procedural warm-up. This suggests that the activation-level targets identified in our work capture properties beyond those reflected in per-tensor parameter statistics.

Furthermore, while matching weight scales and LayerNorm statistics recovers most of the gain for k-Dyck in Table 9, as noted in W1b, we found that this doesn’t hold for shuffled k-Dyck. In this case, matching these statistics recovers only +0.40 pp of the +1.94 pp improvement (now also reported in Table 9). This further supports that additional interventions are needed.

As suggested in W1c, we also added a comparison with mimetic initialization to Figure 18 and the table below. We find that mimetic initialization achieves performance comparable to procedural warm-up, in contrast to Table 2 of Shinnick et al. (https://arxiv.org/abs/2511.13945), which reports that “procedural warm-up produces larger improvements.” We emphasize, however, that the goal of Figures 3 and 18 is not to identify the best initialization method, which is beyond the scope of this paper, but rather to investigate whether the properties identified in Sections 3.1 and 3.2 can be combined.

modeltop-1 (3 seeds)vs. random init
Random init78.08 ± 0.19
k-Dyck D480.09 ± 0.12+2.01
Random init w/ PW residual write ratios80.00 ± 0.14+1.92
Random init w/ PW early-layer targets80.35 ± 0.09+2.28
Combined80.43 ± 0.05+2.35
Mimetic init80.34 ± 0.20+2.26
LayerScale77.76 ± 0.19−0.31
Mean+SD init (blocks)79.20 ± 0.11+1.13
Mean+SD init (blocks + final norm)79.31 ± 0.08+1.23
Pre-attention LN gain init77.46 ± 0.77−0.62

On delayed class decodability (W2, Q1)

We clarified the hypothesized link between delayed class decodability and final accuracy. Both the early- and late-layer interventions shift class readout toward later blocks and improve final accuracy over random initialization, although the gains differ in magnitude. Figure 17 in the revised manuscript shows that models with lower logit-lens accuracy at block 7 tend to achieve higher final accuracy. This negative correlation persists across the training checkpoints we evaluated. At epoch 29, for example, Spearman’s rank correlation was −0.90.

From scratch training limits practical applicability (W3)

We respectfully disagree that the focus on training from scratch limits the practical relevance of our findings. While fine-tuning off-the-shelf models is often effective for downstream tasks, improving the pretraining of vision models remains important to this date. We therefore believe that understanding how initialization affects final performance and generalization remains a valuable research direction.

Further details on attention entropy intervention (Q3)

We matched mean attention entropy. In our intervention, different heads concentrate attention on the same keys, whereas in warmed-up models, different heads typically concentrate attention on different keys. Despite this difference, we find that the intervention recovers the performance benefits, suggesting that matching attention concentration is sufficient without reproducing the precise head-specific attention patterns. We have clarified this and further discussed this in the newly added paragraph (highlighted in blue) in Appendix K.2.

Proofreading (Q4)

Thanks for pointing these out! We’ve corrected them in the revised manuscript.

I thank the authors for the new experiments and clarifications, which address part of my concerns.

W1. The new baselines address my main concern on novelty. The new results show that Wu et al.'s per-tensor statistics recover only part of the gain (+1.1 vs. +2.4 for the combined intervention), and that weight scales and LayerNorm statistics alone are not always enough (+0.4 of +1.9 with shuffled brackets, Table 9).

W2. The new analyses show a consistent link between delayed class decodability and accuracy across initializations, including mimetic initialization (Figs. 17 and 19), but the reason for this link is unclear. The paper suggests that early decodability acts as a shortcut that helps early training but hurts later (l. 172–185), but gives no mechanism for why it would hurt or act as a regularizer. Since delayed decodability looks like a rather arbitrary property, an intuition for this would make the experimental evidence more convincing.

W3. The relevance for pretraining remains unclear. The authors argue that improving pretraining matters, but the study is limited to classification training (as acknowledged in the Limitations), while large-scale vision pretraining mostly relies on self-supervision or image-text pairs. App. P recovers part of the warm-up gain for a language model, but it is unclear whether the proposed initialization would help such vision models, and whether the findings on intermediate class decodability would still hold.

Overall, the new baselines address my main concern on novelty, and the new analyses support a link between delayed class decodability and accuracy. However, the intuition behind these findings and their scope remain unclear, and I would welcome the authors' view on these points (W2, W3).

We thank the reviewer for their reply and are happy to hear that we could already address some of their concerns.

Intuition for why early decodability might hurt generalization (W2)

Our intuition is that a model may use less of its depth if its first several layers already fit the training data well. Consider if a middle-layer can already predict the correct classes with high confidence. Further, say the later layers have near-zero contribution. Then the model already achieves a low training loss and there’s not much pressure to refine it further. The model might be even inclined to “silence” the later layers, as they might disturb the already-working prediction on the training examples. This could hurt generalization in two ways: First, the middle-layer features the model relied on for the training examples might be less reliable on unseen examples. Second, if the later layers learn to correct the few remaining training errors, those corrections may be too specific to those training examples. In that sense, early decodability could indicate that the model relies on a more “shallow” model.

Relevance of pretraining (W3)

We acknowledge that the paper doesn’t cover self-supervised or image-text contrastive training. We observed benefits from procedural pre-pretraining in image classification (and language modeling, added during this rebuttal). In earlier experiments before the original submission, we also initialized CLIP’s vision encoder with the weights from procedural pre-pretraining, but saw no improvement over random initialization (see table below). Since our primary goal was to understand where the observed gains come from, we focused our analysis on the setting in which we saw them.

The reviewer points indeed to an interesting related question: when does procedural warm-up help? Even within pre-pretraining for language modeling, two concurrent ICLR submissions report mixed results: One submission (https://openreview.net/forum?id=73o2bc3VP2) found no gain at matched compute, while another (https://openreview.net/forum?id=P8cMuhbbfz) found gains persisted at scale. We leave this question however for future work.

CC3M

MetricRandomProceduralΔ
ImageNet 0-shot top-114.510.6−3.9
COCO img->txt R@112.47.1−5.3
COCO txt->img R@19.35.6−3.7

CC12M

MetricRandomProceduralΔ
ImageNet 0-shot top-135.432.7−2.7
COCO img->txt R@133.430.7−2.7
COCO txt->img R@122.020.2−1.8

Comments on the paper

1 comment
AuthorsOpenReview ↗

General reply   Change log

We thank the reviewers for their constructive feedback, which helped us improve the paper.

Reviewers highlighted the paper’s “clear and interesting question” (Hw2f), described it as “well written,” and appreciated that “the authors do a good job explaining the motivation of their work” (xqLM). They liked that we addressed our core question “with interventions and not only with correlations” (sGaT), provided an “interesting analysis of training dynamics” (Hw2f), and presented findings that are “interesting for the community also outside procedural warm-up” (sGaT). In response to the reviews, we improved the paper as follows (all references refer to the revision):

  • Added missing seeds for the main experiments on ImageNet, incl. Figure 3.
  • Extended the analysis in Appendix M with experiments for the choice of MLP intervention (rank-one component vs. constant bias offset, Table 12).
  • Added analysis on the choice of layer split point of early- and late-layer interventions in Appendix N.
  • Additional discussion of the choice of attention intervention in Appendix K.2.
  • Added logit lens analysis for the early layer intervention ablation in Appendix M (Figure 17).
  • Added additional baselines (mimetic initialization (https://arxiv.org/abs/2305.09828), layer scale, and three parameter statistic copying variants from Wu et al. (https://arxiv.org/abs/2206.10139)) to Figure 3. For clarity in the main text, we added the variants in the same-style figure in Appendix O (Figure 18).
  • Added experiments on language modeling in Appendix P.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.