- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tries to explain why procedural warm-up (training a ViT on Dyck-language sequences before image training) improves image classification. First the authors show that warmed-up models make classes decodable only in the last layers and memorize fewer corrupted labels, and they read this as a regularizing effect. Then they compare the warmed-up weights with a random initialization and find a few differences: late layers write more to the residual stream, while early and middle layers have sharper attention, suppressed MLPs and different weight and LayerNorm scales. To test whether these differences explain the gain, they impose them on a randomly initialized ViT with two interventions. The late-layer intervention rescales some weights of the last layers so that they write as much to the residual stream as in the warmed-up model. The early-layer intervention modifies the weights and LayerNorm parameters of the early and middle layers to match their attention sharpness, MLP activations and scales. Overall, imposing these 108 scalar statistics recovers most or all of the gain of procedural warm-up on ImageNet and on several distribution shifts. Combining the two interventions gives a model that is on par with or slightly better than procedural warm-up, and that also memorizes fewer corrupted labels.
Critical strengths
-
Clear question and experimental design. The paper asks a concrete question, what procedural warm-up actually changes in the network, and tries to answer it with interventions and not only with correlations. I appreciated several choices: the hybrid models (random early layers, warmed-up late layers) isolate which part of the network carries the gain. The residual write ratios are tested in both directions: scaling a random model up to the procedural ratios and scaling a hybrid model down to the random ratios. The rescaling only touches , and , so that the output scale changes without changing the attention sharpness, and the effect of the larger writes is separated from the smaller relative update size by adjusting the learning rate. The early-layer components are ablated one by one , and the late-layer results are repeated on ViT-S over eight procedural variants (Figure 11). The modified models are also checked to reproduce the logit-lens and write-ratio profiles of procedural warm-up, and they are evaluated on six distribution shifts and not only on ImageNet. The procedures are described precisely, with pseudocode, which makes the work easy to follow and to reproduce.
-
Interesting message on initialization. Initialization is usually designed only to make optimization stable. This paper shows that a few scalar statistics set at initialization can change where learning happens across the depth of the network, and that this effect is still visible after 300 epochs, on distribution shifts and on label memorization. It also has a practical side: 108 scalars are enough to obtain most of the benefit of a full procedural warm-up, which needs all the parameters of the model, and the preliminary result in Appendix L suggests that some of these targets could even be set by hand. This could lead to a simple new initialization scheme, and it is interesting for the community also outside procedural warm-up.
Critical weaknesses
-
Copying the properties of procedural warm-up and getting similar behaviour is somewhat expected. The main contribution measures statistics of the warmed-up model and imposes the same statistics on a random model, so in my opinion it is not very surprising that the modified model behaves like the warmed-up one. More importantly, the experiments do not separate what is specific to procedural warm-up from a generic re-tuning of the initialization. Setting all late-layer write ratios to an arbitrary 1.4 gives almost the same accuracy as the procedural ratios (79.69 vs 79.99, Table 8). The effective weight scales and LayerNorm statistics alone give +1.8 of the +2.3 pp (Table 9). These results show that changing the initialization helps, but not that the specific properties of procedural warm-up are needed: a simpler change might work as well. To check this, the authors could train randomly initialized models with smaller or larger initial weights (e.g. a different standard deviation), without copying anything from the warmed-up model. If this also gives a similar gain, the improvement comes from better initial weight scales in general and not from properties specific to procedural warm-up.
-
The combination of interventions is underexplored. For me the most interesting result is Section 3.4, where combining the early-layer and late-layer interventions gives a model on par with or better than procedural warm-up. This suggests the two interventions may capture different properties, and with some tuning (of the target statistics, of the split layer, or by mixing targets from different procedural variants) the combined initialization could outperform both. But this part is a single short paragraph, tested with one procedural variant and one seed, and the gap over procedural warm-up (80.3 vs 80) is too small to conclude anything. Most of the paper is spent on matching single properties, which is the less surprising part.
-
Possible different explanation of the low decodability of early layers. At initialization the early and middle layers of the warmed-up model write very little to the residual stream (Fig. 2a). The fc1 pre-activations are strongly negative (Fig. 2c), so GELU outputs and gradients are close to zero, as the authors also note in Section 3.3. At the start of training these layers therefore do almost nothing, and the network behaves like a shallower one. This could explain the low logit-lens accuracy of early layers more simply than "delayed class decodability", and it connects the paper to near-identity residual initializations, which are not discussed.
Questions for authors
-
Could the low logit-lens accuracy of early layers be explained by these layers being close to the identity at initialization (low residual writes, suppressed MLPs), instead of by delayed class information?
-
In Section 2.2 you train the warmed-up models with an auxiliary classification loss, which applies the final LayerNorm and classification head to the cls token of layer 7 and adds it to the standard loss with weight 0.3 during the first 50 epochs. Why layer 7, weight 0.3 and 50 epochs? Does the conclusion that earlier decodability hurts final accuracy still hold for other layers (e.g. layer 10), other weights, and over several seeds?
-
Did you try to tune or vary the combined initialization? I would find the following experiments useful:
- different split points between the early-layer and late-layer interventions;
- early and late targets taken from different procedural variants (e.g. early from k-Dyck D4, late from k-Dyck Shuffled D98);
4 comments
Choice of auxiliary-loss settings and robustness (Q2)
We chose block 7 because it shows a pronounced early gap in class readout between random initialization and procedural warm-up (Figure 1a). Applying the auxiliary loss during the first 50 epochs covers this early divergence while letting both conditions finish training under the same objective. The exact cutoff and the weight of 0.3 were pragmatic choices and not optimized. We suspect that we could achieve even sharper effects with larger weights or later cutoff, but find that this wouldn’t add much additional insight. We used two complementary interventions to test whether the depth of class readout matters for final accuracy. Encouraging earlier readout reduced final accuracy for both warm-up variants (Section 2.2). Restricting direct readout to the last three blocks improved final accuracy by 1.3 pp in a randomly initialized ViT (App. D).
I thank the authors for the detailed response. The results on the combined intervention with three seeds and the analysis of the block split points address my concerns.
On W1, the baseline I suggested (changing the standard deviation of the random initialization without using the warmed-up model) is still missing, since the Mean+SD initialization still uses statistics from the warm-up. I recommend expanding the discussion of the combined intervention in the paper, since in my opinion it is the most interesting contribution.
I keep my score.
Thanks for your reply! We are glad that we were able to address most of your concerns.
Regarding W1: Following your suggestion, we ran additional experiments in which we varied the standard deviation of random initialization, without using the warmed-up model. Due to time and computational constraints, we were able to evaluate standard deviations of 0.01 (0.5x of the standard deviation used in all previous experiments) and 0.04 (2x) with one seed each. We also tested 0.08, but training diverged.
The table below shows that neither decreasing or increasing the standard deviation improves performance over the standard initialization. This suggests that simply adjusting the scale of randomly initialized weights is insufficient to reproduce the performance gains. Instead, it supports the importance of the properties of the warmed-up models that we identified.
| Standard deviation | ImageNet top-1 accuracy |
|---|---|
| 0.01 (0.5x) | 77.77 |
| 0.02 (1x, standard) | 78.28/78.04/77.91 |
| 0.04 (2x) | 78.07 |
| 0.08 (4x) | diverged |
We thank the reviewer for the thorough review and the constructive criticisms. Below, we address the remaining points.
We think that it is far from obvious which of the many possible weight- and activation-level statistics and properties are responsible for the performance improvements. Indeed, for brevity and clarity of the paper, we omitted numerous unsuccessful properties that correlated with the improvements but were non-causal. Further, for example, Shinnick et al. (https://arxiv.org/abs/2511.13945) hypothesized that pre-pretraining instills an “abstract computational prior” and further claimed that useful information is “encoded in precise weight structure of both attention and MLP components” (Section 4.4 in Shinnick et al.). Our results challenge these hypotheses. Instead, we find that few weight- and activation-level statistics are sufficient to reproduce the performance improvements.
We believe this is precisely one of the core takeaways of our paper! Our results suggest that re-tuning the initialization, as demonstrated by our interventions in Section 3, can capture the benefits of procedural warm-up. We hope these findings inspire future work to revisit how models are initialized. As the reviewer points out, initialization is typically viewed only as a means of stabilizing optimization. However, we find that it can have lasting effects. The properties we identify offer a promising starting point for designing better and simpler initializations (inferences from Figure 2): large residual writes in the first and final layers, low attention entropy throughout the network, negative FC1 pre-activations, and effective weight scales approximately 1.5× larger than those typically used. However, we leave further investigation for future work.
In our reply to reviewer Hw2f, we did a closely related experiment in which we extracted per-parameter mean and standard deviation statistics from warmed-up models and used them to randomly initialize a new model. However, this recovered only 56-61% of the performance improvement (Figure 18). In contrast, our interventions recovered the full gain and even surpassed the performance of procedural warm-up (Figure 3). While this still requires the warmed-up model, we hope this experiment addresses the spirit of the reviewer’s suggested experiment.
We now report the combined intervention across three seeds (Figure 3). It achieves 80.43 ± 0.05% ImageNet top-1 accuracy, compared with 80.09 ± 0.12% for k-Dyck D4 warm-up and 80.35 ± 0.09% for the early intervention alone. The gains over the early intervention alone are larger on ImageNet-R and ObjectNet. We also varied which blocks receive each intervention (Q3). All tested allocations improved over random initialization (Appendix N, Table 11). These experiments use k-Dyck D4 targets throughout. Whether early and late targets from different warm-up variants work well together remains open.
We agree that near-identity behavior in the early blocks is a plausible explanation for their low logit-lens accuracy at initialization. However, our observation extends into training: early and middle layer readout accuracy remains low after procedural warm-up (Figure 1a), and several early blocks still contribute less to the residual stream after training (Figure 12). Across the early-block ablations (Table 9), block 7 logit-lens accuracy was strongly negatively correlated with final accuracy throughout training (Figure 17). Regarding near-identity initialization (Q1): We trained models with LayerScale with scales set to 1e-4 (as in DeiT-III). However, this yields lower ImageNet top-1 accuracy than random initialization (Figure 18) and shows a weaker effect on delayed class decodability (Figure 19).