acceptodds
Under review as a conference paper at ICLR 2027

Copy the Same, Distill the Difference: Initializing Linear Vision Transformers

Abstract

Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they require from-scratch pre-training and typically underperform the original Softmax version. How to initialize linear ViTs both efficiently and effectively still remains unclear. In this work, we explicitly ask: given that most foundation ViTs are built on the mainstream Softmax attention, can linear ViTs benefit from their pre-trained weights? Recent works on Attention Transfer show that attention is the effective transferable component between Softmax ViTs, suggesting attention alone suffices for such reuse. However, we find the opposite for Softmax-to-linear transfer. The attention weights are operator-specific: copying them barely helps, and is sometimes even worse than random initialization. Instead, the attention's token routing behavior can be recovered through distillation with a proper loss design, letting linear ViTs reduce the gap and even match Softmax ones. In contrast, the MLP weights, which carry the learned representation, are operator-agnostic: they can be transferred by simple direct copying, which already carries most of the benefit of the pre-trained weights. Thus, copying MLPs can serve as an effective foundation for Softmax-to-linear transfer: paired with the distilled attention, linear ViTs eventually close the remaining gap and even surpass Softmax ones. These findings hold consistently across various linear ViT variants, different model sizes, and diverse datasets. We hope this study deepens the understanding of reusing pre-trained weights across attention operators: copy what stays the same and distill what differs, to recover the benefit across the Softmax-to-linear boundary.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Linear Vision Transformers, Linear Attention, Attention Transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YA6KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines which pretrained components remain useful when softmax attention in a ViT is replaced by a linear operator. Across five operator families, copying MLP weights accounts for most of the improvement over random initialization, while copying attention projections alone is much less effective. Distilling attention outputs further improves performance. The experiments cover several classification datasets, with additional detection and segmentation benchmarks.

Critical strengths
  1. For ReLU-Base on ImageNet, MLP copying raises accuracy from 76.5% to 80.2%, and FullCopy with distillation reaches 82.0%. Comparing these settings across several operators gives a clearer picture of transfer than evaluating only the final recipe.

  2. The result extends beyond small classification benchmarks. Tables D and E include COCO and ADE20K, and Table G checks the contribution of patch embeddings, positional embeddings and the class token. These experiments support the practical value of the study.

Critical weaknesses
  1. The operator-specific explanation is confounded with the transfer protocol. Li et al. (NeurIPS 2024), §2.2, inject teacher-computed attention maps throughout student training; this paper initializes projection weights and then trains them on the student's own representations. Appendix A acknowledges the difference, but §4.2 still presents the outcome as a reversal of softmax-to-softmax transfer. A softmax student under the identical weight-copy protocol is the missing control. Table G also shows conditional rather than uniformly ineffective transfer: copying attention changes 76.5% to 75.6% without MLP transfer, but 80.2% to 80.7% with it. A representative softmax/ReLU attention×MLP initialization comparison would test the operator-specific explanation without expanding the dataset suite.

    The TTT result has an additional initialization confound: Appendix B.2 leaves the wider output projection random, although its teacher-compatible submatrix can be copied with the extra columns initially zero. This preserves the output map on the main branch, not the whole TTT function; Net2Net provides the broader function-preserving transfer precedent. This control would refine the TTT-specific conclusion, but is not a prerequisite for crediting the ReLU/ELU evidence.

  2. Output matching does not establish recovery of token routing. Matching attention outputs constrains both aggregation and value representations. Distinct maps can satisfy A1V=A2VA_1V=A_2V, as the non-identifiability analysis of Brunner et al. (ICLR 2020), §3 makes clear. Appendix D.2 additionally describes this objective as directly targeting MLP-input mismatch, although that input depends on the residual stream and normalization. The successful loss is not in question; attention-output matching has an acknowledged precedent in LoLCATs. Narrowing the wording to functional output alignment would address this concern. If the mechanism is retained, measure the actual normalized MLP-input discrepancy on held-out examples, layer by layer.

Questions for authors

Does attention-only weight copying also fail in a softmax student with the same checkpoint, loading mask and training recipe? A representative softmax/ReLU attention×MLP comparison, with per-seed differences, would test the operator-specific claim.

Which output projections, biases and LayerNorm tensors are loaded in each setting? Where is the distillation target taken and how is MSE reduced? Please also identify the data split used to select the loss weight and checkpoints.

Is there evidence of reduced discrepancy at the actual normalized MLP inputs? If not, would you narrow the claim to attention-output alignment? This wording correction is sufficient for my current assessment; a new routing theory is not required.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YA6K (1/2)

We thank the reviewer for the review and the insightful questions. We would like to address your concerns below:

[W1 Q1] Softmax Attention Transfer under our Copy

We thank the reviewer for this precise suggestion. We are happy to provide the requested attention × MLP comparison for both Softmax and ReLU students.

We would like to first note that as the reviewer has pointed out, we have listed the difference between our Copy and Attention Copy in [1] in Sec. A in the Appendix: Attention Copy asks whether the attention alone suffices to transfer a Softmax ViT, while we ask whether the Softmax attention weights can initialize a linear attention operator. Thus, during the downstream training, Attention Copy uses the frozen teacher's attention maps, while our Copy only initializes the attention weights, which are then fine-tuned.

To test whether the failure of Copy comes from the linear operator rather than from the protocol itself, we run the same comparison with a Softmax student under the identical protocol below, where every component of the Softmax model will be fine-tuned:

Acc. (Δ\Delta to Random)CopyFlowers (ViT-Small)Food (ViT-Small)ImageNet-1K (ViT-Base)
SoftmaxNone45.473.278.1
ReLUNone43.064.476.5
SoftmaxAttention67.5 (+22.1)78.7 (+5.5)80.1 (+2.0)
ReLUAttention37.5 (-5.5)65.9 (+1.5)75.6 (-0.9)
SoftmaxMLP67.8 (+22.4)79.5 (+6.3)80.8 (+2.7)
ReLUMLP67.3 (+24.3)78.8 (+14.4)80.2 (+3.7)
SoftmaxBoth91.0 (+45.6)83.8 (+10.6)81.9 (+3.8)

As shown, two trends hold under the identical protocol: (1) Attention-only copying helps the Softmax student consistently on all three datasets, following the same trend as MLP-only copying. In contrast, for the ReLU student, attention-only copying lowers the accuracy on Flowers and ImageNet-1K and gives only a marginal gain on Food. (2) MLP-only copying brings both students to nearly the same accuracy, as expected because MLP weights are operator-agnostic. Note that the larger MLP gains of the ReLU student are due to its lower Random baseline, and distilling the attention on top of the copied MLPs (our recipe) further raises the ReLU student to the Softmax level of the Softmax Both row.

Hence, the failure of attention-only copying comes from the linear operator rather than from the protocol itself. Together with the reviewer's observation from Tab. G that the copied attention does help the ReLU student once its MLPs are copied, this shows that the transfer of attention weights depends on both the operator and the MLP context. We will include the above discussion in the revised version.

[W1] Function-Preserving Initialization for TTT

We thank the reviewer for this suggestion, and we are happy to share the results on TTT with the suggested function-preserving initialization.

Specifically, to test whether the failure of Copy for TTT comes from its random output projection, we copy the teacher's output projection weights into the wider output projection of TTT, with the extra columns set to zero, which preserves the output map on the main branch. We compare it with Random and our Copy on all datasets below:

Acc.CIFAR-10CIFAR-100STL-10FoodFlowersPetsImageNet-1K (ViT-Base)
Random93.370.463.974.341.831.682.2
Copy (Ours)89.263.460.566.334.021.981.0
Copy (Projection-preserving)89.063.360.766.633.722.181.1
Δ\Delta to Copy (Ours)-0.2-0.1+0.2+0.3-0.3+0.2+0.1

As shown, copying the output projection changes the accuracy of Copy only marginally, with both Copy variants falling below Random on all datasets. Hence, the failure of Copy for TTT does not come from the random output projection, but from the linear operator itself, as discussed in [W1 Q1] above. We will add this control experiment in the revised version.

AuthorsOpenReview ↗

Response to Reviewer YA6K (2/2)

[W2 Q3] Functional Output Alignment Rather than Routing Recovery

We thank the reviewer for this insightful comment. We agree that functional output alignment is the more precise framing to describe the effect of our attention distillation, and we are happy to adopt it.

As the reviewer has pointed out, output matching constrains what the attention module computes rather than the attention map that computes it, since distinct maps can produce the same output [2], and in linear attention, distinct queries can even produce the same map and output [3]. This is also why the same output matching loss remains effective for MHLA and TTT, which never form an attention map. Consistently, our loss ablation in Sec. 5.4 shows that matching the attention outputs performs on par with or better than matching the attention maps for ReLU and ELU, and is always better for InLine. Overall, the output matching loss is the operator-general choice: available for every linear operator, and can achieve higher or on-par accuracy than map matching across datasets.

Regarding the normalized MLP inputs, since they also depend on the residual stream and the LayerNorm, we will present the decomposition in Sec. D.2 in the Appendix only as the motivation for where the loss is placed. Therefore, we agree to narrow the claim on the effect of attention distillation from recovery of token routing to functional output alignment, as the reviewer suggested. This narrowing does not change our contributions or results, but makes our findings more precise. We thank the reviewer again for the suggestion, and will update the framing in the revised version.

[Q2] More Details on Weight Loading, Attention Distillation Loss and Data Split

We thank the reviewer for asking for more details on our implementation, and we are happy to share them below:

ItemAnswer
CopyELU, ReLU, InLine and MHLA: the QKV and output projections with both weights and biases of every block. TTT: only the rows of the standard QKV with both weights and biases, since its output projection has a different shape.
MLPCopyMLP of every block with both weights and biases, while leaving all other components randomly initialized.
FullCopyAll pre-trained Softmax weights except the classification head. For components not in the Softmax model, we keep their default initializations.
Distillation TargetThe output of the attention module in each block, after the output projection and before the residual addition. For MHLA and TTT, which have no class token, only the patch tokens are matched when computing the loss.
MSE ComputationThe MSE between the teacher's and student's attention outputs, summed over the 12 blocks, then multiplied by λ\lambda.
Loss Weight λ\lambda50 for every operator, dataset and initialization setting, selected by the λ\lambda sweep in Sec. 5.4.
Data SplitTop-1 Accuracy (%) on the standard test split (validation set for ImageNet-1K), for every setting, including the baselines. The same split is used for selecting λ\lambda.

As shown, beyond the attention and MLP weights, only FullCopy loads the remaining components, including the LayerNorms, the patch embedding, the position embedding and the class token, and Tab. G shows that loading the latter three changes the accuracy only marginally. We will add the above discussions in the Appendix in the revised version.


We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Li et al. On the Surprising Effectiveness of Attention Transfer for Vision Transformers. NeurIPS 2024.

[2] Brunner et al. On Identifiability in Transformers. ICLR 2020.

[3] Han et al. Bridging the Divide: Reconsidering Softmax and Linear Attention. NeurIPS 2024.

Reviewer_FeQXOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to answer a question: what should we initialize or pretrain the linear transformer in order to maintain competitive performance against its softmax attention-based counterpart? The authors first raised an interesting argument that simply copying the softmax attention weights does not help and may even degrade performance on the linear transformer. And then, they argued that only preserving the MLP weights but distilling to get the attention weights is the correct way to do softmax to linear transformer transfer. To validate this, the authors have done substantial experiments across different datasets and models. In the supplementary material, the authors have also provided some interesting analysis and additional results.

Critical strengths
  1. This paper studies a very interesting and important topic: how should we initialize or pretrain the linear transformer to maintain competitive performance compared to the softmax-based attention transformer. And the findings are rather interesting too: the MLP weights should be preserved, and the attention weights should be distilled rather than copied.
  2. The authors have done relatively complete and thorough experiments and comparisons, including ablation studies and some theoretical analysis. Most of the results shown in the paper could back up the main claim.
  3. The paper writing is relatively clear and easy to follow.
Critical weaknesses
  1. Although the authors claimed that the quantitative results across different datasets were maintained, and that simply copying the attention weights did not help or even degrade the results, I think that we still need to discuss this under different dataset sizes. It is clear that in Figure 1, on tiny or small datasets, the claim stands, but when trained on relatively larger datasets such as ImageNet1K, simply copying the attention weights might not degrade the performance. For a complete comparison, the authors might need to include comparisons on even larger datasets, or a separate discussion should be made for different dataset sizes.
  2. The authors could provide complete comparison results on different linear ViT variants, especially in Table 1 and Table 2.
  3. The results in Figure 3 do not show very competitive performance on cifar-100 or flowers datasets. And using only distillation does not yield good results on flowers dataset. The authors have provided an intuitive explanation. However, based on the explanation, the authors could provide more analysis, and dataset size or feature differences should be considered as one possible reason for the performance degradation. Furthermore, more comparisons on larger datasets could be included to strengthen the paper's claim.
  4. I think some analysis in the complementary section D and ablation studies could be moved to the main paper.
Questions for authors

Please see the detailed comments above. My main concern is that not separating discussions/claims for different dataset sizes and the potential lack of larger-scale dataset testing might weaken the arguments in the paper. I would suggest that the authors reorganize the paper logic and try to make a smaller-range claim for the idea. For example, on relatively smaller-scale datasets, copying the same structure weights and distilling for the different structure weights helps the softmax to linear attention transfer. But on larger-scale datasets, we will have a different discussion. Another question is that the authors mentioned that matching the output rather than matching the attention map would yield better performance. However, I think this claim holds only under a specific dataset range or activation functions. For example, Figure 5 does show different performance behavior for different dataset sizes and activations. I think the authors should be more rigorous in this argument. Further analysis could be provided in the supplementary.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FeQX (1/2)

We thank the reviewer for the review and the insightful questions. We would like to address your concerns below:

[W1 Q1] Consistency Across Dataset Scales

We thank the reviewer for this constructive suggestion. We agree that the size of each effect depends on the dataset scale, and we are happy to organize our analysis according to different dataset scale categories.

We would like to first note that the reviewer's observation on ImageNet-1K is consistent with our claim that copying the attention weights barely helps, and is sometimes even worse than random initialization. Specifically, on ImageNet-1K (top row of Fig. 2), copying indeed does not degrade much: the gap between Copy and Random lies between -1.5% and +0.8% in all reported results in Fig. 2 (across five operators), with most of them being negative. However, what does not change with dataset scale is the failure of the expected advantage of Copy: when compared with the attention-only initializations that require no pre-trained weights (Random, Mimetic and Impulse), Copy never stands out on any of the 7 datasets.

We further group the 7 datasets by scale into 3 categories as in [1]: small-scale (Flowers, Pets, STL-10), medium-scale (CIFAR-10, CIFAR-100, Food) and large-scale (ImageNet-1K). For ease of comparison, we report the accuracy gap to Random per category below on ViT-Small. The conclusion above holds in every category.

Δ\Delta vs. Random (ViT-Small, five operators)Small-scaleMedium-scaleLarge-scale
Copy (attention only)-9.8 to -1.8-8.1 to +4.5-1.5 to +0.8
MLPCopy+13.6 to +29.8+1.8 to +20.9+0.7 to +3.2
MLPCopy w/ Distill+27.9 to +67.0+4.4 to +27.7+1.3 to +5.1

As shown, the dataset scale changes how much each component is worth, not which component transfers. At Small-scale, copying the attention alone hurts every operator on every dataset, and distillation adds about as much as the MLPCopy. At Medium-scale, attention-only copying brings no consistent gain over Random, while both MLPCopy and MLPCopy w/ Distill still add clear gains. At Large-scale, attention-only copying barely matters either way, as the reviewer observed, while MLPCopy and the full recipe both still improve over Random on every operator, with consistently positive margins.

We are therefore happy to report every effect per dataset scale, while, as noted above, the conclusion itself holds at every scale: attention-only copying brings no reliable advantage, MLPCopy helps on every dataset, and the recipe's ordering (Random < MLPCopy < MLPCopy w/ Distill) holds throughout, on the six transfer datasets at ViT-Small and on ImageNet-1K, COCO and ADE20K at ViT-Base.

Regarding even larger datasets, we agree that they strengthen this analysis, and we are planning to add iNaturalist 2021, which is about twice the scale of ImageNet-1K, under the same evaluation protocols. Due to the large training cost and the limited time, we will include them in the main results in the final version.

We will add the above discussion, especially the dataset scale-wise findings, in the revised version.

[W2] Additional Results of the Robustness Tests

We thank the reviewer for this suggestion to include the complete list of linear operators in the robustness tests of Tables 1 and 2. We are happy to provide them to further strengthen the robustness claim of our findings.

Below, we complete both tables under the same protocol as the existing entries.

RandomDeiTDINOMoCov3iBOTMAE
InLine78.477.8 (-0.6)77.3 (-1.1)77.7 (-0.7)77.4 (-1.0)78.1 (-0.3)
Δ\Delta Copy vs. RandomELUReLUInLineMHLATTT
IN-A+1.0-0.2+0.3+0.5-0.4
IN-R+0.7+0.1+0.2+0.1-0.1
IN-S+0.6+0.0+0.1+0.4-0.1
IN-V2+0.4-0.1+0.3+0.1-0.2

As shown, the new results follow the same pattern: with InLine, Copy stays below Random for all five pre-trained weights, and under distribution shift, Copy stays within the Random-init level across operators. This further confirms that attention-only copying never stands out in either setting. We will add the highlighted results in the revised version.

AuthorsOpenReview ↗

Response to Reviewer FeQX (2/2)

[W3] Dataset Scale vs. Feature Differences on CIFAR-100 and Flowers

We thank the reviewer for this suggestion. We agree that dataset scale is a key factor, as also discussed in [W1 Q1], and we add a control that separates it from the features of the dataset.

We would like to first note that the blue bars of Fig. 3 isolate attention distillation (Copy w/ Distill) to show what it recovers on its own, and the red bars add the copied MLPs (FullCopy w/ Distill), whereas our full recipe copies the MLPs but not the attention weights (MLPCopy w/ Distill).

To separate the number of training samples per class from the dataset itself, we build CIFAR-100†, a subset of CIFAR-100 with 10 randomly selected images per class (as in Flowers), and test whether the pattern on Flowers reappears:

Δ\Delta vs. Softmax (ViT-Small, ReLU)Random w/ DistillCopy w/ DistillMLPCopy w/ Distill (full recipe)
Flowers (Small-scale)-41.4-33.0+0.1
CIFAR-100 (Medium-scale)-5.9-1.6+0.1
CIFAR-100† (10 images per class)-39.4 ± 1.6-37.4 ± 1.1+0.4 ± 0.2

As shown, distillation alone falls far short on both Flowers and CIFAR-100†, where training samples per class are scarce, while it comes much closer to the Softmax model on the full CIFAR-100. Therefore, the main cause of the gap in Fig. 3 is the number of training samples per class rather than the features of Flowers. Together, these results support our explanation in Sec. 5.1 that, with few training samples, the task signal is too weak to relearn the representation that attention-only supervision cannot supply. Furthermore, copying the MLPs supplies this representation: our full recipe reaches the Softmax level on Flowers, CIFAR-100 and CIFAR-100†, and across the five operators as shown in Figs. A and B in the Appendix, it is at most 2% below the Softmax level, except ELU and TTT on Flowers at ViT-Small. We will add the above discussion together with the controlled experiment in the revised version.

[W4] Suggestions on Moving Analyses from Appendix to Main Text

We thank the reviewer for this suggestion, and we will use the additional page in the final version to include the key analyses and ablation studies, together with the modifications and refinements from all the reviewers.

[Q2] Loss Ablation Across Dataset Scales and Linear Operators

We thank the reviewer for this comment. We agree that the comparison depends on the operator and the dataset, and we are happy to state this claim more precisely. Specifically, comparing the best λ\lambda of each curve in Fig. 5:

Better LossReLU / ELUInLine
ImageNet-1K (Large-scale)on paroutput matching
CIFAR-100 (Medium-scale)on paroutput matching
Flowers (Small-scale)output matchingoutput matching

As shown, for linear operators where the attention maps are proper distributions (ReLU / ELU), the two loss choices are on par on both Medium- and Large-scale datasets and output matching is ahead on the Small-scale dataset. For linear operators where the map is a signed quasi-distribution (InLine), output matching is ahead on all three dataset scales.

Moreover, output matching is the only loss that applies to MHLA and TTT, which never form an attention map. More details are given in Sec. B.4 in the Appendix. Overall, the precise claim is thus that output matching is the operator-general choice: available for every linear operator, and using the output matching loss can achieve higher or on-par accuracy across dataset scales. We will include the above discussion in the revised version.


We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Zheng et al. Structured Initialization for Vision Transformers. NeurIPS 2025.

Reviewer_FeQXOpenReview ↗

Responce to the rebuttal

Thanks for the detailed comments. I have read all the reviewers' comments and the authors' replies. I think the authors' replies have already addressed many of my concerns. I encourage the authors to incorporate these suggestions and add some additional results to the main paper.

Dear Reviewer FeQX,

We are grateful to hear that our reply has addressed your concerns. We will incorporate all the discussions and clarifications in our revised version. Thank you again for your support, engagement and inspiring experiments during the rebuttal period. We wish you all the best!

Reviewer_U3vKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how to initialize Linear Vision Transformers (ViTs) from pretrained Softmax ViTs. Its central finding is that directly copying pretrained attention weights provides limited benefits, whereas copying MLP weights transfers useful representations more effectively. Building on this observation, the authors propose a hybrid initialization strategy that copies MLP weights and distills attention outputs from the pretrained teacher into the linear-attention student. Experiments across multiple linear-attention architectures, model scales, and downstream vision tasks demonstrate the effectiveness of the proposed strategy, suggesting that pretrained representations can be reused more effectively when parameter transfer is combined with functional distillation. Overall, the paper provides an empirical study of component-wise transferability in Vision Transformers and a practical approach to initializing Linear ViTs from pretrained models.

Critical strengths
  1. A useful and potentially generalizable empirical finding. The paper distinguishes the transferability of attention and MLP parameters when transitioning from Softmax to linear attention. This component-wise analysis provides a useful perspective on pretrained weight reuse and challenges the assumption that structurally corresponding parameters should always be directly transferred.

  2. A simple and well-motivated initialization strategy with broad empirical validation. Combining MLP weight copying with attention-output distillation is conceptually straightforward and avoids requiring the student to reproduce the teacher's internal attention maps. Evaluations across multiple linear-attention variants and downstream tasks strengthen the evidence that the approach is not limited to a single architecture or benchmark.

Critical weaknesses
  1. The underlying explanation may be insufficiently established. The observed difference between attention and MLP transferability is interesting, but it does not by itself establish why attention-weight copying fails. The performance gap could depend on differences in attention normalization, feature scaling, optimization dynamics, or architectural compatibility. Without sufficiently controlled analyses that distinguish these explanations, the paper's broader interpretation of component-wise transferability remains somewhat limited.

  2. The practical benefits and generality of the proposed approach require careful qualification. Attention-output distillation introduces additional training computation and requires access to a pretrained teacher. The paper's effectiveness as an initialization method therefore does not necessarily imply lower end-to-end training cost or better deployment efficiency. More importantly, the evidence from the evaluated vision architectures does not establish that the same transferability pattern holds for substantially different settings, such as video Transformers or diffusion Transformers. This limits the scope of the broader conclusions, although it does not invalidate the demonstrated results within the evaluated setting.

Questions for authors
  1. Mechanism of attention-weight transfer failure: Can the authors provide controlled experiments that disentangle the effects of attention normalization, output scale, and optimization dynamics? In particular, does attention-weight copying become more effective when the student and teacher attention outputs are appropriately calibrated before fine-tuning?

  2. Contribution of attention-output distillation: How much of the final improvement comes from MLP weight copying versus attention-output distillation? Are the gains consistent across the evaluated linear-attention variants, and do they remain when the distillation objective is removed after an initial adaptation stage?

  3. Training cost and practical utility: What is the additional training cost of the proposed method, including teacher inference, compared with standard initialization and training from scratch? Does the reduction in training required to reach a given performance compensate for this overhead?

  4. Robustness across model scales and tasks: Are the conclusions consistent across different model scales and downstream tasks, or are there settings in which direct attention-weight transfer is competitive? Such cases would help clarify the conditions under which the proposed strategy is most useful.

5 comments
AuthorsOpenReview ↗

Response to Reviewer U3vK (1/3)

We thank the reviewer for the review and the insightful questions. We would like to address your concerns below:

[W1 Q1] Mechanism of Attention-Only Copying Failure

We thank the reviewer for this insightful comment. We agree that controlled analyses could further distinguish the candidate explanations, and we are happy to provide one for each.

We would like to first note that Sec. D.1 in the Appendix explains the failure functionally: with the same copied QQ and KK, the Softmax map and the normalized kernel map of a linear operator are computed differently, thus simply copying the attention weights does not preserve the pre-trained attention function. Following the reviewer's suggestion, we test each candidate explanation with a controlled analysis below:

Candidate ExplanationControlled AnalysisObservation
Output ScaleSwap only the attention operator of the pre-trained weights from Softmax to linear, then optimally rescale the output of each layer.We measure the relative error of the copied linear attention output to the Softmax one: copying attention weights stays much closer to random-init than to the teacher. This pattern remains consistent across three scaling methods: per-layer scalar, per-channel scale and per-layer linear map.
Attention NormalizationWith the same swap as in Output Scale, compare the attention maps of the copied linear operators with the Softmax ones.The linear normalization flattens the copied attention maps: the ELU/ReLU maps are nearly uniform, as flat as random-init ones, although they keep part of the teacher's key ranking. InLine keeps the ranking but assigns negative weights according to its computation, thus its maps become the furthest from the Softmax ones.
Optimization DynamicsCheck the gap between Copy and Random along the training.Training does not wash out an initial advantage: where Copy ends below Random, it never leads after the warm-up when training becomes stable.
Architectural CompatibilitySoftmax students under the identical weight-copy protocol.Attention-only copying consistently helps the Softmax, but barely helps or even lowers the ReLU.
Calibration before Fine-tuningAlign the copied attention outputs with the teacher's through distillation over the early stage of training.Calibration makes the copied attention more effective: as the calibration covers a gradually longer part of training, the final accuracy also improves gradually.

As shown, neither output scale nor optimization dynamics explains the failure, while attention normalization does, consistent with our explanation that the failure comes from the linear operator itself. Since a linear model differs from the Softmax one only in the attention operator, the Softmax control below isolates how the attention map is normalized.

Regarding architectural compatibility, we run the same comparison with a Softmax student under the identical weight-copy protocol below, where every component of the Softmax model is fine-tuned:

Acc. (Δ\Delta to Random)CopyFlowers (ViT-Small)Food (ViT-Small)ImageNet-1K (ViT-Base)
SoftmaxNone45.473.278.1
ReLUNone43.064.476.5
SoftmaxAttention67.5 (+22.1)78.7 (+5.5)80.1 (+2.0)
ReLUAttention37.5 (-5.5)65.9 (+1.5)75.6 (-0.9)
SoftmaxMLP67.8 (+22.4)79.5 (+6.3)80.8 (+2.7)
ReLUMLP67.3 (+24.3)78.8 (+14.4)80.2 (+3.7)
SoftmaxBoth91.0 (+45.6)83.8 (+10.6)81.9 (+3.8)

As shown, attention-only copying helps the Softmax student consistently on all three datasets, following the same trend as MLP-only copying, while for the ReLU student it lowers the accuracy on Flowers and ImageNet-1K and gives only a marginal gain on Food. Meanwhile, MLP-only copying brings both students to nearly the same accuracy, as expected because MLP weights are operator-agnostic, and distilling the attention on top of the copied MLPs (our recipe) further raises the ReLU student to the Softmax level of the Softmax Both row. Hence, the failure of attention-only copying comes from the linear operator rather than from the protocol itself.

Regarding calibration before fine-tuning, the answer is yes. We apply the attention-output distillation as calibration over the first 10%, 30%, 50% and all of training, and fine-tune without the teacher for the remaining epochs. Results show that the final accuracy improves with the length of the calibration. More details of this experiment are given in [Q2]. Hence, calibrating before fine-tuning makes attention-weight copying more effective, which further supports the attention-output distillation in our recipe.

We will include all the above discussion in the revised version.

AuthorsOpenReview ↗

Response to Reviewer U3vK (2/3)

[W2 Q3] Training Cost and Generality

We thank the reviewer for this comment, and we are happy to share more details on the training cost and generality of the proposed linear ViT initialization recipe.

We would like to first note that, without our recipe, a linear ViT requires from-scratch pre-training, which usually takes hundreds of epochs training on ImageNet-1K, and typically still underperforms the Softmax version. Our recipe skips this stage by reusing the existing Softmax pre-trained weights, which alone saves ~3 days of pre-training for ViT-Base (estimated with 300 epochs on 8 A100 GPUs). Moreover, with our recipe, linear ViTs can match and even surpass Softmax ones.

Regarding training cost, the teacher is the same pre-trained model whose weights are copied, kept frozen and used only in the forward pass during the downstream fine-tuning. It makes one training step about 1.1x longer (measured on one A100 GPU), while the inference cost, parameter count and deployment stay unchanged. Our recipe only requires the downstream fine-tuning, since the distillation is trained jointly with it. Furthermore, the deployed model is the linear ViT itself, thus the efficiency that motivates linear attention is fully kept.

Regarding whether faster training compensates for this overhead, it does by a wide margin, since the skipped pre-training costs far more than the overhead on the downstream fine-tuning. We further find that keeping the teacher for only the first 50% of training lowers the overhead to 1.05x while keeping about 90% of the gain. More details of this experiment are given in [Q2].

Regarding generality, we agree with the reviewer that our conclusions are established within the evaluated setting: five linear operators on six classification tasks across three model sizes, with further validation on large-scale classification (ImageNet-1K) and two dense prediction tasks. Following the reviewer's suggestion, we further test the transfer on Diffusion Transformers, with DiT-S/2 and DiT-B/2 [1] on ImageNet 256×256 using ReLU attention, trained for 400k steps and evaluated by FID-50K:

FID (↓)RandomCopyMLPCopyMLPCopy w/ Distill (Ours)Softmax DiT (reference)
DiT-S/287.5788.2876.4268.8369.14
DiT-B/256.0956.7348.4743.6944.27

As shown, the same pattern holds for Diffusion Transformers at both sizes: copying the attention weights alone is even slightly worse than random initialization, copying the MLPs carries more than half of the gain, and distilling the attention on top of the copied MLPs (our recipe) matches and even surpasses the Softmax DiT. Hence, our conclusions also extend well to Diffusion Transformers for image generation. Video Transformers remain beyond the scope of this work, while the above results are also consistent with LiT [2], which loads every parameter except the linear attention from a pre-trained DiT model.

We will include the above discussion and state the scope explicitly in the revised version.

AuthorsOpenReview ↗

Response to Reviewer U3vK (3/3)

[Q2] Contribution of MLP Copying vs. Attention-Output Distillation

We thank the reviewer for this insightful question, and we are happy to provide the decomposition analysis.

We would like to first note that Fig. 4 in the main text, and Figs. A and B and Tabs. D and E in the Appendix already decompose the final gain across operators, datasets, and model sizes, from Random to MLPCopy to MLPCopy w/ Distill (our recipe), which we summarize below:

Model SizeAvg. Ratio of (Gain by MLPCopy) / (Gain by Full Recipe)
Tiny44.5%
Small51.0%

As shown, the two components contribute comparably for ViT-Tiny and Small on the six transfer datasets, and the gains are consistent across five linear operators: MLPCopy and distillation each help in all ViT-Small tests. Moreover, our results show that once the attention is distilled, copying it as well becomes redundant: MLPCopy w/ Distill stays within 0.8% of FullCopy w/ Distill for all ViT-Small tests.

Regarding removing the distillation after an initial stage, we test it with ReLU on Copy w/ Distill, where the whole gain over Copy comes from the distillation:

Acc. (Δ\Delta to Copy)Flowers (ViT-Small)Food (ViT-Small)ImageNet-1K (ViT-Base)Avg. Gain Recovered
w/o Distill (Copy)37.565.975.6--
w/ Distill for the first 10% of training48.7 (+11.2)73.3 (+7.4)77.6 (+2.0)50.7%
w/ Distill for the first 30% of training53.2 (+15.7)76.9 (+11.0)78.9 (+3.3)76.4%
w/ Distill for the first 50% of training55.9 (+18.4)79.2 (+13.3)79.4 (+3.8)90.0%
w/ Distill for 100% of training (Copy w/ Distill)58.0 (+20.5)80.4 (+14.5)79.9 (+4.3)100%

As shown, the accuracy increases steadily with the length of the distillation on all three datasets: the first 10% of training already recovers about half of the gain, and the first 50% recovers 90% of it. Hence, most of the distillation gain remains after the teacher is removed, so the teacher is not strictly needed throughout training.

Together, the above two discussions further support the effectiveness of the proposed transfer recipe, and we will include the above results in the revised version.

[Q4] Robustness across Model Sizes and Tasks

We thank the reviewer for this insightful question, and we are happy to summarize the robustness tests on model sizes and tasks.

We would like to first note that our conclusions are consistent across model sizes and tasks we test: five linear operators on six classification tasks across three model sizes, with further validation on large-scale classification (ImageNet-1K) and two dense prediction tasks in our main text and Appendix. Following the reviewer's suggestion, we further test our findings on image generation with DiTs of two model sizes in [W2 Q3].

Regarding settings where direct attention-weight transfer is competitive, our results show one case below. With its MLP context: on top of MLPCopy, further copying the attention (FullCopy) broadly helps in all tests, while once the attention is distilled, copying it as well changes the accuracy only marginally.

We will include the above discussion in the revised version.


We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Peebles et al. Scalable Diffusion Models with Transformers. ICCV 2023.

[2] Wang et al. LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation. ICCV 2025.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and constructive feedback.

We have addressed every point in our response to each reviewer individually, with new experiments and analyses where needed. We are currently preparing the revised version of the paper, which will incorporate all these additions, and we will upload it before the end of the discussion period.

We thank you all again for your valuable feedback. If you have any further questions or concerns, we are very happy to answer.

Area_Chair_5zSjOpenReview ↗

Please read the rebuttal and engage in discussion

Dear Reviewers,

The authors have posted their rebuttal. Please read it and engage in discussion, in particular:

  • Whether each of your questions and concerns has been addressed.
  • Whether your rating and recommendation have changed, and why.
  • Any remaining concerns that must be settled before the meta-review.

Best, Your AC

We have uploaded the revised version of our paper, with the main changes marked in orange.

The revised version includes the main additional results and discussions from our replies, mostly in the Appendix. We thank all reviewers again for their constructive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.