- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines which pretrained components remain useful when softmax attention in a ViT is replaced by a linear operator. Across five operator families, copying MLP weights accounts for most of the improvement over random initialization, while copying attention projections alone is much less effective. Distilling attention outputs further improves performance. The experiments cover several classification datasets, with additional detection and segmentation benchmarks.
Critical strengths
-
For ReLU-Base on ImageNet, MLP copying raises accuracy from 76.5% to 80.2%, and FullCopy with distillation reaches 82.0%. Comparing these settings across several operators gives a clearer picture of transfer than evaluating only the final recipe.
-
The result extends beyond small classification benchmarks. Tables D and E include COCO and ADE20K, and Table G checks the contribution of patch embeddings, positional embeddings and the class token. These experiments support the practical value of the study.
Critical weaknesses
-
The operator-specific explanation is confounded with the transfer protocol. Li et al. (NeurIPS 2024), §2.2, inject teacher-computed attention maps throughout student training; this paper initializes projection weights and then trains them on the student's own representations. Appendix A acknowledges the difference, but §4.2 still presents the outcome as a reversal of softmax-to-softmax transfer. A softmax student under the identical weight-copy protocol is the missing control. Table G also shows conditional rather than uniformly ineffective transfer: copying attention changes 76.5% to 75.6% without MLP transfer, but 80.2% to 80.7% with it. A representative softmax/ReLU attention×MLP initialization comparison would test the operator-specific explanation without expanding the dataset suite.
The TTT result has an additional initialization confound: Appendix B.2 leaves the wider output projection random, although its teacher-compatible submatrix can be copied with the extra columns initially zero. This preserves the output map on the main branch, not the whole TTT function; Net2Net provides the broader function-preserving transfer precedent. This control would refine the TTT-specific conclusion, but is not a prerequisite for crediting the ReLU/ELU evidence.
-
Output matching does not establish recovery of token routing. Matching attention outputs constrains both aggregation and value representations. Distinct maps can satisfy , as the non-identifiability analysis of Brunner et al. (ICLR 2020), §3 makes clear. Appendix D.2 additionally describes this objective as directly targeting MLP-input mismatch, although that input depends on the residual stream and normalization. The successful loss is not in question; attention-output matching has an acknowledged precedent in LoLCATs. Narrowing the wording to functional output alignment would address this concern. If the mechanism is retained, measure the actual normalized MLP-input discrepancy on held-out examples, layer by layer.
Questions for authors
Does attention-only weight copying also fail in a softmax student with the same checkpoint, loading mask and training recipe? A representative softmax/ReLU attention×MLP comparison, with per-seed differences, would test the operator-specific claim.
Which output projections, biases and LayerNorm tensors are loaded in each setting? Where is the distillation target taken and how is MSE reduced? Please also identify the data split used to select the loss weight and checkpoints.
Is there evidence of reduced discrepancy at the actual normalized MLP inputs? If not, would you narrow the claim to attention-output alignment? This wording correction is sufficient for my current assessment; a new routing theory is not required.
2 comments
Response to Reviewer YA6K (2/2)
[W2 Q3] Functional Output Alignment Rather than Routing Recovery
We thank the reviewer for this insightful comment. We agree that functional output alignment is the more precise framing to describe the effect of our attention distillation, and we are happy to adopt it.
As the reviewer has pointed out, output matching constrains what the attention module computes rather than the attention map that computes it, since distinct maps can produce the same output [2], and in linear attention, distinct queries can even produce the same map and output [3]. This is also why the same output matching loss remains effective for MHLA and TTT, which never form an attention map. Consistently, our loss ablation in Sec. 5.4 shows that matching the attention outputs performs on par with or better than matching the attention maps for ReLU and ELU, and is always better for InLine. Overall, the output matching loss is the operator-general choice: available for every linear operator, and can achieve higher or on-par accuracy than map matching across datasets.
Regarding the normalized MLP inputs, since they also depend on the residual stream and the LayerNorm, we will present the decomposition in Sec. D.2 in the Appendix only as the motivation for where the loss is placed. Therefore, we agree to narrow the claim on the effect of attention distillation from recovery of token routing to functional output alignment, as the reviewer suggested. This narrowing does not change our contributions or results, but makes our findings more precise. We thank the reviewer again for the suggestion, and will update the framing in the revised version.
[Q2] More Details on Weight Loading, Attention Distillation Loss and Data Split
We thank the reviewer for asking for more details on our implementation, and we are happy to share them below:
| Item | Answer |
|---|---|
| Copy | ELU, ReLU, InLine and MHLA: the QKV and output projections with both weights and biases of every block. TTT: only the rows of the standard QKV with both weights and biases, since its output projection has a different shape. |
| MLPCopy | MLP of every block with both weights and biases, while leaving all other components randomly initialized. |
| FullCopy | All pre-trained Softmax weights except the classification head. For components not in the Softmax model, we keep their default initializations. |
| Distillation Target | The output of the attention module in each block, after the output projection and before the residual addition. For MHLA and TTT, which have no class token, only the patch tokens are matched when computing the loss. |
| MSE Computation | The MSE between the teacher's and student's attention outputs, summed over the 12 blocks, then multiplied by . |
| Loss Weight | 50 for every operator, dataset and initialization setting, selected by the sweep in Sec. 5.4. |
| Data Split | Top-1 Accuracy (%) on the standard test split (validation set for ImageNet-1K), for every setting, including the baselines. The same split is used for selecting . |
As shown, beyond the attention and MLP weights, only FullCopy loads the remaining components, including the LayerNorms, the patch embedding, the position embedding and the class token, and Tab. G shows that loading the latter three changes the accuracy only marginally. We will add the above discussions in the Appendix in the revised version.
We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.
[1] Li et al. On the Surprising Effectiveness of Attention Transfer for Vision Transformers. NeurIPS 2024.
[2] Brunner et al. On Identifiability in Transformers. ICLR 2020.
[3] Han et al. Bridging the Divide: Reconsidering Softmax and Linear Attention. NeurIPS 2024.
Response to Reviewer YA6K (1/2)
We thank the reviewer for the review and the insightful questions. We would like to address your concerns below:
We thank the reviewer for this precise suggestion. We are happy to provide the requested attention × MLP comparison for both Softmax and ReLU students.
We would like to first note that as the reviewer has pointed out, we have listed the difference between our Copy and Attention Copy in [1] in Sec. A in the Appendix: Attention Copy asks whether the attention alone suffices to transfer a Softmax ViT, while we ask whether the Softmax attention weights can initialize a linear attention operator. Thus, during the downstream training, Attention Copy uses the frozen teacher's attention maps, while our Copy only initializes the attention weights, which are then fine-tuned.
To test whether the failure of Copy comes from the linear operator rather than from the protocol itself, we run the same comparison with a Softmax student under the identical protocol below, where every component of the Softmax model will be fine-tuned:
As shown, two trends hold under the identical protocol: (1) Attention-only copying helps the Softmax student consistently on all three datasets, following the same trend as MLP-only copying. In contrast, for the ReLU student, attention-only copying lowers the accuracy on Flowers and ImageNet-1K and gives only a marginal gain on Food. (2) MLP-only copying brings both students to nearly the same accuracy, as expected because MLP weights are operator-agnostic. Note that the larger MLP gains of the ReLU student are due to its lower Random baseline, and distilling the attention on top of the copied MLPs (our recipe) further raises the ReLU student to the Softmax level of the Softmax Both row.
Hence, the failure of attention-only copying comes from the linear operator rather than from the protocol itself. Together with the reviewer's observation from Tab. G that the copied attention does help the ReLU student once its MLPs are copied, this shows that the transfer of attention weights depends on both the operator and the MLP context. We will include the above discussion in the revised version.
We thank the reviewer for this suggestion, and we are happy to share the results on TTT with the suggested function-preserving initialization.
Specifically, to test whether the failure of Copy for TTT comes from its random output projection, we copy the teacher's output projection weights into the wider output projection of TTT, with the extra columns set to zero, which preserves the output map on the main branch. We compare it with Random and our Copy on all datasets below:
As shown, copying the output projection changes the accuracy of Copy only marginally, with both Copy variants falling below Random on all datasets. Hence, the failure of Copy for TTT does not come from the random output projection, but from the linear operator itself, as discussed in [W1 Q1] above. We will add this control experiment in the revised version.