- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Prior work showed that with attention transfer, a ViT can recover most of the pre-trained model's performance by simply copying the attention maps, but this recovery is less robust under distribution shift. The paper claims that previous studies argued that this gap comes from the features instead of the attention transfer itself, but no one had directly measured what the copied attention looks like or how it relates to robustness. The paper therefore asks what attention transfer actually transfers, including the following 3 questions: (1) whether the student's attention really matches the teacher's; (2) whether the robustness gap is real or depends on the transfer training duration; (3) whether changing the transferred attention structure can solve the robustness issue.
With results on ViT-S students with the MAE ViT-S pre-trained weights transferred on ImageNet-100, the paper concludes that: (1) the distilled attention is almost identical to the teacher's and does not change during training; (2) the OOD gap can be reproduced with early-stopped transfer training but mostly closes once the distillation runs the complete training schedule; (3) making the attention more diverse with an attention-diversification loss (ADL) does not detectably reduce this robustness issue. Therefore the paper argues that the remaining deficit lies in the learned features rather than in the attention structure.
Critical strengths
A clear well-posed follow-up question: the paper explores one natural follow-up question of recent work on attention transfer of ViTs and answers it with well-designed controlled ablations. However, the contribution is too narrow with poor experimental validations, which will be discussed in details in Weaknesses.
Critical weaknesses
(1) The contribution is too narrow.
The main empirical finding on this robustness issue largely lands on the short transfer training duration, that the gap can be mostly closed simply with longer training, which is a rather expected solution. This does not explain the gap reported in [1], which trained with fixed budgets without a stopping rule used in this paper, and the proposed "robustness matures later than accuracy" holds only for the paper's ratio-form metric. A training-duration finding can still be a valid contribution, but then the paper should either explain why robustness matures later than accuracy under attention transfer, or provide a solution for reaching fine-tuning-level OOD performance with current short training length.
(2) The experimental validation is not enough.
Only one model size (ViT-S) on ImageNet-100 with MAE pre-trained weights is tested, and robustness is evaluated only on ImageNet-based domain shifts. The authors should test more model sizes (Tiny, Small, Base, Large) with full-scale ImageNet-1K, at least one more OOD setting beyond the ImageNet variants (ObjectNet or other domain-generalization benchmarks), and more pre-trained weights beyond MAE as in [1, 2].
Reference
[1] On the Surprising Effectiveness of Attention Transfer for Vision Transformers.
[2] Attention Transfer Is Not Universally Effective for Vision Transformers.
Questions for authors
Apart from the main concerns in Weaknesses, here are the minor concerns:
-
If the transferred structure is pushed toward fine-tuning (e.g., distilling the fine-tuned model's attention), does effective robustness change at matched accuracy? If it stays flat, the claim that the attention structure does not carry robustness holds; if it rises, the deficit is at least partly in the attention structure.
-
The paper is generally hard to read: terms such as "stopping-lottery", "three clocks", "endpoint / matched-accuracy coordinate", "iso-level", "cap-hit" and "first-class evidence, second-class authority" should be explained in plain language (e.g., in a footnote), and the main findings should be stated more clearly in the introduction and conclusion rather than spread across the appendices.
3 comments
Thanks for the authors' response. However, it agrees with my main concerns without providing any new experiments, while for me additional experimental evidence is of great significance to evaluate the contribution of this work. The authors should include the above-mentioned experiments from all reviews to further improve the quality of this work. I thus keep my score.
Thank you for the reply. The experiments you ask for (more model sizes, training on full ImageNet-1K, and further teachers and shift benchmarks) are beyond the compute available to us. We have explored several options, and none is feasible within the discussion window. We continue to regard the direct measurement of transferred attention structure, and its relation to robustness at this scale, as a noteworthy contribution, though we accept that this venue calls for a more general claim, which requires the broader validation you describe. We will withdraw and revise for a future opportunity. We thank you for the thoughtful feedback and input on making this work stronger.
We thank the reviewer for the review.
On (1), we agree that the paper does not explain why robustness matures later than accuracy under attention transfer. The revision will state this as the open question the finding leaves rather than imply it is resolved. We would distinguish one point. The claim is not only that completing the schedule closes the gap in two of three seeds. It is that the gap tracks the epoch at which each run was stopped, while the transferred attention structure remains fixed to within measurement noise. That is what separates an explanation based on training progress from one based on attention structure. On the gap Li et al. report, the paper does not claim that the stopping-rule result explains it. It reports that their ordering reproduces at this scale under the pre-registered rule, and Section 5 states the fixed-budget implication at their scale as a prediction. On the ratio definition of effective robustness, you are correct. The paper states this at the claim in Section 4.2 and in Appendix G and Section 6: the sign of the correlation between robustness and stopping epoch depends on the definition of effective robustness used. The paper's two pre-registered conclusions rest on matched and near-equal-accuracy comparisons, where the two definitions order models identically. The claims that robustness tracks stopping epoch and matures later than accuracy will be qualified wherever they appear.
On (2), we agree. The study uses one backbone, one dataset, one teacher, and ImageNet-derived shifts only. The paper claims its magnitudes for no other setting and describes portability as argued rather than shown. The extensions you list are the appropriate next step and are outside the scope of this submission.
On your first question, distilling the fine-tuned model's attention is precisely the experiment that would test the opposite direction of change in attention structure, and we agree it is the right one. If robustness at matched accuracy remains flat, the reading that structure does not carry robustness would be supported in both directions. If it rises, part of the deficit lies in the structure. The present null covers only the direction that reduces redundancy. Section 4.4 states this, and the abstract leaves it implicit.
On readability, we agree. Each of the terms you list will be defined in plain language at first use. The three findings will be stated directly in the introduction and conclusion, with the supporting tables in the main text rather than the appendix.