acceptodds
Withdrawn from ICLR 2027

What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structure and tied to robustness. We build that instrumentation for ViT-S students of a self-supervised teacher on ImageNet-100, and report three findings that triangulate one conclusion. First, the transfer is essentially perfect and permanently so: the distilled student's attention ends up roughly two orders of magnitude closer to the teacher's than fine-tuning does, and does not drift with additional training. Second, the gap is real at 14x fewer parameters and 10x less data than previously studied, but it has a time axis. It tracks training maturity, and completing the schedules that the stopping rule interrupted closes it below our pre-registered threshold in two of three seeds, with comparisons at equal accuracy giving the same result. The endpoint gap at this scale is substantially a training-maturity artifact: robustness matures later than accuracy, and stopping rules tuned to accuracy undersample it. Third, forcing cross-row redundancy down by half the structural separation between the distilled and fine-tuned conditions produces no detectable robustness response under two registered ways of matching accuracy. Verified transfer, a gap that closes while the structure never moves, and a null under direct intervention are together consistent with the deficit residing in features, not in the visible attention structure. This is elimination plus intervention, and its scope is the regime we measured. In this regime, attention overlays show where a model looks, not what it knows.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
attention transfer, knowledge distillation, vision transformers, robustness, distribution shift, attention maps, training dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6UoSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Prior work showed that with attention transfer, a ViT can recover most of the pre-trained model's performance by simply copying the attention maps, but this recovery is less robust under distribution shift. The paper claims that previous studies argued that this gap comes from the features instead of the attention transfer itself, but no one had directly measured what the copied attention looks like or how it relates to robustness. The paper therefore asks what attention transfer actually transfers, including the following 3 questions: (1) whether the student's attention really matches the teacher's; (2) whether the robustness gap is real or depends on the transfer training duration; (3) whether changing the transferred attention structure can solve the robustness issue.

With results on ViT-S students with the MAE ViT-S pre-trained weights transferred on ImageNet-100, the paper concludes that: (1) the distilled attention is almost identical to the teacher's and does not change during training; (2) the OOD gap can be reproduced with early-stopped transfer training but mostly closes once the distillation runs the complete training schedule; (3) making the attention more diverse with an attention-diversification loss (ADL) does not detectably reduce this robustness issue. Therefore the paper argues that the remaining deficit lies in the learned features rather than in the attention structure.

Critical strengths

A clear well-posed follow-up question: the paper explores one natural follow-up question of recent work on attention transfer of ViTs and answers it with well-designed controlled ablations. However, the contribution is too narrow with poor experimental validations, which will be discussed in details in Weaknesses.

Critical weaknesses

(1) The contribution is too narrow.

The main empirical finding on this robustness issue largely lands on the short transfer training duration, that the gap can be mostly closed simply with longer training, which is a rather expected solution. This does not explain the gap reported in [1], which trained with fixed budgets without a stopping rule used in this paper, and the proposed "robustness matures later than accuracy" holds only for the paper's ratio-form metric. A training-duration finding can still be a valid contribution, but then the paper should either explain why robustness matures later than accuracy under attention transfer, or provide a solution for reaching fine-tuning-level OOD performance with current short training length.

(2) The experimental validation is not enough.

Only one model size (ViT-S) on ImageNet-100 with MAE pre-trained weights is tested, and robustness is evaluated only on ImageNet-based domain shifts. The authors should test more model sizes (Tiny, Small, Base, Large) with full-scale ImageNet-1K, at least one more OOD setting beyond the ImageNet variants (ObjectNet or other domain-generalization benchmarks), and more pre-trained weights beyond MAE as in [1, 2].

Reference

[1] On the Surprising Effectiveness of Attention Transfer for Vision Transformers.

[2] Attention Transfer Is Not Universally Effective for Vision Transformers.

Questions for authors

Apart from the main concerns in Weaknesses, here are the minor concerns:

  • If the transferred structure is pushed toward fine-tuning (e.g., distilling the fine-tuned model's attention), does effective robustness change at matched accuracy? If it stays flat, the claim that the attention structure does not carry robustness holds; if it rises, the deficit is at least partly in the attention structure.

  • The paper is generally hard to read: terms such as "stopping-lottery", "three clocks", "endpoint / matched-accuracy coordinate", "iso-level", "cap-hit" and "first-class evidence, second-class authority" should be explained in plain language (e.g., in a footnote), and the main findings should be stated more clearly in the introduction and conclusion rather than spread across the appendices.

3 comments

We thank the reviewer for the review.

On (1), we agree that the paper does not explain why robustness matures later than accuracy under attention transfer. The revision will state this as the open question the finding leaves rather than imply it is resolved. We would distinguish one point. The claim is not only that completing the schedule closes the gap in two of three seeds. It is that the gap tracks the epoch at which each run was stopped, while the transferred attention structure remains fixed to within measurement noise. That is what separates an explanation based on training progress from one based on attention structure. On the gap Li et al. report, the paper does not claim that the stopping-rule result explains it. It reports that their ordering reproduces at this scale under the pre-registered rule, and Section 5 states the fixed-budget implication at their scale as a prediction. On the ratio definition of effective robustness, you are correct. The paper states this at the claim in Section 4.2 and in Appendix G and Section 6: the sign of the correlation between robustness and stopping epoch depends on the definition of effective robustness used. The paper's two pre-registered conclusions rest on matched and near-equal-accuracy comparisons, where the two definitions order models identically. The claims that robustness tracks stopping epoch and matures later than accuracy will be qualified wherever they appear.

On (2), we agree. The study uses one backbone, one dataset, one teacher, and ImageNet-derived shifts only. The paper claims its magnitudes for no other setting and describes portability as argued rather than shown. The extensions you list are the appropriate next step and are outside the scope of this submission.

On your first question, distilling the fine-tuned model's attention is precisely the experiment that would test the opposite direction of change in attention structure, and we agree it is the right one. If robustness at matched accuracy remains flat, the reading that structure does not carry robustness would be supported in both directions. If it rises, part of the deficit lies in the structure. The present null covers only the direction that reduces redundancy. Section 4.4 states this, and the abstract leaves it implicit.

On readability, we agree. Each of the terms you list will be defined in plain language at first use. The three findings will be stated directly in the introduction and conclusion, with the supporting tables in the main text rather than the appendix.

Thanks for the authors' response. However, it agrees with my main concerns without providing any new experiments, while for me additional experimental evidence is of great significance to evaluate the contribution of this work. The authors should include the above-mentioned experiments from all reviews to further improve the quality of this work. I thus keep my score.

Thank you for the reply. The experiments you ask for (more model sizes, training on full ImageNet-1K, and further teachers and shift benchmarks) are beyond the compute available to us. We have explored several options, and none is feasible within the discussion window. We continue to regard the direct measurement of transferred attention structure, and its relation to robustness at this scale, as a noteworthy contribution, though we accept that this venue calls for a more general claim, which requires the broader validation you describe. We will withdraw and revise for a future opportunity. We thank you for the thoughtful feedback and input on making this work stronger.

Reviewer_hhp4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes why attention transfer is less robust to ditribution shift than weight transfer. With ViT-S/16 on ImageNet-100, the authors measured ID accuracy and efficient robustness on Taori et al. (2020) benchmark unlike standard setting that uses ImageNet-1K as fine-tuning dataset. Based on the resuls, the authors argue that the robustness degradation is realted to early stopping suggesting that choose the number of epochs based on the robustness not the ID accuracy.

Critical strengths

The paper shows that the robustness gap between attention transfer and weight transfer is realted to early stopping.

Critical weaknesses

Overall, as the authors disclose in the AI use statement, the paper appears to be mostly written by AI, which is allowed at this conference. However, the authors should have revised the paper for better readability.

  1. First of all, the paper is very hard to read, and the writing must be improved.

    1.1 There are strong subjective claims (e.g., most) without proper evidence, such as Attention maps are among computer vision's most trusted windows into a transformer. in L37.

    1.2 The first sentences of many paragraphs are not informative. For example, Attention transfer transfers attention in L191, Two corollaries frame everything downstream in L207, and Everything so far is observational in L326.

    1.3 There are non-academic phrases, such as The gap is real in L20 and in the title of Section 4.2.

    1.4 Many results in the main paper refer to Appendix K, which reduces the readability of the paper. It would be better to move some important results from Appendix K to the main paper.

    1.5 See the Questions section.

  2. In L222, the authors use s2 without any explanation. Only later does it turn out to mean seed 2, so readers cannot understand s2 at that point. Moreover, s1 also refers to a well-known model in the large language model domain (Muennighoff et al., 2025). It would be better to use seed 2 instead of s2 in L222.

  3. As the authors mention in the limitations section, they only use ViT-S/16. It would be better if the authors provided experiments on other backbones, such as ViT-B/16, ViT-L/16, and ViT-H/14.

  4. According to Figures 1, and 9 and Table 1, there are huge discrepancies between random seeds. For example, the stopping epochs of the three distilled seeds differ by up to 145 epochs (Figure 3), and the endpoint gap ranges from 0.0181 to 0.0575 (Table 3).

  5. In the robust fine-tuning domain, one of the key results is the trade-off between ID accuracy and effective robustness (e.g., the ID-OOD accuracy plot of Taori et al. (2020)). However, the paper does not show such a plot, and Figure 9 in Appendix K only shows the per-seed gap at two matched accuracy levels. I recommend the authors add the ID-OOD accuracy plot of all models to the main paper.

  6. The paper mostly compares results seed by seed without summarizing the overall results. Hence, the conclusions can be misleading, especially since some experiments show huge discrepancies between random seeds. I recommend the authors provide the overall results (e.g., mean and standard deviation) for the main results, such as Table 3 and Section 4.3.

  7. [minor] Li et al.'s in L259, L432, and L455 is cited without the year. It should be Li et al. (2024)'s or similar.

[Reference]

Muennighoff, Niklas, et al. "s1: Simple test-time scaling." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors
  1. What does dose mean?

  2. In L151, waht does per-condition cosine horizon mean? Does it mean that the authors used cosine scheduler?

  3. In L200, what does faithfulness mean?

  4. In L236, what does clock mean?

  5. In L243, what does maturity reading mean?

  6. In L313, what does top matched rungs mean? Does it mean saturadted performance?

  7. In L329, what does retired 100-epoch horizon mean?

  8. In L419, what does transplant experiment?

3 comments

We thank the reviewer for the detailed comments on the presentation. We agree with nearly all of them and respond in order.

1.1. The "most trusted windows" sentence will be rewritten as a cited statement rather than an assertion.

1.2. The opening sentences you cite convey nothing to the reader. They will be replaced.

1.3. "The gap is real" will be removed from the abstract, from the section heading, and from its other uses in the text.

1.4. The gap table and the figures on which the results depend will be moved into the main text.

1.5. Answered under your questions, below.

  1. Agreed. It will read "seed 2" throughout.

  2. We agree that ViT-S alone bounds what the paper can claim. The paper claims its magnitudes for no other setting. Larger backbones are the next step.

  3. Two of the spreads you cite are the stopping-rule finding of Section 4.2 rather than noise around it: the stopping epochs in Figure 3 and the range of gaps in Table 3. Under a single stopping rule, runs of the same condition stopped as much as 145 epochs apart, and the gap at those checkpoints tracks the stopping epoch. The spread of gaps at matched accuracy in Figure 9 is different. The paper attributes it to differences in when robustness arrives across the distilled runs without the penalty, and those differences shrink at the highest matched accuracy levels. The matched comparison cannot resolve differences below about one percentage point, and the fine-tuned checkpoints at the highest matched levels are only 75 to 82 percent through their schedules. For those two reasons Section 4.3 places the primary weight on the comparison at completed schedules.

  4. Agreed. An ID-OOD accuracy plot covering all models will be added to the main text.

  5. You are correct that Table 3 reports per-seed values without a mean and standard deviation. Across the three seeds the gap at the stopping-rule checkpoints, fine-tune minus distill in effective robustness, is 0.036 with standard deviation 0.020. At completed schedules it is 0.006 with standard deviation 0.005. The spread largely disappears once the schedules complete, which is what the per-seed presentation was meant to show. Section 4.3 reports the mean gap at the highest matched level. The revision will add the spread there and in Table 3.

  6. The citations without the year will be corrected.

In response to your questions:

  1. Dose is the coefficient on the attention-diversification penalty, λ_adl. It takes the values 0.1, 0.3, 1.0, and 3.0 (Section 3).
  2. Yes, a cosine learning-rate schedule is used. The horizon is the number of epochs the schedule runs over, warmup included: 300 for the random-initialization conditions and 200 for fine-tuning.
  3. Faithfulness is how closely the student's attention matches the teacher's. It is measured as per-layer, per-head KL divergence on held-out images (Section 4.1).
  4. A clock is a timescale. The three clocks are the epochs at which attention structure, ID accuracy, and robustness each reach their final or near-final values within a single run. Structure reaches its final value by epoch 24, accuracy reaches 93 percent of its final value by epoch 119, and robustness reaches its highest measured value at epoch 294 of 300 (Section 4.2).
  5. The maturity reading is the interpretation that the gap at the stopping-rule checkpoints substantially reflects how far along its robustness curve each run was when the rule fired, rather than a difference between the methods. The residual is disclosed in Section 4.2.
  6. The top matched rungs are the highest ID-accuracy levels reached by both the fine-tuned and the distilled trajectory of a seed, approximately 0.88. A rung is one accuracy level in the matched-accuracy comparison. It does not denote saturated performance.
  7. The retired 100-epoch horizon is fine-tuning's original 100-epoch schedule. It was replaced by the 200-epoch schedule under a rule declared before the runs that allowed schedules to be recalibrated after the pilot (Section 3). That rule was used once, for this change. Its endpoint is used once in the main text, as a comparison point in Section 4.3.
  8. The transplant experiment is the distillation condition itself, read as an experiment. The teacher's attention is held fixed in the student by construction, and the question is whether robustness arrives with it.

Each of these terms will be defined where it first appears.

I appreciate the author's response. I will check again once the revision is uploaded. It will be helpful if the authors highlight the changes (e.g., in a different color) such that reviewers and area chairs can easily look at the difference.

Thank you for the reply. The presentation changes described in our response will be made, with the changes marked as you suggest. Together with the additional experiment raised in the reviews, they amount to a new version of the paper rather than a revision within the discussion period, so we are withdrawing the submission to make those changes properly. We thank you for the detailed feedback.

Reviewer_ZbyvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why attention-map distillation can recover in-distribution accuracy while falling short of weight transfer in robustness. Using an MAE ViT-S teacher and ImageNet-100, it compares training from scratch, fine-tuning, and attention distillation. The experiments measure attention fidelity and structure, track robustness across training, and vary an attention diversification loss (ADL). The authors find that attention maps are closely matched, completing the training schedule substantially reduces the robustness gap, and reducing cross-row redundancy with ADL produces no detectable robustness improvement at matched accuracy.

Critical strengths

The paper asks a worthwhile question: what accounts for the robustness difference between attention transfer and weight transfer when their in-distribution performance is similar? Understanding this difference could clarify which benefits of pretraining are captured by attention patterns and which require other aspects of the learned representation.

Critical weaknesses

My main concern is the limited justification for using ADL to explain the robustness gap. The distilled model already has lower cross-row redundancy than the fine-tuned model, despite being less robust at the reported early-stopped endpoints (§4.1). ADL lowers redundancy further, moving away from the fine-tuned model's value (§4.4). Prior work provides a reason to consider diversification as a robustness intervention. But it remains unclear why this direction should address the particular deficit observed here. The lack of a detectable improvement therefore provides limited evidence about the source of that deficit. I recognize that the authors qualify the feature-based explanation as unproven (§6). Even with this qualification, the study does not directly identify which features or mechanisms account for the difference.

Second, the training-duration result does not yet explain the gap reported in the original larger-scale study. The experiments change the model and dataset scale and introduce an early-stopping protocol, whereas Li et al. report fixed-budget results. Completing the schedule shows that premature stopping matters in the present setting, but does not establish that it explains the earlier result. The authors acknowledge this limitation. I am not yet convinced that the result resolves the motivating question beyond the smaller setting studied here.

A minor presentation concern is the frequent use of metaphorical expressions. For example, "first-class evidence, second-class authority" (§3, p.4) and "The dial turns, and it is a scalpel" (§4.4, p.7) describe the status of full-schedule reruns and the selectivity of the ADL intervention, respectively. Although technical explanations are provided, these metaphors make the presentation harder to follow.

Questions for authors

The paper predicts that diversification will narrow the gap and acknowledges that ADL moves redundancy away from the fine-tuned model's value. Given that the distilled model is already less redundant, what additional support does the null result provide for a feature-based explanation of the robustness gap, beyond showing that further diversification produced no detectable improvement in this setting?

3 comments

We thank the reviewer for a careful and fair review.

On the first concern, we agree. ADL was chosen on the finding of Guo et al. (2023) that diversifying ViT attention improves robustness, and on the pre-registered prediction stated in Section 3 that distilled attention would be more overfocused than fine-tuned attention. The measurement reported in Section 4.1 then showed the reverse for cross-row redundancy, the degree to which different tokens attend to the same places. The distilled student already sits at the teacher's redundancy, below the fine-tuned model's, and the penalty moves it further in that direction. The consequence is the one you identify. The experiment cannot say what would happen if redundancy were moved toward the fine-tuned model's value, and it does not identify which features account for the deficit. The first limit is stated in Section 4.4 and the second in Section 6.

What the experiment does establish is narrower. The penalty moved redundancy by half the separation between the two conditions. The student's attention stayed matched to the teacher's, per-row entropy stayed flat, and accuracy at the completed schedule was unchanged. Robustness, averaged over seeds, showed no detectable monotone response at the design's resolution of about one percentage point. That rules out a monotone robustness response to further diversification, in this direction and at that resolution. To answer the question directly, the null's contribution to the features reading is that bound. The reading itself rests on the three findings together, which Sections 4.4 and 6 describe as consistent with the evidence rather than shown by it. The experiment that would test the opposite direction is the one Reviewer 6UoS also proposes, distilling from the fine-tuned model's attention, and we agree it is the right next step.

On the second concern, the paper does not claim that the stopping-rule result explains the gap Li et al. report, as you note. It reports that their ordering reproduces at this scale under the pre-registered rule, and Section 5 states the fixed-budget implication at their scale as a prediction. We agree that "reproduces" invites a stronger reading, and the revision will describe the comparison at the stopping-rule checkpoints without that word. Two further points bear on the relation to Li et al. The completed-schedule comparison in Section 4.2 compares the two methods at their full, completed budgets, fine-tuning at 200 epochs and distillation at 300. Both learning-rate schedules ran to completion. It is the nearest analogue in our setting to a fixed-budget comparison, and it agrees with the matched-accuracy comparisons of Section 4.3. Li et al. train at fixed budgets, and their Table 6 comparison under distribution shift is between a fine-tuned and a distilled model at the same ImageNet-1K accuracy. That is the kind of matched and near-equal-accuracy comparison on which the paper's two pre-registered conclusions rest. The revision will state their protocol in those terms.

On the figurative phrasing, we agree. "First-class evidence, second-class authority" and "the dial turns, and it is a scalpel" will be replaced with direct descriptions of what those paragraphs report. The variant phrasing in Appendix D will be replaced as well.

I appreciate the author's response and clarification of the study's scope. However, as the authors acknowledge, the observed redundancy is opposite to the hypothesis that motivated the ADL intervention. My main concern therefore remains unresolved, and I will maintain my score.

Thank you for the reply. The experiment that would address the direction concern, distilling from the fine-tuned model's attention, is within our means and will be included in the next version. Together with the presentation changes described in our responses, it amounts to a new version of the paper rather than a revision within the discussion period, so we are withdrawing the submission to make those changes properly. We thank you for the careful review.

Comments on the paper

1 comment

We thank the reviewers for their time and their careful reading of the paper. Several concerns are shared across the three reviews. We address those here and respond to each review individually.

We accept the criticism of the presentation. The paper introduces terms of its own without defining them at first use. Several headings and sentences are figurative where a literal statement would serve better. Results on which the main text depends are placed in the appendix. All of this will be corrected in revision.

We also accept two limits of the study. It uses a single architecture, dataset, and teacher. The paper claims its magnitudes for no other setting and describes portability of the mechanism as argued rather than shown. Reviewers also note that the causal intervention moves attention structure in only one direction. That restriction is stated but only in Section 4.4. The abstract and discussion name the direction but do not say that the test is one-sided and moves away from the fine-tuned model's redundancy and they should.

These points do not alter what the experiments show. The distilled attention matches the teacher's, reaches its final form early in training, and does not drift with further training. At the checkpoints selected by the stopping rule, the robustness gap tracks the epoch at which the rule stopped each run, while the transferred attention structure does not change. That relation is measured with the paper's ratio definition of effective robustness, and Appendix G reports how it depends on the definition. At completed schedules the gap compresses below the pre-registered threshold in two of three seeds, and the comparisons at matched accuracy agree.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.