acceptodds
Withdrawn from ICLR 2027

Attention Output Projection Layers Are Strong Targets for Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective adaptation need not update every component or add parameters. Tuning the right subset of existing parameters can be both more effective and more efficient. But where should a pretrained transformer be adapted? We conduct systematic comparisons of transformer components and identify the attention output projection as the strongest individual adaptation target. We attribute this to its role: attention output projections determine how attention features enter the residual stream, so when pretrained features are rich, recombining them can suffice. Our method, Attention Projection Layer Adaptation (APLA), tunes the output projection in each transformer block, or a fixed random subset of its columns, together with the task head. It adds zero parameters, requires no architectural changes, and adds no inference computation. On 21 classification datasets, we achieve state-of-the-art mean accuracy of 91.5% with DINOv2 ViT-B, outperforming 20 baselines (the strongest reaching 90.9%) including full fine-tuning (89.7%). Evaluation on 49 vision tasks in total supports these findings under low-data regimes, distribution shift, segmentation, and detection. With ViT-g, APLA reduces training memory by 53% and training time by 43% relative to full fine-tuning. Our finding extends beyond vision: output projections are the best attention component to adapt on six of eight GLUE language tasks. Lastly, we also find that parameter-adding methods such as LoRA and AdaptFormer benefit from placement at the output projection over their default location.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision transformers, transfer learning, efficient fine-tuning, selective fine-tuning, attention output projections

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P9snOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PEFT methods usually use additional modules such as adapters, prompts or LoRAs, and place them in the frozen models by architectural convention or sensitivity analysis. The paper claims that where to adapt is a primary design decision, but adaptation locations have not been systematically compared across all transformer components. The paper therefore asks where a pretrained transformer should be adapted, with the following 3 questions: (1) which single component is the most effective adaptation target; (2) how much of that component needs to be updated; (3) which parts of this component should be selected.

With evaluations on classification tasks with DINOv2 across model sizes as main results, plus further validations on dense and language tasks, the paper concludes that: (1) the attention output projection WOW_O is the strongest adaptation component, with a slightly higher performance than full fine-tuning, which the authors attribute to it only recombining existing head features into the residual stream; (2) updating only a subset of columns in WOW_O, with the count selected on validation data, downstream accuracy can be further improved; (3) random column selection works effectively compared to gradient-, activation- or magnitude-based selection. Therefore the paper proposes APLA, which tunes WOW_O or a random subset of its columns in every block together with the task head, with no additional parameters or training overhead.

Critical strengths

A simple and practical finding with comprehensive experimental support: the paper proposes a simple but effective PEFT method by only tuning WOW_O, which ranks first among the six attention/MLP matrices on 21 datasets, with further validations across several model sizes, pre-training methods and backbones. However, the novelty over prior work is limited and the fairness of the comparison cannot be verified, which will be discussed in details in the following Weaknesses.

Critical weaknesses

(1) The novelty is limited.

The main finding has already been reported for LLMs in [1], which compared transformer components with a fixed number of trainable parameters and found ''Query/Key ≪ Value/Up/Gate < Output/Down''; [1] then proposed to fine-tune randomly selected channels of the output projections (Output and Down) for PEFT. Although this paper focuses on vision, the core finding and recipe are largely independent of the domain, so the extension to vision alone is a limited contribution.

The same ranking also appears in the equal-budget comparison of [2], where WOW_O was the best single attention matrix, which the paper only describes as ''close to WQW_Q, WVW_V'', and random updates of existing weights were already shown to match importance-based selection or LoRA in [3, 4].

The paper should clearly state its difference and relationship of [1–4], to position itself as a broad empirical study or a new general benchmark on the feature-reuse hypothesis, which is of great significance to evaluate its novelty.

(2) The fairness of the comparison cannot be verified.

Tuning the full WOW_O only matches BitFit and is below AdaptFormer, ARC and Adapter, so the SOTA performance of APLA mainly comes from selecting the number of columns r per dataset on validation data. However, the paper does not report the search spaces or selected ranks/widths of the baselines, so it is unclear whether they were tuned as much as r. The authors should report them and compare under a matched tuning budget.

Reference

[1] S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity.

[2] LoRA: Low-Rank Adaptation of Large Language Models.

[3] RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates.

[4] Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning.

Questions for authors

Apart from the main concerns in Weaknesses, here are the minor concerns:

  • APLA selects the adapted columns randomly, thus a statistical analysis across runs is required, because the performance gain over AdaptFormer is only 0.6% and the gaps between random and informed selection are within 0.2~0.3%. The authors should at least rerun the experiment results in main tables to support the SOTA claims.
Reviewer_JjVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "Attention Projection Layer Adaption", a parameter efficient method to finetuning pretrained transformer models. More specifically, the authors dived deep into the ViT archtecture and find the output projection of self-attentions is the strongest indivisual target for fine-tuning.

By tuning the attention output layer alone, or even a random subset of its columns, the results is often outperforms full finetuning, which requires more trainable components.The authors hypothesis that by adapting this particular activation, it recombines rich and existing pretrained features. and such effect compound with multiple transformer blocks.

Critical strengths
  1. it is really interesting to deep dive into model architecture and find the layer that is mostly easy to strongest individual adaptation target.

  2. "pretrained features are rich, recombining them can suffice" this core hypothsis is conceptually sound. and the authors provides justification on the selected benchmarks and model arch. (please refer to the section below for my concern on the benchmark setup)

  3. I like the idea of directly finetuning a part of model, instead of adding parameters to or around the model. This idea is structural simple, and keep the forward-pass logic entirely unchanged and avoid any inference overhead of complex deployment pipelines.

Critical weaknesses

W1. The paper currently treat the Transformer as a relatively uniform arch, but structural nuances simnifically alter gradient flow and internal features. some differences that came to my mind: different normalization placements (pre-norm, post-norm, etc); and Engram (https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf) and MoE architectures. If the FFNs are rounting parametrs are frozen, it is highly uncertain if updating on the the attention output layer provide enough capacity for finetuning, since the rounting and memory retrieval component will play a different role than regular FFNs in current transformers studied in the paper.

W2. This paper provide an extensive unimodal evaluation (across 49 vision tasks, and 9 language tasks with modern backbones like Llama3 and DINO-v2), but these eval treat vision and languge in complete isolation. These vision / language tasks studied here could be done with a good VLM model. Current VLM model utilize alignment that highly rely on attention mechanism, it is unclear to me that this core observation of the paper could hold on to the multi-modal integration layers.

Questions for authors
  1. while the authors provide a compelling observation on finetuning attention output layer. i wonder will this observation change across diverse structural paradigm? (please refer to W1 above)

  2. to make the finding of this paper truely convincing and useful for current research community, could the authors validate if this observation holds for model VLM arch? this could greatly improve the correctness of this paper's claim. (please refer to W2 above)

Reviewer_DaVUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates transformer adaptation locations and identifies attention output projections as effective tuning targets. The proposed APLA method updates output projection weights or randomly selected columns without adding new parameters.

Critical strengths
  1. The paper provides extensive comparisons across adaptation locations, pretrained backbones, and downstream tasks.
  2. APLA is straightforward to implement and introduces no additional inference parameters or operations.
Critical weaknesses
  1. The proposed adaptation strategy is relatively simple. Updating selected existing weights is not fundamentally new, and the main novelty lies in identifying a favorable parameter location.
  2. The feature-reuse explanation is not directly verified. The paper does not measure whether pretrained attention features are actually preserved or reused more effectively after adaptation.
  3. The theoretical comparison assumes fixed transformer-block inputs. During multi-layer adaptation, earlier updates change the inputs and features of later blocks, limiting the explanatory power of this analysis.
  4. The claim that output projections are the best adaptation targets is stronger than the evidence. Their advantage is most pronounced for image classification and is less consistent on other tasks.
  5. The comparison with full fine-tuning is not uniformly favorable. In Table 1(a), output-projection tuning wins on 10 datasets, loses on 10, and ties on one.
  6. The reported 91.5% mean accuracy for APLA is affected by large gains on individual datasets. For example, the RSSCN7 result improves from 73.3% with full fine-tuning to 86.2% with APLA. The aggregate result would be more informative with an analysis excluding such large outliers.
  7. The results do not report variability across independent training runs or random column selections.
Questions for authors
  1. Can the authors report multiple training runs with different random column selections?
  2. Does the superiority of output projections remain after matching trainable-parameter counts across different adaptation locations?
  3. What is the performance of joint output/value projection adaptation under a fixed parameter budget?
  4. Can feature-reuse measurements support the proposed explanation?
  5. How much does the overall advantage change when datasets with unusually large gains are excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.