- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PEFT methods usually use additional modules such as adapters, prompts or LoRAs, and place them in the frozen models by architectural convention or sensitivity analysis. The paper claims that where to adapt is a primary design decision, but adaptation locations have not been systematically compared across all transformer components. The paper therefore asks where a pretrained transformer should be adapted, with the following 3 questions: (1) which single component is the most effective adaptation target; (2) how much of that component needs to be updated; (3) which parts of this component should be selected.
With evaluations on classification tasks with DINOv2 across model sizes as main results, plus further validations on dense and language tasks, the paper concludes that: (1) the attention output projection is the strongest adaptation component, with a slightly higher performance than full fine-tuning, which the authors attribute to it only recombining existing head features into the residual stream; (2) updating only a subset of columns in , with the count selected on validation data, downstream accuracy can be further improved; (3) random column selection works effectively compared to gradient-, activation- or magnitude-based selection. Therefore the paper proposes APLA, which tunes or a random subset of its columns in every block together with the task head, with no additional parameters or training overhead.
Critical strengths
A simple and practical finding with comprehensive experimental support: the paper proposes a simple but effective PEFT method by only tuning , which ranks first among the six attention/MLP matrices on 21 datasets, with further validations across several model sizes, pre-training methods and backbones. However, the novelty over prior work is limited and the fairness of the comparison cannot be verified, which will be discussed in details in the following Weaknesses.
Critical weaknesses
(1) The novelty is limited.
The main finding has already been reported for LLMs in [1], which compared transformer components with a fixed number of trainable parameters and found ''Query/Key ≪ Value/Up/Gate < Output/Down''; [1] then proposed to fine-tune randomly selected channels of the output projections (Output and Down) for PEFT. Although this paper focuses on vision, the core finding and recipe are largely independent of the domain, so the extension to vision alone is a limited contribution.
The same ranking also appears in the equal-budget comparison of [2], where was the best single attention matrix, which the paper only describes as ''close to , '', and random updates of existing weights were already shown to match importance-based selection or LoRA in [3, 4].
The paper should clearly state its difference and relationship of [1–4], to position itself as a broad empirical study or a new general benchmark on the feature-reuse hypothesis, which is of great significance to evaluate its novelty.
(2) The fairness of the comparison cannot be verified.
Tuning the full only matches BitFit and is below AdaptFormer, ARC and Adapter, so the SOTA performance of APLA mainly comes from selecting the number of columns r per dataset on validation data. However, the paper does not report the search spaces or selected ranks/widths of the baselines, so it is unclear whether they were tuned as much as r. The authors should report them and compare under a matched tuning budget.
Reference
[1] S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity.
[2] LoRA: Low-Rank Adaptation of Large Language Models.
[3] RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates.
[4] Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning.
Questions for authors
Apart from the main concerns in Weaknesses, here are the minor concerns:
- APLA selects the adapted columns randomly, thus a statistical analysis across runs is required, because the performance gain over AdaptFormer is only 0.6% and the gaps between random and informed selection are within 0.2~0.3%. The authors should at least rerun the experiment results in main tables to support the SOTA claims.