acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving *where to place a limited number of adapters to maximize performance* largely open. To investigate this, we introduce **PAGE** (**P**rojected **A**dapter **G**radient **E**nergy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated at a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the **dominant adaptation module** and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose **DomLoRA**, a placement method that places a single adapter at the dominant adaptation module. With only **0.7%** of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across downstream tasks, including instruction following, mathematical reasoning, coding, and multi-turn conversation. This method also matches or improves other LoRA variants and reduces training time by up to **2.74×** compared with broad placement, supporting the dominant adaptation module perspective as a practical placement guideline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Adapter Placement, Gradient-Based Module Selection, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_JU3ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the placement problem of LoRA modules when trainable parameters are limited. A gradient-based metric, PAGE, is proposed to estimate the initial trainable gradient energy of candidate LoRA modules. PAGE is used to identify the downprojected modules of FFNs and designates them as "dominant adaptive modules." Based on this, the DomLoRA algorithm is proposed, which places a single LoRa adapter within this module. Experimental results show that DomLoRA achieves better average performance with a significant reduction in trainable parameters.

Critical strengths
  1. The paper's motivation is reasonable, exploring where LoRA should be placed to achieve better results when the parameter budget is relatively limited.

  2. The paper's method is simple and easy to understand, and easy to follow.

  3. This paper achieves good performance using fewer trainable parameters.

  4. The theoretical derivation is good, explaining why DomLoRA is effective, but the overall logic is slightly rigid, and readability needs improvement.

Critical weaknesses
  1. The method's innovation is relatively limited; using Fisher + gradient to explore module sensitivity is not novel.

  2. The experimental results do not support the paper's main claims; the authors seem to have failed to distinguish between better fit and less capability degradation. In Table 1, I found that on most tasks, the performance of the Domlora-trained model was worse than the base model. What is the point of training it then? The advantage over LoRA mostly comes from the TruthfulQA dataset; even after removing this dataset, Domlora still underperforms the base model. This is puzzling.

  3. The baselines used in the experiments are relatively old; more recent baselines need to be included for a fair comparison.

4.The identity in Theorem 1 is not wrong, but it only characterizes the initial sample-by-sample gradient energy and does not prove that this energy can predict batch optimization progress or final fit quality. The paper needs more comprehensive ranking-to-actual training benefit correlation experiments to extrapolate from sensitivity to placement effectiveness.

Questions for authors

please refer the weaknesses

2 comments

We hope this message finds you well. We are writing to kindly follow up on our previous correspondence regarding the rebuttal for our manuscript. We understand that you have a busy schedule, but we would greatly appreciate any feedback or responses you may have regarding our paper.

We are eager to address any further concerns to help improve the paper. Please let us know if there is anything further we can provide.

Thank you once again for your time and consideration.

Best regards,

Authors of Submission 3099

Dear Reviewer,

Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen our paper.

As the discussion period is approaching its end, we would be very grateful for your feedback on whether our responses and additional experiments have addressed your concerns. If any questions remain, we would sincerely appreciate the opportunity to clarify them before the deadline.

We understand the demands on your time and truly appreciate your consideration.

Best regards, Authors of Submission 3099

Reviewer_vMaTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is trying to identify where to place a few LoRA adapters. This is in contrast with earlier approaches, which choose either layers to apply adapters to, or modules across all the layers. Here, their question is to choose both the module and the layer. They propose a gradient-based score for a weight matrix. Basically, it measures how much signal an adapter there would get at the start of training, and it is computed using a small probe set before training. They find that this score, PAGE, is highest in the early layers and on the feed-forward (FFN) down-projection. They call it the dominant adaptation module. They propose an approach called DomLoRA, where they place an adapter at the module. This also results in fewer parameters compared to LoRA. They evaluate on four broad tasks, including general instruction tuning, math, code, and conversation.

Critical strengths
  • The theory for PAGE looks sound and also the idea of using it to identify the dominant module.
  • The approach to finding this module is simple and cheap. The authors also show savings in training time.
  • The authors showed that the winning module is consistent across training checkpoints, with a module that wins at step 0 also winning at the later steps.
Critical weaknesses
  • In Table 1, the pretrained model itself is quite strong. DomLoRA obtains 0.9 aggregate average improvement over the Pretrained Qwen and 0.8 over Pretrained LLaMA. It can be argued that DomLoRA makes minimal changes on average and thus doesn't disrupt the pretrained model too much. This is in comparison with LoRA which drops the score more on tasks such as TruthfulQA, GSM8k and LogiQA. Thus, it can be argued that the DomLoRA's advantage over LoRA is to avoid LoRA's drops. Table 1 results are not very conclusive to show that DomLoRA adapts better than LoRA.
  • Two model families and four task domains may not be sufficient to show that the approach applies. In addition, Table 2 results for Qwen are not present. We need more families to confirm the general nature of the findings.
  • HumanEval and HumanEval+ contain 164 problems. The differences of a few points could be due to multiple reasons such as seeds, evaluation settings [1]. Even for the other datasets in Table 2, the score difference is not large to be conclusive.

[1] Hochlehnert, A., Bhatnagar, H., Udandarao, V., Albanie, S., Prabhu, A., & Bethge, M. (2025). A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. In Second Conference on Language Modeling (COLM). https://openreview.net/forum?id=90UrTTxp5O

Questions for authors
  • The paper says that Table 1 results are for three seeds. Do you use single seed for table 2? And, if so, why? Single seed results may not be conclusive because the score differences between the models is not large.
1 comment

Dear Reviewer,

Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen our paper.

As the discussion period is approaching its end, we would be very grateful for your feedback on whether our responses and additional experiments have addressed your concerns. If any questions remain, we would sincerely appreciate the opportunity to clarify them before the deadline.

We understand the demands on your time and truly appreciate your consideration.

Best regards, Authors of Submission 3099

Reviewer_EobxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PAGE, a gradient-based probe for selecting LoRA placement, and DomLoRA, which adapts only a chosen shallow FFN down-projection. Experiments show competitive downstream performance while using substantially fewer trainable parameters and reducing training time.

Critical strengths
  1. Attaining competitive performance with a single adapter is of considerable practical value, especially given the substantial reduction in parameters and the measured training speedups.

  2. The initialization-based derivation is clearly presented, and the appendix provides useful evidence that module selection remains stable across probe samples and random seeds.

To me, the strongest potential contribution is the identification of a reusable adaptation interface within a pretrained model: a particular shallow module whose location remains useful across tasks, even as the task-specific updates change.

Critical weaknesses
  1. The evidence does not adequately isolate the benefit of the selected location. Table 5 compares the dominant down-projection with attention projections, but it includes neither alternative down-projection layers nor parameter-matched distributed placements. In addition, PAGE produces the same ranking as raw gradient energy among down-projections of identical dimensions, so the added value of its selection criterion remains unclear.

  2. The benefits of placement are not sufficiently disentangled from reduced forgetting or differences in optimization. In Table 1, broad LoRA underperforms the pretrained models on average, while Appendix H reports hyperparameter sweeps only for DomLoRA. Comparable tuning of the baselines is needed to establish whether the advantage persists beyond the shared training configuration.

Questions for authors
  1. How does the selected module compare with other shallow, middle, and deep FFN down-projections at the same rank? I suggest comparing LLaMA layers 0/1/2/16/31 and Qwen layers 5/6/7 under matched training settings (zero-based indices).

  2. Under an equal parameter budget, does concentrated placement outperform distributed adaptation? For LLaMA, this could be tested by comparing a single rank-64 down-projection adapter against rank-2 down-projection adapters across all 32 layers.

  3. How were the baseline hyperparameters selected, and does the advantage persist when LoRA and DomLoRA are given comparable budgets for learning-rate and rank tuning?

  4. Does PAGE predict actual adaptation gains more accurately than simpler selection criteria, such as the norm of the mean gradient or the magnitude of forward activations?

I find the single-adapter result promising, but the current experimental results do not fully convince me. If the controlled layer comparisons (Q1) support the placement claim and the reported benefits persist under comparable baseline tuning (Q3), I would consider raising my rating, potentially to weak acceptance.

Reviewer_u7atOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces two new concepts, namely: Projected Adapter Gradient Energy (PAGE) which is a gradient-based sensitivity probe for LoRAplacement, it estimates the layer and weight positions where the adapter should be more impactful and Dominant adaptation Module LoRA (DomLoRA) which is the placement of the adapter into this dominant region. The results demonstrate that DomLoRA achieves competitive performance across four tasks and two model families while training up to 2.74x faster and using only 0.7% of the vanilla LoRA parameter count.

Critical strengths

The paper's PAGE mechanism could be beneficial as an explaining tool and its contribution is not restricted to the placement of LoRAadapters.The paper evaluated the consistency of PAGE across various datasets and demonstrated that it is consistent in the selection of the dominant section. DomLoRA achieves competitive performance across four evaluated tasks while training up to 2.74x faster and using only 0.7% of the vanilla LoRA parameter count which is beneficial to the PEFT advances.

Critical weaknesses

Only one of the four experiments of DomLoRA includes Qwen3 which is not clear in the abstract, introduction or conclusion. This could limit the generalization of the method. The paper did not measure the performance in non-training tasks to determine whether the models' other capabilities were preserved.

Questions for authors

The General Instruction Tuning (Sec 4.1) was not evaluated in any instruction benchmark like IFEval, perhaps is better to name it general knowledge tuning? Why is that Qwen 3 is not included in all of DomLoRA experiments? And how can you assert the generalization of the method without, at least, this second model test? An ablation where DomLoRA is trained on up to X of the highest PAGE score regions could further strengthen the results? In appendix I it is demonstrated that PAGE generalizes to seeds and batches sizes, but is not reported the per seed score which may obscure if some of the seeds are dominant from others. Did you check that even without the average all seeds follow the same behavior?

Suggestions:

In Figure 2 it is hard to see the other steps as step 0 is dominant. Table 1 could benefit from reporting the performance difference between DomLoRA and, at least, the best-performing alternative method. The same could be applied to Table 3 by adding a new row with the difference.

2 comments

We hope this message finds you well. We are writing to kindly follow up on our previous correspondence regarding the rebuttal for our manuscript. We understand that you have a busy schedule, but we would greatly appreciate any feedback or responses you may have regarding our paper.

We are eager to address any further concerns to help improve the paper. Please let us know if there is anything further we can provide.

Thank you once again for your time and consideration.

Best regards,

Authors of Submission 3099

Dear Reviewer,

Thank you again for your thoughtful review and constructive suggestions, which have helped us strengthen our paper.

As the discussion period is approaching its end, we would be very grateful for your feedback on whether our responses and additional experiments have addressed your concerns. If any questions remain, we would sincerely appreciate the opportunity to clarify them before the deadline.

We understand the demands on your time and truly appreciate your consideration.

Best regards, Authors of Submission 3099

Comments on the paper

4 comments

Dear reviewers Thank you for your thoughtful review. The authors have submitted their rebuttal, and the discussion period is now open. Please take a moment to read the authors' response and engage with them and your fellow reviewers to address any remaining questions or clarify key points. Your active participation during this phase is vital to reaching a fair and well-informed final recommendation.

Best regards, area chair

Dear Reviewers,

Thank you for your time and thoughtful engagement during the rebuttal period. Your comments and discussion have helped clarify the paper’s strengths and the concerns that remain. With one day left in the discussion period, I encourage reviewers who have yet to weigh in to share their current recommendation and a brief rationale after considering the authors’ response. It would be particularly helpful to hear which concerns you consider resolved and which still inform your recommendation.

Thank you again for your contributions.

Best regards,

Area Chair

Dear Area Chair,

Thank you very much for your careful oversight and continued support throughout the discussion. We sincerely appreciate the reminders you have already sent to encourage reviewer engagement, as well as your efforts to help clarify which concerns remain unresolved.

As the discussion period is approaching its end, some of our latest responses have not yet received feedback. If time permits, we would be very grateful if you could kindly remind the relevant reviewers once more to share whether our responses address their concerns and whether any further clarification is needed.

We understand that you have already devoted considerable time to facilitating this discussion, and we truly appreciate your help and thoughtful consideration.

Best regards, Authors of Submission 3099

We sincerely thank all reviewers and the Area Chair for their time and constructive feedback, especially the Area Chair’s efforts to facilitate discussion. Our responses strengthen the following contributions and practical benefits:

  • Faster training with far fewer parameters. DomLoRA uses 1.0M/1.2M trainable parameters versus LoRA’s 174.6M/167.8M, approximately 0.7% of its parameter budget. Dominant placement delivers 1.6× training speedups for vanilla LoRA and up to 2.7× for DoRA, with only 80–120 seconds of PAGE probing.

  • A reusable adaptation location. PAGE selects the same shallow down-projection across four training datasets and five data-order seeds within each backbone. The new base-model experiments recover the same locations as their instruction-tuned counterparts.

  • Seamless integration with existing methods. DomLoRA changes only adapter placement and integrates seamlessly with existing LoRA variants, preserving their parameterizations and optimization mechanisms.

  • Effective learning beyond capability preservation. New Tülu V3 experiments improve the Qwen/LLaMA base-model averages by 2.6/13.3 points under the same documented zero-shot protocol. DomLoRA exceeds LoRA on Qwen and remains comparable on LLaMA.

  • Controlled evidence for precise placement. At an equal approximately 1.2M-parameter budget, concentrated placement outperforms distributed adaptation, 63.4 versus 60.4. DomLoRA also leads at every tested matched learning-rate/rank configuration, with 63.0 versus 60.2 when each method uses its best tested configuration.

  • Mechanistic insight and stronger evaluation. Appendix C connects concentrated input-channel and weight-gradient energy with strong shallow backward signals. We additionally completed three-seed math/code evaluations on both backbones and alternative-probe comparisons.

Together, these findings establish a simple, efficient, reusable placement rule that complements existing LoRA methods. We respectfully ask the reviewers and Area Chair to consider this evidence alongside our detailed responses.

Best regards,
Authors of Submission 3099

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.