Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving *where to place a limited number of adapters to maximize performance* largely open. To investigate this, we introduce **PAGE** (**P**rojected **A**dapter **G**radient **E**nergy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated at a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the **dominant adaptation module** and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose **DomLoRA**, a placement method that places a single adapter at the dominant adaptation module. With only **0.7%** of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across downstream tasks, including instruction following, mathematical reasoning, coding, and multi-turn conversation. This method also matches or improves other LoRA variants and reduces training time by up to **2.74×** compared with broad placement, supporting the dominant adaptation module perspective as a practical placement guideline.