acceptodds
Withdrawn from ICLR 2027

LoRA in the Last Layers: Score-Constrained Placement Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning of pretrained models, but parameter efficiency does not imply training efficiency. Adaptive methods select modules, parameters, or ranks, often changing adaptation capacity together with layer placement. Yet placement determines how far gradients propa- gate through the frozen backbone: similar parameter budgets can require materially different computation. We analyze this placement-dependent backward cost and introduce LaLoS, a score-constrained search over LoRA placements close to the output. A briefly trained probe and a held-out selection score determine the adapter placement for each task, without prescribing a fixed budget. Across language and vision benchmarks, placement close to the output reduces final fine-tuning compu- tation and elapsed time at matched adapter capacity. LaLoS achieves competitive performance with task-dependent adapter counts. Fixed placements in the last layers remain strong baselines. Together, the analysis and experiments show that layer placement affects fine-tuning efficiency beyond trainable-parameter count.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Low-rank adaptation (LoRA), parameter-efficient fine-tuning (PEFT), adapter placement, layer selection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CjV2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between LoRA placement and training cost and proposes a late-layer placement search based on a frozen probe. The evaluation includes comparisons with matched adapter capacity and experiments on GLUE and VTAB.

Critical strengths

1.The method is well motivated, highlighting that LoRA parameter efficiency does not necessarily translate into training efficiency and examining the role of adapter placement.

2.The paper is well-organized in general, and easy to follow.

3.The experiments are comprehensive, covering commonly used vision and language benchmarks.

Critical weaknesses

1.The methodological novelty is limited, and the advantages over simple baselines remain unclear. The proposed method primarily combines a brief warm-up, adapter masking evaluation, and binary search, offering a limited technical contribution. Compared with the simple Last-K strategy, LaLoS-1.0 scores only approximately 0.10 points higher than Last-18 on GLUE while using more resources for final fine-tuning; on VTAB, Last-80 achieves higher average accuracy.

  1. The specific strategy in the proposed method has been already included in the search space of neural architecture search based LoRA approaches, thus making the contribution questionable.

2.The validity of the probe scores lacks direct validation. After all LoRA adapters have been jointly trained, directly disabling some adapters may disrupt their interactions. Therefore, performance after masking may not predict the performance of the selected configuration after retraining. The paper needs to validate the consistency between probe scores and final training performance.

3.A cost comparison covering the complete pipeline is missing. The main table reports final fine-tuning costs, and Table 2 separately reports warm-up costs. However, the total overhead, including setup, calibration, and selection queries, has not been aggregated. It is therefore difficult to fully assess whether the search mechanism delivers net benefits.

Questions for authors

Please see the critical weaknesses section above.

Reviewer_nVS2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors investigate adaptive LoRA placement at the last layers based on binary search, building on partially trained adapter placement on a held-out dataset. The results show memory savings compared with standard LoRA tuning; however, the advantage over fixed last-K-layer tuning in terms of performance, memory, and computational efficiency is not clearly established. Choosing the last K layers might be a more efficient tuning strategy than other, more elaborate baselines.

Critical strengths

According to the results, choosing the last K layers for adapter placement might be a more efficient tuning strategy than other, more sophisticated baselines such as AdaLoRA or IGU-LoRA.

Critical weaknesses
  1. The proposed adaptive selection does not justify its cost. The simple last-K-layers baseline shows comparable or better results while requiring fewer computations.

  2. The work shows a very modest contribution to the analysis of LoRA placement compared with prior work and amounts to a conventional hyperparameter search.

Questions for authors

Could you explain your rationale for choosing binary search for a potentially non-monotonic decision score?

Reviewer_MH9YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the placement of LoRA adapters affects fine-tuning cost at a fixed adapter count, rank and width. A backward-cost count over a serial chain of linear layers shows that the cost grows linearly with the index b_max of the adapter farthest from the output (Eq. 5), since input gradients have to be propagated down to that layer. This motivates a fixed Last-K baseline (adapters on the output-side half) and LaLoS, which trains a short full-support LoRA probe, uses forward-only masked queries on a held-out split to binary-search the smallest output-side prefix whose score stays within ε of the probe, and then retrains that placement from the pretrained checkpoint. Experiments use DeBERTaV3-base on seven GLUE tasks and DINOv2-G/14 on VTAB-1k with five seeds, against Head-only, Full LoRA, WeightLoRA, PLoP, SPT, AdaLoRA and IGU-LoRA. Last-18/Last-80 match or beat Full LoRA while saving about 26% of FLOPs, 39–47% of wall time and 30–35% of peak memory. LaLoS-1.0 has the best GLUE mean (86.12 vs. 86.02 for Last-18), and Last-80 the best VTAB mean (78.63 vs. 78.24 for LaLoS-0.5).

Critical strengths
  1. The matched-capacity comparison is the most useful result. At the same 221,184 adapter parameters on GLUE, Last-18 saves 26.9% of FLOPs and 47.1% of time at 24.29 GiB, vs. 6.0%/10.6% at 33.97 GiB for WeightLoRA-18, with a slightly higher score. The same pattern holds on VTAB, where Last-80 is also the most accurate method. The message that placement-dependent cost should be reported is clear and practical.

  2. The reporting is honest. The authors say that fixed Last-K remains a strong baseline, report warm-up cost and probe memory separately, and include a sensitivity study. LaLoS also behaves sensibly per task, selecting very few layers where Head-only is already close to Full LoRA (Caltech101 7%, Oxford Pets 4%) and many more on structured tasks (SVHN ~71%).

Critical weaknesses
  1. LaLoS does not beat its own simple baseline. On GLUE it is +0.10 over Last-18 (within one SD), on VTAB it is below Last-80, and on GLUE its final training alone already uses more compute, time and memory than Last-18 (22.8% vs. 26.9% FLOP savings, 25.62 vs. 24.29 GiB).

  2. The efficiency numbers leave out the search. Tables 1 and 9 exclude warm-up and selection. Adding the warm-up cost from Table 2, my estimate is roughly 12–13% FLOP and 33% time savings end-to-end, vs. 26.9%/47.1% for Last-18 (about 21%/35% vs. 25.7%/38.5% on VTAB), before counting the selection queries, which are never reported. Because the probe is full-support LoRA, the peak memory also equals Full LoRA.

  3. Only K = L/2 is tested as a fixed placement, so the value of a per-task K is untested: there is no Last-K sweep, no fixed K at LaLoS's average, and no per-task K chosen on the same held-out split. The selected K is also noisy across seeds (e.g., RTE 53.9 ± 26.3%) and not monotone in the warm-up budget (K increases in 27 of 72 adjacent comparisons in App. H).

  4. The cost analysis is standard reverse-mode accounting, which the paper itself credits to Xu et al. (2026), and it is never checked on the evaluated models. Measured time often diverges from FLOPs (Head-only on GLUE: 52.9% FLOPs vs. 94.1% time; PLoP-Type1 on VTAB: 0.5% vs. 38.4%), so part of the time savings may come from having fewer adapters rather than from placement. The scope is also limited to encoder classification with attention-only LoRA.

Questions for authors
  1. Could you report the end-to-end cost per task (warm-up + selection queries + final training) next to Last-K and Full LoRA? Is there any setting where LaLoS wins end-to-end, or is the intended use amortizing one search over many runs?

  2. What exactly is the rule that replaces low-accuracy VTAB probes with Full-LoRA runs, how many of the 95 task/seed outcomes did it affect, and what are the VTAB results without it?

  3. Several baselines look degenerate: AdaLoRA and IGU-LoRA fall to the Head-only level on MRPC, RTE and CoLA (by my calculation, 74.80 on MRPC is the constant-positive-class score), and all GLUE methods share one learning rate of 5e-5. Was any per-method tuning done?

  4. What explains the gap between FLOP and time savings? A forward/backward/optimizer breakdown and a comparison of Eq. 5 with the measured backward time for a few b_max at fixed K would help, as would a decoder LLM experiment with LoRA on the MLP projections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.