acceptodds
Under review as a conference paper at ICLR 2027

Projection-Level LoRA Placement: An Equal-Parameter Audit Across Models and Tasks

Abstract

Choosing which transformer projections receive LoRA adapters changes both parameter budget and training behavior. We audit placement under matched trainable-parameter budgets using Llama-2-7b-chat-hf and Llama-3.1-8B-Instruct, Alpaca and GSM8K, and three seeds per setting. The resulting 36-run matrix compares up-only, down-only and a training-data-based PLoP allocation. Up-only has the lowest mean response NLL in one of four model–task combinations; down-only wins the other three. These comparisons measure teacher-forced response likelihood. We also audit generated numeric answers at all 18 GSM8K adapters and two base checkpoints on separate internal and official-test subsets; prompt compatibility and termination limit interpretation of raw base scores. An earlier 25-configuration sweep favored up-only on reused WikiText-2 validation of candidate choices, but did not match parameter budgets. On the official-test subset, up-minus-down accuracy is +3.57 percentage points for Llama-2 (pointwise 95% interval [1.30,5.87]) and -0.50 for Llama-3 ([-3.00,1.97]). Historical and matched protocols differ in objective and holdout construction, so their ordering changes are not a placement-only intervention. Our results support task- and metric-dependent evaluation of placement, not a universally best projection or a new module-selection algorithm.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA, parameter-efficient fine-tuning, adapter placement, parameter budgets, language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Tk1QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a very matter-of-fact style paper investigating strategies for placing LoRA adapters within LLMs. Simple heuristics like up- or down-projection only are compared with data-driven methods. The take home message is that neither the simple heuristics nor the data-driven method demonstrates convincing generalization across tasks/metrics/models.

Critical strengths

This paper addresses problems we are all well aware of in empirical works: selection bias, nll vs. downstream metrics, mismatched parameter budgets, overly general claims, etc. The sober look provided in this paper will be relevant for practitioners.

The presentation is very clear. The claims are well supported.

Critical weaknesses

This sort of work is almost always vulnerable to criticism regarding a limited experimental scope - more would almost always be better. This is no exception - 2 tasks and 2 models of the same model family and similar size is very boderline. Another model with a different architecture and/or size would strengthen the paper significantly. But since even this limited experimental setup was already enough to support their call for task- and metric-dependent LoRA placement, I would see this point as a weakness rather than a reason to reject. 


This paper audits existing methods without proposing more robust strategies. Hence, the novelty/innovativeness is limited.

Questions for authors
  • Have you tried other models or tasks?

  • Do you have results on other strategies (top/bottom layer only, attention vs. mlp, etc)?

1 comment

Thank you for recognizing the value of matched budgets and the distinction between response likelihood and downstream performance. We have expanded the model coverage and placement comparisons.

  1. Another model family and size.

We added Qwen2.5-3B-Instruct, extending the study beyond the original 7B/8B Llama models. Generation accuracy is evaluated on the same 1,000-example official-test GSM8K subset.

Qwen2.5-3B / placementMean NLL, two seedsTraining-format accuracy, seed 21604Native-chat accuracy, seed 21604
Up-only0.301463.2%76.2%
Down-only0.295064.4%74.8%

Down-only achieves lower NLL in both seeds. The generation comparison shows that the preferred placement changes with the inference template, reinforcing the importance of evaluating the intended deployment format.

  1. Additional placements and budgets.

We added gate-only and QV-only, comparing Q/V attention projections with individual MLP projections. All entries below are two-seed mean response NLL on Llama-3/GSM8K, with matched parameter budgets within each row.

Adapter budgetUp-onlyDown-onlyGate-onlyQV-only
Original0.42900.42440.43560.4301
Doubled0.43130.4237——

Down-only leads the original-budget comparison and maintains its advantage over up-only at the doubled budget in both seeds.

We also completed three-seed native-template evaluations of the original adapters: Llama-2 favors up-only in all three seeds, while Llama-3 favors down-only in all three. Together, these additions strengthen the paper’s practical guidance: compare placements under equal budgets using the target model, inference format, and evaluation metric.

Reviewer_yz4iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares up-only, down-only, and budget-adapted PLoP placements across two Llama models and two tasks. The main matrix contains 36 runs with matched parameter counts. Down-only wins three of four response-NLL comparisons. A separate GSM8K generation audit shows that the ranking also depends on the model and evaluation metric.

Critical strengths

I cannot identify any meaningful strengths in this submission. The paper lacks methodological novelty, substantial scientific insights, and convincing empirical contributions. Its main findings are largely unsurprising and do not meaningfully advance existing research.

Critical weaknesses

(1) Limited novelty and scientific significance. The paper primarily provides an empirical comparison of existing LoRA placement strategies. Its main conclusion that optimal placement depends on the model, task, and evaluation metric is expected and does not substantially advance prior work.

(2) Insufficient experimental scope and baseline coverage. Despite reporting 36 runs, the main evaluation covers only two closely related model families, two tasks, and three placement configurations. Important equal-budget placement baselines are missing.

(3) Lack of explanatory insights. The study documents differences between placement strategies but provides little explanation for these differences. The spectral analysis is largely disconnected from the main findings, and the results do not lead to a new selection principle or actionable design guidance.

Questions for authors

See weaknesses. I suspect that the authors conducted the entire research process using AI-driven automated research. Reviewing this paper feels like a waste of my time.

1 comment

Thank you for asking for broader comparisons and more actionable insights. We have expanded the evaluation with another model family and size, a second parameter budget, additional equal-budget baselines, and native-chat generation tests.

  1. Broader empirical evidence.

The additions comprise 16 completed training runs and 40 generation evaluations. The new training comparisons use two seeds and exactly matched adapter-parameter budgets within each comparison.

Model / budgetUp-only NLLDown-only NLLOrdering across seeds
Llama-3.1-8B / original0.42900.4244Down wins in 2/2
Llama-3.1-8B / doubled0.43130.4237Down wins in 2/2
Qwen2.5-3B / original0.30140.2950Down wins in 2/2

NLL values are means over two seeds. We additionally evaluated gate-only and QV-only on Llama-3 at the original budget. Their two-seed mean NLL values are 0.4356 and 0.4301, respectively, compared with 0.4244 for down-only.

  1. Concrete guidance for placement selection.

The generation results identify specific mismatches between selection criteria. The following comparisons use seed 21604, the original budget, and the same 1,000-example official-test GSM8K subset.

Model / placementTraining-format accuracyNative-chat accuracy
Qwen2.5-3B / up-only63.2%76.2%
Qwen2.5-3B / down-only64.4%74.8%
Llama-3.1-8B / up-only72.7%69.2%
Llama-3.1-8B / down-only73.8%70.5%
Llama-3.1-8B / gate-only74.3%49.4%
Llama-3.1-8B / QV-only71.7%49.4%

On Qwen, down-only has lower NLL in both seeds and higher training-format accuracy in this generation comparison, but up-only has higher native-chat accuracy. On Llama-3, gate-only leads under the training format, whereas down-only leads under the native template.

These findings provide actionable guidance: select placement using the intended inference template and downstream metric, rather than response NLL alone. Our contribution is the controlled evidence establishing these selection mismatches under equal parameter budgets. We will foreground these findings and their practical implications in the revision.

Reviewer_gpEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares up-only, down-only, and a budget-adapted PLoP at equal trainable-parameter budgets across two Llama models, Alpaca and GSM8K, and three seeds. Down-only has the lowest mean response NLL in three of four settings. On the official-test GSM8K subset, accuracy favors up-only for Llama-2; the up-only/down-only difference for Llama-3 is uncertain.

Critical strengths

The 36-run matrix controls parameter budgets, data splits, and training schedules, allowing a fair comparison across placements.

The authors separate response NLL from generated-answer accuracy and report paired uncertainty, which keeps their conclusions appropriately cautious.

Critical weaknesses

The matched study uses one budget per model and one training recipe, without gate-only or QV-only. This adds limited insight beyond the model/task dependence already discussed in PLoP.

Alpaca is evaluated only through response NLL. GSM8K uses the training-format prompt, and the placement rankings under native chat templates remain untested.

Questions for authors

Do the placement rankings hold at a second parameter budget? How do gate-only and QV-only compare at the same parameter budget? Do the GSM8K rankings hold with native chat templates?

2 comments

Thank you for suggesting a second budget, additional placement baselines, and native-chat evaluation. We have completed these comparisons.

  1. Second budget and additional baselines.

The following results use Llama-3.1-8B on GSM8K, with exactly matched adapter-parameter budgets within each comparison. Response NLL is averaged over two seeds; lower is better.

Adapter budgetUp-onlyDown-onlyGate-onlyQV-only
Original0.42900.42440.43560.4301
Doubled0.43130.4237——

Down-only achieves the lowest NLL among the four placements at the original budget. Its advantage over up-only persists at twice the budget, with the same direction in both seeds.

  1. Native chat templates.

We evaluated the original up/down adapters using native chat templates on the same 1,000-example official-test GSM8K subset. Accuracy is averaged over three seeds.

ModelUp-only accuracyDown-only accuracyOrdering across seeds
Llama-2-7B18.67%17.00%Up wins in 3/3
Llama-3.1-8B65.60%69.80%Down wins in 3/3

The contrasting model-specific rankings therefore persist under native chat templates. These results directly address the budget, baseline, and prompt-format questions and will be incorporated into the revision.

Reviewer_gpEZOpenReview ↗

Response to the Authors' Rebuttal

Thank you for your rebuttal and for addressing my concerns. After considering your responses, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.