- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This is a very matter-of-fact style paper investigating strategies for placing LoRA adapters within LLMs. Simple heuristics like up- or down-projection only are compared with data-driven methods. The take home message is that neither the simple heuristics nor the data-driven method demonstrates convincing generalization across tasks/metrics/models.
Critical strengths
This paper addresses problems we are all well aware of in empirical works: selection bias, nll vs. downstream metrics, mismatched parameter budgets, overly general claims, etc. The sober look provided in this paper will be relevant for practitioners.
The presentation is very clear. The claims are well supported.
Critical weaknesses
This sort of work is almost always vulnerable to criticism regarding a limited experimental scope - more would almost always be better. This is no exception - 2 tasks and 2 models of the same model family and similar size is very boderline. Another model with a different architecture and/or size would strengthen the paper significantly. But since even this limited experimental setup was already enough to support their call for task- and metric-dependent LoRA placement, I would see this point as a weakness rather than a reason to reject.
This paper audits existing methods without proposing more robust strategies. Hence, the novelty/innovativeness is limited.
Questions for authors
-
Have you tried other models or tasks?
-
Do you have results on other strategies (top/bottom layer only, attention vs. mlp, etc)?
Thank you for recognizing the value of matched budgets and the distinction between response likelihood and downstream performance. We have expanded the model coverage and placement comparisons.
We added Qwen2.5-3B-Instruct, extending the study beyond the original 7B/8B Llama models. Generation accuracy is evaluated on the same 1,000-example official-test GSM8K subset.
Down-only achieves lower NLL in both seeds. The generation comparison shows that the preferred placement changes with the inference template, reinforcing the importance of evaluating the intended deployment format.
We added gate-only and QV-only, comparing Q/V attention projections with individual MLP projections. All entries below are two-seed mean response NLL on Llama-3/GSM8K, with matched parameter budgets within each row.
Down-only leads the original-budget comparison and maintains its advantage over up-only at the doubled budget in both seeds.
We also completed three-seed native-template evaluations of the original adapters: Llama-2 favors up-only in all three seeds, while Llama-3 favors down-only in all three. Together, these additions strengthen the paper’s practical guidance: compare placements under equal budgets using the target model, inference format, and evaluation metric.