- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the relationship between LoRA placement and training cost and proposes a late-layer placement search based on a frozen probe. The evaluation includes comparisons with matched adapter capacity and experiments on GLUE and VTAB.
Critical strengths
1.The method is well motivated, highlighting that LoRA parameter efficiency does not necessarily translate into training efficiency and examining the role of adapter placement.
2.The paper is well-organized in general, and easy to follow.
3.The experiments are comprehensive, covering commonly used vision and language benchmarks.
Critical weaknesses
1.The methodological novelty is limited, and the advantages over simple baselines remain unclear. The proposed method primarily combines a brief warm-up, adapter masking evaluation, and binary search, offering a limited technical contribution. Compared with the simple Last-K strategy, LaLoS-1.0 scores only approximately 0.10 points higher than Last-18 on GLUE while using more resources for final fine-tuning; on VTAB, Last-80 achieves higher average accuracy.
- The specific strategy in the proposed method has been already included in the search space of neural architecture search based LoRA approaches, thus making the contribution questionable.
2.The validity of the probe scores lacks direct validation. After all LoRA adapters have been jointly trained, directly disabling some adapters may disrupt their interactions. Therefore, performance after masking may not predict the performance of the selected configuration after retraining. The paper needs to validate the consistency between probe scores and final training performance.
3.A cost comparison covering the complete pipeline is missing. The main table reports final fine-tuning costs, and Table 2 separately reports warm-up costs. However, the total overhead, including setup, calibration, and selection queries, has not been aggregated. It is therefore difficult to fully assess whether the search mechanism delivers net benefits.
Questions for authors
Please see the critical weaknesses section above.