- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for allocating different LoRA ranks across layers based on the stable rank of pretrained weight matrices, without requiring any training data. The authors provide theoretical analysis to support their rank allocation method and compare it with several baselines. The experimental results show some improvements over AdaLoRA, but the improvements over uniform rank allocation are not consistent.
Critical strengths
The paper proposes a simple and interesting method for allocating LoRA ranks across different layers without requiring training data. The authors also provide interesting theoretical insights to support their approach.
Critical weaknesses
- I think the baseline comparisons are not enough. In particular, I would like to see a direct comparison with SR-LoRA, since it uses a similar idea of allocating ranks based on pretrained stable rank.
- I found the paper a bit hard to read, especially with the amount of theoretical and technical details throughout the paper.
- The abstract also includes too many technical details, which makes it harder to understand the main idea and contribution of the paper.
Questions for authors
- Could the authors clarify the empirical advantages of Water-Filling LoRA compared to SR-LoRA?
- Could the authors simplify the abstract and focus more on the main contributions rather than technical details?
Thank you for suggesting a direct comparison with SR-LoRA and a simpler presentation.
On EuroSAT, Water-Filling with standard LoRA updates achieves 94.81% ± 0.12%, versus 93.20% ± 0.61% for the official SR-LoRA implementation including stochastic partial updating (SPU). This is a 1.60 percentage-point improvement, with higher accuracy in all three seeds. Both methods use the same backbone, data split, optimizer, learning rate, batch size, 20-epoch schedule, and allocated adapter budget of 3,889,152 parameters. Results are final-epoch accuracy, reported as mean ± sample standard deviation.
We will simplify the abstract around the allocation problem, our conditional budgeted analysis, and the empirical findings. Detailed derivations and implementation audits will be moved out of the abstract, and the allocation procedure will be presented separately from its proof.