- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper states that a trained LoRA adapter should be reusable across deployment storage budgets. Furthermore, the paper introduces SIFT-LoRA, which compiles Full-LoRA parent into a layer-sparse adapter without additional training or changing model weights. The proposed selector, Conditional SIFT-KL evaluates how much each layer reduces predictive KL when added to the current child. The method lowers predictive KL against baselines in 16 out of 18 settings.
Critical strengths
- The core technique is sound and supported with results
- The method is practical as it does not need to be retrained, and SIFT-LoRA outperforms the baselines evaluated against
Critical weaknesses
Unclear practical motivation and benefit over retraining: The paper does not clearly explain when the budget would become known only after LoRA training, or why one could not wait until the budget is known. The method should be compared with training LoRA using a rank that fits the budget. This comparison should include downstream performance and computational cost. Without it, the practical benefit of the method remains unclear.
Unclear impact on downstream performance: The evaluation focuses on KL divergence and agreement with the parent model. However, it does not clearly show whether better results on these metrics lead to better downstream performance. It is also unclear how performance drops as the budget decreases and when it becomes too poor to be useful. This makes it difficult to judge how well the method works under tight budgets.
Questions for authors
- Could you provide a practical example where the budget is known only after LoRA adapters have been trained? Why would waiting until the budget is known be impractical?
- At what budget p% does performance become poor, and what is the trade-off between budget and performance?
- How does your method compare to retraining LoRA with a rank that satisfies the budget, in terms of performance, computational cost, and wall-clock time?
- Could you report variability or uncertainty in Table 2 to help assess whether the differences are statistically significant, even though no training is involved?
Thank you for your rebuttal and additional experiments.
However, I remain unconvinced of the practical benefit over directly training LoRA to fit the budget. At 12.5%, SIFT-LoRA loses 3.33 percentage points in accuracy, while at 25% it matches accuracy but requires substantially more computation. The latency results are interesting, but their limited scope does not sufficiently resolve my concerns about the overall trade-off. I therefore maintain my original score.