- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies compressed LoRA through a bias–variance framework, deriving conditions for beneficial compression and sparse correction under a fixed parameter budget. It proposes ProLoSA, which combines shared LoRA-factor parameters with sparse residual coordinates selected using warmup SNIP.
Critical strengths
- The matched-budget analysis explicitly accounts for displaced compressed capacity and adaptive-selection noise.
- Controlled experiments include predictive bias–variance measurements and informative failure cases.
- Paired multi-seed GLUE and vision comparisons provide stronger evidence than the quoted baseline results alone.
Critical weaknesses
- The core theory largely builds on classical projection-risk identities; its predictive usefulness for realistic LoRA training remains limited.
- SNIP is only loosely connected to the recovery objective and frequently fails in the controlled selector experiments (Appendix E.1).
- Comparisons beyond Uni-LoRA are less controlled: most GLUE baselines are quoted, while RoSA uses a substantially different budget.
- Math and IFEval gains are small relative to seed variability. Parameter savings also do not yield lower measured training time or peak memory versus LoRA.
- The improvements are marginal based on the provided experimental results, shown in Table 1.
Questions for authors
Can the theory help choose the compressed-to-sparse budget ratio without an extensive hyperparameter search?
Can the theory reduce the search for the budget split?
No. We select d:K by validation and have not shown that the theory reduces the number of splits we need to test. In the equal-rank sequence model, Eq. (8) specializes the classical projection-risk identity to a fixed budget. Moving K parameters to a fixed sparse support lowers risk precisely when joint refitting recovers enough residual signal to outweigh the opportunity cost of shrinking the shared branch. Eq. (29) also accounts for the change in retained noise when the fitting data select the support. We lack estimates of these quantities before fine-tuning on a new task. At B = 23,040, the highest means among the tested splits occurred at d:K = 5:3 on CoLA and 31:1 on MRPC (Table 12).
Warmup SNIP chooses coordinates given K. In Appendix E.1, pointwise paired 95% intervals favored compression alone over SNIP in 7 of 12 conditions for the concentrated teacher and 8 of 12 for the diffuse teacher. A target-informed selector improved on compression in all 12 conditions for each teacher.
What does the empirical evidence support?
On STS-B, increasing Gaussian training-label noise shifted the MSE gap toward compression, as predicted. From zero to the highest noise level, the gap on the original development labels (LoRA minus Uni-LoRA) increased by 0.2740 (five paired training resamples, 95% CI [0.1476, 0.4003]).
Table 1 includes published GLUE baselines. In reruns matched by budget, splits, and seeds, ProLoSA gained 0.511 points over Uni-LoRA on the six-task RoBERTa-large average (five seeds, paired 95% CI [0.042, 0.981], Table 6). Only MRPC's task-level interval excluded zero.
The eight-task mean gains were 1.194 points on ViT-Base and 0.352 on ViT-Large. StanfordCars and FGVC Aircraft accounted for about 95% of the net ViT-Base gain (Table 15). On GSM8K, MATH500, IFEval, and IFBench, all four three-seed paired 95% intervals included zero. RoSA provides a separate-budget reference because its rank-one factors alone exceed our budget on the same RoBERTa-large layers (442,368 versus 23,040 parameters, Appendix D.8).