- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of compressing LoRA updates to maximize the output fidelity using only a few unlabeled calibration data. It does so by first whitening the bottleneck's activations using their uncentered second moment, and then computes the Gram matrix of the Jacobian of the final logits with respect to the whitened activations, and keeping the leading eigenvectors of this matrix as the retained components. At low rank budgets WAS-LoRA (the proposed method) keeps more accuracy with respect to other baselines and methods, while being robust to the calibration set size.
Critical strengths
The paper is very easy to follow, the standard notation helps the understanding as well as the figures, the algorithm and tables. It's generally well written and the narrative is self-contained. The idea is clear and correct and the experimental protocol is sound.
Critical weaknesses
In my opinion, while the experimental protocol is correct, it should be applied to models with a realistic practical gain. As shown in Table 8, data free SVD already matches WAS-LoRA at on these tasks and models.
A graph showing how the cost of the method scales with the number of parameters or the output dimension is not reported.
Bigger models on more significant tasks should be tested and the required time and memory should be highlighted. When does it reach the break even point with training a new adapter?
From my understading theorem one is a reformulation of the Eckart-Young theorem. Can the authors clarify the novelty of this result?
Questions for authors
See weaknesses.
11 comments
Response: Compression cost versus output dimension and backbone size
Response to Weakness (Compression cost vs. output dimension and backbone size)
All manuscript references in this reply are to the revised manuscript.
Following your suggestion, we added time and GPU-memory curves for output dimension and backbone size. For output dimension, we profile Qwen2.5-1.5B-Instruct using four calibration batches, eight probes, and the full output head resident in memory. The timed interval covers sensitivity-statistics collection and CPU whitening/spectral processing. The output-width endpoints are:
| Computed output dimension (logits per token) | Median wall time (s) | Incremental peak allocated GPU memory (MiB) |
|---|---|---|
| 256 | 2.649 | 462.66 |
| 151,936 | 2.700 | 650.35 |
At full output width, increasing the probe count from 1 to 16 raises median time from 0.886 to 5.033 seconds (Appendix E.5.1, Figures 7–8).
For backbone size, we added measurements across four backbones using the same four calibration batches, 151,936 logits per token, and eight probes on one A800 per run in FP32. B4 denotes the four-batch pipeline comprising statistics collection, CPU spectral processing, rank allocation, and factor construction at a parameter budget corresponding to uniform rank four.
| Backbone parameters | B4 time (s), median [min, max] | Peak allocated GPU memory (GiB) | Incremental peak GPU memory (MiB) |
|---|---|---|---|
| 1.54B | 3.726 [3.722, 3.742] | 6.47 | 650 |
| 3.09B | 5.796 [5.705, 5.886] | 12.53 | 931 |
| 7.62B | 5.556 [5.526, 5.575] | 29.83 | 1,177 |
| 14.77B | 12.499 [12.468, 12.523] | 56.97 | 1,717 |
Each configuration uses five timed repeats. Median statistics-collection time increases from 2.693 to 9.192 seconds across these endpoints. Total B4 time is not strictly monotonic: rank allocation takes 2.215 seconds at 3B versus 0.549 seconds at 7B. Adapter tasks and drivers differ across configurations, so these measurements characterize the tested workloads (Appendix E.5.2, Figure 9).
Response: Larger-model evidence, runtime, and memory
Response to Weakness (Larger-model evidence, runtime, and memory)
All manuscript references in this reply are to the revised manuscript.
Following your suggestion, we extended the task evaluation to Qwen2.5-7B-Instruct on GSM8K and ARC-Challenge. WAS-LoRA reduces both centered relative logit error and teacher-to-compressed KL in all 18 matched teacher–budget comparisons against Output-scored Act-SVD. The new experiment covers three independently trained rank-16 teachers per task, three common physical adapter-parameter budgets, and four compression methods, with all 72 compressed configurations evaluated on the full test sets (Appendix F.8, Table 33).
Across the 18 within-teacher pairs, the geometric-mean centered-error and KL ratios are 0.8323 and 0.7843, corresponding to reductions of 16.8% and 21.6%. The centered-error reduction is 3.0% on GSM8K and 28.6% on ARC-Challenge. The table below reports the per-budget fidelity ratios alongside greedy exact match; the task-score differences do not show uniform accuracy gains (Table 34).
| Task | Budget | Centered-error ratio (W/O), geometric mean | KL ratio (W/O), geometric mean | WAS mean exact match (%) | Output-scored mean exact match (%) |
|---|---|---|---|---|---|
| GSM8K | B2 | 0.97933 | 0.95262 | 74.53 | 73.97 |
| GSM8K | B4 | 0.96786 | 0.93493 | 73.74 | 74.07 |
| GSM8K | B8 | 0.96379 | 0.93315 | 74.15 | 74.15 |
| ARC-Challenge | B2 | 0.70609 | 0.67189 | 89.42 | 89.25 |
| ARC-Challenge | B4 | 0.69117 | 0.62524 | 89.16 | 89.08 |
| ARC-Challenge | B8 | 0.74566 | 0.66676 | 89.08 | 89.14 |
W/O denotes WAS-LoRA / Output-scored Act-SVD. B2/B4/B8 are physical parameter budgets equivalent to uniform ranks 2/4/8; WAS-LoRA and Output-scored Act-SVD have identical actual parameter counts within each pair. Each ratio is the geometric mean of three within-teacher ratios, while exact match is an arithmetic mean. Budgets are repeated within six distinct teachers.
We also measured physically compact, exported 7B adapters. Adapter files decrease from 161.53 MB for the rank-16 teachers to 20.26, 40.45, and 80.82 MB, corresponding to 87.5%, 75%, and 50% reductions in adapter parameters. For the fixed 128-input/32-output-token workload on one A800 in FP32, the compressed WAS configurations have per-configuration median batch-one generation times of 1.274–1.378 s (ten timed runs after three warm-ups) and absolute peak allocated GPU memory of approximately 28.59–28.65 GiB across the tested tasks, teachers, and budgets; Table 35 reports the teacher references and both batch sizes. Table 36 separately reports calibration-trajectory generation, sensitivity-statistics collection, and CPU construction. These task-level results complement the fixed-workload cost measurements up to 14.77B parameters in Appendix E.5.2: the larger-model fidelity benefit is now supported by full task evaluation, as well as measurements of deployable adapters.
Response: Practical gains ,the data-free SVD comparison,Theorem 1, Eckart–Young, and the methodological contribution
All manuscript references in this reply are to the revised manuscript.
Response to Weakness (Practical gains and the data-free SVD comparison)
Thank you for these questions about practical value, computational cost, and theoretical novelty. Following your suggestions, we have expanded the evaluations in the revised manuscript.
Our practical objective is to preserve the trained teacher's behavior under a constrained adapter budget. Similar task accuracy does not imply equivalent preservation of the teacher's logits or predictive probabilities. We therefore evaluate centered relative logit error and teacher-to-compressed KL alongside accuracy. At matched adapter budgets, WAS-LoRA reduces overall geometric-mean centered relative logit error by 47.7% relative to data-free Uniform-SVD (Section 4.1, Table 1). This demonstrates a measurable improvement in teacher-output preservation beyond what task accuracy alone captures.
Response to Question (Theorem 1 and Eckart–Young)
Our methodological contribution is to optimize the retained subspace itself in each trained LoRA module, rather than merely selecting or rescoring components of a prescribed local basis. Downstream sensitivity can therefore select combinations of local SVD directions. We agree that the optimization step in Theorem 1 follows the classical Ky Fan variational principle, closely related to Eckart–Young (Appendix A.1); we do not claim a new spectral theorem. From an active-subspace perspective, a conditional Poincaré bound for the conditional-expectation predictor motivates minimizing discarded downstream Jacobian energy in the whitened LoRA bottleneck. Theorem 1 solves this surrogate, and local least squares reconstructs the selected coordinates as standard lower-rank LoRA factors without recovery training (Sections 2–3, Proposition 2). The ridge-free analysis further establishes the local Act-SVD specialization, refactorization invariance on the supported subspace under matched spectral selections, and the IO-SVD correspondence under matched weighting (Corollary 4 and Propositions 5 and 3).
The practical value of selecting the subspace is supported by a controlled ablation. With per-module ranks, whitening maps, and downstream Grams fixed, WAS-LoRA reduces geometric-mean centered relative logit error by 3.5%–28.8% across all 30 task–budget–allocation settings against fixed-basis rescoring (Section 4.3, Table 2, and Appendix C.2.1). This isolates the benefit of adapting the retained directions.