- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new approach for low-rank compression of LLMs. They provide strong experimental results backing their approach.
Critical strengths
1/ The experimental results of the approach seems to be quite strong.
Critical weaknesses
1/ The paper proposes a more computationally expensive method than some of the baselines; it would be interesting to compare both at matched compute and runtime budgets.
2/ The paper’s approach provides limited novelty. Low-rank selection based on pruning singular values in a differentiable way or not is a common approach in this field, leaving very limited novelty ideas in the paper.
Questions for authors
1/ Can the authors provide more experiments matching compute budget or comparing with other methods using finetuning post compression?
2/ Can the authors argue more about the novelty of the approach? I would be happy to reconsider my review given a strong rebuttal.
9 comments
| CR | Method | Remapping | Wiki2 | PTB | C4 | OBQA | ARC-e | ARC-c | WinoG | HellaS | PIQA | MathQA | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Dense | 5.68 | 8.35 | 7.08 | 0.34 | 0.75 | 0.42 | 0.7 | 0.57 | 0.79 | 0.27 | 0.55 | |
| 0.8 | ZS-SVD | X | 6.76 | 11.47 | 10.11 | 0.3 | 0.72 | 0.38 | 0.67 | 0.49 | 0.74 | 0.25 | 0.51 |
| 0.8 | AA-SVD† | X | 7.68 | 16.34 | 14.87 | 0.29 | 0.64 | 0.31 | 0.64 | 0.43 | 0.68 | 0.24 | 0.46 |
| 0.8 | Dobi-SVD‡ | X | 8.54 | 14.83 | 10.01 | 0.26 | 0.59 | 0.31 | 0.66 | 0.44 | 0.7 | 0.23 | 0.46 |
| 0.8 | S&P | X | 6.27 | 10.54 | 9.18 | 0.29 | 0.72 | 0.37 | 0.68 | 0.51 | 0.76 | 0.25 | 0.51 |
| 0.8 | Dobi-SVD | O | 6.08 | 9.02 | 8.07 | 0.33 | 0.73 | 0.4 | 0.69 | 0.55 | 0.77 | 0.27 | 0.53 |
| 0.8 | ZS-SVD | O | 5.9 | 8.83 | 7.56 | 0.34 | 0.75 | 0.42 | 0.7 | 0.56 | 0.78 | 0.26 | 0.55 |
| 0.8 | S&P | O | 5.79 | 8.67 | 7.31 | 0.32 | 0.73 | 0.41 | 0.69 | 0.56 | 0.78 | 0.27 | 0.54 |
| 0.6 | ZS-SVD | X | 11.58 | 43.45 | 34.58 | 0.21 | 0.52 | 0.26 | 0.62 | 0.35 | 0.63 | 0.22 | 0.4 |
| 0.6 | AA-SVD† | X | 12.01 | 47.23 | 34.76 | 0.2 | 0.44 | 0.23 | 0.57 | 0.33 | 0.59 | 0.23 | 0.37 |
| 0.6 | Dobi-SVD‡ | X | 13.54 | 46.38 | 23.54 | 0.22 | 0.41 | 0.27 | 0.58 | 0.34 | 0.61 | 0.23 | 0.38 |
| 0.6 | S&P | X | 8.28 | 25.93 | 19.02 | 0.24 | 0.6 | 0.29 | 0.61 | 0.4 | 0.68 | 0.23 | 0.43 |
| 0.6 | Dobi-SVD | O | 7.69 | 12.36 | 12.15 | 0.27 | 0.65 | 0.34 | 0.68 | 0.48 | 0.73 | 0.24 | 0.48 |
| 0.6 | ZS-SVD | O | 7 | 12.36 | 10.88 | 0.31 | 0.72 | 0.38 | 0.67 | 0.48 | 0.74 | 0.24 | 0.51 |
| 0.6 | S&P | O | 6.35 | 10.68 | 9.59 | 0.29 | 0.71 | 0.37 | 0.66 | 0.51 | 0.75 | 0.25 | 0.51 |
| 0.4 | ZS-SVD | X | 41.76 | 285.46 | 209.11 | 0.16 | 0.31 | 0.2 | 0.55 | 0.28 | 0.56 | 0.21 | 0.32 |
| 0.4 | AA-SVD† | X | 29.44 | 212.28 | 118.48 | 0.15 | 0.32 | 0.19 | 0.5 | 0.27 | 0.54 | 0.22 | 0.31 |
| 0.4 | Dobi-SVD‡ | X | 46.18 | 238.91 | 190.62 | 0.15 | 0.31 | 0.2 | 0.52 | 0.28 | 0.54 | 0.22 | 0.32 |
| 0.4 | S&P | X | 13.75 | 85.9 | 48.73 | 0.2 | 0.44 | 0.24 | 0.55 | 0.32 | 0.6 | 0.22 | 0.37 |
| 0.4 | Dobi-SVD | O | 14.98 | 49.11 | 43.9 | 0.21 | 0.44 | 0.24 | 0.61 | 0.34 | 0.6 | 0.22 | 0.38 |
| 0.4 | ZS-SVD | O | 13.13 | 51.41 | 45.78 | 0.23 | 0.5 | 0.27 | 0.6 | 0.34 | 0.62 | 0.22 | 0.4 |
| 0.4 | S&P | O | 8.2 | 22.75 | 17.89 | 0.26 | 0.6 | 0.28 | 0.61 | 0.41 | 0.67 | 0.23 | 0.44 |
† AA-SVD evaluated without its block-level local refinement step.
‡ Quoted from the Dobi-SVD paper, as no checkpoint is available; Dobi-SVD with remapping was re-run by us.
Our results show that the S&P compressed model is substantially more accurate, with the largest gains under aggressive compression; at CR 0.4, S&P reaches a WikiText-2 perplexity of 13.75, compared with 29.44 for the strongest evaluated no fine-tuning / weight update baseline and zero-shot accuracy average of 0.37 compared to 0.32.
The same one-time-cost argument applies to the baselines: if extra compression time is acceptable, a practitioner could instead spend it fine-tuning a cheaper method after compression. To test this directly, we compare S&P against baselines that use a post-compression update: AA-SVD with block-level local refinement, ZS-SVD with its projected-gradient weight updates, and SVD-LLM with LoRA fine-tuning. We also apply AA-SVD's block-level refinement to S&P, using the same 256 calibration sequences, to check whether S&P's advantage persists once both sides are fine-tuned. All results are on LLaMA-1-7B at CR 0.8, 0.6 and 0.4:
| CR | Method | Fine-tuning / Weight Update | Update Data | Update Samples | Wiki2 | PTB | C4 | OBQA | ARC-e | ARC-c | WinoG | HellaS | PIQA | MathQA | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Dense | 5.68 | 8.35 | 7.08 | 0.34 | 0.75 | 0.42 | 0.70 | 0.57 | 0.79 | 0.27 | 0.55 | |||
| 0.8 | AA-SVD | O | Wiki-2 (Calib. Set) | 256 | 6.90 | 12.32 | 11.43 | 0.30 | 0.71 | 0.36 | 0.66 | 0.48 | 0.72 | 0.25 | 0.50 |
| 0.8 | ZS-SVD 5x‖ | O | Wiki-2 (Calib. Set) | 20 | 6.43 | 11.17 | 10.42 | 0.32 | 0.72 | 0.39 | 0.67 | 0.50 | 0.75 | 0.25 | 0.51 |
| 0.8 | SVD-LLM + LoRA* | O | Alpaca | 50000 | 7.73 | - | 12.23 | 0.33 | 0.67 | - | 0.69 | 0.55 | 0.79 | 0.26 | - |
| 0.8 | S&P | X | X | - | 6.27 | 10.54 | 9.18 | 0.29 | 0.72 | 0.37 | 0.68 | 0.51 | 0.76 | 0.25 | 0.51 |
| 0.8 | S&P + Refinement§ | O | Wiki-2 (Calib. Set) | 256 | 6.10 | 9.73 | 8.84 | 0.30 | 0.72 | 0.38 | 0.67 | 0.52 | 0.77 | 0.26 | 0.52 |
| 0.6 | AA-SVD | O | Wiki-2 (Calib. Set) | 256 | 8.35 | 25.05 | 18.51 | 0.26 | 0.61 | 0.31 | 0.62 | 0.41 | 0.66 | 0.23 | 0.44 |
| 0.6 | ZS-SVD 5x‖ | O | Wiki-2 (Calib. Set) | 20 | 9.45 | 36.52 | 26.20 | 0.23 | 0.56 | 0.29 | 0.62 | 0.38 | 0.66 | 0.22 | 0.42 |
| 0.6 | SVD-LLM + LoRA* | O | Alpaca | 50000 | 9.27 | - | 15.63 | 0.29 | 0.59 | - | 0.68 | 0.52 | 0.69 | 0.20 | - |
| 0.6 | S&P | X | X | - | 8.28 | 25.93 | 19.02 | 0.24 | 0.60 | 0.29 | 0.61 | 0.40 | 0.68 | 0.23 | 0.43 |
| 0.6 | S&P + Refinement§ | O | Wiki-2 (Calib. Set) | 256 | 7.40 | 19.55 | 14.10 | 0.26 | 0.65 | 0.31 | 0.64 | 0.44 | 0.70 | 0.24 | 0.46 |
| 0.4 | AA-SVD | O | Wiki-2 (Calib. Set) | 256 | 13.61 | 74.17 | 45.79 | 0.18 | 0.44 | 0.23 | 0.56 | 0.33 | 0.60 | 0.23 | 0.37 |
| 0.4 | ZS-SVD 10x‖ | O | Wiki-2 (Calib. Set) | 40 | 18.49 | 144.89 | 76.86 | 0.17 | 0.38 | 0.23 | 0.55 | 0.30 | 0.58 | 0.22 | 0.35 |
| 0.4 | SVD-LLM + LoRA* | O | Alpaca | 50000 | 31.79 | - | 43.71 | 0.11 | 0.23 | - | 0.21 | 0.14 | 0.17 | 0.08 | - |
| 0.4 | S&P | X | X | - | 13.75 | 85.90 | 48.73 | 0.20 | 0.44 | 0.24 | 0.55 | 0.32 | 0.60 | 0.22 | 0.37 |
| 0.4 | S&P + Refinement§ | O | Wiki-2 (Calib. Set) | 256 | 10.46 | 47.97 | 27.36 | 0.22 | 0.49 | 0.24 | 0.58 | 0.36 | 0.62 | 0.23 | 0.39 |
* SVD-LLM fine-tuned with LoRA on 50K Alpaca samples; "-" denotes results not reported.
§ S&P followed by AA-SVD's block-level local refinement on the same 256 calibration sequences.
‖ ZS-SVD (ZeroSum SVD) with 5× / 10× denotes the number of correction iterations.
We thank Reviewer ooz4 for the careful and constructive review. We respond to each of the reviewer's concerns in detail below.
1. Comparison of methods with matched compute
We thank the reviewer for raising concerns about compute and runtime cost. Compression is a one-time, pre-deployment cost, whereas the resulting accuracy gains persist for the lifetime of the deployed model. The additional compression time of S&P is therefore well justified. S&P takes 99.4 minutes to compress LLaMA-1-7B on a single H200, compared with 11.7 minutes for SVD-LLM and 8.7 minutes for ZS-SVD (Table 6). For fair comparison, we have re-evaluated relevant strong SVD baselines (AA-SVD, Dobi-SVD, ZS-SVD) alongside applying remapping under identical experimental and evaluation settings; only Dobi-SVD without remapping is quoted from its paper.