- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes LENS (Loss-aligned Directional Salience Calibration), a method aiming to improve existing low-rank compressed LLMs. The main idea of LENS is that low-rank compression decides which directions survive, but not whether those retained directions contribute with the right strength to the language-modeling objective. LENS therefore adds a post-compression calibration step that keeps the compressed low-rank structure and ranks fixed, but rescales the retained directions so their contributions better align with the end-to-end loss. The paper provides supporting mathematical derivations and theoretical guarantees under limited conditions while the experimental results show that LENS is superior against alternative compression methods.
Critical strengths
- The framing of the problem is new and one of the main contributions of this work. The distinction between subspace preservation and directional calibration is genuine and not addressed in prior work to the best of my knowledge.
- The proposed method can be used in tandem with existing low-rank factorization methods to improve the downstream performance of the compressed model.
- The reported experiments cover several LLM backbones, multiple calibration datasets and several low-rank compression methods.
- The theoretical results are appropriately scoped. The authors are explicit that the provided guarantees are local and do not imply loss decrease at arbitrary finite scale.
Critical weaknesses
-
The calibration cost of the method is non-trivial. LENS requires forward/backward passes, per-module eigen-decompositions, a scale search over a geometric grid, and held-out validation. Can the authors provide a full account of the calibration cost including a) wall-clock time for a full calibration phase and b) memory footprint ? Also can the authors report how the calibration cost scales with the model size and the compression rate?
-
The closest related methods are spectral refinement approaches such as SVS and SVD-Surgeon, which also rescale retained spectral components. While the authors argue that LENS is different because it searches within the full retained output subspace rather than along predetermined singular directions, a direct comparison to those closest baselines would strengthen the claim.
-
The gains vary by setting. For example in Table 1 for Qwen 3-8B the average accuracy gain over Duo-SVD is rather small. Also for higher compression ratios than 20%, in Table 3 there are cases where there are no gains or even SVD-LLM outperforms LENS.
-
Given that LENS utilizes a loss-aware strategy, it would be useful to include comparisons with recent loss-aware low-rank compression methods such as LEMS (Advancing SVD-based LLM Compression via Layer-Wise Error Model Search), ICML 2026. It would be even more useful to show whether LENS can be applied on LEMS and show real gains.
-
Table 2 would be more informative if the performance of the original compressed model was included. This would provide a clearer picture on how suppression and amplification affects the original compressed model.
Questions for authors
Please see the weaknesses section.
7 comments
Response to Reviewer U3kN (part 2)
W2: The closest related methods are spectral refinement approaches such as SVS and SVD-Surgeon, which also rescale retained spectral components. While the authors argue that LENS is different because it searches within the full retained output subspace rather than along predetermined singular directions, a direct comparison to those closest baselines would strengthen the claim.
We thank the reviewer for highlighting the importance of comparisons with related spectral refinement methods. We conduct additional experiments to evaluate the effectiveness of loss-responsive direction selection and compare LENS with adaptations of SVD-Surgeon[1] and SVS[2].
(1). Loss-responsive direction selection improves calibration beyond a fixed spectral basis. To isolate the contribution of direction selection, we replace the response eigenbasis with a fixed SVD-basis while retaining the same calibration framework. The results are presented below:
| Backbone | SVD Basis PPL ↓ | Response Eigenbasis PPL ↓ | SVD Basis Avg. ↑ | Response Eigenbasis Avg. ↑ |
|---|---|---|---|---|
| LLaMA-2-7B | 5.75 | 5.61 | 57.56 | 58.07 |
| LLaMA-3.1-8B | 8.21 | 7.95 | 61.58 | 61.86 |
| Mistral-7B | 5.59 | 5.51 | 60.49 | 61.59 |
| Qwen3-8B | 7.32 | 7.34 | 66.57 | 66.63 |
The response eigenbasis improves average downstream accuracy and achieves lower perplexity. These results support the benefit of identifying loss-responsive directions rather than restricting calibration to a fixed spectral basis.
(2). LENS consistently outperforms the evaluated SVD-Surgeon spectral correction. We compare LENS with SVD-Surgeon, which applies second-order spectral correction under a fixed compression budget. The results are presented below:
| Backbone | SVD-Surgeon PPL ↓ | LENS PPL ↓ | SVD-Surgeon Avg. ↑ | LENS Avg. ↑ |
|---|---|---|---|---|
| LLaMA-2-7B | 6.00 | 5.61 | 56.14 | 58.07 |
| LLaMA-3.1-8B | 9.27 | 7.95 | 59.05 | 61.86 |
| Mistral-7B | 6.15 | 5.51 | 59.12 | 61.59 |
| Qwen3-8B | 7.86 | 7.34 | 66.24 | 66.63 |
Compared with this second-order spectral correction baseline, LENS consistently reduces perplexity by 0.39–1.32 and improves average downstream accuracy by 0.39–2.81 percentage points.
(3). Direct SVS scaling is challenging to transfer to compressed LLMs. We evaluate the direct application of SVS square-root singular-value scaling, both without fine-tuning (No FT) and with subsequent short factor-only fine-tuning (FT). The results are presented below:
| Backbone | SVS (No FT) PPL ↓ | SVS (+ FT) PPL ↓ |
|---|---|---|
| LLaMA-2-7B | 17,128.41 | 141.74 |
| LLaMA-3.1-8B | 2,914.87 | 16.15 |
| Mistral-7B | 93.38 | 6.43 |
| Qwen3-8B | 9,639.67 | 5,850.36 |
Direct SVS scaling causes substantial perplexity degradation across all four backbones. Although fine-tuning partially recovers performance, the resulting models remain worse than their pre-scaling counterparts.
Together, these results support the value of LENS's loss-responsive direction selection within the retained output subspace, beyond calibration along predetermined spectral directions.
References:
[1]: SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression, NeurIPS 2026 Workshop.
[2]: Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement, AAAI 2025.
Response to Reviewer U3kN (part 3)
W3: The gains vary by setting. For example in Table 1 for Qwen 3-8B the average accuracy gain over Duo-SVD is rather small. Also for higher compression ratios than 20%, in Table 3 there are cases where there are no gains or even SVD-LLM outperforms LENS.
We thank the reviewer for this observation. We clarify the variation in performance gains and provide additional experiments at higher compression ratios.
(1). LENS consistently improves recovery across all four backbones. Although the average accuracy gain on Qwen3-8B is modest (+0.11 percentage points), LENS consistently improves both perplexity and average downstream accuracy over Duo-SVD across all four backbones, with perplexity reductions of 0.20–0.73 and accuracy gains of 0.11–1.98 percentage points.
(2). LENS further improves SVD-LLM at high compression rates. We acknowledge that SVD-LLM achieves higher downstream accuracy than Duo-SVD+LENS in some high-compression settings. To further investigate this, we directly apply LENS to SVD-LLM-compressed models at 30% and 40% compression. The results are presented below:
| Backbone | Ratio | SVD-LLM PPL ↓ | + LENS PPL ↓ | SVD-LLM Avg. ↑ | + LENS Avg. ↑ |
|---|---|---|---|---|---|
| LLaMA-2-7B | 30% | 7.85 | 7.10 | 52.01 | 52.15 |
| 40% | 9.04 | 8.02 | 49.25 | 49.30 | |
| Mistral-7B | 30% | 8.07 | 7.31 | 55.30 | 55.62 |
| 40% | 9.91 | 8.38 | 50.23 | 50.73 |
Applying LENS to SVD-LLM reduces perplexity by 9.42–15.44% and improves average downstream accuracy across all four settings, demonstrating its effectiveness in further refining highly compressed models.
W4: Given that LENS utilizes a loss-aware strategy, it would be useful to include comparisons with recent loss-aware low-rank compression methods such as LEMS (Advancing SVD-based LLM Compression via Layer-Wise Error Model Search), ICML 2026. It would be even more useful to show whether LENS can be applied on LEMS and show real gains.
We thank the reviewer for this valuable suggestion. We conduct additional experiments comparing LENS with LEMS and further applying LENS to LEMS-compressed models. We would like to explain it as follows.
(1). Duo-SVD+LENS remains competitive with LEMS. We compare Duo-SVD+LENS, LEMS, and LEMS+LENS across four backbones. The results are presented below:
| Backbone | Method | PPL ↓ | Avg. ↑ |
|---|---|---|---|
| LLaMA-2-7B | Duo-SVD+LENS | 5.61 | 58.07 |
| LEMS | 5.66 | 58.41 | |
| LEMS+LENS | 5.32 | 59.93 | |
| LLaMA-3.1-8B | Duo-SVD+LENS | 7.95 | 61.86 |
| LEMS | 7.66 | 55.98 | |
| LEMS+LENS | 7.04 | 59.33 | |
| Mistral-7B | Duo-SVD+LENS | 5.51 | 61.59 |
| LEMS | 5.58 | 58.31 | |
| LEMS+LENS | 5.18 | 61.18 | |
| Qwen3-8B | Duo-SVD+LENS | 7.34 | 66.63 |
| LEMS | 6.97 | 66.81 | |
| LEMS+LENS | 6.82 | 68.42 |
(2). LENS consistently improves LEMS-compressed models across all four backbones. Incorporating LENS reduces perplexity by 2.15–8.09% and improves average downstream accuracy by 1.52–3.35 percentage points across all four backbones. This demonstrates that LENS provides effective and complementary post-compression refinement beyond LEMS.
W5: Table 2 would be more informative if the performance of the original compressed model was included. This would provide a clearer picture on how suppression and amplification affects the original compressed model.
We thank the reviewer for this helpful suggestion. The original compressed model is Duo-SVD, whose results are reported in Table 1. For clarity, we present these results together with the ablation variants from Table 4 below:
| Variant | LLaMA-2-7B PPL ↓ | Avg. ↑ | Qwen3-8B PPL ↓ | Avg. ↑ | LLaMA-3.1-8B PPL ↓ | Avg. ↑ |
|---|---|---|---|---|---|---|
| Duo-SVD (Original) | 5.95 | 56.59 | 7.54 | 66.52 | 8.68 | 60.30 |
| V1 (Amplify only) | 5.76 | 57.87 | 7.40 | 66.34 | 8.00 | 61.63 |
| V2 (Suppress only) | 5.83 | 56.67 | 7.52 | 66.45 | 8.58 | 60.30 |
| V3 (Joint, No reprofiling) | 5.85 | 57.89 | 7.36 | 65.94 | 8.00 | 61.65 |
| V4 (Reverse order) | 5.67 | 57.68 | 7.36 | 66.53 | 8.03 | 61.73 |
| LENS | 5.61 | 58.07 | 7.34 | 66.63 | 7.95 | 61.86 |
Compared with the original Duo-SVD model, both Amplify-only and Suppress-only reduce perplexity, with Amplify providing larger improvements. The complete LENS further achieves the lowest perplexity and highest average downstream accuracy, demonstrating the complementary contributions of both stages and the effectiveness of their coordinated calibration.
Response to Reviewer U3kN (part 1)
We sincerely thank the reviewer for the detailed and constructive feedback. We have also updated the manuscript accordingly, with the revisions highlighted in blue. Below, we address the concerns mentioned in the Weaknesses and Questions.
We thank the reviewer for highlighting the importance of calibration efficiency. We report detailed computational costs across different backbones and compression ratios as follows.
(1). Full LENS calibration takes 86.82–114.74 minutes across four backbones. We measure the complete LENS calibration at 20% compression across four backbones. The stage-wise timings (in minutes) and peak GPU memory (GiB) are reported below:
Direction discovery and response fitting account for 90.6–93.2% of the core calibration time, while scale search and held-out validation contribute substantially less overhead.
(2). Higher compression ratios reduce calibration time and memory consumption. We further evaluate the calibration cost under compression ratios ranging from 10% to 40% on LLaMA-2-7B. The results are presented below:
As the compression ratio increases from 10% to 40%, calibration time decreases by 19.1%, while peak GPU memory decreases by 15.3%. This trend indicates that more aggressive compression does not necessarily incur higher calibration costs.