acceptodds
Under review as a conference paper at ICLR 2027

What Survives Is Not Enough: Loss-Aligned Directional Salience Calibration for LLM Compression

Abstract

Low-rank decomposition facilitates the deployment of large language models (LLMs) by representing dense transformations within compact low-dimensional subspaces. Existing approaches primarily optimize reconstruction objectives to preserve approximation fidelity during compression. However, preserving a direction does not guarantee that its contribution is properly calibrated to the end-to-end language-modeling objective. Towards this end, we propose a novel post-compression framework termed Loss-aligned Directional Salience Calibration (LENS) for LLM compression. The core of LENS is to recalibrate what survives compression through two complementary components: loss-aligned direction identification and state-adaptive salience calibration. In particular, we characterize the first-order loss response to directional rescaling within the retained subspace and aggregate these responses into a response matrix whose eigendirections reveal modes that locally favor amplification or suppression. Subsequently, we estimate direction-specific relative gains using a response-based quadratic surrogate and calibrate their joint magnitude through a shared stage-level scale. The resulting calibration is accepted only when it achieves reliable loss reduction on a held-out validation set and is then folded directly into the existing low-rank factors, preserving the retained subspaces and ranks without introducing additional inference-time parameters or operations. Extensive experiments across multiple LLM backbones and downstream tasks demonstrate the effectiveness of LENS. We further show that LENS can be integrated with different low-rank compressors while improving their performance, supporting its use as a general post-compression calibration framework. The code is available at https://lens1.github.io/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Low-Rank Compression, Post-Compression Calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U3kNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes LENS (Loss-aligned Directional Salience Calibration), a method aiming to improve existing low-rank compressed LLMs. The main idea of LENS is that low-rank compression decides which directions survive, but not whether those retained directions contribute with the right strength to the language-modeling objective. LENS therefore adds a post-compression calibration step that keeps the compressed low-rank structure and ranks fixed, but rescales the retained directions so their contributions better align with the end-to-end loss. The paper provides supporting mathematical derivations and theoretical guarantees under limited conditions while the experimental results show that LENS is superior against alternative compression methods.

Critical strengths
  1. The framing of the problem is new and one of the main contributions of this work. The distinction between subspace preservation and directional calibration is genuine and not addressed in prior work to the best of my knowledge.
  2. The proposed method can be used in tandem with existing low-rank factorization methods to improve the downstream performance of the compressed model.
  3. The reported experiments cover several LLM backbones, multiple calibration datasets and several low-rank compression methods.
  4. The theoretical results are appropriately scoped. The authors are explicit that the provided guarantees are local and do not imply loss decrease at arbitrary finite scale.
Critical weaknesses
  1. The calibration cost of the method is non-trivial. LENS requires forward/backward passes, per-module eigen-decompositions, a scale search over a geometric grid, and held-out validation. Can the authors provide a full account of the calibration cost including a) wall-clock time for a full calibration phase and b) memory footprint ? Also can the authors report how the calibration cost scales with the model size and the compression rate?

  2. The closest related methods are spectral refinement approaches such as SVS and SVD-Surgeon, which also rescale retained spectral components. While the authors argue that LENS is different because it searches within the full retained output subspace rather than along predetermined singular directions, a direct comparison to those closest baselines would strengthen the claim.

  3. The gains vary by setting. For example in Table 1 for Qwen 3-8B the average accuracy gain over Duo-SVD is rather small. Also for higher compression ratios than 20%, in Table 3 there are cases where there are no gains or even SVD-LLM outperforms LENS.

  4. Given that LENS utilizes a loss-aware strategy, it would be useful to include comparisons with recent loss-aware low-rank compression methods such as LEMS (Advancing SVD-based LLM Compression via Layer-Wise Error Model Search), ICML 2026. It would be even more useful to show whether LENS can be applied on LEMS and show real gains.

  5. Table 2 would be more informative if the performance of the original compressed model was included. This would provide a clearer picture on how suppression and amplification affects the original compressed model.

Questions for authors

Please see the weaknesses section.

7 comments
AuthorsOpenReview ↗

Response to Reviewer U3kN (part 1)

We sincerely thank the reviewer for the detailed and constructive feedback. We have also updated the manuscript accordingly, with the revisions highlighted in blue. Below, we address the concerns mentioned in the Weaknesses and Questions.

W1: The calibration cost of the method is non-trivial. LENS requires forward/backward passes, per-module eigen-decompositions, a scale search over a geometric grid, and held-out validation. Can the authors provide a full account of the calibration cost including a) wall-clock time for a full calibration phase and b) memory footprint ? Also can the authors report how the calibration cost scales with the model size and the compression rate?

We thank the reviewer for highlighting the importance of calibration efficiency. We report detailed computational costs across different backbones and compression ratios as follows.

(1). Full LENS calibration takes 86.82–114.74 minutes across four backbones. We measure the complete LENS calibration at 20% compression across four backbones. The stage-wise timings (in minutes) and peak GPU memory (GiB) are reported below:

StageLLaMA-2-7BLLaMA-3.1-8BMistral-7BQwen3-8B
Amplify: Discovery19.3020.9019.6726.06
Amplify: Fitting19.4220.8619.6626.25
Amplify: Scale Search & Validation4.865.495.535.01
Suppress: Discovery19.2920.8619.6526.02
Suppress: Fitting19.4720.8819.6226.27
Suppress: Scale Search & Validation2.302.752.652.57
Full Calibration Time86.8294.2489.09114.74
Peak GPU Memory20.7125.3221.5527.95

Direction discovery and response fitting account for 90.6–93.2% of the core calibration time, while scale search and held-out validation contribute substantially less overhead.

(2). Higher compression ratios reduce calibration time and memory consumption. We further evaluate the calibration cost under compression ratios ranging from 10% to 40% on LLaMA-2-7B. The results are presented below:

Compression RatioTime (min) ↓Peak GPU Mem. (GiB) ↓
10%87.8821.67
20%86.8220.71
30%80.1919.53
40%71.0718.36

As the compression ratio increases from 10% to 40%, calibration time decreases by 19.1%, while peak GPU memory decreases by 15.3%. This trend indicates that more aggressive compression does not necessarily incur higher calibration costs.

AuthorsOpenReview ↗

Response to Reviewer U3kN (part 2)

W2: The closest related methods are spectral refinement approaches such as SVS and SVD-Surgeon, which also rescale retained spectral components. While the authors argue that LENS is different because it searches within the full retained output subspace rather than along predetermined singular directions, a direct comparison to those closest baselines would strengthen the claim.

We thank the reviewer for highlighting the importance of comparisons with related spectral refinement methods. We conduct additional experiments to evaluate the effectiveness of loss-responsive direction selection and compare LENS with adaptations of SVD-Surgeon[1] and SVS[2].

(1). Loss-responsive direction selection improves calibration beyond a fixed spectral basis. To isolate the contribution of direction selection, we replace the response eigenbasis with a fixed SVD-basis while retaining the same calibration framework. The results are presented below:

BackboneSVD Basis PPL ↓Response Eigenbasis PPL ↓SVD Basis Avg. ↑Response Eigenbasis Avg. ↑
LLaMA-2-7B5.755.6157.5658.07
LLaMA-3.1-8B8.217.9561.5861.86
Mistral-7B5.595.5160.4961.59
Qwen3-8B7.327.3466.5766.63

The response eigenbasis improves average downstream accuracy and achieves lower perplexity. These results support the benefit of identifying loss-responsive directions rather than restricting calibration to a fixed spectral basis.

(2). LENS consistently outperforms the evaluated SVD-Surgeon spectral correction. We compare LENS with SVD-Surgeon, which applies second-order spectral correction under a fixed compression budget. The results are presented below:

BackboneSVD-Surgeon PPL ↓LENS PPL ↓SVD-Surgeon Avg. ↑LENS Avg. ↑
LLaMA-2-7B6.005.6156.1458.07
LLaMA-3.1-8B9.277.9559.0561.86
Mistral-7B6.155.5159.1261.59
Qwen3-8B7.867.3466.2466.63

Compared with this second-order spectral correction baseline, LENS consistently reduces perplexity by 0.39–1.32 and improves average downstream accuracy by 0.39–2.81 percentage points.

(3). Direct SVS scaling is challenging to transfer to compressed LLMs. We evaluate the direct application of SVS square-root singular-value scaling, both without fine-tuning (No FT) and with subsequent short factor-only fine-tuning (FT). The results are presented below:

BackboneSVS (No FT) PPL ↓SVS (+ FT) PPL ↓
LLaMA-2-7B17,128.41141.74
LLaMA-3.1-8B2,914.8716.15
Mistral-7B93.386.43
Qwen3-8B9,639.675,850.36

Direct SVS scaling causes substantial perplexity degradation across all four backbones. Although fine-tuning partially recovers performance, the resulting models remain worse than their pre-scaling counterparts.

Together, these results support the value of LENS's loss-responsive direction selection within the retained output subspace, beyond calibration along predetermined spectral directions.

References:

[1]: SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression, NeurIPS 2026 Workshop.

[2]: Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement, AAAI 2025.

AuthorsOpenReview ↗

Response to Reviewer U3kN (part 3)

W3: The gains vary by setting. For example in Table 1 for Qwen 3-8B the average accuracy gain over Duo-SVD is rather small. Also for higher compression ratios than 20%, in Table 3 there are cases where there are no gains or even SVD-LLM outperforms LENS.

We thank the reviewer for this observation. We clarify the variation in performance gains and provide additional experiments at higher compression ratios.

(1). LENS consistently improves recovery across all four backbones. Although the average accuracy gain on Qwen3-8B is modest (+0.11 percentage points), LENS consistently improves both perplexity and average downstream accuracy over Duo-SVD across all four backbones, with perplexity reductions of 0.20–0.73 and accuracy gains of 0.11–1.98 percentage points.

(2). LENS further improves SVD-LLM at high compression rates. We acknowledge that SVD-LLM achieves higher downstream accuracy than Duo-SVD+LENS in some high-compression settings. To further investigate this, we directly apply LENS to SVD-LLM-compressed models at 30% and 40% compression. The results are presented below:

BackboneRatioSVD-LLM PPL ↓+ LENS PPL ↓SVD-LLM Avg. ↑+ LENS Avg. ↑
LLaMA-2-7B30%7.857.1052.0152.15
40%9.048.0249.2549.30
Mistral-7B30%8.077.3155.3055.62
40%9.918.3850.2350.73

Applying LENS to SVD-LLM reduces perplexity by 9.42–15.44% and improves average downstream accuracy across all four settings, demonstrating its effectiveness in further refining highly compressed models.

W4: Given that LENS utilizes a loss-aware strategy, it would be useful to include comparisons with recent loss-aware low-rank compression methods such as LEMS (Advancing SVD-based LLM Compression via Layer-Wise Error Model Search), ICML 2026. It would be even more useful to show whether LENS can be applied on LEMS and show real gains.

We thank the reviewer for this valuable suggestion. We conduct additional experiments comparing LENS with LEMS and further applying LENS to LEMS-compressed models. We would like to explain it as follows.

(1). Duo-SVD+LENS remains competitive with LEMS. We compare Duo-SVD+LENS, LEMS, and LEMS+LENS across four backbones. The results are presented below:

BackboneMethodPPL ↓Avg. ↑
LLaMA-2-7BDuo-SVD+LENS5.6158.07
LEMS5.6658.41
LEMS+LENS5.3259.93
LLaMA-3.1-8BDuo-SVD+LENS7.9561.86
LEMS7.6655.98
LEMS+LENS7.0459.33
Mistral-7BDuo-SVD+LENS5.5161.59
LEMS5.5858.31
LEMS+LENS5.1861.18
Qwen3-8BDuo-SVD+LENS7.3466.63
LEMS6.9766.81
LEMS+LENS6.8268.42

(2). LENS consistently improves LEMS-compressed models across all four backbones. Incorporating LENS reduces perplexity by 2.15–8.09% and improves average downstream accuracy by 1.52–3.35 percentage points across all four backbones. This demonstrates that LENS provides effective and complementary post-compression refinement beyond LEMS.

W5: Table 2 would be more informative if the performance of the original compressed model was included. This would provide a clearer picture on how suppression and amplification affects the original compressed model.

We thank the reviewer for this helpful suggestion. The original compressed model is Duo-SVD, whose results are reported in Table 1. For clarity, we present these results together with the ablation variants from Table 4 below:

VariantLLaMA-2-7B PPL ↓Avg. ↑Qwen3-8B PPL ↓Avg. ↑LLaMA-3.1-8B PPL ↓Avg. ↑
Duo-SVD (Original)5.9556.597.5466.528.6860.30
V1 (Amplify only)5.7657.877.4066.348.0061.63
V2 (Suppress only)5.8356.677.5266.458.5860.30
V3 (Joint, No reprofiling)5.8557.897.3665.948.0061.65
V4 (Reverse order)5.6757.687.3666.538.0361.73
LENS5.6158.077.3466.637.9561.86

Compared with the original Duo-SVD model, both Amplify-only and Suppress-only reduce perplexity, with Amplify providing larger improvements. The complete LENS further achieves the lowest perplexity and highest average downstream accuracy, demonstrating the complementary contributions of both stages and the effectiveness of their coordinated calibration.

Reviewer_iXsbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SVD-based compression methods attempt to reduce the size and inference latency of large models by decomposing large weight matrices into a multiplication of two low rank matrices. Existing methods primarily focus on finding low rank decompositions that achieve better performance at similar compression ratios. This work instead introduces a cheap algorithm for improving the performance of a model after undergoing SVD-based compression. The authors do this by developing a method which amplifies or suppresses the direction that survive the original SVD compression. The authors validate their method by applying their method to models compressed by a variety of SVD-based compression methods. Across multiple model architectures and compression methods, they show that their approach consistently improves the performance of the compressed model.

Critical strengths
  • Lightweight correction steps for post-training compression methods, to my knowledge, is relatively under explored. The method as presented seems both novel and a reasonable approach for improving the performance of SVD compression.
Critical weaknesses
  • While the experiments show that the proposed method consistently improves the performance of SVD-based compression methods, the authors do not justify their method by comparing with methods similar to theirs. For example, a straightforward method of improving the performance after compression would be to use PEFT. This is an approach that is explored in SVD-LLM. Another recently published SVD-based compression method, Zero-sum SVD, also introduces a lightweight post-compression stage to improve performance. In my view, the authors must compare the performance improvement of their proposed method with other strategies while additionally comparing the amount of time and compute necessary for each.
Questions for authors

Can the authors provide a comparison of their method with other methods that attempt to improve the performance after compression?

6 comments
AuthorsOpenReview ↗

Response to Reviewer iXsb

We sincerely thank the reviewer for the constructive comments. We have also updated the manuscript accordingly, with the revisions highlighted in blue. Below, we address the concerns raised in the Weaknesses and Questions.

W1 & Q1: While the experiments show that the proposed method consistently improves the performance of SVD-based compression methods, the authors do not justify their method by comparing with methods similar to theirs. For example, a straightforward method of improving the performance after compression would be to use PEFT. This is an approach that is explored in SVD-LLM. Another recently published SVD-based compression method, Zero-sum SVD, also introduces a lightweight post-compression stage to improve performance. In my view, the authors must compare the performance improvement of their proposed method with other strategies while additionally comparing the amount of time and compute necessary for each.

We thank the reviewer for this valuable suggestion. We conduct additional experiments with three representative PEFT methods, including LoRA[1], AdaLoRA[2], and DoRA[3], on Mistral-7B at 20% compression. We would like to explain it as follows.

(1). LENS achieves better perplexity and downstream accuracy than PEFT refinements. We compare the perplexity and average accuracy across seven downstream tasks. The results are presented below:

MethodPPL ↓Avg. ↑
LoRA7.0953.88
AdaLoRA5.9557.63
DoRA7.0453.79
LENS5.5161.59

LENS achieves the lowest perplexity and highest average downstream accuracy. Compared with the strongest PEFT baseline, LENS further reduces perplexity by 0.44 and improves average accuracy by 3.96 percentage points.

(2). LENS achieves stronger recovery with lower offline time. We further measure refinement time and peak GPU memory. The results are presented below:

MethodTime (min) ↓Peak GPU Mem. (GiB) ↓
LoRA119.1819.25
AdaLoRA140.7820.31
DoRA180.3126.54
LENS89.0921.55

LENS requires 25.2% less offline time than LoRA. These results demonstrate the effectiveness of LENS in balancing post-compression recovery and offline calibration cost.

References:

[1]: LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022.

[2]: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023.

[3]: DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer iXsb,

Thank you for your valuable feedback. We have provided our responses and would be happy to address any remaining questions.

Best regards,

The Authors

I thank the authors for their time and for providing additional experiments comparing their method against PEFT methods. The results seem encouraging. However, I have a few remaining concerns.

  1. Could the authors provide PPL and accuracy numbers prior to applying PEFT/LENS? The performance of LoRA and DoRA seem fairly bad so it would be helpful to understand why that is the case.
  2. Would it be possible to compare with the correction step outlined in ZeroSum-SVD? I feel that this is one of the most relevant baselines for this method as it's a recently published SVD-based compression method which optionally uses task gradients to further improve performance after compression.
Reviewer_4PvPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LENS, a post-compression calibration framework for low-rank LLMs. LENS uses a symmetric activation–gradient response matrix to identify directions that locally favor amplification or suppression, filters them using independent response estimates, and selects calibrated gains through a response-based surrogate and held-out scale validation.

Critical strengths
  1. Identifying loss-aligned eigendirections is of interest.

  2. Reasonable theoretical and algorithmic grounding. The response matrix has a direct first-order interpretation, while Theorems 3.1 and 3.2 justify the use of its eigenbasis for the considered symmetric perturbations and establish local descent under the stated selection conditions.

  3. Broad and mostly consistent empirical support.

Critical weaknesses
  1. The practical cost of post-compression calibration is insufficiently quantified. LENS incurs offline gradient profiling, eigendecompositions, and multiple candidate-scale loss evaluations in each of two stages. Although zero additional inference-time overhead is a clear advantage, the paper does not report sufficiently explicit incremental runtime and peak memory relative to the underlying compressor.

  2. The contribution of the new calibration directions is not isolated. The key distinction from singular-value calibration is the response-eigenbasis search, yet the reported ablations mainly isolate amplification, suppression, ordering, and reprofiling. They do not directly compare the proposed eigendirections with loss-aware rescaling along the original SVD basis while holding the gain estimator, reliability tests, data, and scale search fixed. Thus, the reported improvements may partly arise from additional loss-gradient calibration rather than the proposed choice of directions.

  3. A few generality claims need qualification. The discussion should better qualify the gains (when will it be large, and when will it be small) and clarify evaluation details.

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4PvP (part 1)

We sincerely thank the reviewer for the constructive and insightful comments. We have also updated the manuscript accordingly, with the revisions highlighted in blue. Below, we address the concerns raised in the Weaknesses and Questions.

W1: The practical cost of post-compression calibration is insufficiently quantified. LENS incurs offline gradient profiling, eigendecompositions, and multiple candidate-scale loss evaluations in each of two stages. Although zero additional inference-time overhead is a clear advantage, the paper does not report sufficiently explicit incremental runtime and peak memory relative to the underlying compressor.

We thank the reviewer for highlighting the need to quantify calibration overhead. We conducted additional measurements to report the complete offline calibration cost of LENS as follows.

(1). LENS introduces a relatively small additional calibration cost compared with the underlying Duo-SVD compression. We measure the complete calibration time and peak GPU memory on four backbones at 20% compression using a single RTX 5090. The results are presented below:

BackboneDuo-SVD TimeLENS TimeRelative Overhead (%)Peak GPU Mem. (GiB)
LLaMA-2-7B6h 44m86.82m21.4920.71
LLaMA-3.1-8B14h 08m94.24m11.1125.32
Mistral-7B6h 12m89.09m23.9521.55
Qwen3-8B18h 25m114.74m10.3827.95

LENS requires 86.82–114.74 minutes of additional offline calibration, corresponding to at most 10.38%–23.95% of the underlying Duo-SVD construction time. The peak GPU memory ranges from 20.71 to 27.95 GiB.

(2). Gradient-based discovery and fitting account for the majority of calibration time. We further break down the computational cost. The results are reported in minutes below:

StageLLaMA-2-7BLLaMA-3.1-8BMistral-7BQwen3-8B
Amplify: Discovery19.3020.9019.6726.06
Amplify: Fitting19.4220.8619.6626.25
Amplify: Scale Search & Validation4.865.495.535.01
Suppress: Discovery19.2920.8619.6526.02
Suppress: Fitting19.4720.8819.6226.27
Suppress: Scale Search & Validation2.302.752.652.57

Discovery and fitting account for 90.6%–93.2% of the core calibration time across all four backbones, while scale search and validation contribute substantially less.

(3). Higher compression ratios reduce LENS calibration time and memory consumption. We further measure the full calibration cost on LLaMA-2-7B across compression ratios from 10% to 40%. The results are presented below:

Compression RatioTime (min) ↓Peak GPU Mem. (GiB) ↓
10%87.8821.67
20%86.8220.71
30%80.1919.53
40%71.0718.36

As the compression ratio increases from 10% to 40%, calibration time decreases by 19.1%, while peak GPU memory decreases by 15.3%. These results indicate that more aggressive compression does not incur additional calibration overhead.

W2: The contribution of the new calibration directions is not isolated. The key distinction from singular-value calibration is the response-eigenbasis search, yet the reported ablations mainly isolate amplification, suppression, ordering, and reprofiling. They do not directly compare the proposed eigendirections with loss-aware rescaling along the original SVD basis while holding the gain estimator, reliability tests, data, and scale search fixed. Thus, the reported improvements may partly arise from additional loss-gradient calibration rather than the proposed choice of directions.

We thank the reviewer for this valuable suggestion. We conducted a direction-basis ablation to isolate the contribution of the proposed response eigenvectors as follows.

(1). Response eigenvectors achieve stronger overall performance than fixed SVD directions. We compare the proposed response-eigenbasis calibration with a fixed SVD-basis variant using the same calibration strategy. The results at 20% compression are presented below:

BackboneSVD Basis PPL ↓Response Eigenbasis PPL ↓SVD Basis Avg. ↑Response Eigenbasis Avg. ↑
LLaMA-2-7B5.755.6157.5658.07
LLaMA-3.1-8B8.217.9561.5861.86
Mistral-7B5.595.5160.4961.59
Qwen3-8B7.327.3466.5766.63

Response eigenvectors improve average downstream accuracy by 0.06–1.10 percentage points across all four backbones and achieve lower perplexity on three of them. These results demonstrate that the proposed direction selection contributes to the performance gains beyond loss-aware rescaling alone.

AuthorsOpenReview ↗

Response to Reviewer 4PvP (part 2)

W3: A few generality claims need qualification. The discussion should better qualify the gains (when will it be large, and when will it be small) and clarify evaluation details.

We thank the reviewer for this helpful suggestion. We clarify the variation in performance gains and the evaluation protocol as follows.

(1). LENS consistently improves both perplexity and downstream accuracy across four backbones. At 20% compression, LENS reduces perplexity by 0.20–0.73 and improves average downstream accuracy by 0.11–1.98 percentage points across all four backbones (Table 1 and Appendix Table 7). These results demonstrate consistent benefits across model families.

(2). LENS maintains a clear perplexity advantage across compression ratios, particularly under aggressive compression. Across 10%–40% compression on LLaMA-2-7B and Mistral-7B, LENS consistently achieves the lowest PPL among all compared methods. Its PPL advantage over the strongest competing baseline is 0.18–0.23 at 10% and 0.49–1.42 at 40%, while maintaining competitive downstream accuracy (Figure 6(a) and Appendix Table 8).

(3). LENS consistently improves performance across different low-rank compressors. When applied to BoLaCo, SoLA, and SVD-LLM across four backbones, LENS reduces perplexity in all configurations and improves average downstream accuracy, demonstrating its broad compatibility with different compression methods (Figure 3 and Appendix Table 10).

(4). Evaluation protocol. We use a default 20% compression ratio, evaluate WikiText-2 PPL with 4096-token blocks, and report average accuracy across seven downstream tasks (MMLU 5-shot; others zero-shot). Further details are provided in Section 4.1 and Appendix B.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer 4PvP,

Thank you for your valuable feedback. We have provided our responses and would be happy to address any remaining questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.