acceptodds
Under review as a conference paper at ICLR 2027

Loss Aware Truncation with Gradients for Low-Rank Decomposition of Large Language Models

Abstract

The continued scaling of large language models (LLMs) has resulted in rapidly growing memory requirements. Singular Value Decomposition (SVD) provides a principled approach to reducing this burden through low-rank compression. However, existing state-of-the-art SVD-based methods have not fully characterized the relationship between singular components and the performance degradation induced by compression. In this work, we show through the first-order Taylor expansion that the local gradient of each layer provides a direct connection between singular value importance and the resulting loss degradation of the compressed model. Building on this insight, we propose Loss-aware Truncation with Gradients (LTG), which transforms the weight space according to local loss sensitivity. This transformation redistributes the singular spectrum such that components with greater influence on the model objective are preferentially retained during truncation. We demonstrate the effectiveness of LTG over related state-of-the-art approaches across three LLMs families and standard benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, compression, low rank

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gKwoOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Loss-aware Truncation with Gradients (LTG), a low-rank compression method for large language models. The main idea is that traditional SVD methods focus on minimizing weight reconstruction error, which doesn't necessarily preserve model performance. To address this, the authors use gradient information to identify more important weight directions and guide the SVD truncation process. They also introduce a self-calibration strategy that generates calibration data directly from the original model, reducing the need for external datasets. Experiments across several LLM families show that LTG generally outperforms existing SVD-based compression methods, especially at higher compression ratios.

Critical strengths
  1. Well-motivated approach with solid mathematical derivation. Traditional SVD methods focus on minimizing the reconstruction error caused by low-rank truncation, but components with small singular values can still be important for the model's performance. This paper addresses this issue by incorporating gradient information to better capture how different components affect the model's loss. I particularly appreciate the mathematical derivation, which uses first-order approximation and upper-bound optimization to gradually introduce gradient information into the transformation matrix. The overall approach is well-motivated and mathematically grounded.

  2. Comprehensive experiments with strong results. The paper provides extensive experiments across multiple LLM families, compression ratios, and downstream tasks.

  3. Another strength is the self-calibration approach, which allows the model to generate its own calibration data instead of relying on an external calibration dataset. This makes the calibration process more flexible and practical.

Critical weaknesses
  1. Self-calibration and robustness: While self-calibration is a nice idea, I'm wondering whether model-generated data can really cover diverse real-world distributions, especially rare or long-tail cases. Could this affect the robustness of the compressed model? It would be helpful to see some experiments on out-of-distribution tasks.

  2. Approximation errors: The mathematical derivation is solid, but it relies on a first-order Taylor approximation and then optimizes an upper bound rather than the actual loss change. I'm curious how well this surrogate objective reflects the true loss degradation, especially at higher compression ratios. Could the authors provide more analysis of this gap (Theo & Experiment)?

  3. Computational and memory overhead: Since LTG requires collecting per-sample gradients and computing gradient covariance matrices, I'm concerned about the additional computational and memory costs. How does the method scale with larger batch sizes and model sizes? It would be useful to report compression time and peak GPU memory usage compared to existing methods.

  4. For expression (8) in paper, I think the authors should provide more theoretical analysis to show whether the minimum of the proposed upper bound can be sufficiently small.

Questions for authors

See in Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer gKwo (Part 1/2)

Dear Reviewer gKwo,

Thank you for your insightful and constructive feedback, which will help us further strengthen our work. Below, we provide responses to each of your concerns in 02 parts.

We will revise the manuscript accordingly to incorporate your suggestions.


[W1. Self-calibration and robustness: While self-calibration is a nice idea, I'm wondering whether model-generated data can really cover diverse real-world distributions, especially rare or long-tail cases. Could this affect the robustness of the compressed model? It would be helpful to see some experiments on out-of-distribution tasks.]

We thank the reviewer for this insightful question.

We consider "out-of-distribution" (OOD) in two way as below:

Case 1: OOD relative to the calibration distribution. Our experiments use only 256 calibration samples (consistent with SVD-LLM, ASVD, FWSVD), while evaluating the compressed models across diverse distributions and tasks, including language modeling (C4 perplexity), commonsense reasoning (WinoGrande, HellaSwag, and PIQA), question answering (OpenBookQA and ARC-Easy), and mathematical reasoning (MathQA and GSM8K). These evaluations differ substantially from the self-generated calibration data, providing empirical evidence that LTG preserves model capabilities beyond the calibration distribution.

Case 2: out-of-distribution of pretraining data Self-calibration aims to approximate the model's learned pretraining distribution rather than explicitly capture unseen distributions. Therefore, we acknowledge that its effectiveness on distributions substantially different from the pretraining data is not guaranteed and may be limited. However, identifying genuinely OOD benchmarks is challenging, particularly since the complete pretraining corpora of the evaluated models are not publicly available. We therefore consider rigorous evaluation under such distribution shifts an important direction for future work and would welcome the reviewer's suggestions regarding suitable OOD benchmarks.


[W3. Computational and memory overhead: Since LTG requires collecting per-sample gradients and computing gradient covariance matrices, I'm concerned about the additional computational and memory costs. How does the method scale with larger batch sizes and model sizes? It would be useful to report compression time and peak GPU memory usage compared to existing methods.]

Thank you for raising this concern. Please refer to our response to Reviewer PEgS, [W2.2, W3, Q1.2, and Q2: Systematic evaluations on more recent and larger-scale LLMs], where we provide detailed comparisons of compression time, peak GPU VRAM, and CPU RAM usage against SVD-LLM and other baselines.

Importantly, compression incurs only a one-time offline cost and can be performed on a high-performance computing platform before deploying the compressed model to resource-constrained devices. Therefore, this overhead does not affect inference-time computational or memory requirements.

AuthorsOpenReview ↗

Response to Reviewer gKwo (Part 2/2)

[W2. Approximation errors: The mathematical derivation is solid, but it relies on a first-order Taylor approximation and then optimizes an upper bound rather than the actual loss change. I'm curious how well this surrogate objective reflects the true loss degradation, especially at higher compression ratios. Could the authors provide more analysis of this gap (Theo & Experiment)?]

and

[W4. For expression (8) in paper, I think the authors should provide more theoretical analysis to show whether the minimum of the proposed upper bound can be sufficiently small.]

Theory. We provide further theoretical analysis to characterize the minimum achievable value of the proposed upper bound in Eq. (8).

Specifically, our optimization objective in Eq. (9) is the aforementioned upper bound. As established in Eq. (13), the minimum of the gradient-weighted reconstruction objective under a rank-rr constraint is given by

\min_{\operatorname{rank}(\widetilde{\mathbf W})\le r} \Vert(\mathbf W-\widetilde{\mathbf W})\mathbf G_{\mathbf W}^{\top}\Vert_F^2 = \sum_{j>r}\sigma_j^2,$$ where $\sigma_j$ denotes the $j$-th singular value of $\mathbf W\mathbf G$, with $\mathbf G\mathbf G^\top=\mathbf G_{\mathbf W}^\top\mathbf G_{\mathbf W}$ obtained through Cholesky decomposition. Since the $\sigma_j$ are nonnegative and ordered in descending magnitude, $\mathcal{G}^*$ decreases monotonically as the retained rank $r$ increases, reaching zero when all $\sigma_j$ are retained. Therefore, lower compression ratios yield smaller upper bounds on the first-order loss approximation, providing theoretical justification for better model performance preservation at less aggressive compression levels. **Experiments.** Moreover, as shown in Fig. 4 and Tables 1, 3, and 5, LTG remains effective even at aggressive compression ratios of up to 80%. Furthermore, Fig. 3(a) directly evaluates the true loss degradation induced by removing singular components. Specifically, we divide the singular components into 10 equally sized chunks, ordered by decreasing singular value magnitude, and measure the actual loss change relative to the original model after removing each chunk. The results show that LTG's gradient-aware transformation makes singular value magnitudes better reflect their importance to the model loss. In particular, smaller singular values consistently locally correspond to lower loss degradation.
Reviewer_dmwnOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper presents LTG, a post-training SVD compression method for LLMs that uses gradient-based whitening instead of the activation-based transformation employed by SVD-LLM. By deriving a loss-aware approximation objective from a first-order Taylor expansion, LTG aims to preserve singular components that have a greater impact on the model's loss. It estimates the required gradients using sequences generated by the original model, avoiding dependence on external calibration data, and applies LoRA fine-tuning to recover performance after compression. Experiments on five LLMs across three model families show that LTG consistently achieves lower C4 perplexity than the evaluated SVD baselines, although its improvements in downstream task accuracy over SVD-LLM are generally modest.

Critical strengths

1. Clear presentation: The paper is well organized and easy to follow. The derivation connecting the loss-aware objective to an SVD-based formulation in the transformed space is presented step by step.

2. Simple and practical design: LTG makes a relatively small modification to the SVD-LLM pipeline by replacing activation-based whitening with gradient-based whitening.

3. Effective use of self-calibration: Adapting the self-generated calibration strategy of Williams et al. (2025) to SVD-based compression is a reasonable extension of an existing idea. It makes the method more convenient to apply when calibration data are unavailable.

Critical weaknesses

1. Confounded main comparison: LTG uses self-generated calibration data, while SVD-LLM uses WikiText-2. Table 5 (before reconstruction) shows that on LLaMA-7B at 40%, SVD-LLM reaches 50.5 perplexity, LTG† 49.4, and LTG 26.9. Most of the gain therefore comes from the calibration data rather than the gradient criterion. An SVD-LLM baseline that uses the same self-generated data is needed.

2. No cost analysis: LTG generates 256 × 2048 tokens and computes per-sample gradients, but the paper reports no compression time, memory, or inference latency compared with SVD-LLM. The limitations section acknowledges the higher memory use but never quantifies it.

3. Limited evaluation: The main text has only one results table. Only 7B models from older families are evaluated, with no Llama 3 or Qwen, and all ablations use OPT-350M. Recent SVD baselines (SVD-LLM V2, Dobi-SVD, AdaSVD, Zero-Sum SVD) are missing, and LTG uses only uniform rank allocation.

4. Incremental novelty: The method replaces SVD-LLM's activation covariance with a gradient covariance. Gradient-weighted SVD already exists (FWSVD), and the self-calibration step is taken directly from Williams et al. (2025).

5. Marginal downstream gains: In Table 1, accuracy and GSM8K differences against SVD-LLM, an older baseline, are within 0.01 in most settings (12 of 20 accuracy and 18 of 20 GSM8K entries), and SVD-LLM still wins in some. Main results use a single fixed seed. Self-calibration also underperforms external data on Vicuna, the only instruction-tuned model.

6. Instability at high compression: At 80% compression without reconstruction, LTG reaches 85,156 perplexity on Vicuna versus 2,649 for SVD-LLM, as the paper itself acknowledges (Appendix C). Its reported robustness depends on LoRA reconstruction.

Questions for authors

1. Calibration data vs. gradient criterion: How does SVD-LLM perform with the same self-generated calibration data as LTG? Without this comparison, the contribution of the gradient criterion cannot be separated from that of the calibration data.

2. Computational cost: What are the compression time and peak memory of LTG compared with SVD-LLM, including the cost of generating calibration sequences and computing per-sample gradients?

3. Reporting details: Why is the original LLaMA-7B C4 perplexity 7.12 in Table 1 but 7.34 in Table 8? Is Table 1 reported after LoRA reconstruction, and if so, were SVD, FWSVD, and ASVD reconstructed with the same procedure?

5 comments
AuthorsOpenReview ↗

Response to Reviewer dmwn (Part 1/3)

Dear Reviewer dmwn,

Below, we address the weaknesses and questions you raised in 3 parts.


[W1. Confounded main comparison: LTG uses self-generated calibration data, while SVD-LLM uses WikiText-2. Table 5 (before reconstruction) shows that on LLaMA-7B at 40%, SVD-LLM reaches 50.5 perplexity, LTG† 49.4, and LTG 26.9. Most of the gain therefore comes from the calibration data rather than the gradient criterion. An SVD-LLM baseline that uses the same self-generated data is needed.]

and

[Q1. Calibration data vs. gradient criterion: How does SVD-LLM perform with the same self-generated calibration data as LTG? Without this comparison, the contribution of the gradient criterion cannot be separated from that of the calibration data.]

We respectfully disagree with the reviewer's assessment that most of the improvement comes from self-generated calibration data rather than our gradient-aware criterion.

Evidence from the original submission. We believe that drawing a conclusion from a single case (LLaMA-7B at 40% compression) does not fairly reflect the overall trend. For instance, in the same Table 5, LTG† reduces PPL from 673.7 to 379.1 (294.6 lower) on Vicuna-7B at 60% compression, and from 12,696 to 7,716 (4,980 lower) on OPT-6.7B at 80% compression, despite using the same external calibration data as SVD-LLM.

Additional experiments with matched calibration data. To further address this concern, we conduct additional experiments on two recent LLMs, Qwen3-1.7B and Llama-3.2-1B, comparing LTG against SVD-LLM using self-generated calibration data (SVD-LLM_self) under the same experimental settings. All results are reported before reconstruction.

RatioMethodPPL.Qwen3PPL.Llama3.2
0%Original14.4313.08
********
20%SVD-LLM_self53.65120.92
LTG41.54104.78
********
40%SVD-LLM_self168.88659.04
LTG153.10401.52
********
60%SVD-LLM_self1310.222545.12
LTG1072.321360.96
********
80%SVD-LLM_self9741.957295.46
LTG5559.624636.70

LTG achieves lower PPL in all 8/8 settings.


[W2 and Q2. No cost analysis.]

Thank you for raising this concern. Please refer to our response to Reviewer PEgS, [W2.2, W3, Q1.2, and Q2: Systematic evaluations on more recent and larger-scale LLMs], where we provide detailed comparisons of compression time, peak GPU VRAM, and CPU RAM usage against SVD-LLM and other baselines, along with a discussion of inference efficiency.


[W3. Limited evaluation: The main text has only one results table. Only 7B models from older families are evaluated, with no Llama 3 or Qwen, and all ablations use OPT-350M. Recent SVD baselines (SVD-LLM V2, Dobi-SVD, AdaSVD, Zero-Sum SVD) are missing, and LTG uses only uniform rank allocation.]

Only one results table in the main text. The main text includes 02 results figures and 01 big results table, covering 06 models from 03 model families across 24 experimental settings and 08 widely used LLM benchmarks. In addition, we provide 07 supplementary tables in the appendix to further support our findings.

Older models. We follow the publicly available SVD-LLM benchmark and implementation to ensure fair and reproducible comparisons with established baselines under consistent evaluation settings.

Only on 7B models. As explicitly stated in Line 355, our experiments are conducted on a single NVIDIA L40S GPU with 48 GB of VRAM. Given these computational constraints, we prioritize evaluating LTG across five models spanning three LLM families to examine its generalizability across different architectures.

Importantly, smaller models tend to be harder to compress. Prior studies have demonstrated that larger Transformer models are generally more robust to compression, potentially due to their greater parameter redundancy [1, 2]. Therefore, evaluating across diverse 7B models provides a meaningful and challenging benchmark.

Comparison on newer and larger models. Please refer to our response to Reviewer PEgS, [W2.2, W3, Q1.2, and Q2: Systematic evaluations on more recent and larger-scale LLMs], where we provide additional experiments on Qwen3-1.7B, Llama-3.2-1B, and Llama-2-13B, including comparisons with recent baselines KFAC-SVD and GFWSVD.

Why not SVD-LLM V2, Dobi-SVD, AdaSVD, Zero-Sum SVD and why uniform rank only. Because these methods focus on different goal rather than ours. While we aim to find a better way to transform the weights before low-rank truncation, these methods aim to find a better rank allocation strategy. Comparisons with them would therefore not isolate the core contribution of our work.

We will acknowledge and discuss this line of research in the revised Related Work section.

AuthorsOpenReview ↗

Response to Reviewer dmwn (Part 2/3)

[W4. Incremental novelty: The method replaces SVD-LLM's activation covariance with a gradient covariance. Gradient-weighted SVD already exists (FWSVD), and the self-calibration step is taken directly from Williams et al. (2025).]

We respectfully disagree with this assessment.

While LTG may appear to simply replace activation covariance with gradient information from an engineering perspective, the key novelty lies in the theoretical derivation that justifies why and how gradients should guide weight transformation before low-rank truncation. To our knowledge, LTG is the first to explicitly derive such a transformation directly from local loss gradients, with theoretical justification rather than relying on heuristic importance metrics or restrictive assumptions (please see our response to Reviewer PEgS, [W1 and Q1.1: Fundamental differences between LTG and KFAC-SVD and GFWSVD.] for further details).

We acknowledge that self-calibration builds upon Williams et al. (2025), but this is not our core contribution.


[W5. Marginal downstream gains: In Table 1, accuracy and GSM8K differences against SVD-LLM, an older baseline, are within 0.01 in most settings (12 of 20 accuracy and 18 of 20 GSM8K entries), and SVD-LLM still wins in some. Main results use a single fixed seed. Self-calibration also underperforms external data on Vicuna, the only instruction-tuned model.]

Marginal downstream gains. We respectfully disagree with this assessment. Table 1 shows that LTG improves downstream accuracy in 17/20 settings, with several substantial gains of 3 to 5%. Moreover, LTG consistently achieves lower perplexity in 20/20 settings and higher GSM8K accuracy in 19/20 settings compared with SVD-LLM.

We also wish to clarify that the reviewer's threshold of 0.01 represents an absolute difference of 1%. Such differences should not automatically be considered negligible, particularly on multi-step mathematical reasoning benchmarks such as GSM8K, where one percentage point corresponds to approximately 13 additional correctly solved problems on the 1319 examples in the test set.

Self-calibration on Vicuna-7B. Self-calibration underperforms external calibration in perplexity only at aggressive compression ratios of 60% and 80%, as already acknowledged and explained in Appendix C (lines 952 to 957). This may be attributed to the instruction-tuned nature of Vicuna-7B. Importantly, the performance degradation is substantially mitigated after reconstruction. At 80% compression, LTG initially underperforms SVD-LLM (Table 5) but achieves lower perplexity after reconstruction (53.62 vs. 60.00 - Table 1). Therefore, this observation does not undermine our core contribution regarding gradient-aware weight transformation.

Random seeds. Similar to prior SVD-based compression studies (SVD-LLM, ASVD, FWSVD), our main results are reported without multi-seed averaging. Results across multiple random seeds are provided in Table 6, showing negligible variance. This is expected since the random seed only affects calibration data sampling, while all baselines require neither model initialization nor retraining.


[W6. Instability at high compression: At 80% compression without reconstruction, LTG reaches 85,156 perplexity on Vicuna versus 2,649 for SVD-LLM, as the paper itself acknowledges (Appendix C). Its reported robustness depends on LoRA reconstruction.]

We have explicitly acknowledged and discussed this behavior in lines 952 to 957. Importantly, this degradation disappears after reconstruction, where LTG achieves lower PPL than SVD-LLM (53.62 vs. 60.00), suggesting that the subspace retained by LTG remains effective for preserving model performance after adaptation.

We also disagree that LTG's robustness solely depends on LoRA reconstruction. As shown in Table 5, LTG outperforms SVD-LLM in 19/20 settings even without reconstruction, while LTG† achieves lower PPL in all 20/20 settings, including Vicuna-7B at 80% compression (2,152 vs. 2,649). Moreover, both LTG and SVD-LLM use the same reconstruction protocol. Therefore, the improvement cannot be attributed solely to LoRA reconstruction.


[Q3. Reporting details: Why is the original LLaMA-7B C4 perplexity 7.12 in Table 1 but 7.34 in Table 8? Is Table 1 reported after LoRA reconstruction, and if so, were SVD, FWSVD, and ASVD reconstructed with the same procedure?]

All experiments were conducted on an HPC cluster managed by Slurm with an NVIDIA L40S GPU, except for Table 8, which was evaluated on a personal laptop with an RTX 5090 GPU. We apologize for this inconsistency and thank the reviewer for pointing it out. We will clarify the experimental settings in the revised version.

There are no reconstrution phase in SVD, FWSVD, and ASVD. We follow the same benchmark of SVD-LLM to ensure this fair comparision.

AuthorsOpenReview ↗

Response to Reviewer dmwn (Part 3/3)

References

[1] Li, Zhuohan, et al. "Train big, then compress: Rethinking model size for efficient training and inference of transformers." International Conference on machine learning. PMLR, 2020.

[2] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International conference on machine learning. PMLR, 2023.

Reviewer_PEgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

his paper proposes LTG, a gradient-aware low-rank compression method for Large Language Models (LLMs). By incorporating gradient information into the weight transformation process, LTG ensures that singular components sensitive to the loss function are preserved during the Singular Value Decomposition (SVD) truncation phase. Furthermore, the method introduces self-generated calibration data and employs a reconstruction technique based on LoRA. Experimental results demonstrate that this approach achieves performance improvements over various SVD-based compression baselines.

Critical strengths

(1) The paper addresses the limitation that "minimizing weight reconstruction error does not necessarily preserve model performance." The theoretical derivation is relatively sound, and the logic is coherent.

(2) The method achieves competitive results across various LLM families and different compression ratios; notably, it outperforms SVD-LLM in terms of perplexity.

Critical weaknesses

(1) The paper overlooks several recent gradient-based and Fisher-information-based low-rank compression methods (such as GFWSVD and KFAC-SVD) and fails to provide a direct comparison with them. Since the use of gradient-based sensitivity to guide low-rank decomposition has already been explored in existing literature, what are the specific advantages of LTG over methods like KFAC-SVD or GFWSVD? (2) The experiments focus primarily on older, relatively smaller-scale models (e.g., OPT, LLaMA-1/2, and Vicuna), all with approximately 7 billion parameters. The paper lacks evaluations on more recent, larger-scale LLMs (such as Llama-3 or Qwen3). (3) The execution pipeline of LTG involves self-generated calibration data, backpropagation, and the calculation of gradient covariance; however, the paper lacks a systematic comparison regarding compression time, peak memory consumption, and actual inference efficiency.

Questions for authors

Q1. What are the fundamental differences between LTG and recent methods based on Fisher information (such as GFWSVD and KFAC-SVD)? Could the authors provide a direct comparison under the same compression budget and calibration settings? Q2. Could the authors evaluate LTG on newer, larger-scale large language models (LLMs) and provide detailed measurements regarding compression time, peak memory usage, and inference efficiency?

[1] Chekalina V, Moskovskiy D, Cherniuk D, et al. Generalized fisher-weighted svd: Scalable kronecker-factored fisher approximation for compressing large language models[J]. arXiv preprint arXiv:2505.17974, 2025. [2] Thoma M, Groezinger M, Forstenhäusler M, et al. Advancing SVD-based LLM Compression via Layer-Wise Error Model Search[C]//Forty-third International Conference on Machine Learning. 2026.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PEgS (Part 1/4)

Dear Reviewer PEgS,

Thank you for your time and effort in reviewing our paper. We sincerely appreciate your valuable feedback. Below, we address the weaknesses and questions you raised in 4 parts.


[W1 and Q1.1: Fundamental differences between LTG and KFAC-SVD and GFWSVD.]

KFAC-SVD and GFWSVD rely on additional approximations and assumptions. KFAC-SVD and GFWSVD derive their optimization objectives from a second-order approximation of the model loss, replacing the loss Hessian with the Fisher Information Matrix and approximating the latter using a Kronecker-factored structure. These approximations are not guaranteed for neural networks [1]. The authors of GFWSVD also acknowledge this limitation in Sec. 5 and Appendix A of their paper [2].

In contrast, LTG derives its optimization objective directly from the first-order perturbation of the model loss, without relying on either assumption. More detailed comparison is as follows:

Comparison with KFAC-SVD. Let E=W−W~\mathbf E=\mathbf W-\widetilde{\mathbf W} denote the compression error. KFAC-SVD minimizes the following Fisher-weighted reconstruction objective:

\left\| \mathbf B^{1/2}\mathbf E\mathbf A^{1/2} \right\|_F^2, \tag{A}$$ where $$\mathbf A=\mathbb E[\mathbf x\mathbf x^\top], \qquad \mathbf B=\mathbb E[\boldsymbol\delta\boldsymbol\delta^\top],$$ and $\mathbf x$ and $\boldsymbol\delta$ denote the input activation and output gradient, respectively. To examine the relationship between this objective and the first-order loss perturbation, consider Eq. (7) of our submission: $$\Delta\mathcal L^2= \left\langle\mathbf E,\mathbf G_W\right\rangle_F^2, \tag{7}$$ where $\mathbf G_W=\partial\mathcal L/\partial\mathbf W$. Without loss of generality, consider a calibration objective averaged over $N$ token-wise contributions, we have $$\mathbf G_W= \frac1N\sum_{i=1}^N \boldsymbol\delta_i\mathbf x_i^\top.$$ Substituting this into Eq. (7) gives $$\begin{aligned} \Delta\mathcal L^2 &= \frac1{N^2} \left( \sum_{i=1}^N \boldsymbol\delta_i^\top\mathbf E\mathbf x_i \right)^2\\ &= \frac1{N^2} \sum_{i=1}^N (\boldsymbol\delta_i^\top\mathbf E\mathbf x_i)^2+ \frac1{N^2} \underbrace{ \sum_{i\neq j} (\boldsymbol\delta_i^\top\mathbf E\mathbf x_i) (\boldsymbol\delta_j^\top\mathbf E\mathbf x_j) }_{\text{Cross terms}}. \end{aligned} \tag{B}$$ However, expanding the KFAC-SVD objective (Eq. (A)) yields $$\begin{aligned} \mathcal J_{\mathrm{KFAC}} &= \operatorname{Tr} \left( \mathbf E^\top\mathbf B\mathbf E\mathbf A \right)\\ &= \frac1{N^2} \sum_{i,j=1}^N (\boldsymbol\delta_i^\top\mathbf E\mathbf x_j)^2. \end{aligned} \tag{C}$$ Eqs. (B) and (C) are not mathematically equivalent in general. The first-order loss perturbation explicitly accounts for cross terms between sample-wise loss perturbations, whereas KFAC-SVD does not preserve these terms. **Comparison with GFWSVD.** GFWSVD minimizes a Fisher-weighted quadratic objective, but estimates its Kronecker factors from per-sample weight gradients. Let $\mathbf g_i=\operatorname{vec}(\mathbf G_{W,i})$ and $\mathbf e=\operatorname{vec}(\mathbf E)$. The underlying empirical Fisher is defined as $$\mathbf F=\frac1N\sum_{i=1}^N\mathbf g_i\mathbf g_i^\top,$$ which GFWSVD subsequently approximates using a Kronecker-factored representation. Even before this approximation, the Fisher-weighted objective expands to $$\begin{aligned} \mathcal J_{\mathrm{Fisher}} &=\mathbf e^\top\mathbf F\mathbf e\ &=\frac1N\sum_{i=1}^N(\mathbf e^\top\mathbf g_i)^2\ &=\left(\mathbf e^\top\frac1N\sum_{i=1}^N\mathbf g_i\right)^2 +\operatorname{Var}_i(\mathbf e^\top\mathbf g_i). \end{aligned}$$ Clearly, this Fisher objective differs from the first-order loss perturbation derived in Eq. (7) of our submission. Furthermore, GFWSVD introduces an additional Kronecker approximation that does not guarantee preservation of the Fisher quadratic form. Consequently, its resulting optimization objective is even further removed from the first-order loss perturbation induced by compression. Numerical comparisons with KFAC-SVD and GFWSVD can be found in part 3/4 of our response.
AuthorsOpenReview ↗

Response to Reviewer PEgS (Part 2/4)

[W2.1: Why are comparisons conducted only on older 7B models?]

Older models. We follow the publicly available SVD-LLM benchmark and implementation to ensure fair and reproducible comparisons with established baselines under consistent evaluation settings.

Only on 7B models. Our experiments are conducted on a single NVIDIA L40S GPU with 48 GB of VRAM, as explicitly stated in Line 355 of our submission. Given these computational constraints, we prioritize evaluating LTG across five models spanning three different LLM families, rather than focusing on a smaller number of larger models. This allows us to examine the generalizability of LTG across different model families within our available computational budget. We acknowledge that extending the evaluation to larger and more recent models would further strengthen our findings. Comparison with larger model can be found in our next response.

Importantly, smaller models tend to be harder to compress. Prior studies have demonstrated that larger Transformer models are generally more robust to compression, potentially due to their greater parameter redundancy [3, 4]. Therefore, evaluating across diverse 7B models provides a meaningful and challenging benchmark.

AuthorsOpenReview ↗

Response to Reviewer PEgS (Part 3/4)

[W2.2, W3, Q1.2 and Q2: Systematic evaluations on more recent and larger scale LLMs (such as Llama-3 or Qwen3).]

Comparisons on LLama-3 and Qwen3. We conduct additional experiments on Qwen3-1.7B and Llama-3.2-1B, comparing LTG against several baselines, including KFAC-SVD and GFWSVD. All experiments follow the same configuration as our main experiments. For methods involving reconstruction, we perform a single pass over the Alpaca-50K dataset.

The following table reports perplexity (PPL), compression time (seconds, including reconstruction when applicable), and peak GPU and CPU memory usage (GB) during compression. SVD-LLM* and LTG* denote results before reconstruction, while their counterparts without * include reconstruction.

RatioMethodPPL.Qwen3Times.Qwen3Peak VRAM.Qwen3Peak RAM.Qwen3PPL.Llama3.2Times.Llama3.2Peak VRAM.Llama3.2Peak RAM.Llama3.2
0%Original14.43---13.08---
********************
20%SVD697854745.230.286.0317996323.150.364.87
FWSVD11260447111.1912.985.997979962.049.134.88
ASVD10355865.124.026.17830435.882.874.91
GFWSVD198991500.8613.20103.7960931041.228.9277.25
KFAC-SVD47.783280.1113.7735.1681.452558.779.4032.46
SVD-LLM*85.61265.1710.6414.16241.0134.158.5212.24
LTG*41.541313.6824.2814.14104.8719.3417.4112.26
SVD-LLM26.363077.53--35.761650.26--
LTG23.794070.07--30.132265.52--
********************
60%SVD7348674349.730.284.9247851419.530.363.72
FWSVD118042298105.3512.984.9336398258.809.133.78
ASVD958021564.034.024.866638033.382.873.74
GFWSVD174731498.0313.20103.79132621040.838.9277.25
KFAC-SVD15013277.5313.7735.1618772555.479.4032.46
SVD-LLM*3806263.1510.6414.163602131.418.5212.24
LTG*10721310.7924.2814.141361715.6517.4112.26
SVD-LLM192.43016.24--163.31631.49--
LTG135.83911.75--101.32225.87--

LTG consistently achieves the lowest PPL among the evaluated compression methods. Peak memory consumption is largely independent of the compression ratio, as it is primarily determined by calibration and decomposition rather than the retained rank.


Larger model. We extend our evaluation to Llama-2-13B. Since our experiments are conducted under limited computational resources, we compress each layer sequentially to reduce peak memory consumption, at the cost of additional compression time.

We also attempted to include KFAC-SVD and GFWSVD. However, their Fisher-based computations require collecting and processing high-dimensional gradient statistics and Kronecker-factored matrices, which exceeded our available memory budget for this model (as proven in the previous table). Therefore, we report results for the remaining baselines under the same evaluation settings.

RatioMethodPPL.Llama2-13BTimes.Llama2-13BPeak VRAM.Llama2-13BPeak RAM.Llama2-13B
0%Original6.55-- 
************
20%SVD6112.03730.221.6026.23
FWSVD36915.234765.7525.6728.01
ASVD16.97827.8226.4626.31
SVD-LLM*14.031422.2213.5778.38
LTG*10.676096.0039.6278.33
************
60%SVD71559.44688.291.5825.92
FWSVD18870.754719.8525.6728.01
ASVD4395.99779.6426.4626.08
SVD-LLM*265.431371.3613.5778.00
LTG*119.936022.7339.6277.95

Inference efficiency. As clarified in Lines 1015 to 1019 of our submission, compression is a one-time offline process performed before deployment. Therefore, the additional compression cost does not translate into inference overhead on the target device. LTG retains the same low-rank factorized architecture as other SVD-based approaches. We refer to Figure 3 of SVD-LLM [5] for empirical inference throughput measurements of low-rank compressed models.


We will include these comparisons to the revised version of our submission.

AuthorsOpenReview ↗

Response to Reviewer PEgS (Part 4/4)

References

[1] Kunstner, Frederik, Philipp Hennig, and Lukas Balles. "Limitations of the empirical fisher approximation for natural gradient descent." Advances in neural information processing systems 32 (2019).

[2] Chekalina, Viktoriia A., et al. "Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity." Forty-third International Conference on Machine Learning.

[3] Li, Zhuohan, et al. "Train big, then compress: Rethinking model size for efficient training and inference of transformers." International Conference on machine learning. PMLR, 2020.

[4] Frantar, Elias, and Dan Alistarh. "Sparsegpt: Massive language models can be accurately pruned in one-shot." International conference on machine learning. PMLR, 2023.

[5] Wang, Xin, et al. "Svd-llm: Truncation-aware singular value decomposition for large language model compression." International Conference on Learning Representations. Vol. 2025. 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reminder to engage in discussion

Dear Reviewers,

We have provided detailed clarifications and additional experimental results addressing all of your concerns. Please let us know if you have any further questions or concerns. We would be happy to provide additional explanations or evidence as needed.

As the rebuttal deadline is approaching, we kindly invite you to review our responses and engage in further discussion.

Thank you for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of the discussion phase

Dear Public Readers, Program Chair, Area Chair, and Reviewers,

Since the discussion phase will end in 3 hours, we would like to summarize it as follow.


Discussion Status. 2/3 reviewers have not engaged, while the remaining reviewer responded only 17 hours before the discussion deadline. Our rebuttal has been available for over a week, and we sent a reminder two days ago.


Reviewer gKwo (4) requested compression-time measurements and further theoretical analysis of our proposed method. We have addressed these requests with additional experimental results and theoretical clarification.


Reviewer PEgS (2) requested evaluation on two recent models, comparisons with two recent baselines, and compression-time measurements. We have fully addressed these requests with additional experiments and theoretical analysis. The resulting evidence even further strengthens our method.


Reviewer dmwn (2) provided inaccurate assessments:

  • Limited and confounded comparisons (W1, W3): Our main results already cover 24 experimental settings across 06 models and 08 widely used LLM benchmarks, with 07 additional tables provided in the appendix.
  • Incremental novelty (W4): To our knowledge, we are the first to theoretically establish a gradient-aware weight transformation based on the Cholesky factor of the local weight-gradient Gram matrix before low-rank truncation. Both Reviewers PEgS and gKwo acknowledged the novelty of our approach in their Critical Strengths.
  • Marginal improvements and instability at high compression ratios (W5, W6): Our results demonstrate substantial and consistent improvements over existing methods, including at high compression ratios, contrary to the reviewer's assessment. These improvements were also acknowledged by Reviewers PEgS and gKwo in their Critical Strengths.

In their follow-up, Reviewer dmwn acknowledged our rebuttal and increased their score by one point, but continued to express baseless negative assessments without providing specific supporting evidence as we pointed out in our immediate response. We have not received any further response from the reviewer since then.


We have provided detailed rebuttals direcly address all concerns. We will incorporate the relevant clarifications and experimental results into the revised manuscript if the paper is accepted.

Moreover, we would like to raise concern regarding the reviews from PEgS and dmwn, both of which were flagged as 100% AI-generated by Pangram. We respectfully ask the Area Chairs and Program Chairs to take this concern into consideration when assessing the reviews.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.