- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Loss-aware Truncation with Gradients (LTG), a low-rank compression method for large language models. The main idea is that traditional SVD methods focus on minimizing weight reconstruction error, which doesn't necessarily preserve model performance. To address this, the authors use gradient information to identify more important weight directions and guide the SVD truncation process. They also introduce a self-calibration strategy that generates calibration data directly from the original model, reducing the need for external datasets. Experiments across several LLM families show that LTG generally outperforms existing SVD-based compression methods, especially at higher compression ratios.
Critical strengths
-
Well-motivated approach with solid mathematical derivation. Traditional SVD methods focus on minimizing the reconstruction error caused by low-rank truncation, but components with small singular values can still be important for the model's performance. This paper addresses this issue by incorporating gradient information to better capture how different components affect the model's loss. I particularly appreciate the mathematical derivation, which uses first-order approximation and upper-bound optimization to gradually introduce gradient information into the transformation matrix. The overall approach is well-motivated and mathematically grounded.
-
Comprehensive experiments with strong results. The paper provides extensive experiments across multiple LLM families, compression ratios, and downstream tasks.
-
Another strength is the self-calibration approach, which allows the model to generate its own calibration data instead of relying on an external calibration dataset. This makes the calibration process more flexible and practical.
Critical weaknesses
-
Self-calibration and robustness: While self-calibration is a nice idea, I'm wondering whether model-generated data can really cover diverse real-world distributions, especially rare or long-tail cases. Could this affect the robustness of the compressed model? It would be helpful to see some experiments on out-of-distribution tasks.
-
Approximation errors: The mathematical derivation is solid, but it relies on a first-order Taylor approximation and then optimizes an upper bound rather than the actual loss change. I'm curious how well this surrogate objective reflects the true loss degradation, especially at higher compression ratios. Could the authors provide more analysis of this gap (Theo & Experiment)?
-
Computational and memory overhead: Since LTG requires collecting per-sample gradients and computing gradient covariance matrices, I'm concerned about the additional computational and memory costs. How does the method scale with larger batch sizes and model sizes? It would be useful to report compression time and peak GPU memory usage compared to existing methods.
-
For expression (8) in paper, I think the authors should provide more theoretical analysis to show whether the minimum of the proposed upper bound can be sufficiently small.
Questions for authors
See in Weaknesses
2 comments
Response to Reviewer gKwo (Part 2/2)
[W2. Approximation errors: The mathematical derivation is solid, but it relies on a first-order Taylor approximation and then optimizes an upper bound rather than the actual loss change. I'm curious how well this surrogate objective reflects the true loss degradation, especially at higher compression ratios. Could the authors provide more analysis of this gap (Theo & Experiment)?]
and
[W4. For expression (8) in paper, I think the authors should provide more theoretical analysis to show whether the minimum of the proposed upper bound can be sufficiently small.]
Theory. We provide further theoretical analysis to characterize the minimum achievable value of the proposed upper bound in Eq. (8).
Specifically, our optimization objective in Eq. (9) is the aforementioned upper bound. As established in Eq. (13), the minimum of the gradient-weighted reconstruction objective under a rank- constraint is given by
\min_{\operatorname{rank}(\widetilde{\mathbf W})\le r} \Vert(\mathbf W-\widetilde{\mathbf W})\mathbf G_{\mathbf W}^{\top}\Vert_F^2 = \sum_{j>r}\sigma_j^2,$$ where $\sigma_j$ denotes the $j$-th singular value of $\mathbf W\mathbf G$, with $\mathbf G\mathbf G^\top=\mathbf G_{\mathbf W}^\top\mathbf G_{\mathbf W}$ obtained through Cholesky decomposition. Since the $\sigma_j$ are nonnegative and ordered in descending magnitude, $\mathcal{G}^*$ decreases monotonically as the retained rank $r$ increases, reaching zero when all $\sigma_j$ are retained. Therefore, lower compression ratios yield smaller upper bounds on the first-order loss approximation, providing theoretical justification for better model performance preservation at less aggressive compression levels. **Experiments.** Moreover, as shown in Fig. 4 and Tables 1, 3, and 5, LTG remains effective even at aggressive compression ratios of up to 80%. Furthermore, Fig. 3(a) directly evaluates the true loss degradation induced by removing singular components. Specifically, we divide the singular components into 10 equally sized chunks, ordered by decreasing singular value magnitude, and measure the actual loss change relative to the original model after removing each chunk. The results show that LTG's gradient-aware transformation makes singular value magnitudes better reflect their importance to the model loss. In particular, smaller singular values consistently locally correspond to lower loss degradation.
Response to Reviewer gKwo (Part 1/2)
Dear Reviewer gKwo,
Thank you for your insightful and constructive feedback, which will help us further strengthen our work. Below, we provide responses to each of your concerns in 02 parts.
We will revise the manuscript accordingly to incorporate your suggestions.
[W1. Self-calibration and robustness: While self-calibration is a nice idea, I'm wondering whether model-generated data can really cover diverse real-world distributions, especially rare or long-tail cases. Could this affect the robustness of the compressed model? It would be helpful to see some experiments on out-of-distribution tasks.]
We thank the reviewer for this insightful question.
We consider "out-of-distribution" (OOD) in two way as below:
Case 1: OOD relative to the calibration distribution. Our experiments use only 256 calibration samples (consistent with SVD-LLM, ASVD, FWSVD), while evaluating the compressed models across diverse distributions and tasks, including language modeling (C4 perplexity), commonsense reasoning (WinoGrande, HellaSwag, and PIQA), question answering (OpenBookQA and ARC-Easy), and mathematical reasoning (MathQA and GSM8K). These evaluations differ substantially from the self-generated calibration data, providing empirical evidence that LTG preserves model capabilities beyond the calibration distribution.
Case 2: out-of-distribution of pretraining data Self-calibration aims to approximate the model's learned pretraining distribution rather than explicitly capture unseen distributions. Therefore, we acknowledge that its effectiveness on distributions substantially different from the pretraining data is not guaranteed and may be limited. However, identifying genuinely OOD benchmarks is challenging, particularly since the complete pretraining corpora of the evaluated models are not publicly available. We therefore consider rigorous evaluation under such distribution shifts an important direction for future work and would welcome the reviewer's suggestions regarding suitable OOD benchmarks.
[W3. Computational and memory overhead: Since LTG requires collecting per-sample gradients and computing gradient covariance matrices, I'm concerned about the additional computational and memory costs. How does the method scale with larger batch sizes and model sizes? It would be useful to report compression time and peak GPU memory usage compared to existing methods.]
Thank you for raising this concern. Please refer to our response to Reviewer PEgS, [W2.2, W3, Q1.2, and Q2: Systematic evaluations on more recent and larger-scale LLMs], where we provide detailed comparisons of compression time, peak GPU VRAM, and CPU RAM usage against SVD-LLM and other baselines.
Importantly, compression incurs only a one-time offline cost and can be performed on a high-performance computing platform before deploying the compressed model to resource-constrained devices. Therefore, this overhead does not affect inference-time computational or memory requirements.