acceptodds
Under review as a conference paper at ICLR 2027

GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression

Abstract

Large language models (LLMs) demonstrate strong generalization across a broad range of reasoning and generation tasks, but their large parameter counts impose substantial memory and computation costs, hindering deployment in practical resource-constrained settings. One-shot low-rank compression is a promising solution since it reduces model size and inference cost without requiring custom kernels. However, existing SVD-based methods primarily target multiple-choice benchmarks and often fail to preserve performance on generation tasks. Moreover, their rank allocation strategies are typically heuristic. We propose GA-SVD, a generation-aware low-rank compression framework for LLMs. First, we introduce Generation-Aware Weight Whitening (GAWW), which leverages gradients from on-policy knowledge distillation to construct whitening transformations aligned with generative behavior. Second, we develop an evolutionary global rank allocation method that searches for effective layer-wise rank configurations under a given compression ratio. Across extensive experiments on Mistral, Qwen, and Llama models, evaluated on both multiple-choice and generative benchmarks, GA-SVD consistently outperforms existing SVD-based baselines and achieves state-of-the-art results. We further demonstrate scalability by applying our method to models with up to 70B parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SVD-based LLM compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_waMEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GA-SVD, which compresses LLMs via SVD with bidirectional whitening, and it also applies evolutionary search for layer-wise rank allocation.

Critical strengths

The paper evaluates SVD-based compression beyond perplexity and multiple-choice benchmarks, covering generation tasks such as code generation, mathematical reasoning and summarization.

The empirical coverage is broad, spanning several model families and sizes up to 70B parameters.

The authors also show that the method can be combined with quantization for further compression, and they provide sensitivity analyses of the warm-up ratio, calibration set size and rollout length.

Critical weaknesses

The paper lacks novelty compared to prior art.

The bidirectional whitening has already been proposed in OBD-LLM (Li et al., 2026) and GFWSVD (Chekalina et al., 2025); in particular, OBD-LLM’s whitening, SVD and recoloring steps are identical to Eqs. 6–7 and 33 here. Neither work is cited or compared against.

The evolutionary search for rank allocation is also a standard technique, and for its small modifications (specific mutation operators, etc.) the paper does not compare against a standard GA to show a significant benefit.

The paper also does not compare its rank allocation, on the same whitened weights, against prior rank allocation methods such as ASVD’s STRS (Yuan et al., 2023), AdaSVD (Li et al., 2025), or PGSVD (Solgi et al., 2025).

There also seem to be theoretical gaps. In the derivation of generation-aware whitening, the authors drop the first-order term after Eq. 15. Prior work (e.g., OBD-LLM) justifies this by assuming the gradient of the pretrained model is small. Here, however, the gradients are computed on the student, which is already compressed by the input-based warm-start step, so they may not be small (or near zero).

Also, the expansion in Eq. 14 is taken around the teacher’s output, while the statistics are computed from the student. The authors need to justify this derivation.

LM Loss and Prefix KD are used as alternative loss functions in the ablation but are not clearly defined.

From both the contribution perspective and writing clarity, the paper requires substantial revision.

References Y. Li, et al. Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation. 2026.

V. Chekalina, et al. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. 2025.

R. Solgi, et al. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM. 2025.

Z. Yuan, et al. ASVD: Activation-Aware Singular Value Decomposition for Compressing Large Language Models. 2023.

Z. Li, et al. AdaSVD: Adaptive Singular Value Decomposition for Large Language Models. 2025.

Questions for authors

In Table 5, do the LM Loss and Prefix KD variants use ERA? If not, please explain how a bidirectional whitening method achieves worse results than GA-SVD w/o GAWW.

What is the novelty of this work compared with prior two-sided whitening methods? If using the one-sided compressed student to compute the gradients is the only difference, then the proposed solution seems less rigorous and suboptimal, as some of its assumptions may not hold in practice. What, then, makes this method better?

Reviewer_JkNrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GA-SVD improves low-rank LLM compression with two components: bi-sided weight transformations using activations and reverse-KL gradients from a warm-start student’s own rollouts, and evolutionary search over ranks and near-boundary singular components. It evaluates coding, math, multiple-choice tasks, and perplexity on models from 7B to 70B. Results improve over the tested SVD methods, although substantial gaps to the dense models remain.

Critical strengths
  1. Evaluating actual generation is worthwhile. Table 1 compares several SVD baselines on Mistral-7B, and Table 2 extends the comparison with SVD-LLM to Qwen3 and Llama-3.3-70B. The Qwen3-14B coding average improves from 19.0 to 44.1, which is a meaningful gain within this compression setting.

  2. The weighted low-rank solution is clearly derived once the quadratic metric is assumed. Table 5 also separates whitening from rank search, and the appendix checks rollout length, calibration size, summarization, and compatibility with GPTQ. These experiments make the approach promising.

Critical weaknesses
  1. The link between the proposed sensitivity matrix and generation loss needs stronger justification. Appendix A.1 drops the first-order term and substitutes reverse-KL gradient outer products for curvature. At a warm-start student, stationarity is not established, and this gradient second moment is not automatically the Gauss-Newton matrix. The transformed SVD solves the chosen metric, but that does not validate the metric as a local loss approximation.

  2. The global-budget comparison is hard to verify. Equation (11) transfers equal rank increments between matrices, although their parameter costs r(d_in+d_out) can differ. The text calls budget preservation approximate, but does not specify a repair or rejection rule or report achieved parameter counts. Since rank search contributes substantially in Table 5, this omission makes it difficult to assess comparisons at equal compression.

Questions for authors
  1. Clarification and validation: At which model’s activations and weights are the reverse-KL gradients evaluated? Why is dropping the linear term reasonable there, and how well does the quadratic score predict measured loss changes for candidate compressions?

  2. Clarification: How are offspring that exceed the exact parameter budget handled? Please report achieved parameter counts for each method and the rule used after rank mutation. A matched-budget, repeated-seed comparison of GAWW against input-only whitening would also help establish the generation-specific benefit.

Reviewer_RycWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GA-SVD, a low-rank compression framework designed to preserve the generation capabilities of large language models. Its first component, Generation-Aware Weight Whitening (GAWW), collects input activations and reverse-KL distillation gradients on continuations generated by a warm-start compressed student. These statistics define a double-sided weighted metric for SVD-based compression. Its second component uses evolutionary search to allocate ranks across weight matrices and adjust retained singular components near the truncation boundaries under a global parameter budget. Experiments on Mistral-7B, Qwen3, and Llama-3.3-70B evaluate code generation, mathematical reasoning, multiple-choice tasks, and perplexity. The authors report improvements over the evaluated SVD baselines, together with reduced memory usage and increased inference throughput.

Critical strengths
  1. Motivated design for LLM compression. The paper addresses the degradation of autoregressive generation under low-rank compression. Using distillation gradients from the compressed student’s own rollouts is a motivated way to align the compression metric with the behavior encountered during generation.

  2. Broad empirical evaluation and practical measurements. The experiments cover several model families, including models up to 70B parameters, across coding, mathematical reasoning, multiple-choice tasks, and perplexity. The reported improvements support the overall pipeline, while memory, throughput, and compression-time measurements provide evidence of its practical applicability.

Critical weaknesses
  1. The methodological advance over existing approaches appears limited. Double-sided sensitivity-aware SVD and evolutionary allocation have close precedents in GFWSVD[R1]/IO-SVD[R2] and EvoPress[R3]/DarwinLM, respectively. GA-SVD mainly adds sensitivity statistics from student rollouts and restricted boundary selection. Table 5 implies GAWW gains of 1.6/2.1/0.4 points on HumanEval/MBPP/GSM8K; without GAWW, the method already slightly exceeds the compressed baselines in Table 1 on these metrics. Although GAWW improves the scores, the robustness of its incremental benefit remains unclear without variability across runs.

  2. The empirical comparisons do not sufficiently establish competitiveness and generality. GFWSVD and IO-SVD are omitted, while the larger-model and 40% compression results compare only against SVD-LLM among compressed baselines. Gains are also not uniform: on Llama-3.3-70B, GA-SVD trails SVD-LLM on MBPP (51.8 vs. 56.9). Comparisons should match realized parameter counts and calibration conditions and account for rollout generation, teacher scoring, student backward passes, and search costs. Budget enforcement also needs clarification, since equal-rank mutations preserve parameter cost only approximately.

  3. The sensitivity metric is not sufficiently justified as a curvature approximation. Appendix A.1 expands around the original layer output, whereas the collected gradients appear to come from the warm-start student; this evaluation point needs clarification. It also drops the first-order term without establishing stationarity and substitutes gradient outer products for curvature. Reverse-KL gradients can vanish when predictions match, while KL curvature can remain nonzero, so the matrix is not generally Fisher/Gauss–Newton curvature. The appendix establishes an exact SVD solution for the chosen surrogate, but its ability to predict actual generation-loss changes, especially after recompression and rank reallocation, remains unvalidated.

[R1] Chekalina et al. Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models. arXiv:2505.17974, 2025.
[R2] Abbasi, Thrash, Qin, Pirsiavash, and Kolouri. IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression. arXiv:2605.15626, 2026.
[R3] Sieberling, Kuznedelev, Kurtic, and Alistarh. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. arXiv:2410.14649, 2024.

Questions for authors
  1. Statistics and calibration: Are the input activations used in Σx\Sigma_x collected from the teacher or the warm-start student? Please clarify the respective roles of the 1024 × 2048 calibration samples and the student rollouts in estimating Eq. (3), and whether the statistics are refreshed after recompression or rank reallocation.

  2. Search protocol: How does ERA enforce the global parameter budget and evaluate candidates? Please specify how infeasible mutations are handled, report actual parameter counts, and clarify the fitness KL direction and the source and reuse of evaluation sequences.

  3. Clarity and consistency: What does the “warmup ratio” in Table 9 measure, and how does it relate to the warm-start initialization and final compression ratio? Please define this term explicitly and check numerical consistency across tables, particularly the dense-model PIQA score of 28.1 in Table 8 versus 82.1 in Table 1.

  4. Smaller models: At 20% compression, Qwen3-8B retains only 16.5 of the dense model’s 78.4 coding average (Table 2). How does GA-SVD perform on smaller models (e.g., 1–3B) at moderate compression ratios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.