acceptodds
Under review as a conference paper at ICLR 2027

IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

Abstract

Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constrained and latency-sensitive settings. SVD-based post-training compression offers a hardware-agnostic way to reduce model size and improve inference efficiency through low-rank factorization. However, existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression. We propose Input-Output Whitened SVD (IO-SVD), a post-training compression method that forms a KL-aware double-sided whitening space for model weights. Using a second-order expansion of the KL loss over the top- token probabilities, IO-SVD constructs an output-side metric that captures predictive sensitivity, while input whitening captures activation statistics. We further introduce an efficient heterogeneous rank-allocation strategy that scores whitened singular components using first-order calibration loss estimates and prunes the least sensitive components under a global budget. Inspired by prior work that combines SVD truncation with quantization, we improve hybrid SVD-quantization compression through loss-aware remapping, which selects low-rank factor rows for 8-bit quantization based on the predicted loss change incurred by quantizing them. Across LLM and VLM families, we show that double-sided whitening preserves the model's predictive output distribution more closely than input-only whitening at matched ranks. This improvement is also reflected in stronger free-form generation and higher speculative-decoding acceptance, while our inference analysis characterizes the memory and throughput benefits of low-rank compression across different serving settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language model compression, low-rank approximation, singular value decomposition, post-training compression, double-sided whitening, adaptive rank allocation, loss-aware compression, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GCJdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a SVD compression method that combines double-sided whitening using a second-order KL-divergence approximation, a heterogeneous rank-allocation strategy and a loss-aware remapping for hybrid low-rank compression and quantization. The evaluation demonstrates decent performance compared to selected baselines, while providing several component ablations.

Critical strengths
  • Although doubled-sided whitening itself is established in prior work [1,2,3], the proposed KL-based double-sided whitening presents a compelling approach to SVD compression and is supported by evidence to be beneficial over input-only whitening.
  • Additional evaluation on VLMs and structured pruning providing insights beyond perplexity and accuracy.

[1] Chekalina et al., Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. ICML 2026

[2] Thoma et al., Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. ICML 2026

[3] Li et al., Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation. Preprint arXiv 2026

Critical weaknesses

Evaluation:

  • Limited comparison against critical related work: Although Tables 2 and 6 demonstrate the benefits of double-sided whitening in small ablations, closely related double-sided whitening methods remain limited or are omitted entirely, for example, GFWSVD [1] (although referenced) and KFAC-SVD [2] are not evaluated. OBD-LLM [3] appears in Table 6 (on a single model and a single dataset), where improvements are small.
  • Insufficient attribution of search strategy and decomposition: The evaluation, in particular with respect to the double-sided whitening and the heterogeneous rank allocation, does not clearly differentiate between the impact of these two components nor does it demonstrate whether each component independently advances recent baselines. Table 6 only provides a limited ablation in this regard. Pairing a decomposition with a rank-allocation strategy has been explored by recent works, so a clear attribution of the contribution of each component is necessary to establish novelty. Similarly, a direct comparison of the proposed rank-allocation strategy against existing strategies, such as Swift-SVD [4], LEMS [2], ACIP [5], ASVD [6], SVD-LLMv2 [7], ARS [8], ATP [9] will be needed to provide stronger evidence that the proposed method indeed is beneficial over current baselines.
  • Lack of model diversity and fragmented model coverage: The experimental evaluation mainly focuses on the Llama-family, specifically the first-generation Llama, which is outdated. Other, newer model families like Mistral, Qwen3 or Gemma are completely omitted from the evaluation. Additionally, while the main experiments and ablations concentrate on Llama, Llama-3 is evaluated for free-form generation while Llama-2 is evaluated for reasoning and throughput speed, limiting a consistent assessment of the proposed method. Moreover, additional models in Table 5 omit datasets from the global benchmark (PTB, C4, ARC_C). The evaluation would benefit from consistent use of the complete benchmark, since it is unclear why these datasets were omitted.

Positioning and claims:

  • Limited novelty and differentiation from prior work: The authors argue in the abstract that “..., existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression.” (line 014). However, several recent related methods, except SVD-LLM [11], have introduced a search strategy for allocating compression budgets (cf. [2,4,6,7,10]), e.g., ZS-SVD employs a greedy search, while singular values as error proxies was already explored by SVD-LLMv2 [7]. Therefore, the main potential novelty lies primarily in the KL-divergence approximation used for double-sided whitening. The paper should more clearly establish and demonstrate what the proposed approach adds over existing methods.
  • The conclusion states that "Experiments show that IO-SVD better preserves perplexity and downstream accuracy than prior SVD-based methods" (line 467). However, the approach only shows minor improvements over ZS-SVD in zero-shot accuracy while mainly performing on par with ZS-SVD. Given the limited evaluation against critical related work and other model families, claims as such must be substantially tempered.

[4] Qi et al., Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. ICML 2026

[5] Genzel et al., Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation. arXiv 2025

[6] Yuan et al., ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv 2025

[7] Wang et al., SVD-LLM v2: Optimizing singular value truncation for large language model compression. NAACL 2025

[8] Gao et al., Adaptive rank selections for low-rank approximation of language models. NAACL 2024

[9] Huang et al., Determining layer-wise sparsity for large language models through a theoretical perspective. ICML 2025

[10] Abbasi et al., Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. ICML 2026

[11] Wang et al., SVD-LLM: Truncation-aware singular value decomposition for large language model compression. ICLR 2025

Questions for authors
  1. Could the authors elaborate on the model-selection rationale and why modern model families, such as Qwen3 or Mistral, are not evaluated?

  2. While I acknowledge that some relevant baselines [2,4,10] were only recently published at ICML 2026, ZS-SVD [10] is still evaluated. I am curios why the others were omitted from the evaluation? Comparing them seems important to establish performance improvements of the proposed method, given that OBD-LLM [3] in Table 6 is the only comparison to a double-sided whitening approach. Additionally, since GFWSVD [1] and OBD-LLM [3] are explicitly acknowledged by the authors as related work on which their method builds upon (line 059), why are they omitted from the evaluation altogether (apart from OBD-LLM's appearance in Table 6)?

  3. Line 221 in the manuscript mentions a `calibration loss', but its precise definition is unclear. Could the authors please provide the specific loss function?

  4. The loss-aware remapping generally provides modest additional improvements over standard remapping in Table 8. Are these gains consistent across calibration seeds and model families? Is there additional time-cost associated with the loss-aware remapping compared to standard remapping?

Reviewer_wRWBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IO-SVD, a compression framework for LLMs that uses SVD with top-K KL-based bidirectional whitening. It also proposes loss-aware rank allocation and int8 remapping, improving perplexity over several SVD baselines.

Critical strengths

Strengths:

  • The paper offers a distinct perspective on double-sided whitening: the output-side curvature is derived from the KL divergence to the original model's predictive distribution, rather than from the loss against ground-truth labels.

  • It integrates double-sided whitening, adaptive rank allocation, and loss-aware hybrid SVD-quantization (remapping) into a single framework.

  • Experiments span several LLM and VLM families and multiple task types.

  • Ablations show the benefit of bidirectional over input-only whitening and of adaptive over uniform rank allocation.

Critical weaknesses
  • My main concern is that the novelty is incremental. As the authors state (Appendix E.1), the core contribution is the KL-based output-side curvature. Double-sided whitening for SVD compression has already been proposed (OBD-LLM; Kronecker-factored Fisher in Chekalina et al., 2025). The proposed formulation differs mainly in the curvature estimator: the expected (true) Fisher over the model's top-K predictions instead of the empirical Fisher from data labels. Studying whether matching the model's distribution rather than the labels helps is interesting, but the paper does not convincingly show that this change yields a significant benefit.

  • The most relevant comparison or baseline for this KL contribution, isolating its effect against other double-sided SVD (i.e. OBD-LLM), appears only in Table 6: one model (LLaMA-7B), one metric (WikiText-2 perplexity), with marginal gains. It is unclear whether these gains hold across models, datasets, and downstream tasks.

  • The strongest evidence for distribution preservation (output KL in Table 2, GSM8K in Table 3, speculative decoding in Table 4) compares against input-only whitening or ZS-SVD, not against double-sided whitening. These results therefore support double-sided whitening in general, which was already known, rather than the KL formulation specifically.

  • Compression time is not compared against the empirical fisher. The top-K curvature requires K backward probes per token versus one backward pass for the empirical fisher (OBD-LLM), so the proposed method is likely more expensive. It is unclear whether this cost is justified by the small gains.

  • The rank-allocation ablation compares only against uniform ranks, not against existing loss-aware allocators (e.g., SVD-LLMv2, Dobi-SVD, ZS-SVD) applied in the same whitened space.

  • The related work on rank allocation could be updated with other relevant methods such as ARS, ARA, AdaSVD, and PGSVD

Gao et al. Adaptive Rank Selections for Low-Rank Approximation of Language Models. 2024.

Xv et al. ARA: Adaptive Rank Allocation for Efficient Large Language Model SVD Compression. 2025.

Li et al. AdaSVD: Adaptive Singular Value Decomposition for Large Language Models. 2025.

Solgi et al. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM. 2025.

Questions for authors
  • Setting aside rank allocation and remapping, what is the theoretical or empirical case for the KL (true-Fisher) formulation over OBD-LLM's empirical-Fisher curvature?

  • Can the authors report Tables 2-4 with an OBD-LLM-curvature baseline at matched ranks?

Reviewer_qTjbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies low rank LLM compression. Its goal is to preserve the predictive distribution of the original model, not only to reconstruct the output of each linear layer. IO-SVD derives a weighted reconstruction objective from a second order approximation to KL divergence. It combines input activation statistics with output sensitivity estimated from the top K token probabilities. Then it applies SVD in the resulting space. After that, it allocates ranks across layers using first order calibration loss scores. For hybrid compression, the method also uses predicted loss changes to select rows of the retained factors for quantization to INT8.

The evaluation covers language models and vision language models. It also includes experiments on GSM8K generation and speculative decoding acceptance. The comparison at identical ranks shows lower output KL than whitening based only on input statistics. The broader results show improvements in perplexity over several SVD baselines. My main concern is whether these results establish the value of the particular KL metric and allocation rules, rather than the general benefit of including output sensitivity.

Critical strengths
  1. Preserving the predictive distribution is a sensible objective. Similar local reconstruction errors can have different effects on the final predictions. Table 2 supports this motivation. It shows that double sided whitening lowers output KL at identical ranks even though local reconstruction error is slightly higher.

  2. The evaluation goes beyond perplexity and multiple choice accuracy. It also examines free form generation and speculative decoding acceptance. In particular, the large GSM8K improvement over ZS-SVD suggests that the difference between the methods matters for generated responses. However, the compressed models still lose considerable accuracy relative to the original model.

  3. Tables 6 and 8 make a useful effort to separate whitening, rank allocation, and remapping. Appendix H reports compression time for the individual stages. The authors also distinguish serialized storage from inference memory. They acknowledge that latent key and value caching is not specific to IO-SVD. This helps clarify the scope of the efficiency results.

Critical weaknesses
  1. The paper establishes the benefit of double sided whitening much more clearly than the benefit of its particular KL metric. Table 6 shows only small improvements over OBD-LLM under heterogeneous ranks: 6.45 to 6.41 at ratio 0.8, and 28.19 to 27.70 at ratio 0.4. However, this comparator is completely absent from the output KL, generation, and acceptance experiments in Tables 2 to 4. So the strongest evidence for preserving predictive behavior does not actually establish what IO-SVD adds beyond an existing formulation that already includes output sensitivity. This is a major gap for me.

  2. More fundamentally, Section 3.2 does not define the calibration loss used for the first order rank allocation scores. This is a problem even though Algorithm 1 evaluates the gradients before truncation. If this is the KL objective in Equation 1, then the gradients vanish at the original model, as shown in Appendix B.1. If it is a different loss, then the paper needs to explain how rank allocation relates to the stated goal of preserving the predictive distribution. Either way, this needs to be clarified.

  3. The top K approximation also leaves an important type of distribution change unmeasured. Renormalizing the selected probabilities makes the reduced KL metric insensitive to a common shift of the selected logits relative to the omitted vocabulary. Figure 3 studies the choice of K, but it reports normalized perplexity rather than agreement with changes in KL over the full vocabulary. So it does not show how this omission affects the metric used for compression.

  4. The hybrid comparison is also hard to interpret. Equation 68 defines the savings still required after SVD truncation, but it does not specify how the initial ranks are chosen for a given final maintenance ratio. Retaining more singular components can itself improve quality. Therefore, Tables 1 and 8 really need the actual factor ranks and serialized sizes to show whether the benefit of loss aware remapping comes from row selection or just a different division of compression.

Questions for authors
  1. I think a direct comparison with OBD-LLM would be quite informative. If the ranks, calibration data, and remapping strategy are held fixed, does the proposed KL metric still show a clear advantage on output KL and GSM8K? Also, showing variation across calibration seeds would help, especially since the perplexity gaps in Table 6 are quite small.

  2. I found the rank allocation is hard to follow. What exactly is the calibration loss in Section 3.2, and at which model state are the gradients evaluated? I ask because if this loss is the same as Equation 1, the gradients would vanish at the original model (as noted in Appendix B.1). If it is a different loss, how does it still connect to preserving the predictive distribution?

  3. The top K approximation also seems tricky. Does it track changes in KL over the full vocabulary on held-out data? Also, is Table 2 computed over the entire vocabulary? Reporting how much probability mass the selected K actually covers would make the approximation much easier to judge.

  4. For the hybrid results, how is the initial SVD budget chosen before remapping? I also couldn't tell if the standard and loss-aware variants start from the exact same factors and end up with the same serialized size. Providing the retained ranks, the number of FP16/INT8 rows, and the final bytes would really help separate the effect of the row selection from just changing how compression is divided.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.