- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a SVD compression method that combines double-sided whitening using a second-order KL-divergence approximation, a heterogeneous rank-allocation strategy and a loss-aware remapping for hybrid low-rank compression and quantization. The evaluation demonstrates decent performance compared to selected baselines, while providing several component ablations.
Critical strengths
- Although doubled-sided whitening itself is established in prior work [1,2,3], the proposed KL-based double-sided whitening presents a compelling approach to SVD compression and is supported by evidence to be beneficial over input-only whitening.
- Additional evaluation on VLMs and structured pruning providing insights beyond perplexity and accuracy.
[1] Chekalina et al., Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. ICML 2026
[2] Thoma et al., Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. ICML 2026
[3] Li et al., Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation. Preprint arXiv 2026
Critical weaknesses
Evaluation:
- Limited comparison against critical related work: Although Tables 2 and 6 demonstrate the benefits of double-sided whitening in small ablations, closely related double-sided whitening methods remain limited or are omitted entirely, for example, GFWSVD [1] (although referenced) and KFAC-SVD [2] are not evaluated. OBD-LLM [3] appears in Table 6 (on a single model and a single dataset), where improvements are small.
- Insufficient attribution of search strategy and decomposition: The evaluation, in particular with respect to the double-sided whitening and the heterogeneous rank allocation, does not clearly differentiate between the impact of these two components nor does it demonstrate whether each component independently advances recent baselines. Table 6 only provides a limited ablation in this regard. Pairing a decomposition with a rank-allocation strategy has been explored by recent works, so a clear attribution of the contribution of each component is necessary to establish novelty. Similarly, a direct comparison of the proposed rank-allocation strategy against existing strategies, such as Swift-SVD [4], LEMS [2], ACIP [5], ASVD [6], SVD-LLMv2 [7], ARS [8], ATP [9] will be needed to provide stronger evidence that the proposed method indeed is beneficial over current baselines.
- Lack of model diversity and fragmented model coverage: The experimental evaluation mainly focuses on the Llama-family, specifically the first-generation Llama, which is outdated. Other, newer model families like Mistral, Qwen3 or Gemma are completely omitted from the evaluation. Additionally, while the main experiments and ablations concentrate on Llama, Llama-3 is evaluated for free-form generation while Llama-2 is evaluated for reasoning and throughput speed, limiting a consistent assessment of the proposed method. Moreover, additional models in Table 5 omit datasets from the global benchmark (PTB, C4, ARC_C). The evaluation would benefit from consistent use of the complete benchmark, since it is unclear why these datasets were omitted.
Positioning and claims:
- Limited novelty and differentiation from prior work: The authors argue in the abstract that “..., existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression.” (line 014). However, several recent related methods, except SVD-LLM [11], have introduced a search strategy for allocating compression budgets (cf. [2,4,6,7,10]), e.g., ZS-SVD employs a greedy search, while singular values as error proxies was already explored by SVD-LLMv2 [7]. Therefore, the main potential novelty lies primarily in the KL-divergence approximation used for double-sided whitening. The paper should more clearly establish and demonstrate what the proposed approach adds over existing methods.
- The conclusion states that "Experiments show that IO-SVD better preserves perplexity and downstream accuracy than prior SVD-based methods" (line 467). However, the approach only shows minor improvements over ZS-SVD in zero-shot accuracy while mainly performing on par with ZS-SVD. Given the limited evaluation against critical related work and other model families, claims as such must be substantially tempered.
[4] Qi et al., Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. ICML 2026
[5] Genzel et al., Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation. arXiv 2025
[6] Yuan et al., ASVD: Activation-aware singular value decomposition for compressing large language models. arXiv 2025
[7] Wang et al., SVD-LLM v2: Optimizing singular value truncation for large language model compression. NAACL 2025
[8] Gao et al., Adaptive rank selections for low-rank approximation of language models. NAACL 2024
[9] Huang et al., Determining layer-wise sparsity for large language models through a theoretical perspective. ICML 2025
[10] Abbasi et al., Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. ICML 2026
[11] Wang et al., SVD-LLM: Truncation-aware singular value decomposition for large language model compression. ICLR 2025
Questions for authors
-
Could the authors elaborate on the model-selection rationale and why modern model families, such as Qwen3 or Mistral, are not evaluated?
-
While I acknowledge that some relevant baselines [2,4,10] were only recently published at ICML 2026, ZS-SVD [10] is still evaluated. I am curios why the others were omitted from the evaluation? Comparing them seems important to establish performance improvements of the proposed method, given that OBD-LLM [3] in Table 6 is the only comparison to a double-sided whitening approach. Additionally, since GFWSVD [1] and OBD-LLM [3] are explicitly acknowledged by the authors as related work on which their method builds upon (line 059), why are they omitted from the evaluation altogether (apart from OBD-LLM's appearance in Table 6)?
-
Line 221 in the manuscript mentions a `calibration loss', but its precise definition is unclear. Could the authors please provide the specific loss function?
-
The loss-aware remapping generally provides modest additional improvements over standard remapping in Table 8. Are these gains consistent across calibration seeds and model families? Is there additional time-cost associated with the loss-aware remapping compared to standard remapping?