- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Multi-Subspace Compression (HMC), a post-training low-rank compression framework for large language models (LLMs), which achieves improved compression performance over global low-rank baselines under the same parameter budget. Specifically, the authors first partition the rows of weight matrices into multiple groups using k-subspaces clustering, allowing each group to be represented by an independent low-rank subspace rather than a shared global subspace. Then, an adaptive rank allocation strategy is introduced to assign different ranks to the groups according to their activation-aware spectral energy gains and parameter costs under a fixed budget. To optimize the low-rank factors, the authors further propose an activation-aware joint fitting method that accounts for cross-group activation correlations and derives exact block updates based on truncated singular value decomposition. Furthermore, a hierarchical representation is constructed by recursively repartitioning and approximating the residual weight matrices, followed by cross-level refinement to jointly improve the reconstruction quality. Experiments on five LLMs demonstrate that HMC generally achieves lower perplexity and higher zero-shot accuracy than existing global low-rank compression baselines across different compression ratios. However, I have some concerns about this paper. My detailed comments are as follows.
Critical strengths
- The authors propose a hierarchical multi-subspace low-rank compression framework for LLMs, which improves the flexibility and parameter efficiency of low-rank representations by allowing different groups of weight rows to be approximated using distinct subspaces.
- The proposed activation-aware joint fitting method explicitly accounts for cross-group activation correlations and derives exact block-wise updates via truncated SVD, providing a principled optimization approach for the multi-subspace representation.
- The hierarchical residual approximation with adaptive rank allocation enables the method to capture additional low-rank structures under a fixed parameter budget, while further improving reconstruction quality through cross-level refinement.
- Extensive experiments on five LLMs across multiple compression ratios, including perplexity and zero-shot evaluations, demonstrate the effectiveness of HMC over existing global low-rank compression baselines. The ablation studies further validate the contributions of several key components.
Critical weaknesses
- Insufficient validation of the local low-rank structure. The paper mainly uses pairwise cosine similarity visualizations in Figure 1 to motivate the multi-subspace representation. However, cosine similarity does not directly demonstrate the existence of distinct low-dimensional subspaces. The authors are encouraged to provide more convincing evidence to better support this key motivation.
- Insufficient ablation of adaptive rank allocation. The paper proposes an adaptive rank allocation strategy, but its independent contribution is not clearly evaluated. A comparison with simpler strategies, such as uniform rank allocation under the same parameter budget, would help demonstrate its effectiveness.
- Insufficient analysis of the regularization strategy. The paper introduces a damping term to ensure the positive definiteness of the activation covariance matrix for Cholesky decomposition. However, prior work such as SVD-LLM V2 [A] has explored SVD-based alternatives that can handle singular covariance matrices. Since damping modifies the reconstruction objective, the authors are encouraged to investigate its necessity and sensitivity, or compare it with an SVD-based alternative.
- Limited comparison with recent SVD-based compression methods. The paper mainly compares HMC with conventional global low-rank compression baselines. However, recent methods such as SVD-LLM V2 [A] and Dobi-SVD [B] have demonstrated the effectiveness of adaptive rank allocation across different weight matrices. Since HMC primarily allocates ranks within each weight matrix, comparisons with these methods under the same model-level parameter budget would help better establish its competitiveness.
- Limited comparison with parameter-efficient low-rank compression methods. The paper mainly compares HMC with conventional global low-rank methods, while other structured compression approaches, such as MoDeGPT and Pivoting Factorization, also aim to improve the parameter efficiency of low-rank representations. Comparisons with these methods under comparable compression budgets would help better demonstrate the effectiveness and competitiveness of HMC.
- Limited analysis of compression time overhead. HMC introduces additional computational steps, including subspace clustering, joint fitting, and cross-level refinement. Although the paper analyzes optimization convergence, it does not report the actual end-to-end compression time or compare it with existing baselines.
[A] SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. 2025 NAACL. [B] Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. 2025 ICLR. [C] MoDeGPT: Modular Decomposition for Large Language Model Compression. 2025 ICLR. [D] Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models. 2026 ICML.
Questions for authors
See weaknesses.
I thank the authors for performing the additional comparisons and for the provided clarifications. Most of my concerns have been now adequately addressed so I am therefore leaning toward a positive recommendation.