acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Multi-Subspace Low-Rank Compression for Large Language Models

Abstract

Low-rank model compression represents a weight matrix in a global low-rank subspace, which can limit its ability to preserve distinct local low-rank structures existing in the pretrained weight. We propose Hierarchical Multi-subspace low-rank Compression (HMC), a post-training method that partition the weight rows into different groups, where each group can separately approximate their local subspace using low-rank factorization. We further construct a hierarchical representation by repartitioning and approximating successive residuals to capture local structure beyond a single partition. To optimize the local factor matrices, HMC adaptively allocates ranks and jointly fits the local factors across different groups. Under subspace and partition assumptions, our analysis establishes when local factors reduce reconstruction error relative to a global factorization and when two levels reduce the factor cost of exact reconstruction relative to a single level. Experimental results across five LLMs, HMC generally lowers WikiText-2 and C4 perplexity and improves accuracy on six down-stream tasks over the low-rank compression baselines. Controlled ablations further support the benefits of multi-subspace partition, joint fitting, and hierarchical representation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-rank Approximation, Model Compression, Multi-subspace Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mY2yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Multi-Subspace Compression (HMC), a post-training low-rank compression framework for large language models (LLMs), which achieves improved compression performance over global low-rank baselines under the same parameter budget. Specifically, the authors first partition the rows of weight matrices into multiple groups using k-subspaces clustering, allowing each group to be represented by an independent low-rank subspace rather than a shared global subspace. Then, an adaptive rank allocation strategy is introduced to assign different ranks to the groups according to their activation-aware spectral energy gains and parameter costs under a fixed budget. To optimize the low-rank factors, the authors further propose an activation-aware joint fitting method that accounts for cross-group activation correlations and derives exact block updates based on truncated singular value decomposition. Furthermore, a hierarchical representation is constructed by recursively repartitioning and approximating the residual weight matrices, followed by cross-level refinement to jointly improve the reconstruction quality. Experiments on five LLMs demonstrate that HMC generally achieves lower perplexity and higher zero-shot accuracy than existing global low-rank compression baselines across different compression ratios. However, I have some concerns about this paper. My detailed comments are as follows.

Critical strengths
  1. The authors propose a hierarchical multi-subspace low-rank compression framework for LLMs, which improves the flexibility and parameter efficiency of low-rank representations by allowing different groups of weight rows to be approximated using distinct subspaces.
  2. The proposed activation-aware joint fitting method explicitly accounts for cross-group activation correlations and derives exact block-wise updates via truncated SVD, providing a principled optimization approach for the multi-subspace representation.
  3. The hierarchical residual approximation with adaptive rank allocation enables the method to capture additional low-rank structures under a fixed parameter budget, while further improving reconstruction quality through cross-level refinement.
  4. Extensive experiments on five LLMs across multiple compression ratios, including perplexity and zero-shot evaluations, demonstrate the effectiveness of HMC over existing global low-rank compression baselines. The ablation studies further validate the contributions of several key components.
Critical weaknesses
  1. Insufficient validation of the local low-rank structure. The paper mainly uses pairwise cosine similarity visualizations in Figure 1 to motivate the multi-subspace representation. However, cosine similarity does not directly demonstrate the existence of distinct low-dimensional subspaces. The authors are encouraged to provide more convincing evidence to better support this key motivation.
  2. Insufficient ablation of adaptive rank allocation. The paper proposes an adaptive rank allocation strategy, but its independent contribution is not clearly evaluated. A comparison with simpler strategies, such as uniform rank allocation under the same parameter budget, would help demonstrate its effectiveness.
  3. Insufficient analysis of the regularization strategy. The paper introduces a damping term to ensure the positive definiteness of the activation covariance matrix for Cholesky decomposition. However, prior work such as SVD-LLM V2 [A] has explored SVD-based alternatives that can handle singular covariance matrices. Since damping modifies the reconstruction objective, the authors are encouraged to investigate its necessity and sensitivity, or compare it with an SVD-based alternative.
  4. Limited comparison with recent SVD-based compression methods. The paper mainly compares HMC with conventional global low-rank compression baselines. However, recent methods such as SVD-LLM V2 [A] and Dobi-SVD [B] have demonstrated the effectiveness of adaptive rank allocation across different weight matrices. Since HMC primarily allocates ranks within each weight matrix, comparisons with these methods under the same model-level parameter budget would help better establish its competitiveness.
  5. Limited comparison with parameter-efficient low-rank compression methods. The paper mainly compares HMC with conventional global low-rank methods, while other structured compression approaches, such as MoDeGPT and Pivoting Factorization, also aim to improve the parameter efficiency of low-rank representations. Comparisons with these methods under comparable compression budgets would help better demonstrate the effectiveness and competitiveness of HMC.
  6. Limited analysis of compression time overhead. HMC introduces additional computational steps, including subspace clustering, joint fitting, and cross-level refinement. Although the paper analyzes optimization convergence, it does not report the actual end-to-end compression time or compare it with existing baselines.

[A] SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. 2025 NAACL. [B] Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. 2025 ICLR. [C] MoDeGPT: Modular Decomposition for Large Language Model Compression. 2025 ICLR. [D] Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models. 2026 ICML.

Questions for authors

See weaknesses.

1 comment

I thank the authors for performing the additional comparisons and for the provided clarifications. Most of my concerns have been now adequately addressed so I am therefore leaning toward a positive recommendation.

Reviewer_1vY1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HMC (Hierarchical Multi-Subspace Low-Rank Compression) , a post-training compression method, which partitions the rows of each weight matrix into groups via k-subspace clustering and approximates each group with it's own low-rank factorization. The motivation behind this strategy is that pretrained weight matrices contain multiple distinct local low-rank structures that a global low-rank factorization doesn't have the capacity to represent. This partitioning strategy is extended hierarchically by repartitioning and approximating successive residuals over levels, where ranks are allocated within each level via Knapsack. The authors report experiments on five models at different compression ratios and compare against ASVD, SVD-LLM and Swift-SVD

Critical strengths
  1. The paper motivates well the multi subspace compression strategy. Figure 1 shows that row vectors of pretrained weight matrices exhibit strong local similarity structure after reordering, and Figure 8 extends this to residuals. This makes the central claim that a single global low-rank subspace can be too restrictive, concrete.

  2. The reported ablations are well designed and make it easier to attribute the gains to the proposed components

  3. The evaluation of the proposed method is broad enough across models and budgets. It covers five LLMs at multiple retention ratios, while both WikiText-2 and C4 perplexity and six zero-shot downstream tasks are reported. The comparison against ASVD, SVD-LLM, and Swift-SVD is relevant to the activation-aware global low-rank line of work.

Critical weaknesses
  1. The main baselines are ASVD, SVD-LLM, and Swift-SVD. These are indeed representative methods for activation-aware global low-rank compression, but the paper should compare to cross-layer sharing low-rank methods such as Basis Sharing [1], multilevel low-rank approximation [Parshakova et. al. 2025], and more recent and stronger baselines that utilize rank-allocation methods such as SVD-LLM V2, Dobi-SVD, HiSVD, LEMS [2] and Rocket [5].

  2. While [3] and it's follow-up works [4, 5, 6] are relatively recent methods and thus is not entirely necessary to be included in the comparisons, they definitely need to be discussed. These methods also argue that global low-rank methods do not have the capacity for rich representations and instead propose and utilize a union-of-subspace representation motivated by dictionary learning techniques.

  3. Four out of the five models that are used for validation span two closely related LLaMA-derived lineages (LLaMA-2/3 and Yi). The authors completely omit comparisons on more recent models such as Qwen, Gemma or Phi and model sizes outside the range 6B-13B, so it is not entirely clear if the same row-block low-rank structure patterns appear in other model families and sizes. Does the Fig. 1 multi-subspace row structure hold for Qwen2.5/Qwen3 or Gemma weights? Please show the cosine-similarity analysis and HMC results for at least one Qwen model at >= 8B scale.

  4. The budget accounting and practical efficiency are not convincingly evaluated. The theoretical and experimental budgets exclude partition metadata and padding, while Algorithm 2 uses padded grouped execution. The paper does not report actual memory footprint, compression time, clustering cost, or end-to-end latency for the two-level model. Figure 7(b) reports throughput only for dense and single-level decoding, and even single-level is slower than dense on the Yi models. Since the claimed benefit is deployment-oriented, the absence of two-level throughput and total memory savings is a significant omission.

  5. The proposed method is effectively an additive sum of level-wise low-rank approximations with residual repartitioning, not a tree-structured hierarchy. The paper shows that a second level helps but there is no analysis of when residual repartitioning is preferable to simply increasing K or the rank at a single level. A controlled comparison against a single-level model with the same total number of subspaces/factors would strengthen the claim that hierarchy itself is responsible for the gains rather than the extra partitioning.

References

  1. Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression, ICLR 2025 (https://openreview.net/forum?id=gp32jvUquq)

  2. Advancing SVD-based LLM Compression via Layer-Wise Error Model Search, ICML 2026 (https://icml.cc/virtual/2026/poster/64908)

  3. CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning, https://arxiv.org/abs/2509.22075

  4. COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/150918, https://arxiv.org/abs/2602.15200

  5. ROCKET: Rapid Optimization via Calibration-guided Knapsack-Enhanced Truncation for Efficient Model Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/154690, https://arxiv.org/abs/2602.11008

  6. GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/151461, https://arxiv.org/abs/2609.25963

Questions for authors

Please see weaknesses section.

Reviewer_KLLGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors aim to mitigate the error caused by low-rank factorization approaches and propose a hierarchical multi-subspace low-rank compression method, HMC, to partition weight rows into groups to preserve distinct local subspaces. A hierarchical representation is used to recursively repartition and approximate successive residuals. The experimental results show that HMC can achieve lower perplexity and higher zero-shot accuracy compared with baselines.

Critical strengths
  1. The paper identifies a meaningful limitation of global low-rank approximation, which requires different regions of a weight matrix to share the same low-dimensional representation.
  2. The paper evaluates five LLMs across several compression ratios and includes comparisons with representative global low-rank compression methods.
Critical weaknesses
  1. The core idea of partitioning matrix rows into multiple subspaces and performing separate low-rank approximations has already been explored in prior work.
  2. Although the paper proposes a hierarchical multi-level representation as its main method, Figure 7(b) evaluates inference throughput only for the single-level implementation.
Questions for authors
  1. Could the authors clarify the key methodological differences between HMC and existing multi-subspace or multilevel low-rank methods?
  2. Figure 7(b) reports inference throughput for the single-level implementation. Do the authors have corresponding measurements for the two-level HMC used in the main experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.