Hierarchical Multi-Subspace Low-Rank Compression for Large Language Models

Low-rank model compression represents a weight matrix in a global low-rank subspace, which can limit its ability to preserve distinct local low-rank structures existing in the pretrained weight. We propose Hierarchical Multi-subspace low-rank Compression (HMC), a post-training method that partition the weight rows into different groups, where each group can separately approximate their local subspace using low-rank factorization. We further construct a hierarchical representation by repartitioning and approximating successive residuals to capture local structure beyond a single partition. To optimize the local factor matrices, HMC adaptively allocates ranks and jointly fits the local factors across different groups. Under subspace and partition assumptions, our analysis establishes when local factors reduce reconstruction error relative to a global factorization and when two levels reduce the factor cost of exact reconstruction relative to a single level. Experimental results across five LLMs, HMC generally lowers WikiText-2 and C4 perplexity and improves accuracy on six down-stream tasks over the low-rank compression baselines. Controlled ablations further support the benefits of multi-subspace partition, joint fitting, and hierarchical representation.