acceptodds
Under review as a conference paper at ICLR 2027

ARAS: ADAPTIVE RANK-ALLOCATED SHARED SUBSPACES FOR LLM COMPRESSION

Abstract

Deploying foundation models across diverse hardware platforms requires reducing model size without substantially degrading model quality. Existing low-rank compression methods often rely on global approximations, complete rows or columns, predefined spatial blocks, or fixed sharing patterns. We observe that Transformer weight matrices contain finer low-dimensional structure: spatially distant fine-grained weight blocks can often be represented by the same subspace. Based on this observation, we introduce Adaptive Rank-Allocated Shared Subspaces (), a post-training compression framework that learns which fine-grained weight blocks should share a low-rank basis, independent of their locations within the matrix. further allocates rank adaptively using a small calibration set under a global storage budget. Across OPT models from 125M to 30B parameters and Llama models from 1B to 8B parameters and compression levels, reduces the overall model size by 67% while maintaining competitive perplexity and consistently outperforms the evaluated low-rank baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Compression, Low-Rank Compression, Subspace Clustering, Adaptive Rank Allocation, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NusmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes adaptive rank-allocated shared subspaces (ARAS), a post-training compression method that groups fine-grained weight blocks into learned shared subspaces independently of their spatial locations. Additionally, ARAS adaptively allocates ranks across the weight matrices under a global storage budget.

Critical strengths
  • The paper's proposition, which allows spatially distant blocks to share subspaces, reduces reconstruction error compared to fixed spatial sharing under otherwise matched settings. This is an interesting finding and contribution.
  • ARAS consistently outperforms the evaluated baselines across multiple model sizes, including aggressive compression settings. Particularly, it consistently outperforms Basis Sharing [1].

[1] Wang et al., Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression. ICLR 2025

Critical weaknesses

Evaluation:

  • Limited rank-allocation ablation: Rank allocation strategies for LLM compression are well established now. Although Table 5 shows the benefit of the proposed allocation policy compared to random and reconstruction-based policies, the overall evaluation omits important related work, including SVD-LLMv2 [2], ACIP [3], LEMS [4], Swift-SVD [5]; in particular ZS-SVD [6], which already employs a gradient-informed greedy allocation strategy. Thus, the proposed ablation does not establish sufficient evidence that the proposed method is beneficial over existing related work. Additionally, SVD-LLM uses a uniform rank-allocation, so the comparison is a bit skewed (in Table 1), and the work should be compared to the aforementioned methods that also propose rank-allocation strategies. Moreover, important baselines discussed in the related work section, such as BLAST [7], are absent from the main evaluation.
  • Missing inference/efficiency evaluation: No inference speed or throughput ablations are provided, only a small ablation on memory. Consequently, it remains unclear whether the proposed method induces an execution overhead or delivers practical inference benefits.
  • Missing datasets: Common datasets from zero-shot benchmarking (e.g., PIQA, OpenBookQA , HellaSwag, ARC-Easy/ARCChallenge, etc.) are missing.
  • Badly tuned baselines: The perplexity results for Basis Sharing, SVD-LLM and ASVD seem substantially worse than reported in other works [2, 4, 5, 6]. Differences in experimental settings may explain these discrepancies, but they require clarification.

Writing:

  • Repeated misused citation style throughout the paper, e.g., lines 031, 034, 037, 039, 041, 051, 075, 121, ...

  • I believe the caption in Table 1 (line 271) should be '... retention ratio' rather than '... reduction ratio'.

  • The claim of comparison against 'state-of-the-art compression methods' (line 254) needs to be tempered, since evaluated baselines are somewhat out-of-date as recent advances [2,4,5,6] are not considered.


[2] Wang et al., SVD-LLM v2: Optimizing singular value truncation for large language model compression. NAACL 2025

[3] Genzel et al., Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation. arXiv 2025

[4] Thoma et al., Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. ICML 2026

[5] Qi et al., Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. ICML 2026

[6] Abbasi et al., Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. ICML 2026

[7] Lee et al., BLAST: Block-level adaptive structured matrices for efficient deep neural network inference. NeurIPS 2024

Questions for authors
  1. Why are only the results of WikiText-2 shown in Table 1 and the full results, including PTB and C4, only in the Appendix (Table 7)?

  2. Why are zero-shot datasets not benchmarked? Could you provide the zero-shot accuracies for ARAS?

  3. Could you reconcile the reported baseline perplexities with those in prior work, as they seem to be insufficiently tuned? The discrepancies also appear for Llama models; differences in model families alone seem like an insufficient explanation.

  4. How does the proposed rank allocation strategy compare against propositions from related work?

  5. How does BLAST [8] compare to ARAS on the perplexity and zero-shot benchmarks?

Reviewer_hnF7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ARAS, a post-training compression framework that learns shared low-rank bases for fine-grained weight blocks regardless of their spatial locations and adaptively allocates ranks under a global storage budget. Experiments across OPT and Llama models show that ARAS reduces overall model size by 67% while maintaining competitive perplexity and consistently outperforming the evaluated low-rank baselines.

Critical strengths
  1. The method is well motivated, combining learned, spatially unrestricted subspace sharing with sensitivity-aware rank allocation to exploit weight redundancy while preserving important information under a storage budget.
  2. Results across multiple model scales and compression levels demonstrate strong perplexity retention, while controlled ablations support the contributions of learned sharing and gradient-based rank allocation.
Critical weaknesses
  1. Evaluation focuses primarily on perplexity, with limited evidence of downstream capability preservation. Efficiency analysis concerns offline compression rather than end-to-end inference, leaving practical latency and memory benefits unverified.
  2. The spatially unrestricted sharing scheme introduces irregular basis lookups and block reconstruction, which may incur additional computational overhead during inference. Consequently, storage savings may not translate into practical speedups.
Questions for authors
  1. Several baselines exhibit severe perplexity degradation even at 60% storage retention. Could you clarify their settings and confirm whether actual storage costs are matched across methods?
  2. How is the compressed representation executed during inference: through direct computation with shared bases or by reconstructing dense weights? Could you report end-to-end latency, throughput, and peak GPU memory?
  3. Does ARAS preserve downstream performance on knowledge or reasoning benchmarks in addition to perplexity?
  4. How robust is rank allocation to the choice and size of the calibration set?
Reviewer_BAXGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies LLM compression using low-rank weight matrix factorization with shared bases block-wise within the same layer. The authors propose to a) split the weight matrix into blocks and group blocks using k-means++ and then b) for each group they find a common basis using an alternating minimization strategy. The per group rank is selected based on a loss-aware sensitivity score to choose between two options. This paper seems to combine of ideas from prior work such as Basis Sharing (https://arxiv.org/abs/2410.03765), BLAST (https://arxiv.org/html/2410.21262v1), SVD-LLM, and LEMS (https://icml.cc/virtual/2026/poster/64908).

Critical strengths
  1. This work studies an important topic with real-world applications. Training-free model compression is highly relevant since it enables wider usage of LLM models by reducing the memory requirements and computational resources.
  2. The reported results show significant gains against the selected baseline benchmarks and models.
Critical weaknesses
  1. Limited novelty: The suggested method is a combination of techniques from prior work on model compression via weight-factorization. Clear contributions are currently missing from this version of this paper.
  2. Certain choices made by the authors are lacking adequate motivation. Specifically there are several questions that are currently unanswered: a) Why k-means is the best strategy for grouping? b) Why is the normalization direction more important than the magnitude? c) Why alternating minimization is necessary? Furthermore, there is an arbitrary selection of parameters that differs across models. For example K, the number of groups, is different across models. This requires running expensive tuning for each new model, which limits the practical value of the proposed method. In addition, the proposed rank-allocation strategy is not clearly explained while clarity is also missing regarding the choice of the two rank options.
  3. In its current state, the paper is missing several highly relevant and strong baselines on weight factorization [1, 2, 3, 4, 5] and adaptive rank allocation [1, 3]. Therefore, the current comparisons are not very representative of the evolution of the field of model compression via weight factorization. While some of the above methods are quite recent to be compared against, they still need to be discussed while [1, 2] need to be included in the comparisons.
  4. Can the authors explain why the first-order approximation of the model loss is the best criterion for rank selection? Usually, advanced sensitivity methods such as [1] use second-order Taylor approximations that involve either the diagonal of the empirical Fisher information matrix (EFIM) or its KFAC approximation. In addition, given that there are few prior works that utilize the Taylor approximation of the model loss [1, 6] for weight-factorization and rank allocation, these works should be mentioned and compared against.
  5. There is inconsistent use of notations. In line 208 rhorho is defined as the retention parameter, while in line 270 is defined as the reduction ratio.
  6. WikiText is used by the authors both as calibration data and as the main perplexity metric. This raises concerns about the validity of the reported results
  7. The benchmarks used to validate the performance of the proposed method are outdated. LM-eval harness is missing as well as task-specific-benchmarks, while the reported models are relatively old. The authors should consider newer models such as Gemma, Qwen, Phi, etc and report results on benchmarks from the lm-eval harness, agent bechmarks and new/old LLM leaderbords. Perplexity alone is not indicative of the method's performance. There are many methods that can preserve perplexity but lead to huge degradation of the reasoning capabilities of the models they are applied on.

** References **

  1. Advancing SVD-based LLM Compression via Layer-Wise Error Model Search, ICML 2026 (https://icml.cc/virtual/2026/poster/64908)

  2. Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression, ICML 2026, https://icml.cc/virtual/2026/poster/61765

  3. COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/150918, https://arxiv.org/abs/2602.15200

  4. ROCKET: Rapid Optimization via Calibration-guided Knapsack-Enhanced Truncation for Efficient Model Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/154690, https://arxiv.org/abs/2602.11008

  5. GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression, NeurIPS 2026, https://neurips.cc/virtual/2026/loc/sydney/poster/151461, https://arxiv.org/abs/2609.25963

  6. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity, ICML 2026, https://icml.cc/virtual/2026/poster/61515

Questions for authors

Please refer to weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.