- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes adaptive rank-allocated shared subspaces (ARAS), a post-training compression method that groups fine-grained weight blocks into learned shared subspaces independently of their spatial locations. Additionally, ARAS adaptively allocates ranks across the weight matrices under a global storage budget.
Critical strengths
- The paper's proposition, which allows spatially distant blocks to share subspaces, reduces reconstruction error compared to fixed spatial sharing under otherwise matched settings. This is an interesting finding and contribution.
- ARAS consistently outperforms the evaluated baselines across multiple model sizes, including aggressive compression settings. Particularly, it consistently outperforms Basis Sharing [1].
[1] Wang et al., Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression. ICLR 2025
Critical weaknesses
Evaluation:
- Limited rank-allocation ablation: Rank allocation strategies for LLM compression are well established now. Although Table 5 shows the benefit of the proposed allocation policy compared to random and reconstruction-based policies, the overall evaluation omits important related work, including SVD-LLMv2 [2], ACIP [3], LEMS [4], Swift-SVD [5]; in particular ZS-SVD [6], which already employs a gradient-informed greedy allocation strategy. Thus, the proposed ablation does not establish sufficient evidence that the proposed method is beneficial over existing related work. Additionally, SVD-LLM uses a uniform rank-allocation, so the comparison is a bit skewed (in Table 1), and the work should be compared to the aforementioned methods that also propose rank-allocation strategies. Moreover, important baselines discussed in the related work section, such as BLAST [7], are absent from the main evaluation.
- Missing inference/efficiency evaluation: No inference speed or throughput ablations are provided, only a small ablation on memory. Consequently, it remains unclear whether the proposed method induces an execution overhead or delivers practical inference benefits.
- Missing datasets: Common datasets from zero-shot benchmarking (e.g., PIQA, OpenBookQA , HellaSwag, ARC-Easy/ARCChallenge, etc.) are missing.
- Badly tuned baselines: The perplexity results for Basis Sharing, SVD-LLM and ASVD seem substantially worse than reported in other works [2, 4, 5, 6]. Differences in experimental settings may explain these discrepancies, but they require clarification.
Writing:
-
Repeated misused citation style throughout the paper, e.g., lines 031, 034, 037, 039, 041, 051, 075, 121, ...
-
I believe the caption in Table 1 (line 271) should be '... retention ratio' rather than '... reduction ratio'.
-
The claim of comparison against 'state-of-the-art compression methods' (line 254) needs to be tempered, since evaluated baselines are somewhat out-of-date as recent advances [2,4,5,6] are not considered.
[2] Wang et al., SVD-LLM v2: Optimizing singular value truncation for large language model compression. NAACL 2025
[3] Genzel et al., Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation. arXiv 2025
[4] Thoma et al., Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. ICML 2026
[5] Qi et al., Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. ICML 2026
[6] Abbasi et al., Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression. ICML 2026
[7] Lee et al., BLAST: Block-level adaptive structured matrices for efficient deep neural network inference. NeurIPS 2024
Questions for authors
-
Why are only the results of WikiText-2 shown in Table 1 and the full results, including PTB and C4, only in the Appendix (Table 7)?
-
Why are zero-shot datasets not benchmarked? Could you provide the zero-shot accuracies for ARAS?
-
Could you reconcile the reported baseline perplexities with those in prior work, as they seem to be insufficiently tuned? The discrepancies also appear for Llama models; differences in model families alone seem like an insufficient explanation.
-
How does the proposed rank allocation strategy compare against propositions from related work?
-
How does BLAST [8] compare to ARAS on the perplexity and zero-shot benchmarks?