- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LRCC, a conditional computation framework for low-rank compression of pretrained LLMs. It constructs nested low-rank paths using existing decomposition and rank allocation methods, and trains lightweight block-level routers to dynamically select ranks for each token while keeping the low-rank factors frozen. Experiments on Llama and Qwen models demonstrate improved predictive performance over static low-rank compression methods under matched average active-parameter budgets.
Critical strengths
-
The paper presents a straightforward and computationally efficient framework for introducing token-dependent rank allocation into pretrained LLMs without fine-tuning the underlying model weights.
-
The proposed approach is compatible with existing low-rank decomposition techniques, offering flexibility in integrating conditional computation into compressed models.
-
The experiments cover multiple model families and compression budgets, demonstrating consistent improvements over static low-rank baselines in predictive performance. The paper also evaluates actual decoding latency rather than relying solely on active-parameter counts.
Critical weaknesses
-
Unclear practical advantages over established quantization methods. The practical motivation and competitiveness of LRCC remain questionable. As shown in Tables 2–4 (Appendix D.1), reducing the active-parameter ratio to approximately 50% leads to substantial performance degradation. For instance, the average downstream accuracy of Llama-2-7B drops from 66.6% to 37.6%. In contrast, widely adopted 8-bit quantization methods can reduce FP16/BF16 model storage by approximately 50% while typically preserving most of the original performance, with efficient inference kernels readily available. What practical advantages does LRCC offer over such mature quantization techniques? The paper does not provide a convincing justification for why dynamic low-rank compression would be preferable in realistic deployment scenarios. Comparisons under matched memory budgets and actual inference latency are necessary to establish its practical relevance.
-
Insufficient comparisons with existing adaptive rank selection methods. Adaptive low-rank computation has already been investigated in prior work, including FlexRank [1] and prompt-aware dynamic rank selection [2]. Although these works are mentioned in the Related Work section, direct experimental comparisons are missing. In particular, [2] is closely related to LRCC, as both adapt low-rank computation according to the input. Comparisons with these methods would help establish the benefits of token-level routing over existing adaptive rank selection strategies.
-
Insufficient validation across low-rank decomposition methods. LRCC is presented as a general framework that can be applied to different low-rank compression methods. However, the experiments use COALA's QR-based decomposition for Llama models and Swift-SVD's Gram-matrix-based decomposition for Qwen3-4B. Appendix E.1 states that COALA failed to converge on Qwen3-4B, without further investigating the reason. Since LRCC only trains lightweight routers over frozen low-rank factors, its effectiveness may depend on the underlying decomposition. The authors should evaluate LRCC with different compatible decomposition methods on the same backbone models to demonstrate its generality and isolate the benefits of dynamic routing.
References
[1] Zaccone, Riccardo et al. FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment. ICML 2026.
[2] Zhu, Hengyi et al. Different Prompts, Different Ranks: Prompt-Aware Dynamic Rank Selection for SVD-Based LLM Compression. arXiv:2605.08568, 2026.
Questions for authors
See weaknesses
5 comments
Thank you for the response and additional experiments. However, my main concerns remain:
-
Comparison with quantization: The new experiments show that quantization can be applied to LRCC, but they do not explain why LRCC would be a better choice than standard INT8 quantization, which achieves substantial memory savings with much smaller accuracy degradation.
-
Comparison with adaptive rank methods: The additional baselines help demonstrate the benefits of token-aware routing, but they do not show whether LRCC performs better than existing input-adaptive methods such as PARSE.
-
Impact of decomposition methods: Since LRCC uses COALA for Llama and Swift-SVD for Qwen, it remains unclear whether the reported improvements are consistent across different decomposition methods or depend on the specific decomposition used.
These concerns remain unresolved, so I maintain my original rating.
We sincerely thank the reviewer for the additional comments and continued engagement with our work. We address each point below.
1. Comparison with quantization
Regarding the comparison with quantization, we note that previous works on low-rank compression emphasize that both methods can be combined to introduce further gains, rather than arguing that low-rank compression should generally replace quantization. This is the case with previously published works such as Dobi-SVD [1], ASVD [2] and SVD-LLM [3]. At the same time, COALA [4] and Swift-SVD [5] do not report experiments comparing their methods with quantization or combining them with quantization. In line with the previous work, our results support that LRCC is orthogonal to quantization.
[1] Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives.
[2] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models.
[3] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression.
[4] COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation.
[5] Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression.
2. Comparison with adaptive rank methods
We thank the reviewer for the suggestion of additional baseline. We were not able to find PARSE checkpoints nor a reference implementation. Therefore, we reimplemented the approach with the information described in the preprint. However, to the best of our efforts, we were not able to obtain high predictive performances with our reimplementation of PARSE on a Llama-3.2-1B. This makes a direct comparison with the method unsuitable.
For reference, our reimplementation applied PARSE on top of a COALA decomposition of Llama-3.2-1B, we selected the number of ranks using the information-density approach cited by the paper, and trained routers to select rank-1 component masks at each linear projection using its reported reconstruction loss function. As for the dataset, we tried both our calibration dataset (256 WikiText-2 samples of 2048 tokens) and 5,000 C4 samples, matching their reported training-set size.
For evaluation, we considered both an activation-dependent routing, which lets the routers at the linear layers select the mask during the forward pass, and the frozen selection, which reuses each projection's subset selected for the last prompt token throughout decoding. In the table below, we show the results obtained by evaluating the approaches on 16 samples from the WikiText-2 test set, each containing 2,048 tokens. We used 128 tokens as the prompt, and evaluated perplexity on the remaining 1,920 tokens.
| Retained factor ratio | Online selection PPL | Frozen selection PPL |
|---|---|---|
| 0.7 | 1,601.65 | 2,224.91 |
| 0.9 | 482.90 | 575.30 |
3. Impact of decomposition methods
The reviewer is correct in pointing out that we had not demonstrated empirically that LRCC worked across other low-rank decomposition methods. In order to improve the paper in this respect, we have now included an ablation on the low-rank decomposition method using Llama-3.2-1B. Specifically, we applied LRCC-5 on top of the decompositions of the other evaluated baselines, namely SVD-LLM and Swift-SVD (besides COALA, which is used in the main results). The results on the test sets are shown in the Table below for the 0.9 target cost. Additional target costs and dataset-specific performances are presented in the paper in the newly added Appendix D.8.
| Target cost | Method | WikiText-2 test perplexity ↓ | Mean downstream accuracy (%) ↑ |
|---|---|---|---|
| 0.90 | COALA+Swift | 19.77 | 45.33 |
| 0.90 | COALA+LRCC-5 | 11.81 | 51.66 |
| 0.90 | SVD-LLM+Swift | 21.05 | 43.47 |
| 0.90 | SVD-LLM+LRCC-5 | 11.81 | 52.06 |
| 0.90 | Swift-SVD | 21.05 | 41.45 |
| 0.90 | Swift-SVD+LRCC-5 | 15.16 | 50.37 |
For all decompositions across the measured target costs, including LRCC-5 improved the results. This provides evidence that the method can be applied across different decompositions.
We thank the reviewer for its comments. We address the weaknesses in the following.
1. Unclear practical advantages over established quantization methods
As with standard low-rank decompositions, LRCC can also be combined with quantization. We added experiments using INT8 and NF4 quantization to Appendix E, comparing how it affects the performance of LRCC and COALA+Swift without additional training.
At nominal targets 0.9, 0.7 and 0.5 across both models, LRCC-5 retains 98.3 to 100.2% of its BF16 mean downstream accuracy with INT8 and 94.5 to 99.5% with NF4. The corresponding ranges for the static low-rank baseline, COALA+Swift, are 97.8 to 100.7% and 88.2 to 100.2%. Each proportion is relative to the same method's BF16 result at the same target cost. For example, at target 0.9 on Llama-2-7B, the accuracy of LRCC-5 changes from 62.85% in BF16 to 62.96% with INT8 and 61.90% with NF4, while Wiki2 perplexity changes from 6.17 to 6.22 and 6.47. COALA+Swift's accuracy changes from 55.25% to 54.26% and 53.15%, while perplexity changes from 7.27 to 7.51 and 8.24.
These results show that quantization is complementary to LRCC, as it is to static low-rank compression.
2. Insufficient comparisons with existing adaptive rank selection methods
We clarify that the adaptiveness of LRCC comes from activation-awareness. That is, the selected path (and therefore the corresponding number of ranks) are dynamically adapted based on the input activations to a Transformer block. FlexRank adaptiveness is related to the fact that the number of ranks depends on the layer that is decomposed, but not on the input activations. This form of adaptiveness is the same studied by Swift-SVD, which we compare against. PARSE, on the other hand, ultimately also fixes the number of ranks activated during the decoding steps, whereas LRCC shows how to use lightweight routers at block-level to retain activation awareness. Hence, the manuscript provides comparisons with these forms of adaptiveness, and LRCC provides a method whose combination with particular contributions of FlexRank and PARSE, particularly its customized kernels, could lead to interesting future works.
We have also included additional control baselines, namely Clean, Block-Swift and B-Aware, described and evaluated in Section 3.1 and Appendix D.2. All three use the same candidate allocation paths as LRCC-5, including the dense option. Clean learns activation-independent path logits, Block-Swift selects a fixed path per block by minimizing reconstruction error under a hard deployment-parameter budget, and B-Aware conditions a lightweight linear router on a learned budget representation rather than token activations. Clean and B-Aware follow the same training and evaluation protocol as LRCC. These controls do not generally recover LRCC-5's predictive performance, providing additional evidence for the relevance of token awareness beyond static allocation and lightweight adaptation alone.
3. Insufficient validation across low-rank decomposition methods
We thank the reviewer for this comment. We would like to clarify that the paper does not claim that LRCC can be applied to other low-rank decomposition approaches. Its generality across decomposition methods is not investigated in this manuscript, and we leave such experiments to future work. Nevertheless, there is a theoretical basis for arguing that LRCC could be more generally applied, since decompositions such as COALA's and Swift-SVD's are optimally solving the same optimization problem and obtaining similar solutions.
Regarding COALA's convergence failure on Qwen3-4B, we note that this is unrelated to the router training, since the failure happens during the decompositions of the layers, while solving the activation-aware SVD problem. The decomposition is a step performed independently of router training and prior to the introduction of any routing. As such, we would argue that exploring this is also not critical for supporting our main results.