acceptodds
Under review as a conference paper at ICLR 2027

LRCC: Generalizing Low-rank Compression with Conditional Computation

Abstract

Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers’ path choices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
conditional computation, low-rank compression, modular deep learning, large language models, dynamic inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nZ6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LRCC, a conditional computation framework for low-rank compression of pretrained LLMs. It constructs nested low-rank paths using existing decomposition and rank allocation methods, and trains lightweight block-level routers to dynamically select ranks for each token while keeping the low-rank factors frozen. Experiments on Llama and Qwen models demonstrate improved predictive performance over static low-rank compression methods under matched average active-parameter budgets.

Critical strengths
  1. The paper presents a straightforward and computationally efficient framework for introducing token-dependent rank allocation into pretrained LLMs without fine-tuning the underlying model weights.

  2. The proposed approach is compatible with existing low-rank decomposition techniques, offering flexibility in integrating conditional computation into compressed models.

  3. The experiments cover multiple model families and compression budgets, demonstrating consistent improvements over static low-rank baselines in predictive performance. The paper also evaluates actual decoding latency rather than relying solely on active-parameter counts.

Critical weaknesses
  1. Unclear practical advantages over established quantization methods. The practical motivation and competitiveness of LRCC remain questionable. As shown in Tables 2–4 (Appendix D.1), reducing the active-parameter ratio to approximately 50% leads to substantial performance degradation. For instance, the average downstream accuracy of Llama-2-7B drops from 66.6% to 37.6%. In contrast, widely adopted 8-bit quantization methods can reduce FP16/BF16 model storage by approximately 50% while typically preserving most of the original performance, with efficient inference kernels readily available. What practical advantages does LRCC offer over such mature quantization techniques? The paper does not provide a convincing justification for why dynamic low-rank compression would be preferable in realistic deployment scenarios. Comparisons under matched memory budgets and actual inference latency are necessary to establish its practical relevance.

  2. Insufficient comparisons with existing adaptive rank selection methods. Adaptive low-rank computation has already been investigated in prior work, including FlexRank [1] and prompt-aware dynamic rank selection [2]. Although these works are mentioned in the Related Work section, direct experimental comparisons are missing. In particular, [2] is closely related to LRCC, as both adapt low-rank computation according to the input. Comparisons with these methods would help establish the benefits of token-level routing over existing adaptive rank selection strategies.

  3. Insufficient validation across low-rank decomposition methods. LRCC is presented as a general framework that can be applied to different low-rank compression methods. However, the experiments use COALA's QR-based decomposition for Llama models and Swift-SVD's Gram-matrix-based decomposition for Qwen3-4B. Appendix E.1 states that COALA failed to converge on Qwen3-4B, without further investigating the reason. Since LRCC only trains lightweight routers over frozen low-rank factors, its effectiveness may depend on the underlying decomposition. The authors should evaluate LRCC with different compatible decomposition methods on the same backbone models to demonstrate its generality and isolate the benefits of dynamic routing.

References

[1] Zaccone, Riccardo et al. FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment. ICML 2026.

[2] Zhu, Hengyi et al. Different Prompts, Different Ranks: Prompt-Aware Dynamic Rank Selection for SVD-Based LLM Compression. arXiv:2605.08568, 2026.

Questions for authors

See weaknesses

5 comments

We thank the reviewer for its comments. We address the weaknesses in the following.

1. Unclear practical advantages over established quantization methods

As with standard low-rank decompositions, LRCC can also be combined with quantization. We added experiments using INT8 and NF4 quantization to Appendix E, comparing how it affects the performance of LRCC and COALA+Swift without additional training.

At nominal targets 0.9, 0.7 and 0.5 across both models, LRCC-5 retains 98.3 to 100.2% of its BF16 mean downstream accuracy with INT8 and 94.5 to 99.5% with NF4. The corresponding ranges for the static low-rank baseline, COALA+Swift, are 97.8 to 100.7% and 88.2 to 100.2%. Each proportion is relative to the same method's BF16 result at the same target cost. For example, at target 0.9 on Llama-2-7B, the accuracy of LRCC-5 changes from 62.85% in BF16 to 62.96% with INT8 and 61.90% with NF4, while Wiki2 perplexity changes from 6.17 to 6.22 and 6.47. COALA+Swift's accuracy changes from 55.25% to 54.26% and 53.15%, while perplexity changes from 7.27 to 7.51 and 8.24.

These results show that quantization is complementary to LRCC, as it is to static low-rank compression.

2. Insufficient comparisons with existing adaptive rank selection methods

We clarify that the adaptiveness of LRCC comes from activation-awareness. That is, the selected path (and therefore the corresponding number of ranks) are dynamically adapted based on the input activations to a Transformer block. FlexRank adaptiveness is related to the fact that the number of ranks depends on the layer that is decomposed, but not on the input activations. This form of adaptiveness is the same studied by Swift-SVD, which we compare against. PARSE, on the other hand, ultimately also fixes the number of ranks activated during the decoding steps, whereas LRCC shows how to use lightweight routers at block-level to retain activation awareness. Hence, the manuscript provides comparisons with these forms of adaptiveness, and LRCC provides a method whose combination with particular contributions of FlexRank and PARSE, particularly its customized kernels, could lead to interesting future works.

We have also included additional control baselines, namely Clean, Block-Swift and B-Aware, described and evaluated in Section 3.1 and Appendix D.2. All three use the same candidate allocation paths as LRCC-5, including the dense option. Clean learns activation-independent path logits, Block-Swift selects a fixed path per block by minimizing reconstruction error under a hard deployment-parameter budget, and B-Aware conditions a lightweight linear router on a learned budget representation rather than token activations. Clean and B-Aware follow the same training and evaluation protocol as LRCC. These controls do not generally recover LRCC-5's predictive performance, providing additional evidence for the relevance of token awareness beyond static allocation and lightweight adaptation alone.

3. Insufficient validation across low-rank decomposition methods

We thank the reviewer for this comment. We would like to clarify that the paper does not claim that LRCC can be applied to other low-rank decomposition approaches. Its generality across decomposition methods is not investigated in this manuscript, and we leave such experiments to future work. Nevertheless, there is a theoretical basis for arguing that LRCC could be more generally applied, since decompositions such as COALA's and Swift-SVD's are optimally solving the same optimization problem and obtaining similar solutions.

Regarding COALA's convergence failure on Qwen3-4B, we note that this is unrelated to the router training, since the failure happens during the decompositions of the layers, while solving the activation-aware SVD problem. The decomposition is a step performed independently of router training and prior to the introduction of any routing. As such, we would argue that exploring this is also not critical for supporting our main results.

Thank you for the response and additional experiments. However, my main concerns remain:

  1. Comparison with quantization: The new experiments show that quantization can be applied to LRCC, but they do not explain why LRCC would be a better choice than standard INT8 quantization, which achieves substantial memory savings with much smaller accuracy degradation.

  2. Comparison with adaptive rank methods: The additional baselines help demonstrate the benefits of token-aware routing, but they do not show whether LRCC performs better than existing input-adaptive methods such as PARSE.

  3. Impact of decomposition methods: Since LRCC uses COALA for Llama and Swift-SVD for Qwen, it remains unclear whether the reported improvements are consistent across different decomposition methods or depend on the specific decomposition used.

These concerns remain unresolved, so I maintain my original rating.

We sincerely thank the reviewer for the additional comments and continued engagement with our work. We address each point below.

1. Comparison with quantization

Regarding the comparison with quantization, we note that previous works on low-rank compression emphasize that both methods can be combined to introduce further gains, rather than arguing that low-rank compression should generally replace quantization. This is the case with previously published works such as Dobi-SVD [1], ASVD [2] and SVD-LLM [3]. At the same time, COALA [4] and Swift-SVD [5] do not report experiments comparing their methods with quantization or combining them with quantization. In line with the previous work, our results support that LRCC is orthogonal to quantization.

[1] Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives.

[2] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models.

[3] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression.

[4] COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation.

[5] Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression.

2. Comparison with adaptive rank methods

We thank the reviewer for the suggestion of additional baseline. We were not able to find PARSE checkpoints nor a reference implementation. Therefore, we reimplemented the approach with the information described in the preprint. However, to the best of our efforts, we were not able to obtain high predictive performances with our reimplementation of PARSE on a Llama-3.2-1B. This makes a direct comparison with the method unsuitable.

For reference, our reimplementation applied PARSE on top of a COALA decomposition of Llama-3.2-1B, we selected the number of ranks using the information-density approach cited by the paper, and trained routers to select rank-1 component masks at each linear projection using its reported reconstruction loss function. As for the dataset, we tried both our calibration dataset (256 WikiText-2 samples of 2048 tokens) and 5,000 C4 samples, matching their reported training-set size.

For evaluation, we considered both an activation-dependent routing, which lets the routers at the linear layers select the mask during the forward pass, and the frozen selection, which reuses each projection's subset selected for the last prompt token throughout decoding. In the table below, we show the results obtained by evaluating the approaches on 16 samples from the WikiText-2 test set, each containing 2,048 tokens. We used 128 tokens as the prompt, and evaluated perplexity on the remaining 1,920 tokens.

Retained factor ratioOnline selection PPLFrozen selection PPL
0.71,601.652,224.91
0.9482.90575.30

3. Impact of decomposition methods

The reviewer is correct in pointing out that we had not demonstrated empirically that LRCC worked across other low-rank decomposition methods. In order to improve the paper in this respect, we have now included an ablation on the low-rank decomposition method using Llama-3.2-1B. Specifically, we applied LRCC-5 on top of the decompositions of the other evaluated baselines, namely SVD-LLM and Swift-SVD (besides COALA, which is used in the main results). The results on the test sets are shown in the Table below for the 0.9 target cost. Additional target costs and dataset-specific performances are presented in the paper in the newly added Appendix D.8.

Target costMethodWikiText-2 test perplexity ↓Mean downstream accuracy (%) ↑
0.90COALA+Swift19.7745.33
0.90COALA+LRCC-511.8151.66
0.90SVD-LLM+Swift21.0543.47
0.90SVD-LLM+LRCC-511.8152.06
0.90Swift-SVD21.0541.45
0.90Swift-SVD+LRCC-515.1650.37

For all decompositions across the measured target costs, including LRCC-5 improved the results. This provides evidence that the method can be applied across different decompositions.

Reviewer_rjzSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LRCC, a framework that aims to dynamically route tokens to different ranks to overcome the limitation of static low-rank compression. This construct nested low-rank paths that share underlying factor matrices. While keeping the backbone weight frozen, it trains lightweight linear routers to allocate rank that satisfies the budget. Experimental results demonstrate that LRCC consistently outperforms static low-rank baselines under the same active-parameter budget setting.

Critical strengths

1. Mathmatically Sound Formulation

The paper intorduces an activation-aware decomposition that minimizes the reconstruction error ∣∣XbℓWbℓ−XbℓW~bℓ,r∣∣F||X_{bℓ}W_{bℓ} − X_{bℓ} \tilde{W}_{bℓ,r}||_F using right singular vectors. Therefore, the resulting nested MRL like property allows multiple dimensions to be accessed without parameter duplication. Moreover, the block-level routing and a differentiable softplus cost cap provides a mathematical sound framework.

2. Strong Empirical Gains

Without finetuning the underlying low-rank factors, LRCC consistently outperformed static compression methods under the same active parameter budget. This gain was shown in PPL and multiple commonsense reasoning benchmarks. Moreover, analysis on number of paths and per-token allocation well demonstrated the effectiveness of the proposed work, while showing interesting intuition of the trained model.

Critical weaknesses

1. Practical Deployment and System Limitations.

As mentioned in Sec 3.2, the evaluation of the latency was done using batch size of 1, leaving out practical scenarios with batch size being larger than 1. With larger batch size, the per-token selection might cause break in GEMM batching and introduce computational bottlenecks. Also, because tokens can dynamically route to almost dense-like rank, the total parameter in GPU memory during inference can be similar to the original dense model especially as the batch size grows, therefore, fails to reduce the total static memory footprint.

2. Fairness of Adaptation Budget In the experiment section, all static baselines are evaluated without any training, whereas LRCC trains its router on the calibration dataset. Even though the author claims the comparison is fair, due to the frozen backbone weights, the performance gap might narrow down, if the static baselines were also allowed to optimized using a comparably small number of parameters under the same training budget.

Questions for authors

1. Prefill and Batched Decoding Latency

How does LRCC handle token-wise divergent path selections during the prefill phase or under batched decoding with batch size > 1, and how does its wall-clock latency compare to static baselines and the dense model in these regimes?

2. Total Static Memory Footprint

What is the exact peak GPU memory footprint and total stored parameters of a deployed LRCC model compared to the original dense model and static low-rank baselines?

** 3. Parameter-Matched Adaptation for Static Baselines**

If static baselines are given the exact same training budget and calibration data to optimize a comparable number of lightweight parameters using the LLM loss, how much of LRCC's performance gain remains?

5 comments

We thank the reviewer for the comments. We address the questions in the following.

1. Prefill and Batched Decoding Latency

LRCC selects paths independently for tokens within a batch. As reported in Appendix D.5, however, the current prefill and multi-token implementation computes the first factor multiplication at the maximum stored rank, masks each token's intermediate representation according to its selected path, and applies the second factor. This is essentially a reference implementation, which does not try to take advantage of the reduced FLOPs in larger batches. Appendix A identifies optimized implementations for prefill and larger batches as future work. However, exploiting the reduced FLOPs in these regimes would likely require custom kernels.

As a reference taken from the original manuscript, on the same settings of the decoding measures, but during prefill, the latencies are:

ModelDense (ms)COALA+Swift (ms)LRCC-5 (ms)
Llama-2-7B244.82260.14294.18
Llama-3.2-1B52.7055.2258.13

We note that grouping tokens by selected path through specialized grouped kernels offers a promising way to avoid masked work.

2. Total Static Memory Footprint

At target 0.9, LRCC-5 stores 7.56B parameters, compared with 6.74B in dense Llama-2-7B. We included Table 2 in Section 3.2 of the paper, reporting storage and peak inference memory at every evaluated target. Additional details on the memory protocol were added to Appendix D.3. The Llama-2-7B results are:

TargetLRCC-5 parameters (B)COALA+Swift parameters (B)Tail Pruning 0.5% parameters (B)Tail Pruning 5% parameters (B)LRCC-5 peak GBCOALA+Swift peak GB
0.97.5616.0727.5616.92316.23113.256
0.78.2654.7947.1035.99517.64210.711
0.58.8533.5004.3973.63718.8198.143
0.39.7922.2043.5222.93920.7005.532

Paths share factors, but each factorized projection must retain the ranks needed by its highest-cost path. These factors can exceed the original dense weight count, even when rarely activated, leading to additional stored parameters and correspondingly an increase in peak memory.

These observations also served as motivation for our new experiments with Tail Pruning, which removes rarely selected high-cost tails and truncates the factors without fine-tuning. At target 0.7, a 0.5% cutoff reduces storage by 14.1%, from 8.27B to 7.10B parameters, with unchanged Wiki2 perplexity of 9.49 and a 0.12 percentage-point decrease in mean accuracy. Section 4.4 and the appendix give additional details and results, supporting a useful way to reduce retained capacity without additional training.

3. Parameter-Matched Adaptation for Static Baselines We thank the reviewer for suggesting this comparison. We added B-Aware, which retains LRCC's linear router but conditions it on a learned budget representation rather than token activations. It introduces a similar, slightly higher number of lightweight trainable parameters and follows the same training and evaluation protocol as LRCC, using the same calibration data and LLM loss, with hyperparameters selected by WikiText-2 validation performance. This control directly tests whether comparable trainable capacity and adaptation can recover LRCC's gains without token dependence.

We also added Clean and Block-Swift, built on the same candidate allocation paths produced using Swift-SVD, including the dense option. Clean learns activation-independent path logits using the same training and evaluation protocol. Block-Swift selects a fixed path per block by minimizing reconstruction error under a hard deployment-parameter budget. Together, these controls compare learned static allocation and reconstruction-based allocation over the same paths used by LRCC-5.

The table below summarizes the test results at target 0.9 for both models:

ModelBudgetMethodWiki2 PPL ↓Mean accuracy (%) ↑
Llama-2-7B0.9Clean--
Llama-2-7B0.9B-Aware27.4537.2
Llama-2-7B0.9Block-Swift39.9437.5
Llama-2-7B0.9LRCC-56.1762.6
Llama-3.2-1B0.9Clean18.2950.5
Llama-3.2-1B0.9B-Aware19.7743.1
Llama-3.2-1B0.9Block-Swift56.3340.9
Llama-3.2-1B0.9LRCC-511.8149.9

The controls do not generally recover LRCC-5's predictive performance. At target 0.9 on Llama-2-7B, B-Aware obtains Wiki2 perplexity 27.45 and 37.2% mean accuracy, compared with LRCC-5's 6.17 and 62.6%. Block-Swift reaches perplexity 39.94 and 37.5% mean accuracy. On Llama-3.2-1B, B-Aware reaches 19.77 and 43.1%, versus LRCC-5's 11.81 and 49.9%. Clean slightly exceeds LRCC-5's mean accuracy on this model, 50.5% versus 49.9%, while its perplexity remains worse at 18.29. These results support the contribution of token-dependent routing beyond lightweight adaptation alone. The control definitions and full results are reported in Section 3.1 and Appendix D.2.

We thank the authors for the detailed response and the additional experiments included in the revision. After reviewing the rebuttal and the updated manuscript, we would like to share our follow-up thoughts on the three points:

Prefill and Batched Decoding Latency (Q1): We understand that the current reference implementation relies on maximum-rank multiplication with masking for multi-token workloads, and we acknowledge that developing custom grouped kernels could help translate the reduced FLOPs into actual speedups for prefill and batch size > 1 regimes in future work.

Total Static Memory Footprint (Q2): While LRCC aims to improve inference efficiency under a given parameter/compute budget, the results in Table 2 show that the total stored parameters (7.56B to 9.79B) and peak GPU memory (16.23 GB to 20.70 GB) exceed those of the original dense Llama-2-7B model (6.74B), and increase further at lower target budgets. Even with the 0.5% Tail Pruning cutoff at target 0.7 (7.10B), the memory footprint remains larger than the dense baseline. This appears to run counter to the practical goal of model compression under constrained memory budgets.

Parameter-Matched Adaptation for Static Baselines (Q3): While we appreciate the newly added static path-selection controls (Clean, B-Aware, and Block-Swift), optimizing discrete path choices over candidate factorizations seems unstable on Llama-2-7B. For a direct and fair comparison, we still believe it is necessary to evaluate the existing static baselines (e.g., COALA+Swift) by attaching a comparable number of lightweight continuous parameters (such as low-rank adapters or scaling vectors) and optimizing them under the exact same calibration data and LLM loss budget.

Reviewer_FpuhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LRCC adds token-dependent computation to a low-rank-compressed LLM by factorizing each linear layer once with a nested activation-aware SVD, defining a few whole-model rank allocations as per-block “paths,” and training one lightweight router per Transformer block (weights frozen, 256 WikiText-2 sequences) to pick a path per token under an adaptive compression penalty, with target-cost annealing producing a family of checkpoints in one run. The contributions are the routing-over-nested-factors design, the annealing/soft-cap training scheme, and analyses showing five paths suffice, routers stay token-dependent, and deployment cost tracks the target on unseen data. On Llama-2-7B, Llama-3.2-1B, and Qwen3-4B against SVD-LLM, COALA, and Swift-SVD at matched active-parameter ratios, LRCC improves perplexity and seven-task accuracy (62.7% vs. 55.1% on Llama-2-7B at 90%), but at matched batch-size-1 decoding latency the gain holds only on Llama-3.2-1B and is mixed on Llama-2-7B.

Critical strengths
  1. It separates “fewer active parameters” from “faster inference” and tests both. Table 1 and Figure 2 match active-parameter budgets; Figure 3 matches measured decode latency with a careful protocol (compiled baselines, identical inputs, five repeats). The protocol is good enough to show LRCC is slower than COALA+Swift at equal target cost on Llama-2-7B, and the paper reports and explains it. .
  2. The method is minimal and the analysis shows the minimal thing works. Only a linear router per block is trained on 256 sequences; nestedness yields 28 budget checkpoints from one annealed run. Five paths suffice (Figure 4), and the routers find structure without a balancing loss: a dense backbone, multi-path use elsewhere, cheaper paths for non-word tokens (Figure 5). This is the evidence that token-dependent rank allocation is a usable primitive on pretrained models.
Critical weaknesses
  1. The paper does not show that token-dependent routing is what produces the gain. LRCC can leave a block dense at cost C_b and trains its routers on the LM loss; the baselines must factorize every layer and get no training. Figure 5(a) shows most Llama-2-7B blocks stay dense at 90% and many routed blocks are effectively static. The missing control is a static per-block path choice over the same nested paths with the same dense option. Without it, I cannot tell whether the 7.6 points come from routing or from not factorizing the wrong layers.
  2. The reported budget is active parameters per token, which no one deploys against. Stored parameters and memory are not reported, and keeping dense blocks next to factored ones may cost more memory than the baseline. The one deployment number given, decode latency, has LRCC slower than COALA+Swift at every target on Llama-2-7B (Table 5), and at 50% its perplexity is worse than every baseline (41.9 vs. 32.1), which the text does not mention. On the largest model, the nominal savings do not turn into memory or speed.
Questions for authors
  1. Please run the static control: for each block, pick one path from the same nested set (dense included), chosen to meet the budget with no token dependence, and report Table 1 numbers for it. Two variants would settle the attribution: the path chosen by a simple criterion (e.g., per-block reconstruction error), and the path chosen by training the same router but forcing one output per block.
  2. At the 90% target on Llama-2-7B, what fraction of blocks are deployed dense, and what is the accuracy of a static model that keeps exactly those blocks dense and uses COALA+Swift for the rest at the same active-parameter ratio?
  3. Report stored parameters and peak memory for LRCC at each target, alongside the static baselines. Does LRCC at the 90% target store more parameters than the dense Llama-2-7B?
  4. Why is LRCC’s perplexity at the 50% budget on Llama-2-7B worse than every baseline (41.9 vs. 32.1) while accuracy is slightly higher? Is this a convergence failure of the annealing at that target, and does it recur on Qwen3-4B?
  5. How many trainable parameters do the routers add, and were the baselines given any matched-budget adaptation? If SVD-LLM’s fine-tuning were restored and limited to the same number of trainable parameters or training tokens, how do the numbers change?
  6. Table 5 shows LRCC slower than COALA+Swift at every target on Llama-2-7B. What is the per-token overhead of the router and dispatch alone, and how much of the gap closes if the dense-deployed blocks are counted at their actual dense latency rather than routed?
2 comments

Question 1 and Weakness 1 - Additional Controls

We address Weakness 1 and Question 1 together, as they are closely related.

We thank the reviewer for suggesting these additional controls. Following this suggestion, we added both variants over the same five nested paths as LRCC-5, including dense. Block-Swift selects a fixed path per block by minimizing reconstruction error under a hard deployment-parameter budget. Clean learns activation-independent path logits with the same training and evaluation protocol as LRCC. We additionally introduced B-Aware, which retains the linear router but conditions it on a learned budget representation, testing whether additional router capacity can recover the gains without activation dependence. For each control, we tuned the hyperparameters and selected the best results based on WikiText-2 validation performance, as in LRCC. Representative new test results are given in the table below, while the full results can be seen in the revised manuscript:

ModelBudgetMethodWiki2 PPL ↓Mean accuracy (%) ↑
Llama-2-7B0.9Clean--
Llama-2-7B0.9B-Aware27.4537.2
Llama-2-7B0.9Block-Swift39.9437.5
Llama-2-7B0.9LRCC-56.1762.6
Llama-3.2-1B0.9Clean18.2950.5
Llama-3.2-1B0.9B-Aware19.7743.1
Llama-3.2-1B0.9Block-Swift56.3340.9
Llama-3.2-1B0.9LRCC-511.8149.9

Clean did not reach the reported Llama-2-7B budgets despite tuning and the remaining controls did not show better results for this model. On Llama-3.2-1B, the controls showed some improvements against the static baselines. The best control result was obtained at 0.9 on Llama-3.2-1B, where Clean reaches 50.5% accuracy versus LRCC-5's 49.9%. However, LRCC-5 still achieves substantially better perplexity, 11.81 versus 18.29.

Taken together, these additional experiments strengthen the evidence for the contribution of token awareness. We added the results and method descriptions to Section 3.1, and to Appendix D.2.

Question 2 - Blocks deployed dense

At the 0.9 target, 25 of 32 blocks are deployed fully dense (78.1%), leaving seven factorized blocks. This count holds for both selected LRCC-5 and LRCC-9.

To further examine the contribution of dense blocks, our new controls allow the static method to choose which blocks remain dense, rather than inheriting LRCC's choices. Block-Swift and B-Aware obtain 37.5% and 37.2% mean accuracy at the 0.9 target, compared with LRCC-9's 62.7%. This supports the benefit of token-dependent routing beyond access to dense blocks.

Interestingly, Clean improves downstream accuracy over the static baselines on Llama-3.2-1B and slightly exceeds LRCC at 0.9, while its perplexity remains notably worse. These results are reported in Section 3.1 and Appendix D.2 gives the fuller comparison.

Question 3 - Stored Parameters and Peak Memory

We thank the reviewer for pointing out the stored parameters and peak memory metrics. At target 0.9, LRCC-5 stores 7.56B parameters, compared with 6.74B in dense Llama-2-7B. We included Table 2 in the Section 3.2 of the paper, reporting storage and peak inference memory at every evaluated target. The Llama-2-7B results are:

TargetLRCC-5 parameters (B)COALA+Swift parameters (B)Tail Pruning 0.5% parameters (B)Tail Pruning 5% parameters (B)LRCC-5 peak GBCOALA+Swift peak GB
0.97.5616.0727.5616.92316.23113.256
0.78.2654.7947.1035.99517.64210.711
0.58.8533.5004.3973.63718.8198.143
0.39.7922.2043.5222.93920.7005.532

Paths share factors, but each factorized projection must retain the ranks needed by its highest-cost path. These factors can exceed the original dense weight count, even when rarely activated.

The reviewer's observation motivated our new experiments with Tail Pruning, which removes rarely selected high-cost tails and truncates the factors without fine-tuning. At target 0.7, a 0.5% cutoff reduces storage by 14.1%, from 8.27B to 7.10B parameters, with unchanged Wiki2 perplexity of 9.49 and a 0.12 percentage-point decrease in mean accuracy. The new Section 4.4 and corresponding results in the appendix give the tradeoffs. These results identify a useful way to reduce retained capacity without additional training.

Question 4 - Poor Performance at the 50% Ratio LRCC's 50% budget on Llama-2-7b was not a convergence failure. Instead, it corresponds to a suboptimal solution. At this budget, the router assigns 95.4% of decisions in factorized blocks to the cheapest path, approaching a static allocation. Together with the effective number of paths in Figure 5 and the token-type analyses, this is consistent with a suboptimal routing solution where the router showed a tendency to collapse towards a static allocation.

Question 5 - Matched-budget Adaptation

LRCC-5 adds approximately 655,000 trainable router parameters for Llama-2-7B and 164,000 for Llama-3.2-1B, close to 0.010% and 0.013% of the respective dense models.

Regarding matched-budget adaptation for the static baselines, we introduced Clean, Block-Swift and B-Aware as controls, which provide learned allocations. B-Aware, in particular, also introduces a similar number of additional trainable parameters through its learned budget representation and the linear router. We find that these additional controls provide a more direct comparison of router adaptation. As discussed in the previous responses and updated manuscript, these controls do not generally recover LRCC-5's predictive performance.

Compared to SVD-LLM finetuning, rank-8 adaptation of both low-rank factors in SVD-LLM would introduce approximately 24-28M trainable parameters for the corresponding Llama-2-7B allocations, about 37-43 times the LRCC-5 router count.

Question 6 - Router Latency

We thank the reviewer for this helpful suggestion. However, both LRCC and COALA+Swift are already benchmarked with projections materialized in dense form where this is cheaper than factorized execution, and fully dense LRCC blocks do not execute a router. We have not separately profiled the router and dispatch components alone. Such profiling would help guide future implementation improvements, however, the reported end-to-end measurements are sufficient to directly establish the quality-latency tradeoffs studied in this paper.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful comments and constructive suggestions, which helped strengthen the experimental evidence of our claims. Following this feedback, we revised the manuscript with three main additions:

  • Additional static controls: a method that selects one fixed path per block based on block reconstruction error, a router that directly learns path scores independently of token activations, and a router conditioned on learned vectors representing the current budget. All use the same candidate paths and dense option to isolate the contribution of token activations as router inputs from the effects of router training and parameter count.
  • Stored-parameter counts and peak inference-memory measurements, together with Tail Pruning, a training-free approach that reduces retained factors by removing rarely selected high-cost paths.
  • INT8 and 4-bit NF4 experiments showing that LRCC can be combined with quantization while retaining much of its predictive performance.

These additions strengthen the evaluation of LRCC's predictive benefits and make its storage and execution costs more explicit. The responses to the reviewers summarize the relevant results and indicate where the corresponding changes appear in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.