acceptodds
Under review as a conference paper at ICLR 2027

RAC-LoRA: Router-Aware Combinatorial LoRA for Adaptive Parameter-Efficient Fine-Tuning

Abstract

Low-Rank Adaptation (LoRA) provides an efficient approach to parameter-efficient fine-tuning, while recent MoE-LoRA methods introduce multiple LoRA experts and input-dependent routing to improve adaptation capacity. However, existing designs commonly impose fixed relationships between the input-side and output-side low-rank factors, and the router mainly acts as an external selector rather than directly participating in expert computation. These constraints limit the flexibility of expert composition and weaken the coupling between routing and expert specialization. We propose RAC-LoRA (Router-Aware Combinatorial LoRA), which constructs implicit experts by independently parameterizing and factors and dynamically routing among their combinations. In addition to expert selection, each router defines an input direction that conditions the corresponding expert while retaining the original representation. This enables the router to participate directly in expert computation without excessive information loss. We aggregate selected experts additively to allow complementary adaptation components to be jointly incorporated. Experiments on Llama-3.2-3B and Llama-3.1-8B show that RAC-LoRA achieves the highest average accuracy in the single-domain setting and the highest multi-task accuracy on both backbones among all compared methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Mixture-of-Experts, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TKN1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAC-LoRA, which constructs implicit LoRA experts by independently combining input-side matrices (A_i) and output-side matrices (B_j). Instead of maintaining fixed expert pairs, the method uses Top-K routing over (N_A N_B) possible combinations. It further introduces router-conditioned input transformations using (M_{ij}=(1-\lambda)I+\lambda u_{ij}u_{ij}^{T}), and aggregates the selected experts through direct summation. Experiments on Llama-3.2-3B and Llama-3.1-8B show improvements over LoRA, LoRAMoE, HydraLoRA, and CoLA across several downstream tasks.

Critical strengths
  1. The combinatorial factor design is a reasonable extension of existing MoE-LoRA architectures. Decoupling the (A) and (B) factors allows different input-side and output-side transformations to be reused across implicit experts without independently parameterizing every combination. The router-conditioned transformation also creates a direct connection between routing and expert computation. This is more interesting than simply increasing the number of LoRA experts, and Appendix C shows that RAC-LoRA uses a slightly smaller trainable parameter budget than the compared methods.

  2. The experimental improvements are supported by targeted component ablations. On Llama-3.1-8B, RAC-LoRA improves MMLU accuracy from 55.98% with CoLA to 59.57%. Table 4 further shows that fixed pairing reduces accuracy from 59.57% to 57.85%, while removing router conditioning reduces it to 58.41%. These results provide some evidence that both combinatorial routing and the input transformation are useful.

Critical weaknesses
  1. The aggregation ablation is significantly affected by differences in effective update magnitude. In Table 4, hard summation achieves 59.57%, while masked softmax and hard averaging only reach 51.68% and 52.11%. However, with Top-K = 6, summation can produce an update with a substantially different magnitude from a normalized mixture. This makes it difficult to attribute the large performance gap to the aggregation mechanism itself. Although Figure 3 includes scale control when varying K, an equivalent scale-controlled comparison between hard summation and softmax aggregation is missing. This is important because the paper treats direct summation as one of its central design choices.

  2. The additional computational complexity is not fully justified by the performance improvements. RAC-LoRA introduces (N_A N_B) router vectors and activates six implicit experts per token under the default configuration. Appendix A discusses a linear-scaling router, but this requires restricting the available factor combinations. The theoretical analysis in Section 3.1 establishes a larger linear span, which does not directly guarantee better optimization or performance under hard Top-K routing. Meanwhile, the actual improvements are uneven: the gains on GSM8K are quite small, the 3B FineVal result is below CoLA, and the BBH improvements over CoLA are only 0.61 and 0.49 percentage points on the two backbones. Figure 5 also indicates approximately 1.91 times the wall-clock training time of standard LoRA. I am not fully convinced that the resulting accuracy-efficiency trade-off is sufficiently strong.

Post-rebuttal assessment (corrected): I retain my original rating of 2. The K-scaled softmax and random-direction controls improve the component analysis, but do not establish a sufficiently strong overall accuracy-compute trade-off. As Reviewer m61i notes, independently learned directions are comparable or slightly better (47.75 versus 47.40), leaving parameter efficiency as the demonstrated benefit of tying directions to routers. The theory still concerns a linear-span relaxation. Reviewer xe9r's concern about held-out hyperparameter selection is not explicitly resolved; the new learning-rate sweep also omits LoRAMoE and HydraLoRA. Seed variability for the new ablations is not reported, and decode latency remains 2.08 times merged LoRA. These unresolved issues outweigh the improved ablations in my recommendation.

Questions for authors
  1. Could the authors compare hard summation, masked softmax, and hard averaging under matched effective LoRA update scales? For example, multiplying the softmax mixture by K or tuning the adapter scaling separately would help determine whether the large differences in Table 4 reflect aggregation quality or simply update magnitude.

  2. Does router conditioning still provide a clear benefit when compared against a uniformly scaled input transformation (M=(1-\lambda)I), or against conditioning with fixed random directions? This would help isolate the contribution of learned router directions from the effect of suppressing input components orthogonal to those directions.

2 comments

We thank the reviewer for the constructive comments and insightful questions. We address the concerns regarding aggregation, computational efficiency, and router conditioning below.

W1 & Q1: Aggregation and output-scale differences

We replace the two original aggregation ablation variants with a scale-controlled Scaled Softmax variant:

$ \alpha_i = K\cdot \frac{\exp(s_i)} {\sum_{j\in\mathcal{T}_K}\exp(s_j)}. $

This ensures that the total coefficient mass equals KK, matching that of hard summation.

Aggregation strategyMMLU Accuracy
Scaled Softmax58.14
Hard Summation (Aggregation)59.57

Scaled Softmax still underperforms hard summation by 1.43 percentage points. This indicates that the advantage of hard summation cannot be fully explained by its larger output scale. Rather, uniformly summing the selected expert outputs also provides benefits over the input-dependent weighting introduced by softmax.

These additional results will be included in the revised manuscript to provide a more comprehensive analysis of the aggregation mechanism.

W2: Computational overhead and efficiency trade-off

We acknowledge that RAC-LoRA incurs additional wall-clock training time compared with standard LoRA. This overhead primarily comes from the additional router computation. Importantly, this is a general computational cost associated with router-based LoRA architectures, rather than an overhead unique to RAC-LoRA's combinatorial factorization.

More importantly, we argue that the efficiency of RAC-LoRA should not be evaluated solely in terms of training time. Its main advantage lies in adaptively combining expert parameters through input-dependent routing, allowing it to achieve competitive or even better performance with fewer trainable parameters than standard LoRA. This can also reduce peak GPU memory consumption while maintaining model performance.

To further investigate this trade-off, we conduct additional experiments comparing LoRA with different ranks and RAC-LoRA configurations. Specifically, we include a RAC-LoRA variant whose trainable parameter budget is approximately half that of LoRA (r=32).

MethodTrainable ParametersPeak GPU Memory (Relative)Average Accuracy
LoRA (r=16)0.52×0.99×42.84
LoRA (r=32)1.03×1.00×44.61
RAC-LoRA(small)0.58×0.84×45.08
RAC-LoRA0.91×0.85×47.40

The results show that standard RAC-LoRA achieves higher accuracy than LoRA (r=32) while using fewer trainable parameters and less peak GPU memory. More notably, even with approximately half the parameter budget of LoRA (r=32), RAC-LoRA retains competitive performance and still outperforms both LoRA configurations.

Regarding the theoretical analysis in Section 3.1, we clarify that the linear-span inclusion analysis establishes the representational advantage of allowing multiple A and B factors to form combinatorial connections. The aggregation strategy addresses a complementary aspect: it combines the selected expert outputs after the input has been dynamically projected toward their corresponding router directions. We will clarify these complementary roles in the revised manuscript.

Q2: Effectiveness of learned router-conditioned directions

To isolate the contribution of learned router directions, we conduct additional experiments comparing the original router-conditioned transformation with Uniform Scaling and Fixed Random Directions.

Conditioning StrategyMMLU Accuracy
Uniform Scaling (M=(1−λ)IM=(1-\lambda)I)57.41
Fixed Random Directions58.57
Router-Conditioned Learned Directions59.57

These findings indicate that the improvement does not simply come from suppressing input components or introducing an additional transformation. Instead, the directions learned through the router provide useful input-dependent information for conditioning expert computation. This supports the design choice of coupling expert routing with the corresponding input transformation.

We will incorporate these additional results into the revised manuscript to further validate the contribution of router-conditioned directions.

Reviewer_TKN1OpenReview ↗

Follow-up

Correction to my earlier score update: I retain my original rating of 2. The scaled-softmax and random-direction controls improve the ablations, but m61i's distinction between soundness and contribution is relevant: independently learned directions reach 47.75 versus 47.40 for router-tied directions, supporting parameter savings rather than an accuracy advantage from tying. The linear-span analysis still does not cover the trained hard-routing model. The reply to xe9r reports a learning-rate sweep for only three methods and does not specify held-out hyperparameter selection. With small BBH gains, 2.08 times merged-LoRA decode latency, and no variability reported for the new controls, the overall accuracy-compute case remains insufficient for me to change the recommendation.

Reviewer_m61iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RAC-LoRA, which forms implicit experts through dynamically selected combinations of independently parameterized LoRA A/B factors. Each router vector additionally conditions its expert's input through a residual directional projection, and the selected Top-K experts are summed. Experiments on two Llama backbones across single-domain and multi-task adaptation report improved average accuracy over the selected LoRA/MoE-LoRA baselines.

Critical strengths
  1. Simple and potentially useful design. Sharing A/B factors across routed combinations while coupling the router to expert computation offers flexible adaptation with a relatively small trainable parameter budget.
  2. Promising empirical results. The method achieves competitive average performance across two model sizes and several language tasks, with ablations providing initial support for its pairing and conditioning choices.
Critical weaknesses
  1. Insufficiently established novelty and theoretical support. Cross-factor composition and low-rank mixing are closely related to existing methods such as CoLA and MoSLoRA. The linear-span inclusion argument concerns a relaxed model and does not establish advantages in expressivity, optimization, or generalization for the actual hard-routed architecture. The incremental contribution over prior work therefore remains unclear.
  2. Unresolved attribution and practical trade-offs. The experiments do not isolate the benefit of tying conditioning directions to router vectors. Moreover, Hard Sum and Hard Mean differ only by scale for fixed K, confounding the aggregation ablation. Input-dependent routing also prevents full static merging into the backbone, yet inference latency and throughput are not reported, leaving the accuracy–cost trade-off uncertain.
Questions for authors
  1. What is RAC-LoRA's distinctive contribution beyond CoLA's fully collaborative A/B factors and MoSLoRA's intermediate mixing matrix? Can the authors provide parameter-matched comparisons with the most relevant compositional/dynamic LoRA baselines?
  2. Can the authors compare router-tied conditioning against independently learned conditioning directions, and revisit Hard Sum versus Hard Mean under matched effective update scales, to isolate the proposed mechanisms from extra modulation or scaling effects?
  3. How do prefill/decode latency and throughput compare with merged standard LoRA and competing dynamic adapters? Does the accuracy gain justify this inference overhead?
2 comments

We thank the reviewer for the thoughtful comments and constructive suggestions.

W1 & Q1: Novelty and comparison with existing compositional LoRA methods

CoLA and MoSLoRA employ input-independent combination weights. Once trained, their factor compositions are fixed, and the resulting adaptation can be represented as a linear transformation of the input. In contrast, RAC-LoRA uses an input-dependent router to select different combinations of A/B factors for different inputs.

This distinction can be illustrated mathematically. For input-independent factor composition, the adaptation can be expressed in the general form:

$\Delta y_{\mathrm{static}}(x)=\left(\sum_{i,j}c_{ij}B_jA_i\right)x, $

where the combination coefficients cijc_{ij} are learned but independent of the current input.

In contrast, RAC-LoRA can be expressed schematically as:

$ \Delta y_{\mathrm{RAC}}(x)=\sum_{(i,j)\in T_K(x)}h_{ij}(x)B_jA_iM_{ij}x, $

where both the selected expert set and router-conditioned directions depend on the input. Consequently, the transformation involves input-dependent second-order interactions, rather than merely applying a fixed linear transformation to the input. To provide additional empirical evidence, we include MoSLoRA as a new baseline, while retaining the CoLA results already reported in the original manuscript.

MethodInput-dependent compositionAverage AccuracyTrainable Parameters
CoLANo43.741.03%
MoSLoRANo44.621.03%
RAC-LoRAYes47.400.91%

This provides empirical evidence that the improvement is not simply due to combining low-rank factors, but also benefits from constructing input-dependent expert combinations.

Regarding the theoretical analysis in Section 3.1, we clarify that the linear-span inclusion analysis establishes the representational advantage of allowing multiple A and B factors to form combinatorial connections. The aggregation strategy addresses a complementary aspect: it combines the selected expert outputs after the input has been dynamically projected toward their corresponding router directions. We will clarify these complementary roles in the revised manuscript.

W2 & Q2: Router-conditioned directions and aggregation mechanisms

We compare the original router-tied conditioning with an alternative in which each implicit expert is assigned an independently learned conditioning direction. The results are shown below.

Conditioning StrategyAverage AccuracyTrainable Parameters
Independently Learned Directions47.751.05%
Router-Tied Directions (RAC-LoRA)47.400.91%

The performance difference is negligible, while reusing router directions avoids introducing separate conditioning parameters.

We further investigate the aggregation mechanism. The two original variants, Masked Softmax and Hard Averaging, are replaced with a scale-controlled Scaled Softmax variant:

$ \alpha_i= K\cdot \frac{\exp(s_i)} {\sum_{j\in\mathcal{T}_K}\exp(s_j)}, $

where sis_i denotes the routing score of expert ii. This ensures that the sum of aggregation coefficients is KK, matching hard summation.

Aggregation strategyMMLU Accuracy
Scaled Softmax58.14
Hard Summation (Aggregation)59.57

This indicates that the advantage of hard summation cannot be fully attributed to its larger output scale, and that uniformly aggregating the selected experts also plays an important role.

W2 & Q3: Inference efficiency and practical trade-offs

We measure prefill latency, average decode latency per token, generation throughput, and peak GPU memory under the same inference hardware and serving configuration. The results are normalized relative to merged LoRA.

MethodPrefill LatencyDecode Latency
Merged LoRA1.001.00
CoLA1.081.22
LoRAMoE1.452.06
HydraLoRA1.482.14
RAC-LoRA1.452.08

The results show that RAC-LoRA introduces additional inference overhead compared with static, merged LoRA methods. However, its overhead remains comparable to the competing dynamic LoRA baselines in our experiments.

More importantly, inference latency alone does not fully characterize the practical efficiency of a parameter-efficient adaptation method. The comparison should also account for trainable parameter requirements and GPU memory consumption. As shown in our analysis, RAC-LoRA achieves higher accuracy with fewer trainable parameters and GPU memory than the standard LoRA and CoLA configurations.

We will incorporate these additional experiments and clarify the accuracy-efficiency trade-off in the revised manuscript.

Thank you for the additional experiments and analysis. I believe these results have addressed some of my previous concerns and provided a clearer understanding of the method's effectiveness and the contributions of its individual components. The inference efficiency experiments also help clarify the practical trade-off between performance and computational overhead.

However, I still have some reservations about the core novelty and broader significance of this work. I acknowledge the distinction the authors make between RAC-LoRA and CoLA/MoSLoRA in terms of input-dependent composition, and I appreciate how this paper integrates low-rank factor combinations, dynamic routing, and feature modulation in a coherent and effective way. Nevertheless, these ideas have already been explored to varying degrees in prior work, and the current results mainly demonstrate the effectiveness of this specific architecture. I am not yet fully convinced that it offers a sufficiently distinctive methodological contribution that can generalize to broader settings. Additionally, the independently learned conditioning directions achieve comparable or even slightly better performance, suggesting that the main benefit of reusing router directions lies in parameter efficiency. The existing theoretical analysis also does not directly cover the full dynamic routing mechanism.

Overall, I consider this a solid piece of work with practical value. The additional experiments have strengthened my confidence in the experimental findings and alleviated some of my concerns about the soundness of the method and the adequacy of its evaluation. Therefore, I will increase soundness score.

Reviewer_xe9rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studied input-dependent specialization without storing an independent low-rank factor pair for every expert. It maintains banks of input-side A and output-side B factors, assigns a router vector to each candidate pair and selects K pair products per token. Selected outputs are summed. Each expert additionally transforms its input using the router direction. The proposed optimization reuses factor computations and routing logit

Experiments compare two llama backbones with LoRA, LoRAMoE, HydraLoRA and CoLA across five single-domain settings and OpenOrca-to-BBH adaptation. RAC-LoRA has the highest reported aggregate means with fewer reported adapter parameters. Component ablations support investigating pair selection and moderate conditioning. However, the expressivity analysis concerns a fixed-factor linear-span relaxation, not the full learned hard-routing class.

Critical strengths

This paper provided a useful accuracy/storage tradeoff. Table 2 reports averages 37.77 versus CoLA's 36.55 on 3B and 47.51 versus 45.84 on 8B. MMLU-Overall improves by 2.08/3.59 points. Table 3 favors RAC on BBH by 0.61/0.49 points. Table 9 reports adapter parameters of 1.32%/0.91%, versus CoLA's 1.49%/1.03%. Figure 5 discloses training time of 1.79× LoRA, below CoLA's 2.82× but above LoRAMoE's 1.53×.

Critical weaknesses

The comparisons do not separate architecture from optimization settings. All methods share one learning rate (5×10−55\times10^{-5}), with no per-method tuning reported. Yet averaging instead of summing the same selected experts costs 7.46 pp and a masked softmax 7.89 pp (Table 4). I feel the accuracy at this learning rate is highly scale-sensitive. The normalized-mixture baselines LoRAMoE and HydraLoRA score 29.01 and 28.64 on 3B MMLU(Table 2). Also, λ\lambda, KK, NA/NBN_A/N_B and the aggregation rule were chosen on Dolly→MMLU with Llama-3.1-8B, whose 59.57 is the headline Table 2 entry. No held-out split or un-adapted backbone result is reported.

Questions for authors

Was the learning rate tuned for any method? A small sweep (e.g., {5×10−5,10−4,2×10−4}\{5\times10^{-5}, 10^{-4}, 2\times10^{-4}\}) for all five methods on Dolly→MMLU with Llama-3.1-8B, with the learning rate selected on a held-out split, would settle above mentioned weakness. If RAC-LoRA's margin over LoRA and CoLA survives, I would consider raise my score.

6 comments

We thank the reviewer for raising the concern about the potential influence of learning rates and output scaling on the reported performance. We have conducted additional experiments to investigate these issues.

1. Learning rate sensitivity.

To examine the sensitivity to learning rate, we conduct additional experiments on Dolly-to-MMLU with Llama-3.1-8B, evaluating LoRA, CoLA, and RAC-LoRA under three learning rates: 5×10−55\times10^{-5}, 1×10−41\times10^{-4}, and 2×10−42\times10^{-4}. Each configuration is independently trained with three random seeds, and we report the mean MMLU accuracy.

lr5×10−55\times10^{-5}1×10−41\times10^{-4}2×10−42\times10^{-4}
LoRA48.6454.8152.77
CoLA55.9856.6657.29
RAC-LoRA59.5760.1459.45

The results show that RAC-LoRA consistently outperforms both LoRA and CoLA across all three learning rates. Its advantage over CoLA remains above 2 percentage points, suggesting that the performance improvement is not specific to the originally adopted learning rate.

2. Aggregation and output-scale control.

We acknowledge that the original aggregation ablation was affected by differences in output scales. Specifically, conventional Softmax normalizes the selected expert weights to sum to 1, whereas our Aggregation operation directly sums the outputs of all K selected experts, resulting in a total coefficient mass of K. Therefore, the original comparison does not adequately isolate the effect of the aggregation mechanism.

To address this issue, we replace the two original aggregation ablation variants, namely Masked Softmax and Hard Averaging, with a scale-controlled Scaled Softmax variant:

$ \alpha_i = K\cdot \frac{\exp(s_i)} {\sum_{j\in\mathcal{T}_K}\exp(s_j)}, $

where TK\mathcal{T}_K denotes the selected top-K experts. This modification ensures that the total coefficient mass equals K, matching that of our Aggregation operation.

The revised comparison yields the following results:

Aggregation StrategyMMLU Accuracy
Scaled Softmax58.14
Aggregation (Sum)59.57

Although Scaled Softmax substantially improves the scale-controlled comparison, it still underperforms Aggregation by 1.43 percentage points. This suggests that the advantage of Aggregation cannot be explained solely by its larger output magnitude. Even when the total coefficient mass is matched, directly summing the selected expert outputs remains more effective than assigning them input-dependent normalized weights. These results further support our choice of Aggregation in RAC-LoRA.

We will incorporate the additional learning rate experiments and the revised aggregation ablation into the manuscript to provide a more comprehensive evaluation of the proposed method.

Thanks for provide the learning rate sweep. But I found the number doesn't match Table 2. LoRA's 55.98 is CoLA's number. And CoLA's number 55.46 match neither.

Also, CoLA improves at every step. Maybe the best learning rate is not reached. one more learning rate could close the gap further.

Another thing is that, Meta's model card put llama 3.1 8B at 66.7 on 5 shot MMLU, way above reported 8B result in the paper. Author may need explain this.

Thank you for correcting.

  1. The value of lr 5e-5 was copied incorrectly, and we have corrected it. It can be viewed in the revised manuscript.
  2. A larger learning rate does not solve the problem. For example, LoRA shows a decrease in performance at a larger learning rate, while the average of CoLA is only 54.79 at lr 5e-4. RAC-LoRA is the most robust to changes in the learning rate. Moreover, CoLA and RAC-LoRA both adopt the form of aggregation, and the performance gap cannot be simply attributed to scale.
  3. All methods are evaluated in a zero-shot setting, whereas Meta reports the Llama-3.1-8B results under a 5-shot setting. This difference in evaluation settings may contribute to the performance gap. We have clarified this distinction in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.