- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RAC-LoRA, which constructs implicit LoRA experts by independently combining input-side matrices (A_i) and output-side matrices (B_j). Instead of maintaining fixed expert pairs, the method uses Top-K routing over (N_A N_B) possible combinations. It further introduces router-conditioned input transformations using (M_{ij}=(1-\lambda)I+\lambda u_{ij}u_{ij}^{T}), and aggregates the selected experts through direct summation. Experiments on Llama-3.2-3B and Llama-3.1-8B show improvements over LoRA, LoRAMoE, HydraLoRA, and CoLA across several downstream tasks.
Critical strengths
-
The combinatorial factor design is a reasonable extension of existing MoE-LoRA architectures. Decoupling the (A) and (B) factors allows different input-side and output-side transformations to be reused across implicit experts without independently parameterizing every combination. The router-conditioned transformation also creates a direct connection between routing and expert computation. This is more interesting than simply increasing the number of LoRA experts, and Appendix C shows that RAC-LoRA uses a slightly smaller trainable parameter budget than the compared methods.
-
The experimental improvements are supported by targeted component ablations. On Llama-3.1-8B, RAC-LoRA improves MMLU accuracy from 55.98% with CoLA to 59.57%. Table 4 further shows that fixed pairing reduces accuracy from 59.57% to 57.85%, while removing router conditioning reduces it to 58.41%. These results provide some evidence that both combinatorial routing and the input transformation are useful.
Critical weaknesses
-
The aggregation ablation is significantly affected by differences in effective update magnitude. In Table 4, hard summation achieves 59.57%, while masked softmax and hard averaging only reach 51.68% and 52.11%. However, with Top-K = 6, summation can produce an update with a substantially different magnitude from a normalized mixture. This makes it difficult to attribute the large performance gap to the aggregation mechanism itself. Although Figure 3 includes scale control when varying K, an equivalent scale-controlled comparison between hard summation and softmax aggregation is missing. This is important because the paper treats direct summation as one of its central design choices.
-
The additional computational complexity is not fully justified by the performance improvements. RAC-LoRA introduces (N_A N_B) router vectors and activates six implicit experts per token under the default configuration. Appendix A discusses a linear-scaling router, but this requires restricting the available factor combinations. The theoretical analysis in Section 3.1 establishes a larger linear span, which does not directly guarantee better optimization or performance under hard Top-K routing. Meanwhile, the actual improvements are uneven: the gains on GSM8K are quite small, the 3B FineVal result is below CoLA, and the BBH improvements over CoLA are only 0.61 and 0.49 percentage points on the two backbones. Figure 5 also indicates approximately 1.91 times the wall-clock training time of standard LoRA. I am not fully convinced that the resulting accuracy-efficiency trade-off is sufficiently strong.
Post-rebuttal assessment (corrected): I retain my original rating of 2. The K-scaled softmax and random-direction controls improve the component analysis, but do not establish a sufficiently strong overall accuracy-compute trade-off. As Reviewer m61i notes, independently learned directions are comparable or slightly better (47.75 versus 47.40), leaving parameter efficiency as the demonstrated benefit of tying directions to routers. The theory still concerns a linear-span relaxation. Reviewer xe9r's concern about held-out hyperparameter selection is not explicitly resolved; the new learning-rate sweep also omits LoRAMoE and HydraLoRA. Seed variability for the new ablations is not reported, and decode latency remains 2.08 times merged LoRA. These unresolved issues outweigh the improved ablations in my recommendation.
Questions for authors
-
Could the authors compare hard summation, masked softmax, and hard averaging under matched effective LoRA update scales? For example, multiplying the softmax mixture by K or tuning the adapter scaling separately would help determine whether the large differences in Table 4 reflect aggregation quality or simply update magnitude.
-
Does router conditioning still provide a clear benefit when compared against a uniformly scaled input transformation (M=(1-\lambda)I), or against conditioning with fixed random directions? This would help isolate the contribution of learned router directions from the effect of suppressing input components orthogonal to those directions.
2 comments
Follow-up
Correction to my earlier score update: I retain my original rating of 2. The scaled-softmax and random-direction controls improve the ablations, but m61i's distinction between soundness and contribution is relevant: independently learned directions reach 47.75 versus 47.40 for router-tied directions, supporting parameter savings rather than an accuracy advantage from tying. The linear-span analysis still does not cover the trained hard-routing model. The reply to xe9r reports a learning-rate sweep for only three methods and does not specify held-out hyperparameter selection. With small BBH gains, 2.08 times merged-LoRA decode latency, and no variability reported for the new controls, the overall accuracy-compute case remains insufficient for me to change the recommendation.
We thank the reviewer for the constructive comments and insightful questions. We address the concerns regarding aggregation, computational efficiency, and router conditioning below.
W1 & Q1: Aggregation and output-scale differences
We replace the two original aggregation ablation variants with a scale-controlled Scaled Softmax variant:
$ \alpha_i = K\cdot \frac{\exp(s_i)} {\sum_{j\in\mathcal{T}_K}\exp(s_j)}. $
This ensures that the total coefficient mass equals K, matching that of hard summation.
Scaled Softmax still underperforms hard summation by 1.43 percentage points. This indicates that the advantage of hard summation cannot be fully explained by its larger output scale. Rather, uniformly summing the selected expert outputs also provides benefits over the input-dependent weighting introduced by softmax.
These additional results will be included in the revised manuscript to provide a more comprehensive analysis of the aggregation mechanism.
W2: Computational overhead and efficiency trade-off
We acknowledge that RAC-LoRA incurs additional wall-clock training time compared with standard LoRA. This overhead primarily comes from the additional router computation. Importantly, this is a general computational cost associated with router-based LoRA architectures, rather than an overhead unique to RAC-LoRA's combinatorial factorization.
More importantly, we argue that the efficiency of RAC-LoRA should not be evaluated solely in terms of training time. Its main advantage lies in adaptively combining expert parameters through input-dependent routing, allowing it to achieve competitive or even better performance with fewer trainable parameters than standard LoRA. This can also reduce peak GPU memory consumption while maintaining model performance.
To further investigate this trade-off, we conduct additional experiments comparing LoRA with different ranks and RAC-LoRA configurations. Specifically, we include a RAC-LoRA variant whose trainable parameter budget is approximately half that of LoRA (r=32).
The results show that standard RAC-LoRA achieves higher accuracy than LoRA (r=32) while using fewer trainable parameters and less peak GPU memory. More notably, even with approximately half the parameter budget of LoRA (r=32), RAC-LoRA retains competitive performance and still outperforms both LoRA configurations.
Regarding the theoretical analysis in Section 3.1, we clarify that the linear-span inclusion analysis establishes the representational advantage of allowing multiple A and B factors to form combinatorial connections. The aggregation strategy addresses a complementary aspect: it combines the selected expert outputs after the input has been dynamically projected toward their corresponding router directions. We will clarify these complementary roles in the revised manuscript.
Q2: Effectiveness of learned router-conditioned directions
To isolate the contribution of learned router directions, we conduct additional experiments comparing the original router-conditioned transformation with Uniform Scaling and Fixed Random Directions.
These findings indicate that the improvement does not simply come from suppressing input components or introducing an additional transformation. Instead, the directions learned through the router provide useful input-dependent information for conditioning expert computation. This supports the design choice of coupling expert routing with the corresponding input transformation.
We will incorporate these additional results into the revised manuscript to further validate the contribution of router-conditioned directions.