- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CAST‑LoRA, a multi‑task MoE‑LoRA framework for parameter‑efficient multi‑task adaptation. The core designs include annealed cluster‑guided soft routing, role‑separated expert bank, and a single‑query cross‑attentive task encoder. The authors conduct extensive experiments across seven benchmarks against ten baselines.
Critical strengths
- Annealed cluster‑to‑soft routing is a practically valuable design. By only leveraging semantic clusters as temporary training guidance and gradually transferring control toward input‑dependent soft routing, the method avoids requiring explicit task labels during inference. This circumvents a common practical limitation of many existing task‑conditioned MoE‑LoRA approaches.
- Overall empirical performance is competitive. On seven multi‑task benchmarks, CAST‑LoRA achieves the best average accuracy and substantially lowers gradient conflict compared with strong MoE‑LoRA baselines, demonstrating its effectiveness for mitigating multi‑task interference. Comprehensive ablation and causal intervention studies are provided to validate different components.
- The cross‑attentive task encoder brings favorable complexity reduction. Replacing full self‑attention based task encoder with a single‑query cross‑attention design reduces attention complexity from to , yielding lower latency and peak memory for the encoder module, especially for long input sequences.
Critical weaknesses
- Concerns about parameter‑efficiency and unfair comparison against PEFT baselines. The paper positions CAST‑LoRA as a parameter‑efficient fine‑tuning method. The total trainable parameters reach 60.34M (3.76% of the Qwen2‑1.5B backbone). Compared with vanilla LoRA, LoRA+, MixLoRA and LoRAMoE, which only maintain LoRA‑expert parameters without such a heavy standalone encoder, the parameter budget is significantly higher.
- Training‑side time efficiency is insufficiently reported. The paper thoroughly reports inference latency of the isolated task encoder, but largely omits training overhead for the full model. CAST‑LoRA introduces multiple computation overheads: annealing interpolation between cluster assignments and learned routing, auxiliary regularization losses for shared experts, backward pass of the heavy cross‑attentive encoder, and offline prompt clustering preprocessing.
- Unexplained mismatch: seven evaluation benchmarks while selecting 6 task‑specific experts. Empirical ablation shows that setting obtains peak average performance, while increasing to 7 task‑specific experts degrades performance. Given there are seven downstream evaluation benchmarks, it is natural to wonder why adding one extra task‑specific expert does not bring benefits. One plausible hypothesis is that some benchmarks contain highly semantically similar tasks, leading to redundant expert capacity and higher risk of expert under‑specialization. The paper does not provide further analysis on this phenomenon. I encourage the authors to compute pairwise task / benchmark semantic similarities, analyze the behavior when , and explain the performance drop.
Questions for authors
See Weakness
3 comments
Part Ⅱ
- W3. “Six task experts for seven benchmarks; why does a seventh expert not help?”
The six experts partition 1,820 training tasks, not seven evaluation datasets; adding a seventh expert produces less balanced specialization without an accuracy gain. In a matched 10,000-update comparison:
| Diagnostic | 6 task experts | 7 task experts |
|---|---|---|
| Seven-benchmark accuracy | 64.95 | 64.18 |
| Task-expert gate-mass CV (lower is more balanced) | 0.0615 | 0.2112 |
| Mean shared routing mass | 29.732% | 29.729% |
Six experts are selected on validation data and are better than seven; the seventh expert increases routing imbalance without improving accuracy. At 10,000 updates, six experts give 64.95 vs. 64.18 for seven experts. The load CV of task-expert routing increases from 0.0615 to 0.2112 (3.44×), while shared weight remains almost unchanged (29.732% vs. 29.729%). Several benchmarks are semantically close: ARC-E/ARC-C centroid similarity is 0.985, and MMLU/MMLU-Pro is 0.883. Expert count should match training-task structure, not the number of evaluation benchmark names. The seventh expert adds 7.6% parameters but lowers accuracy by 0.77.
In summary, CAST-LoRA’s principal benefit is measurable mitigation of task interference and more effective allocation of specialized/shared capacity, with competitive accuracy at a clearly reported parameter/latency cost.
References
[1] HMoRA: Making LLMs More Effective with Hierarchical Mixture of LoRA Experts. ICLR, 2025.
[2] MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts. arXiv, 2024.
[3] Hybrid Routing for a Mixture of LoRA Experts. AAAI, 2026.
[4] LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. ACL, 2024.
Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.
If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.
Thank you again for your constructive review.
Part Ⅰ
Dear Reviewer 54L6,
Thank you for recognizing the practical value of annealed routing and for raising important questions about parameter budget, end-to-end cost, and expert granularity. CAST-LoRA targets two specific limitations of multi-task adaptation: conflicting task updates and insufficiently differentiated shared versus task-specific knowledge. Its contribution is supported by 1.09% gradient conflict versus 5.07% for MixLoRA, 26.8% lower expert-update redundancy at matched capacity, and the highest seven-benchmark mean accuracy (64.95%) (Tables 1–2, Fig. 2).
Against the same-budget HMoRA-LW, CAST-LoRA is clearly better. Against lighter MixLoRA, the larger budget buys a large reduction in interference and redundancy. CAST-LoRA and HMoRA-LW [1] both use 60.34M trainable parameters, but CAST-LoRA improves Avg from 62.56 to 64.95 (+2.39). Thus, the gain is not merely from a larger budget.
Compared with MixLoRA [2], CAST-LoRA uses a larger budget, but it reduces gradient conflict from 5.07% to 1.09% and expert redundancy by 26.8%. More importantly, Flat8-TOP4 holds expert count, trainable budget, and four active branches fixed: CAST-LoRA reduces pairwise update cosine from 0.414 to 0.303 (−26.8%), and expert-output cosine from 0.467 to 0.343 (−26.5%) (Fig. 2), while improving the mean accuracy from 64.33 to 64.59. This is the intended trade-off: more capacity is used to separate task-specific and shared directions, not to simply increase the adapter size.
The encoder is substantially cheaper, and the complete model remains deployable without task labels; the end-to-end training-time trade-off is real and is now quantified. On a single RTX PRO 6000 Blackwell Max-Q (BF16), with identical profiling workloads:
CAST-LoRA is not the fastest end-to-end, but it has lower peak training memory and lower inference latency than HoTMoE while achieving better accuracy and much lower conflict. The isolated encoder latency drops by 86.9%. End-to-end, compared with HotMoE [3], CAST-LoRA uses 8.79 vs. 25.58 GiB peak training memory, 83.95 vs. 117.05 ms MC latency, and 2.143 vs. 2.815 s generation latency. Training latency is 3.400 vs. 2.880 s/update. The encoder’s benefit is largest for long prompts. The end-to-end training cost is the price of broader adaptation and routing. We will add these measurements.
MiniLM embeddings and spherical k-means over 1,820 training-task centroids are a one-time offline preprocessing stage; their complete wall-clock cost was not recorded and is not included above. No cluster labels or MiniLM inference are required at serving time. Thus, the intended trade-off is improved multi-task separation and shared-knowledge utilization at the cost of a manageable increase in training latency.