acceptodds
Under review as a conference paper at ICLR 2027

COACH:ASYMMETRIC ANCHOR–HELPER ROUTING IN MIXTURES OF LORA EXPERTS

Abstract

Combining Low-Rank Adaptation (LoRA) with Mixture-of-Experts (MoE) has been widely studied for parameter-efficient multi-task adaptation of large language models (LLMs). However, the existing routing methods have not fully explored dimension-wise discriminative information across expert regions or candidate utility conditioned on selected experts, potentially limiting multi-task adaptation performance. To address this limitation, this paper proposes **COACH**, an asymmetric Anchor–Helper routing approach for mixtures of LoRA experts based on **C**onditioning **o**n **A**nchors for **C**alibrating **H**elpers. Specifically, COACH comprises two complementary modules: Prototype-Guided Dimension Reweighting (PDR) exploits dimension-wise expert discriminability to stabilize base matching, while Cluster Residual Collaboration (CRC) uses an Anchor-conditioned residual-response prior to recalibrate Helper selection, addressing the lack of selected-expert conditioning in one-shot routing. Together, these two modules calibrate routing without introducing additional trainable parameters or auxiliary training objectives, while preserving the prescribed sparse activation budget. From a feature-weighted clustering perspective, the theoretical analysis derives the optimal dimension weights for PDR, proves that more discriminative dimensions receive higher weights, and applies this result to calibrate base routing. Extensive experiments across seven tasks show that COACH achieves optimal performance in both macro accuracy (equally averaged over tasks) and micro accuracy (computed over all evaluation examples) among the compared methods, exceeding the equally weighted mean performance of six external baselines by **1.87** and **2.11** percentage points, respectively. Further mechanistic experiments demonstrate that PDR improves routing separation and perturbation stability, while CRC selects more useful Helpers through Anchor-conditioned calibration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of LoRA Experts, Expert Routing, Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Multi-Task Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PSM8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes COACH, an asymmetric Anchor–Helper routing method for mixtures of LoRA experts. The method consists of two components: Prototype-Guided Dimension Reweighting (PDR), which uses expert prototypes to reweight input dimensions for Anchor selection, and Cluster Residual Collaboration (CRC), which uses Anchor-conditioned expert responses to calibrate Helper selection. Both components are designed to improve expert routing without introducing additional trainable parameters or auxiliary training objectives. The paper also provides a theoretical analysis of PDR from a prototype-separation perspective. Experiments on seven NLP tasks using Qwen2-1.5B demonstrate improvements over existing baselines, together with ablation studies and analyses of routing behavior.

Critical strengths
  • Experiments are comprehensive
  • The research question is interesting and meaningful.
Critical weaknesses
  • The main concern is the paper's presentation. I have tried my best, but I still find it hard to follow. The main text lacks lots of important formulations, explanations, and even intuition. Few technical details are explained in Sec.3. I recommend that the authors polish their writing.
  • Another concern is the algorithm design. Based on my understanding, this method requires computation for each token, layer, and expert, making it very costly. Given that, the performance improvement is marginal and there is no cost is reported in the paper. I am concerned about the practicability of this method.

I may misunderstand this paper, and I am willing to raise my score if my concerns are addressed.

Questions for authors
  • The paper lacks the necessary background and formulations. E.g., what are the EE experts in Sec.3? Are they LoRA adapters trained on specific domains, respectively?
  • As one of the core design elements, μe\mu_e is critical in the algorithm, so it should be defined in the main text instead of the appendix.
  • I am unclear about the meaning of αe(l)\alpha_e^{(l)}.
  • Given dd is a large hidden dimension, dpedp_e would be very large as well, making ρl\rho_l hard to control the magnitude.
  • Does Eq.3 mean that for each token and layer we have to compute such a routing weight? This is very costly
  • When computing μ\mu, the similarity is computed between the token representation and router weight, which I think does not make sense. Can you explain the intuition behind this?
  • I am not clear why Hi,i=0H_{i,i}=0.
  • Qwen2 is a very old model. I suggest using at least Qwen3.
Reviewer_KQtoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes COACH, an asymmetric Anchor–Helper routing method for mixtures of LoRA experts. Prototype-Guided Dimension Reweighting (PDR) calibrates input–expert matching, while Cluster Residual Collaboration (CRC) adjusts Helper selection conditioned on the chosen Anchor. Experiments on seven tasks using Qwen2-1.5B show improvements over the evaluated baselines, supported by component ablations and routing analyses.

Critical strengths
  1. The Anchor–Helper formulation addresses a meaningful limitation of one-shot Top-K routing: a candidate expert’s utility can depend on its co-activated expert. Component ablations and controlled Helper-identity interventions provide useful evidence for the proposed approach.

  2. PDR and CRC preserve the sparse activation budget without adding trainable parameters or auxiliary losses, offering a relatively lightweight way to incorporate prototype information and conditional expert selection.

Critical weaknesses
  1. CRC’s response-magnitude proxy is not sufficiently connected to expert complementarity. It measures a candidate’s output norm on the Anchor prototype without directly considering the Anchor’s output, redundancy, or joint loss reduction. The experiments support its usefulness but do not fully distinguish collaboration-aware selection from an Anchor-region-dependent preference for larger expert responses.

  2. The empirical evidence does not adequately establish comparative advantage or practical generality. Baselines differ in training-example budgets, parameter counts, adaptation placement, and optimization settings. COACH uses 25.86M trainable parameters versus mtLoRA’s 13.07M for an approximately 0.5-point aggregate accuracy gain. Evaluation is also restricted to one 1.5B backbone and multiple-choice tasks, without adequately quantifying the runtime and memory overhead of the additional routing statistics.

Questions for authors
  1. Does COACH retain its advantage over mtLoRA under matched training-example budgets, comparable parameter counts, and comparable hyperparameter-search budgets?

  2. After controlling for candidates’ overall output magnitudes, does the Anchor-conditioned CRC score still predict joint loss improvement? How does it compare with an explicit output-complementarity criterion?

  3. Do the gains transfer to a larger backbone or a generative task, and what are the training-throughput, inference-latency, and memory overheads?

Reviewer_LuqyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed COACH is an asymmetric Anchor–Helper routing method for MoE-LoRA multi-task adaptation. The authors motivate it with two shortcomings of existing routing. First, base matching does not make full use of dimension-wise differences between expert regions. Second, later expert choices are not conditioned on the expert already selected. PDR tackles the first issue by using running expert prototypes to reweight routing dimensions, which stabilizes base matching and Anchor selection. CRC tackles the second by recalibrating Helper scores with an Anchor-conditioned prior built from residual responses. Neither component adds trainable parameters or auxiliary objectives, and the sparse activation budget is preserved. The paper also provides a theoretical analysis indicating that PDR assigns larger weights to more discriminative dimensions. Across seven tasks, COACH achieves the best macro and micro accuracy among the compared methods. Mechanistic analyses further suggest that PDR improves routing separation and perturbation stability, while CRC tends to select more useful Helpers.

Critical strengths

The motivation for this paper is strong.

Neither PDR nor CRC introduces additional trainable parameters or auxiliary training objectives, and both preserve the original sparse activation budget.

The ablation study is clear. Comparing Base, PDR-only, CRC-only, and full COACH shows that each module helps on its own, and the two together work best.

Critical weaknesses

The main experiments only use Qwen2-1.5B and seven MCQA tasks, so both the evaluated models and the evaluated tasks are limited. I suggest increasing the number of experimental models, since Qwen2-1.5B alone is not sufficient to demonstrate the generalizability of some of the methods.

The paper lists asymmetric Anchor–Helper routing as a main contribution (p. 2, line 83). However, the reported comparison provides limited evidence for the performance advantage of this asymmetric design. The "fix the main expert first" strategy achieves 76.29% accuracy, while "select both experts together" achieves 76.11%, a difference of only 0.18 percentage points. Reporting results across multiple random seeds or providing further clarification would help readers assess the robustness of this improvement.

Questions for authors

The main experiments are currently conducted only on Qwen2-1.5B. Could the authors add results for at least one additional model, such as Llama 3.2 1B or Gemma 3 1B, under the same data and activation budget?

All seven benchmarks are currently multiple-choice question answering tasks. If the authors could provide a set of experiments showing that COACH generalizes to tasks beyond MCQA, this would help strengthen the paper's claim of broad effectiveness.

Regarding Weakness 2, I suggest that the authors report results from multiple random seeds to assess the robustness of the 0.18 percentage point improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.