- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes COACH, an asymmetric Anchor–Helper routing method for mixtures of LoRA experts. The method consists of two components: Prototype-Guided Dimension Reweighting (PDR), which uses expert prototypes to reweight input dimensions for Anchor selection, and Cluster Residual Collaboration (CRC), which uses Anchor-conditioned expert responses to calibrate Helper selection. Both components are designed to improve expert routing without introducing additional trainable parameters or auxiliary training objectives. The paper also provides a theoretical analysis of PDR from a prototype-separation perspective. Experiments on seven NLP tasks using Qwen2-1.5B demonstrate improvements over existing baselines, together with ablation studies and analyses of routing behavior.
Critical strengths
- Experiments are comprehensive
- The research question is interesting and meaningful.
Critical weaknesses
- The main concern is the paper's presentation. I have tried my best, but I still find it hard to follow. The main text lacks lots of important formulations, explanations, and even intuition. Few technical details are explained in Sec.3. I recommend that the authors polish their writing.
- Another concern is the algorithm design. Based on my understanding, this method requires computation for each token, layer, and expert, making it very costly. Given that, the performance improvement is marginal and there is no cost is reported in the paper. I am concerned about the practicability of this method.
I may misunderstand this paper, and I am willing to raise my score if my concerns are addressed.
Questions for authors
- The paper lacks the necessary background and formulations. E.g., what are the experts in Sec.3? Are they LoRA adapters trained on specific domains, respectively?
- As one of the core design elements, is critical in the algorithm, so it should be defined in the main text instead of the appendix.
- I am unclear about the meaning of .
- Given is a large hidden dimension, would be very large as well, making hard to control the magnitude.
- Does Eq.3 mean that for each token and layer we have to compute such a routing weight? This is very costly
- When computing , the similarity is computed between the token representation and router weight, which I think does not make sense. Can you explain the intuition behind this?
- I am not clear why .
- Qwen2 is a very old model. I suggest using at least Qwen3.