- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Expert-Aware Router (EAR), which constructs an expert-derived routing component by projecting each expert’s gate matrix through a shared learnable vector and combining it with an independent router. The aim is to strengthen router–expert coupling while retaining standard linear routing at inference. The paper evaluates EAR at two model scales and reports improvements in average downstream accuracy, expert utilization, and training efficiency.
Critical strengths
- The idea seems intuitive and simple. The expert-derived component can be merged into the independent router after training, avoiding additional inference-time routing computation.
- The initial analysis of router–expert mismatch and expert utilization is interesting, and the experiments involve substantial training budgets, including a 16B model trained on 500B tokens.
- I think the paper is written by humans, not AI.
Critical weaknesses
- I think the authors have a clear discussion, but with the current baselines and results, it is hard to identify the contribution of the paper and its advantages. The most extensive experiment, at 16B/500B tokens, compares only against vanilla MoE and EAR without the independent router; ERC is absent, and AoE is evaluated only at the smaller scale. Comparisons with stronger balancing strategies are also important because load balance accounts for a substantial part of the claimed benefit. As presented, the results establish an improvement over the chosen baseline more clearly than an advance over existing approaches.
- Some ablations are missing for the main components of the method. The gradient-flow and projection-source ablations remove the independent router, although the paper argues that this component is important to EAR’s effectiveness. Consequently, these experiments do not establish whether a learned probe, gate-based coupling, or routing gradients into expert weights are necessary for the gains of the actual hybrid method. Controls using an untied additional router branch or coupling to a different expert’s weights would also help distinguish meaningful router–expert coupling from a generic parameterization effect. Also, the role of /alfa should be investigated as well.
- The paper moves between improving router–expert alignment, reducing inactive neurons, balancing expert loads, improving predictive performance, and reducing training cost. These objectives need not agree. For example, the variant without the independent router achieves higher output-norm overlap but worse downstream performance and validation loss. Similarly, freezing the probe improves language-modeling loss in the gradient ablation despite worsening balance. These results suggest trade-offs that deserve more analysis. Output norm is also only a proxy for expert usefulness, and larger activations or fewer inactive neurons do not directly demonstrate better specialization. The paper needs clearer evidence connecting its diagnostic improvements to useful model behavior.
Questions for authors
- What justifies sharing (\alpha) across experts, and how does this compare with fixed, expert-specific, or grouped probes?
- Can the main gradient-flow and projection ablations be repeated with the independent router retained? How much do the probe, expert-derived component, and independent router each contribute during training?
- Does EAR improve expert usefulness or specialization under a direct measurement, such as the loss impact of expert replacement or removal, rather than activation magnitude or output-norm overlap alone?
4 comments
Response to Questions
Q1. Why share α across experts?
We share across experts within each MoE layer, with a separate vector per layer. This learns a common projection of expert gate matrices while preserving expert-specific routing through their distinct weights. It is also parameter-efficient, requiring only 128 parameters per layer, compared with 20,480 for expert-specific probes (160 experts).
To investigate the role of , we compare learned and fixed uniform probes. In matched 1B-model 100B-token experiments:
| Probe | Val. Loss ↓ | Load Vio ↓ | Acc_norm ↑ |
|---|---|---|---|
| learning | 2.69029 | 0.3502 | 47.23% |
| Per-expert learned | 2.69209 | 0.3635 | 47.09% |
| Fix | 2.68842 | 0.3463 | 47.03% |
The shared learned probe achieves the highest downstream accuracy. However, the fixed probe achieves slightly better validation loss and load balance.
We further analyze the learned from two additional EAR runs with seeds 25 and 26. Its layer-averaged cosine similarities with the uniform initialization are −0.549 and −0.587, respectively, indicating that learns a substantially different projection rather than simply preserving uniform averaging.
Overall, sharing provides a more parameter-efficient design without sacrificing downstream performance. Learning offers a modest accuracy improvement, while expert-specific probes do not provide additional benefits in our experiments.
Q3. Direct measurement of expert usefulness
We further evaluate expert usefulness through removal and replacement experiments on the 16B/500B models.
We randomly intervene on 10% of experts across all 19 MoE layers, using three seeds and both general and specialized texts. Removal masks the affected experts and refills the top-K slots, while replacement swaps their FFNs without changing routing weights.
The following table reports the resulting increase in loss (Loss).
| Model | Removal (General) | Replacement (General) | Removal (Specialized) | Replacement (Specialized) |
|---|---|---|---|---|
| MoE | 0.002471 | 0.031986 | 0.002007 | 0.026297 |
| EAR (w/o Indep.) | 0.005420 | 0.017439 | 0.002792 | 0.008037 |
| EAR | 0.003152 | 0.004217 | 0.002378 | 0.002986 |
These results suggest that EAR is more robust to expert replacement while remaining sensitive to expert removal. However, lower replacement sensitivity may also reflect greater redundancy among experts, rather than stronger specialization.
These results provide direct evidence of expert usefulness and replacement robustness.
Thank you for the additional experiments and clarifications. The full-model ablations and probe comparisons are helpful and partially address my concerns. In particular, the detached-gate result indicates that bidirectional gradient co-adaptation is not necessary for the reported gains. Please reflect this distinction in the manuscript.
My main concern about practical significance remains. The two-seed results provide additional evidence of a modest accuracy improvement, but validation loss is unchanged at the reported precision. I think the authors should provide stronger baselines. Otherwise, the current improvement is not significant enough compared to the naive MoE. You can find many papers in this area that show similar gains compared to other methods or MoE, such as CompeteSMoE, auxiliary-loss-free balancing, and Union-of-Experts. You can find more if you search. They claimed the same amount of gain compared to naive MoE. I understand these experiments are computationally expensive, and hyperparameters play a role (AoE is better than the naive baseline in the original paper and Union-of-Experts, but it is worse than the naive baseline in this paper!). Still, this paper does not have enough theoretical contribution to be accepted based only on comparisons with a naive baseline.
We thank the reviewer for the follow-up comments and suggestions.
We have rerun AoE with an adjusted low-rank dimension to better accommodate our highly sparse MoE architecture (160 experts, Top-8 routing). The updated results are reported below.
| Benchmark | MoE | AoE | ERC | EAR (w/o Indep.) | EAR |
|---|---|---|---|---|---|
| LAMBADA | 41.68 | 41.99 | 40.87 | 41.98 | 42.97 |
| HellaSwag | 41.86 | 40.16 | 42.04 | 41.88 | 41.59 |
| WinoGrande | 51.22 | 52.49 | 51.85 | 50.59 | 52.64 |
| ARC-E | 48.57 | 39.74 | 48.53 | 50.08 | 48.48 |
| ARC-C | 26.62 | 28.76 | 26.37 | 27.65 | 28.50 |
| PIQA | 69.70 | 68.12 | 69.48 | 69.53 | 69.21 |
| Avg.₆ ↑ | 46.61 | 45.21 | 46.52 | 46.95 | 47.23 |
UoE and CompeteSMoE have not been evaluated with as many experts as EAR. Our 1B model uses 160 experts with Top-8 routing (5% active), while our 16B model uses 256 experts with Top-8 routing (3.125% active). Besides, UoE modifies expert architectures, while CompeteSMoE requires additional expert evaluations during training. Auxiliary-loss-free balancing focuses on load balancing without explicitly coupling experts and routers. In contrast, EAR directly couples routing with expert parameters while introducing no additional inference overhead.
Response to Weaknesses
We thank the reviewer for the detailed feedback and helpful suggestions.
W1. Comparison with stronger baselines
We agree that stronger baselines would make our contribution clearer.
In Table 2, we compare EAR with ERC at both 1B and 16B scales, and with AoE at the 1B scale. AoE requires evaluating unselected experts, making it expensive for our highly sparse MoE configurations.
For the 16B/500B setting, we acknowledge that the current comparison is limited. We are also evaluating standard MoE with different load-balancing coefficients and bias-based balancing strategies to better understand whether EAR's improvements come mainly from better load balance.
We conduct additional 1B experiments with two random seeds seeds (25 and 26) and incorporat the architectural settings used in our 16B model. Under these matched settings, EAR achieves higher average downstream accuracy across the two seeds. We report the average results across the two seeds below.
W2 & Q2. Ablations with the independent router
Our original ablations were conducted on EAR(w/o Indep.) to isolate the expert-derived routing component and better understand the mechanisms introduced beyond standard MoE.
Due to computational constraints, we trained grad ablations on EAR using a single seed. Under the same evaluation protocol, EAR, EAR with detached gate gradients, and EAR with a frozen probe achieve average normalized accuracies of 47.22%, 47.67%, and 46.66%, respectively, compared with 45.90% for MoE. These results do not establish bidirectional gradient co-adaptation as necessary for the gains. Although detaching gate gradients obtains a higher score in this single-seed comparison, we retain full EAR as our default formulation because it preserves end-to-end gradient flow through the routing branch without introducing an explicit stop-gradient operation.
W3. Different objectives and their trade-offs
As shown in Table 3, EAR (w/o Indep.) achieves higher output-norm overlap but worse downstream performance and load balance. This suggests that stronger alignment alone does not guarantee better routing. The independent router provides additional flexibility, while the expert-derived component introduces expert information into routing.
We view output-norm overlap and inactive neurons as diagnostic metrics rather than direct measures of expert usefulness. Our primary goal is to improve predictive performance and training efficiency, rather than optimize these diagnostic metrics independently.