acceptodds
Under review as a conference paper at ICLR 2027

Expert-Aware Router: Coupling Routers and Experts in Mixture-of-Experts

Abstract

Sparse Mixture-of-Experts (MoE) models scale Transformer capacity by activating only a small subset of experts for each token. However, standard MoE routers are parameterized independently from the experts they select, so routing scores do not directly reflect evolving expert representations. Existing expert-aware methods strengthen this coupling using expert-side activations, requiring activating many or all experts before routing, introducing substantial training overhead. We propose Expert-Aware Router (EAR), a lightweight routing mechanism that derives each expert's routing representation directly from its gate projection matrix through a shared learnable probe vector. The resulting expert-derived routing matrix is combined with the standard learned router, allowing updates to expert parameters to directly affect routing decisions while retaining flexibility in token assignment. This coupling enables the router and experts to co-adapt during training while preserving the standard linear routing formulation. Across MoE models from 1B to 16B parameters and training budgets up to 500B tokens, EAR consistently improves downstream performance, increasing average accuracy by up to 0.76 points over standard MoE. % Under Long Horizon 16B training, For the 16B model trained for 500B tokens, it reduces the maximum expert-load violation by 71.7%, increases router–expert top- overlap from 19.45% to 21.72%, and substantially reduces inactive neurons. EAR also improves token efficiency, achieving a 1.12 gain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Mixture of Experts, Router

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jtanOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Expert-Aware Router (EAR), which constructs an expert-derived routing component by projecting each expert’s gate matrix through a shared learnable vector and combining it with an independent router. The aim is to strengthen router–expert coupling while retaining standard linear routing at inference. The paper evaluates EAR at two model scales and reports improvements in average downstream accuracy, expert utilization, and training efficiency.

Critical strengths
  • The idea seems intuitive and simple. The expert-derived component can be merged into the independent router after training, avoiding additional inference-time routing computation.
  • The initial analysis of router–expert mismatch and expert utilization is interesting, and the experiments involve substantial training budgets, including a 16B model trained on 500B tokens.
  • I think the paper is written by humans, not AI.
Critical weaknesses
  • I think the authors have a clear discussion, but with the current baselines and results, it is hard to identify the contribution of the paper and its advantages. The most extensive experiment, at 16B/500B tokens, compares only against vanilla MoE and EAR without the independent router; ERC is absent, and AoE is evaluated only at the smaller scale. Comparisons with stronger balancing strategies are also important because load balance accounts for a substantial part of the claimed benefit. As presented, the results establish an improvement over the chosen baseline more clearly than an advance over existing approaches.
  • Some ablations are missing for the main components of the method. The gradient-flow and projection-source ablations remove the independent router, although the paper argues that this component is important to EAR’s effectiveness. Consequently, these experiments do not establish whether a learned probe, gate-based coupling, or routing gradients into expert weights are necessary for the gains of the actual hybrid method. Controls using an untied additional router branch or coupling to a different expert’s weights would also help distinguish meaningful router–expert coupling from a generic parameterization effect. Also, the role of /alfa should be investigated as well.
  • The paper moves between improving router–expert alignment, reducing inactive neurons, balancing expert loads, improving predictive performance, and reducing training cost. These objectives need not agree. For example, the variant without the independent router achieves higher output-norm overlap but worse downstream performance and validation loss. Similarly, freezing the probe improves language-modeling loss in the gradient ablation despite worsening balance. These results suggest trade-offs that deserve more analysis. Output norm is also only a proxy for expert usefulness, and larger activations or fewer inactive neurons do not directly demonstrate better specialization. The paper needs clearer evidence connecting its diagnostic improvements to useful model behavior.
Questions for authors
  • What justifies sharing (\alpha) across experts, and how does this compare with fixed, expert-specific, or grouped probes?
  • Can the main gradient-flow and projection ablations be repeated with the independent router retained? How much do the probe, expert-derived component, and independent router each contribute during training?
  • Does EAR improve expert usefulness or specialization under a direct measurement, such as the loss impact of expert replacement or removal, rather than activation magnitude or output-norm overlap alone?
4 comments
AuthorsOpenReview ↗

Response to Weaknesses

We thank the reviewer for the detailed feedback and helpful suggestions.

W1. Comparison with stronger baselines

We agree that stronger baselines would make our contribution clearer.

In Table 2, we compare EAR with ERC at both 1B and 16B scales, and with AoE at the 1B scale. AoE requires evaluating unselected experts, making it expensive for our highly sparse MoE configurations.

For the 16B/500B setting, we acknowledge that the current comparison is limited. We are also evaluating standard MoE with different load-balancing coefficients and bias-based balancing strategies to better understand whether EAR's improvements come mainly from better load balance.

We conduct additional 1B experiments with two random seeds seeds (25 and 26) and incorporat the architectural settings used in our 16B model. Under these matched settings, EAR achieves higher average downstream accuracy across the two seeds. We report the average results across the two seeds below.

BenchmarkMoEEAR
LAMBADA38.2939.40
HellaSwag43.0042.79
WinoGrande51.6253.83
ARC-E46.5845.88
ARC-C28.2628.93
PIQA70.0270.59
Avg.₆ ↑46.3046.90
Val loss1.8131.813

W2 & Q2. Ablations with the independent router

Our original ablations were conducted on EAR(w/o Indep.) to isolate the expert-derived routing component and better understand the mechanisms introduced beyond standard MoE.

Due to computational constraints, we trained grad ablations on EAR using a single seed. Under the same evaluation protocol, EAR, EAR with detached gate gradients, and EAR with a frozen probe achieve average normalized accuracies of 47.22%, 47.67%, and 46.66%, respectively, compared with 45.90% for MoE. These results do not establish bidirectional gradient co-adaptation as necessary for the gains. Although detaching gate gradients obtains a higher score in this single-seed comparison, we retain full EAR as our default formulation because it preserves end-to-end gradient flow through the routing branch without introducing an explicit stop-gradient operation.

W3. Different objectives and their trade-offs

As shown in Table 3, EAR (w/o Indep.) achieves higher output-norm overlap but worse downstream performance and load balance. This suggests that stronger alignment alone does not guarantee better routing. The independent router provides additional flexibility, while the expert-derived component introduces expert information into routing.

We view output-norm overlap and inactive neurons as diagnostic metrics rather than direct measures of expert usefulness. Our primary goal is to improve predictive performance and training efficiency, rather than optimize these diagnostic metrics independently.

AuthorsOpenReview ↗

Response to Questions

Q1. Why share α across experts?

We share α\alpha across experts within each MoE layer, with a separate vector per layer. This learns a common projection of expert gate matrices while preserving expert-specific routing through their distinct weights. It is also parameter-efficient, requiring only 128 parameters per layer, compared with 20,480 for expert-specific probes (160 experts).

To investigate the role of α\alpha, we compare learned and fixed uniform probes. In matched 1B-model 100B-token experiments:

ProbeVal. Loss ↓Load Vio ↓Acc_norm ↑
learning α\alpha2.690290.350247.23%
Per-expert learned α\alpha2.692090.363547.09%
Fix α\alpha2.688420.346347.03%

The shared learned probe achieves the highest downstream accuracy. However, the fixed probe achieves slightly better validation loss and load balance.

We further analyze the learned α\alpha from two additional EAR runs with seeds 25 and 26. Its layer-averaged cosine similarities with the uniform initialization are −0.549 and −0.587, respectively, indicating that α\alpha learns a substantially different projection rather than simply preserving uniform averaging.

Overall, sharing α\alpha provides a more parameter-efficient design without sacrificing downstream performance. Learning α\alpha offers a modest accuracy improvement, while expert-specific probes do not provide additional benefits in our experiments.

Q3. Direct measurement of expert usefulness

We further evaluate expert usefulness through removal and replacement experiments on the 16B/500B models.

We randomly intervene on 10% of experts across all 19 MoE layers, using three seeds and both general and specialized texts. Removal masks the affected experts and refills the top-K slots, while replacement swaps their FFNs without changing routing weights.

The following table reports the resulting increase in loss (Δ\DeltaLoss).

ModelRemoval (General)Replacement (General)Removal (Specialized)Replacement (Specialized)
MoE0.0024710.0319860.0020070.026297
EAR (w/o Indep.)0.0054200.0174390.0027920.008037
EAR0.0031520.0042170.0023780.002986

These results suggest that EAR is more robust to expert replacement while remaining sensitive to expert removal. However, lower replacement sensitivity may also reflect greater redundancy among experts, rather than stronger specialization.

These results provide direct evidence of expert usefulness and replacement robustness.

Thank you for the additional experiments and clarifications. The full-model ablations and probe comparisons are helpful and partially address my concerns. In particular, the detached-gate result indicates that bidirectional gradient co-adaptation is not necessary for the reported gains. Please reflect this distinction in the manuscript.

My main concern about practical significance remains. The two-seed results provide additional evidence of a modest accuracy improvement, but validation loss is unchanged at the reported precision. I think the authors should provide stronger baselines. Otherwise, the current improvement is not significant enough compared to the naive MoE. You can find many papers in this area that show similar gains compared to other methods or MoE, such as CompeteSMoE, auxiliary-loss-free balancing, and Union-of-Experts. You can find more if you search. They claimed the same amount of gain compared to naive MoE. I understand these experiments are computationally expensive, and hyperparameters play a role (AoE is better than the naive baseline in the original paper and Union-of-Experts, but it is worse than the naive baseline in this paper!). Still, this paper does not have enough theoretical contribution to be accepted based only on comparisons with a naive baseline.

We thank the reviewer for the follow-up comments and suggestions.

We have rerun AoE with an adjusted low-rank dimension to better accommodate our highly sparse MoE architecture (160 experts, Top-8 routing). The updated results are reported below.

BenchmarkMoEAoEERCEAR (w/o Indep.)EAR
LAMBADA41.6841.9940.8741.9842.97
HellaSwag41.8640.1642.0441.8841.59
WinoGrande51.2252.4951.8550.5952.64
ARC-E48.5739.7448.5350.0848.48
ARC-C26.6228.7626.3727.6528.50
PIQA69.7068.1269.4869.5369.21
Avg.₆ ↑46.6145.2146.5246.9547.23

UoE and CompeteSMoE have not been evaluated with as many experts as EAR. Our 1B model uses 160 experts with Top-8 routing (5% active), while our 16B model uses 256 experts with Top-8 routing (3.125% active). Besides, UoE modifies expert architectures, while CompeteSMoE requires additional expert evaluations during training. Auxiliary-loss-free balancing focuses on load balancing without explicitly coupling experts and routers. In contrast, EAR directly couples routing with expert parameters while introducing no additional inference overhead.

Reviewer_huBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Expert-Aware Router (EAR), a parameter-sharing approach that couples MoE routing with expert parameters. For each expert, a shared learnable probe projects the expert’s SwiGLU gate matrix into a routing vector. This expert-derived vector is added to the conventional independently learned router vector. During training, expert updates therefore directly affect routing scores, while routing gradients can also update expert gate parameters. After training, the two routing components can be merged into a standard linear router without additional inference computation. The paper evaluates 1B and 16B MoE models trained on 100B–500B tokens. EAR improves reported average downstream accuracy over vanilla MoE by 0.62, 0.17, and 0.76 percentage points across the three settings. Additional analyses examine expert load balance, inactive neurons, router–expert alignment, and the roles of the independent router, shared probe, and source projection.

Critical strengths
  1. The method is simple, cheap, and leaves the inference unchanged. The expert-derived router costs one matrix–vector product per expert per step, independent of the number of tokens. It shards naturally under expert parallelism with an O(Nd) all-gather. After training it folds into a single router matrix, so the inference graph is identical to a standard MoE. Unlike AoE or RoutingFree, it never runs any part of an unselected expert on a token.
  2. Evaluation extends to a 16B model trained on 500B tokens, and the reported average downstream improvements are positive in all three main settings.
Critical weaknesses
  1. Table 7 in Appendix E reports 47.65 average accuracy for the selected vanilla MoE configuration, and the text states that this configuration is adopted as the baseline. Table 2 instead reports 46.61 for the small-model MoE baseline and 47.23 for EAR. These results may use different checkpoints, training budgets, or evaluation settings, but the manuscript does not present that distinction. If directly comparable, the selected baseline would outperform EAR.
  2. The paper treats top-K overlap with output-norm rankings as a measure of routing quality. In a standard MoE, however, an expert receives no gradient from tokens the router does not send to it. Its output norm on those tokens is not trained to reflect anything, so low overlap is not by itself evidence of poor routing. The trained MoE and EAR routers are negatively rank-correlated with output norms, as in Figure 6. The most aligned variant, EAR (w/o Indep.), is the worst on validation loss, Avg.15, and MaxVio.
Questions for authors
  1. How large is the expert-derived term at the end of training, such as ‖R_EAR‖ relative to ‖W_r‖ per layer over training, the fraction of tokens whose top-K set changes when either term is dropped at inference, and what α converges to?
  2. Several 16B MoE scores fall from 100B to 500B tokens: LAMBADA 63.19 → 57.05, ARC-E 68.31 → 64.18, PIQA 75.19 → 73.72. The six-task average computed from Table 3 is 57.83, below the 58.43 at 100B. Meanwhile, HellaSwag rises about 10 points for both models. Is this the unannealed constant schedule, or were Tables 2 and 3 produced with different evaluation settings?
  3. How to comprehend Table 7’s selected baseline accuracy of 47.65 and Table 2’s MoE accuracy of 46.61? Please specify the training budget, checkpoint, benchmark aggregation, and evaluation protocol for both, and identify the directly matched comparison with EAR.
  4. How are the 1.12× token-efficiency/effective-speedup result and the 1.22× iteration-time speedup calculated?
  5. Do the benefits persist when routing gradients into the gate matrices are detached, or when the probe is frozen, while retaining the independent router? These controls would directly test the proposed co-adaptation mechanism in the best-performing configuration.
4 comments
AuthorsOpenReview ↗

Response to Weaknesses

We thank the reviewer for the detailed feedback and helpful suggestions.

W1. Performance differences between Table 2 and Table 3

The main difference between Table 2 and Table 3 is the learning rate schedule. Table 2 uses a decaying learning-rate schedule, while Table 3 uses a constant learning rate. This explains why some downstream scores differ. Our main focus is to compare MoE and EAR under the same training settings within each experiment.

W2. Output norm as a measure of routing quality

We agree that output norm is not a perfect measure of routing quality.

One motivation comes from viewing MoE as a sparse approximation of a large FFN. Under certain assumptions, selecting neurons with larger output norms can reduce the approximation error. This idea is also discussed in prior work, such as AoE, ERC, and CompeteSMoE.

However, this is only a proxy, not a guarantee of optimal routing. Our experiments also show that better alignment with output norms does not always lead to better downstream performance. In particular, EAR without the independent router achieves better alignment but worse performance.

This suggests that output-norm alignment alone is insufficient. We believe these findings also provide useful insights for future research. Nevertheless, motivated by this perspective, we developed a new routing mechanism that achieves promising empirical results.

AuthorsOpenReview ↗

Responses to Questions

Q1. Magnitude of the expert-derived router

We observe that the relative magnitude of REARR_{\mathrm{EAR}} changes throughout training. Its proportion first increases, then decreases, and slightly recovers toward the end. It peaks at 20.50% around 10B tokens, drops to 9.54% around 60B tokens, and finally reaches 11.89%.

Removing either REARR_{\mathrm{EAR}} or WrW_r leads to worse downstream performance. In particular, removing REARR_{\mathrm{EAR}} changes over 55% of Top-8 expert sets. This suggests that although its impact on downstream accuracy is relatively small, REARR_{\mathrm{EAR}} plays an important role in routing behavior and expert selection.

ConfigurationTop-8 Change RateLM Loss ↓Vio ↓Avg. acc_norm ↑
Full EAR01.967720.58946.90%
w/o REARR_{\mathrm{EAR}}55.67%1.968960.82646.61%
w/o WrW_r≈100%6.3816811.49431.79%

Since these experiments are conducted at the 1B scale, the performance differences are not very pronounced. We expect the contribution of REARR_{\mathrm{EAR}} to become clearer at larger model scales and plan to investigate this in future experiments.

To investigate the role of (\alpha), we compare learned and fixed uniform probes. In matched 1B-model 100B-token experiments:

ProbeVal. Loss ↓Load Vio ↓Acc_norm ↑
learning α\alpha2.690290.350247.23%
Per-expert learned α\alpha2.692090.363547.09%
Fix α\alpha2.688420.346347.03%

The shared learned probe achieves the highest downstream accuracy. However, the fixed probe achieves slightly better validation loss and load balance.

We further analyze the learned α\alpha from two additional EAR runs with seeds 25 and 26. Its layer-averaged cosine similarities with the uniform initialization are −0.549 and −0.587, respectively, indicating that α\alpha learns a substantially different projection rather than simply preserving uniform averaging.

Overall, sharing α\alpha provides a more parameter-efficient design without sacrificing downstream performance. Learning α\alpha offers a modest accuracy improvement, while expert-specific probes do not provide additional benefits in our experiments.

Q2. Performance differences between 100B and 500B tokens

As explained in W1, Table 2 uses learning rate decay, while Table 3 uses a constant learning rate. Both tables use the same evaluation protocol, but their results are not directly comparable due to the different training schedules. Within each experiment, MoE and EAR are trained and evaluated under identical settings.

Q3. Difference between Table 7 and Table 2

Table 7 and Table 2 use the same training budget and evaluation protocol, but evaluate different checkpoints and benchmark sets. Table 7 includes seven benchmarks, while Table 2 excludes BoolQ. When evaluated on the same six benchmarks, the selected MoE baseline from Table 7 achieves 46.6279%, closely matching the 46.61% reported in Table 2. Therefore, the apparent discrepancy mainly comes from the different benchmark aggregation rather than a substantial difference in baseline performance.

We will clarify these differences in the revised paper.

Q4. Calculation of the two speedups

1.12× token efficiency: We fit validation-loss curves using data from 300B to 500B training tokens. We then compare the number of tokens required by MoE and EAR to reach the same validation loss: Stoken=NMoE(L)NEAR(L)≈1.12×S_{\mathrm{token}}=\frac{N_{\mathrm{MoE}}(L^)}{N_{\mathrm{EAR}}(L^)}\approx 1.12\times

1.22× iteration-time speedup: We use 690 matched logged steps from 300B to 500B training tokens. We calculate the average iteration time for MoE and EAR over these matched steps, then take their ratio: Siter=T‾MoET‾EAR≈1.22×S_{\mathrm{iter}}=\frac{\overline{T}{\mathrm{MoE}}}{\overline{T}{\mathrm{EAR}}}\approx1.22\times

Q5. Additional ablations on routing gradients

Due to computational constraints, we conducted gradient-flow ablations on EAR using a single seed. Under the same evaluation protocol, EAR, EAR with detached gate gradients, and EAR with a frozen probe achieve average normalized accuracies of 47.22%, 47.67%, and 46.66%, respectively, compared with 45.90% for MoE. These results do not establish bidirectional gradient co-adaptation as necessary for the gains. Although detaching gate gradients obtains a higher score in this single-seed comparison, we retain full EAR as our default formulation because it preserves end-to-end gradient flow through the routing branch without introducing an explicit stop-gradient operation.

Thanks to the authors for the detailed rebuttal. I have carefully read the rebuttal & the other reviewers' comments. First, I noticed that some of my concerns are shared by the other reviewers.

  • Similar to my W1/Q3, Reviewer PAW8 also ask about the gap between the selected baseline in Table 7 and the MoE baseline in Table 2.
  • Similar to my W2, Reviewer jtan also note that output norm is only a proxy for expert usefulness.
  • Similar to my Q5, Reviewer jtan also ask for the gradient-flow ablations with the independent router kept in place.
  • Similar to my Q1, Reviewer jtan also ask about the role of the shared probe α compared with fixed and expert-specific probes.

The rebuttal has addressed some of my concerns, but I still has 3 concerns:

  • The new ablations do not support the proposed mechanism. Ex: detaching gate gradients scores higher than full EAR / A fixed probe gives better validation loss and load balance than the learned probe, with similar accuracy / (much more)
  • Also, the improvement is small. For example: More than half of the 0.60-point average gain comes from WinoGrande, which is close to chance at this scale.
  • Last, the baseline may be under-tuned + the 16B/500B evidence is preliminary.

Overall, I appreciate the additional experiments. However, the evidence still shows only a poor gain over a possibly under-tuned baseline + it does not support the claimed co-adaptation mechanism. Therefore, I will keep my score.

We thank the reviewer for the follow-up and would like to clarify the remaining concerns.

Concern 1 Detach Gate only blocks gradients, without changing the EAR formulation. These additional results are from a single-seed 1B experiment, where the differences are relatively small. In contrast, the EAR (w/o Indep.) ablations show clearer differences in training dynamics and load balancing, which may better reveal the role of gradient coupling.

Concern 2 To avoid relying on a single experiment, we evaluated EAR under three settings: 1B/100B, 16B/100B, and 16B/500B. EAR outperforms MoE in all three. Notably, at 16B/500B, EAR improves Avg.15 by 0.76 points, even under a constant learning rate schedule rather than the better-performing decaying schedule.

Concern 3 We selected the 1B baseline through a grid search over eight configurations to ensure a competitive comparison. We also compared auxiliary load-balancing coefficients of 0.005, 0.01, and 0.05 on a related 16B MoE backbone. At the same training budget, their training LM losses were 2.154, 2.158, and 2.207, respectively. The coefficient of 0.005, as used in our paper, achieved the lowest training LM loss while maintaining acceptable load balance.

The 16B/500B results come from a complete training run, not an early-stage estimate. Beyond accuracy, EAR also improves validation loss, load balancing, and training efficiency. We believe these results support the effectiveness of EAR.

Reviewer_PAW8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors argue that a standard MoE router is learned separately from the experts it selects. As a result, the router may pick experts that respond weakly to a token. The authors show this mismatch in four open-source MoE models: the router's top-k experts overlap with the experts that have the largest outputs on only 29% to 67% of picks. Prior methods fix this by running all experts before routing (AoE, RoutingFree), which is expensive, or by adding an extra loss (ERC).

The authors propose the Expert-Aware Router (EAR). For each expert, EAR builds a second routing vector from the expert's own gate projection weights, using one learned vector α that all experts share. EAR adds this vector to the standard router vector. Because the routing score now depends on the expert's weights, updating an expert also changes how often it is picked. At inference, the two parts merge into one standard router, so EAR adds no cost.

The authors pretrain MoE models at 1B (160 experts) and 16B (256 experts) parameters, both with top-8 routing, on 100B and 500B tokens. At 100B tokens, EAR raises average accuracy from 46.61 to 47.23 at 1B and from 58.43 to 58.60 at 16B. At 500B tokens on the 16B model, EAR raises average accuracy from 41.45 to 42.21 and lowers validation loss from 2.311 to 2.308. EAR also improves expert load balance and cuts dead neurons from 0.318% to 0.043%. The authors report that training steps run 1.22 times faster due to the better balance. In ablations, removing the standard router part hurts, and the gate projection works better than the up or down projections.

Critical strengths

The method is cheap, easy to add, and tested at a realistic scale. EAR adds one shared vector per layer, does not need to run any extra experts, and merges into a standard router at inference (Sec. 4.3). The authors test it on models with 160 and 256 small experts and top-8 routing, up to 16B parameters and 500B tokens. This setup is much closer to current MoE models than prior work on linking routers and experts.

EAR greatly improves load balance in the largest run. On the 16B model trained for 500B tokens, MaxVio drops from 3.67 to 1.04 and dead neurons drop from 0.318% to 0.043% (Table 3). The authors also report that training steps run 1.22 times faster. If this result holds against a well-tuned baseline, it is the most useful result in the paper, because uneven expert load directly slows down MoE training across GPUs.

Critical weaknesses

The accuracy and loss gains are small, come from single runs, and may be noise. EAR improves average accuracy by 0.62 points (1B), 0.17 points (16B, 100B tokens) and 0.76 points (16B, 500B tokens). The validation loss gap is only 0.003. However, the authors' own baseline grid search shows that average accuracy varies by 1.6 points across very similar settings, which is larger than any EAR gain. The 1.12 times "effective speedup" also comes from this 0.003 loss gap on one run.

Questions for authors

How much do average accuracy and validation loss vary across seeds for the standard MoE and for EAR? Can you add at least two more seeds at 1B?

Why does the selected baseline setting score 47.65 in Table 7 but 46.61 in Table 2? Are these the same model and the same benchmarks?

What load-balancing loss weight do you use, and did you tune it for the standard MoE? Does the standard MoE match EAR's load balance with a tuned weight or with bias-based balancing?

1 comment

We thank the reviewer for the constructive feedback.

Q1. Multi-seed experiments

We thank the reviewer for the suggestion. We conduct additional 1B experiments with two random seeds seeds (25 and 26) and incorporat the architectural settings used in our 16B model. Under these matched settings, EAR achieves higher average downstream accuracy across the two seeds. We report the average results across the two seeds below.

BenchmarkMoEEAR
LAMBADA38.2939.40
HellaSwag43.0042.79
WinoGrande51.6253.83
ARC-E46.5845.88
ARC-C28.2628.93
PIQA70.0270.59
Avg.₆ ↑46.3046.90
Val loss1.8131.813

Q2. Difference between Table 2 and Table 7

Table 7 and Table 2 use the same training budget and evaluation protocol, but evaluate different checkpoints and benchmark sets. Table 7 includes seven benchmarks, while Table 2 excludes BoolQ. When evaluated on the same six benchmarks, the selected MoE baseline from Table 7 achieves 46.63%, closely matching the 46.61% reported in Table 2. Therefore, the apparent discrepancy mainly comes from the different benchmark aggregation rather than a substantial difference in baseline performance.

Q3. Load-balancing coefficient and baseline tuning

We use the same load-balancing coefficient, λ=0.005\lambda=0.005, for all main experiments.

Since EAR's routing scores depend on expert parameters, the auxiliary balancing loss can also affect expert representations through the router. We therefore use a relatively small coefficient to reduce its interference with language modeling and expert learning.

We tested different load-balancing coefficients for standard MoE. The results are shown below.

CoefficientTrain LossFinal Eval LossAvg. Acc. (%)MeanMaxVioMeanMinVio
0.0052.44112.440647.280.2750-0.1896
0.012.43602.436047.810.1485-0.1521
0.052.44022.439646.670.1201-0.1326
0.12.44322.442946.500.1204-0.1122
0.52.47602.475046.280.0976-0.0863

The results show that our chosen coefficient (λ=0.005\lambda=0.005) performs reasonably close to the best setting (λ=0.01\lambda=0.01). Increasing the coefficient generally improves load balance, but a larger coefficient does not always lead to better downstream accuracy. This suggests a trade-off between load balance and model performance.

We will include these results and the bias-based balancing comparison in the revised paper once the remaining experiments are complete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.