acceptodds
Under review as a conference paper at ICLR 2027

HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

Abstract

Mixture-of-Experts (MoE) architectures have become a standard way to scale model capacity while keeping computation sparse, yet routing remains a key determinant of MoE quality and training behavior. Prior empirical studies suggest that MoE routing reflects input semantics and upstream computation across depth, but standard routers do not explicitly use the routing distributions produced by preceding layers. We propose HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers. The key idea is simple: HERO-MoE preserves the original token-conditioned routing branch and adds a residual historical routing contribution before the standard softmax and top- dispatch. To stabilize this historical signal, HERO-MoE introduces a scale-preserving fusion mechanism that matches the magnitude of historical routing memory to the current hidden representation and accounts for the number of visible historical layers, without introducing an auxiliary routing loss or a fusion-specific tuning parameter. By reusing routing distributions already computed by preceding MoE layers, HERO-MoE improves training-loss reduction with modest end-to-end overhead. The resulting router remains compatible with standard sparse dispatch, including top- and group-limited routing, and can be inserted into existing MoE backbones with minimal architectural changes. Experiments on an approximately 8B-parameter MoE model with 0.5B active parameters, trained from scratch on 100B tokens, show that HERO-MoE reduces the final loss from 1.6393 to 1.6184, while peak memory and FLOPs increase by only 0.44% and 0.64%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MOE, Pre-training, Historical Expert Routing

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_fHQQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HERO-MoE augments the current-token router with detached, dense routing distributions from preceding MoE layers within the same pipeline-parallel stage. It projects this history into residual routing logits and uses RMS scaling and depth-aware initialization to control their magnitude while preserving standard sparse dispatch. On an approximately 8B-total/0.5B-active-parameter model trained on 100B tokens, the method reduces training loss from 1.6393 to 1.6184 and improves the average score across 13 benchmarks by 0.85 points, with a 3.85% increase in training time. The paper also presents history-window and scaling ablations, routing analyses, and supplementary Qwen3 results.

Critical strengths
  1. HERO-MoE preserves the current-token routing branch and standard sparse dispatch without introducing an additional auxiliary loss. Restricting history to the same pipeline stage avoids additional cross-stage communication.
  2. The 100B-token experiment shows lower training loss and improvements on 9 of 13 downstream tasks. The ablations and supplementary Qwen3 results provide supporting evidence, while the reported 3.85% training-time overhead suggests a potentially useful quality-cost tradeoff.
Critical weaknesses
  1. RMoE also uses cross-layer routing information, but is not evaluated. Without this comparison, the practical advantage of HERO-MoE over existing approaches remains unclear.
  2. The ablations do not separate the benefit of token-specific history from increased router capacity, or establish whether depth compensation improves upon simple RMS alignment.
  3. The average downstream improvement is 0.85 points, with four tasks declining. Without repeated-run results or uncertainty estimates, it is difficult to assess the reliability of these gains.
Questions for authors
  1. What evaluation harness, prompt templates, few-shot settings, and decoding configurations were used for Table 1? What task and metric does “PIPA” denote?
  2. What load-balancing settings were used for the baseline and HERO-MoE, and how was the RCP regularization coefficient selected?
  3. Which Qwen3 model and initialization were used in Appendix C, and what was the training-token budget?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fHQQ

We thank the reviewer for the detailed and constructive questions.

W1 — Comparison with RMoE

  1. The RMoE paper itself states that it is "orthogonal to most existing router designs." HERO-MoE modifies only the router logits and can be stacked directly on top of RMoE, so the two methods are complementary and can be combined.
  2. RMoE propagates an implicit GRU state learned from hidden states, whereas HERO-MoE propagates explicit routing distributions under stop-gradient. The two address different research questions: HERO-MoE asks whether routing history itself is useful, while RMoE asks whether a cross-layer recurrent representation is useful.
  3. HERO-MoE is a strict extension of the standard router. It exactly recovers the baseline when W_R = 0 (Prop. 1), introduces no new module, and leaves dispatch unchanged. RMoE adds a projector and a GRU and replaces the router input.
  4. Fig. 2 already compares HERO-MoE with RCP, a cross-layer routing method, under identical settings. RCP reduces the loss by 0.0193 and HERO-MoE by 0.0209. HERO-MoE's advantage also grows faster in late training, and it requires no hyperparameter tuning.
  5. We will implement RMoE in our setting and include the comparison in the revised version.

W2 — History vs. capacity, and depth compensation

  1. HERO-MoE adds only 3.44M parameters. This is comparable in size to the router matrices, but it is only 0.04% of the total model parameters (8B).
  2. We randomly shuffled the routing history across tokens before feeding it to the router. This keeps the parameters and architecture identical but removes the token-specific history. The gain vanishes completely: the final loss is 1.6401, essentially the same as the baseline (1.6393). This shows that the gain comes from historical routing information, not from increased router capacity.
  3. In Fig. 2, using only the most recent 5 or 10 layers already achieves most of the gain. Extending the window to all visible layers adds router capacity but brings no further improvement.
  4. We agree that ablating ρ is important and added the following experiment. We denote the two factors of Eq. (10) as ρ₁ (RMS alignment) and ρ₂ (depth compensation).
Variantρ₁ (RMS alignment)ρ₂ (depth compensation)Final loss
No scaling✗✗1.6264
ρ₂ only✗✓1.6221
ρ₁ only✓✗1.6219
ρ₁·ρ₂ (HERO-MoE)✓✓1.6184

Used alone, the two factors perform similarly. Combining them gives the best result: adding ρ₂ on top of RMS alignment reduces the loss by a further 0.0035, which confirms that depth compensation improves upon RMS alignment alone. The loss curves also show that ρ₁ drives the sustained late-stage gain, whereas ρ₂ mainly improves early training: over the last 15,000 steps, the gain of ρ₁-only grows by about 0.003 more than that of ρ₂-only.

W3 — Reliability of downstream gains

  1. A loss reduction of ~0.02 at 100B training tokens is substantial at this scale, and the downstream gains are considerable for this training scale.
  2. The non-monotonic changes arise because different benchmarks emphasize different abilities. With the same training data and length, scores on knowledge-oriented benchmarks (e.g., MMLU, CMMLU) stay close between the two models, which dilutes the average. The gains concentrate on reasoning, math, and code (HumanEval +3.66, ARC-C +2.31, MATH +1.46, HellaSwag +1.39), indicating that HERO-MoE improves the model's reasoning ability.
  3. lm-evaluation-harness uses deterministic decoding, so repeated evaluations give identical scores. Instead, we report the standard errors from lm-evaluation-harness and will add them for all tasks in the revised version (e.g., GSM8K: HERO-MoE 12.21 ± 0.90, baseline 10.99 ± 0.86).
  4. All experiments in the paper use seed 1234. Repeating the main experiment with three additional seeds (2025, 2026, 2027), the final losses over 4 seeds are 1.6407 ± 0.0028 (baseline) and 1.6193 ± 0.0023 (HERO-MoE). The gap (0.0214) is about 8× the seed-to-seed standard deviation, so the gain is robust.

Q1 — Evaluation details

  1. All results in Table 1 use the default evaluation settings of lm-evaluation-harness, including prompt templates, few-shot settings, and decoding configurations. We will list the per-task settings in the revised version.
  2. We apologize: "PIPA" is a typo for PIQA (Bisk et al., "PIQA: Reasoning about Physical Commonsense in Natural Language," AAAI 2020). We will correct it in the revised version.

Q2 — Load balancing and the RCP coefficient

  1. We apologize for not stating this clearly. The baseline and HERO-MoE use the same and only load-balancing strategy: auxiliary-loss-free bias balancing (Wang et al., 2024), as adopted in DeepSeek-V3.
  2. As shown in Appendix Fig. 5, adding HERO-MoE has almost no impact on load balancing.
  3. RCP was an important reproduction target in our early experiments. We searched all coefficients from 1e+2 to 1e-4 with a factor of 0.1 between successive values, and the best result was obtained at 1e-1. HERO-MoE, by contrast, requires no such tuning.

Q3 — Qwen3 setup

  1. The Qwen3-architecture model is trained from scratch with random initialization; it is not continued from any released Qwen3 checkpoint.
  2. All models use standard architectures. The router initialization is described in Section 3.2, and the MoE configuration is given in Section 4.1.
  3. The training budget is 100B tokens.

Dear Reviewer fHQQ,

Thank you again for your constructive feedback. We have posted responses addressing your concerns about the comparison with RMoE, separating token-specific history from router capacity, the role of depth compensation, the reliability of the downstream gains, and the evaluation, load-balancing, RCP, and Qwen3 setup details. In particular, we added a shuffled-history control, a ρ ablation showing that depth compensation improves upon RMS alignment alone, results over 4 random seeds together with evaluation standard errors, and clarifications of the evaluation settings, the PIQA typo, the RCP coefficient search, and the Qwen3 training setup.

If you have a chance, we would greatly appreciate any further feedback on whether there are any remaining concerns or unclear points.

Thank you for your time.

Reviewer_UpQhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors augment the router input with processed dense routing probabilities from preceding MoE layers for the same token within the same pipeline stage. They introduce a scale-preserving fusion mechanism to stabilize the incorporation of this historical information. Experiments on an MoE model with approximately 8B total parameters and 0.5B active parameters show improvements in model quality with modest overall overhead.

Critical strengths
  1. The residual enhancement reuses existing routing probabilities and is simple to integrate. The reported experiments demonstrate improvements in model quality with modest additional overhead under the evaluated settings.
Critical weaknesses
  1. The experimental coverage remains limited and does not establish broad applicability across wider and deeper models, more fine-grained expert configurations, or higher sparsity levels.
  2. The strength and representativeness of the baseline are not sufficiently established. The main experiments use softmax-based top-4 routing, but the load-balancing configuration, including whether auxiliary-loss-free bias balancing is enabled, is not clearly specified. It is therefore unclear whether the gains would persist with stronger routing and load-balancing baselines or depend on the particular configuration evaluated.
Questions for authors
  1. Have you examined whether the top-k expert sets selected for the same token across layers within a pipeline stage remain highly consistent after aligning expert identities using a fixed, layer-specific permutation shared across all tokens? Such an analysis would help distinguish whether historical routing primarily reflects stable token groupings across depth or captures more complex, depth-dependent expert selection patterns.
2 comments
AuthorsOpenReview ↗

Response to Reviewer UpQh

We thank the reviewer for the insightful comments, especially the analysis suggested in Q1.

W1 — Broader applicability

  1. Our configuration is representative of current large-scale MoE models. The activation ratio of 4/128 (3.1%) is the same as in DeepSeek-V3/R1 (8/256) and gpt-oss-120b (4/128), and sparser than in Qwen3-30B-A3B (8/128). Our experiments also cover both DeepSeek-style and Qwen3 architectures. Because of limited compute, we will validate HERO-MoE on larger models with more experts in future work.
  2. We additionally trained the same model on 200B tokens. HERO-MoE reduces the loss from 1.5824 to 1.5571 (−0.0253), larger than the 0.0209 gain at 100B, showing that the benefit grows with longer training.

W2 — Baseline strength and load balancing

  1. We apologize for not stating this clearly. The baseline and HERO-MoE use the same and only load-balancing strategy: auxiliary-loss-free bias balancing (Wang et al., 2024), as adopted in DeepSeek-V3. We will state this explicitly in Section 4.1.
  2. As shown in Appendix Fig. 5, adding HERO-MoE has almost no impact on load balancing.

Q1 — Stable token groupings vs. depth-dependent patterns

We thank the reviewer for this insightful question. We performed the suggested analysis. For each layer pair within a pipeline stage (Stage 0, E = 128, k = 4, 12,283 tokens), we built the expert co-occurrence matrix, used the Hungarian algorithm to find the optimal token-independent, layer-specific permutation, and then computed the aligned top-k overlap.

Layer Distance (Δ=l2−l1\Delta = l_2 - l_1)Random Floor (k/Ek/E)HERO-MoE (Raw / Unaligned)Baseline (Permutation-Aligned)HERO-MoE (Permutation-Aligned)Gap (HERO − Base)
Δ=1\Delta = 1 (Adjacent)3.12%3.08%32.62%34.72%+2.10%
Δ=2\Delta = 23.12%3.15%31.66%33.24%+1.58%
Δ=3\Delta = 33.12%2.94%30.18%31.46%+1.28%
Δ=4\Delta = 43.12%3.21%29.39%30.30%+0.91%
Δ=5\Delta = 53.12%2.89%27.96%28.79%+0.83%
Δ=10\Delta = 103.12%3.01%24.65%25.42%+0.77%
Δ=14\Delta = 14 (Max in Stage)3.12%3.11%23.41%24.51%+1.10%

Even under the optimal alignment, adjacent-layer overlap is only 34.7%, and it decreases monotonically to 24.5% at Δ = 14. Routing history therefore contains a stable component well above chance, but most expert choices (over 65%) change with depth in a way that no fixed permutation can explain. This is exactly why HERO-MoE learns a mapping W_R instead of copying or sharing routing decisions across layers. HERO-MoE also slightly increases the aligned consistency (+0.8% to +2.1%) without making routing rigid, acting as a soft trajectory prior.

Overall, HERO-MoE reinforces existing expert pathways (an idea also shared by RCP) while capturing more complex, depth-dependent expert selection patterns.

Dear Reviewer UpQh,

Thank you again for your constructive feedback. We have posted responses addressing your concerns about the broader applicability of HERO-MoE, the strength of the baseline and its load-balancing configuration, and the cross-layer routing analysis. In particular, we compared our sparsity configuration with widely used MoE models, added a 200B-token training run, clarified that both models use the same auxiliary-loss-free bias balancing, and performed the permutation-aligned overlap analysis you suggested to distinguish stable token groupings from depth-dependent expert selection patterns.

If you have a chance, we would greatly appreciate any further feedback on whether there are any remaining concerns or unclear points.

Thank you for your time.

Reviewer_L3AiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HERO-MoE, a routing method that incorporates historical routing information from preceding MoE layers into the current router through a residual branch. The method uses RMS-based Scaling and Depth-Aware Initialization to control the scale of historical features while preserving standard Top-k Routing and Sparse Dispatch.

The authors evaluate HERO-MoE on an 8B MoE model trained on 100B tokens. The results show lower training loss (1.6393 to 1.6184) and improvements on 9 of 13 downstream benchmarks, with a 3.85% increase in total training time. The paper also includes ablations on historical window sizes and scaling, additional results on Qwen3, and an analysis of cross-layer routing patterns.

Critical strengths
  1. Simple and Practical Design: HERO-MoE uses historical routing information through a residual branch while keeping standard Top-k Routing and Sparse Dispatch unchanged, without adding auxiliary losses.

  2. Well-Motivated Scaling Mechanism: RMS-based Scaling and Depth-Aware Initialization are supported by theoretical analysis showing depth-independent initial router logit variance under specific assumptions.

  3. Promising Experimental Results: On an 8B MoE model trained with 100B tokens, HERO-MoE reduces training loss and improves 9 of 13 benchmarks, with only 3.85% additional training time.

Critical weaknesses
  1. Limited Novelty and Baseline Comparisons: The idea overlaps with RMoE, but the paper lacks direct comparisons to establish its advantages over existing methods.

  2. Incomplete Ablation Studies: The experiments do not fully separate the benefits of historical routing from increased router capacity, and alternative history representations are not explored.

  3. Limited Experimental Validation: The paper lacks multi-seed results and inference benchmarks. The additional Qwen3 experiment mainly reports training loss, leaving broader generalization unclear.

Questions for authors
  1. Comparison with RMoE: How does HERO-MoE compare with RMoE under similar model sizes and training budgets? A direct comparison would help clarify its advantages over existing cross-layer routing methods.

  2. Source of Performance Gains: Does HERO-MoE still outperform the baseline when the number of router parameters is matched? This would help determine whether the improvements come from historical routing information or simply increased router capacity.

4 comments
AuthorsOpenReview ↗

Response to Reviewer L3Ai

We thank the reviewer for the positive assessment and the constructive suggestions.

W1, Q1 — Comparison with RMoE

  1. The RMoE paper itself states that it is "orthogonal to most existing router designs." HERO-MoE modifies only the router logits and can be stacked directly on top of RMoE, so the two methods are complementary and can be combined.
  2. RMoE propagates an implicit GRU state learned from hidden states, whereas HERO-MoE propagates explicit routing distributions under stop-gradient. The two address different research questions: HERO-MoE asks whether routing history itself is useful, while RMoE asks whether a cross-layer recurrent representation is useful.
  3. HERO-MoE is a strict extension of the standard router. It exactly recovers the baseline when W_R = 0 (Prop. 1), introduces no new module, and leaves dispatch unchanged. RMoE adds a projector and a GRU and replaces the router input.
  4. Fig. 2 already compares HERO-MoE with RCP, a cross-layer routing method, under identical settings. RCP reduces the loss by 0.0193 and HERO-MoE by 0.0209. HERO-MoE's advantage also grows faster in late training, and it requires no hyperparameter tuning.
  5. We will implement RMoE in our setting and include the comparison in the revised version.

W2, Q2 — Historical information vs. router capacity

  1. HERO-MoE adds only 3.44M parameters. This is comparable in size to the router matrices, but it is only 0.04% of the total model parameters (8B).
  2. To separate the two factors directly, we randomly shuffled the routing history across tokens before feeding it to the router. This keeps the parameters and architecture identical but removes the token-specific history. The gain vanishes completely:
VariantExtra router paramsFinal loss
Baseline01.6393
Shuffled history3.44M1.6401
HERO-MoE3.44M1.6184

This shows that the gain comes from historical routing information, not from increased router capacity.

  1. In Fig. 2, using only the most recent 5 or 10 layers already achieves most of the gain. Extending the window to all visible layers adds router capacity but brings no further improvement.

W2 — Alternative history representations

  1. Because this research direction is new, we favored a simple concat + residual design, which works well in practice.
  2. To improve stability in early training, we also explored other history representations:
    • Low-rank W_R. Representing the 128 × 128 expert-to-expert relations with fewer parameters slows the late-stage loss improvement, and the gain drops sharply to 0.012.
    • History pruning. As shown in Fig. 2, using only the most recent 5 layers removes weakly related distant layers, which improves training stability without reducing the gain.
    • Future work. We will explore weighting the history with a decay coefficient based on layer distance.

W3 — Seeds and Qwen3 generalization

  1. All experiments in the paper use seed 1234. We repeated the main experiment (Fig. 2) with three additional seeds (2025, 2026, 2027):
ModelFinal loss (mean ± std over 4 seeds)
Baseline1.6407 ± 0.0028
HERO-MoE1.6193 ± 0.0023
Δloss−0.0214

The gap (0.0214) is about 8× the seed-to-seed standard deviation, so the gain is robust to random seeds.

  1. The small gain on Qwen3 in Appendix C came from that experiment's use of a lower-quality open-source dataset. Retrained on the same data as the main experiment, HERO-MoE reduces the loss from 1.6307 to 1.6191 (Δ = 0.0116, above 1e-2).
  2. The gain is smaller than on the DeepSeek-style architecture. We attribute this to QK-Norm in Qwen3, which smooths token features and makes the model fluctuate more in early training. Restricting the history window to the most recent 5 layers reduces this noise and further lowers the loss to 1.6157.
  3. Downstream results under the Qwen3 architecture also show clear gains on reasoning benchmarks:
ModelHumanEvalARC-CMATHAvg.(3)
Qwen3 baseline (1.6307)16.4636.188.1020.25
Qwen3 + HERO-MoE (1.6157)19.5137.979.4822.32
Δ+3.05+1.79+1.38+2.07

Dear Reviewer L3Ai,

Thank you again for your constructive feedback. We have posted responses addressing your concerns about the comparison with RMoE, the source of the performance gains, alternative history representations, multi-seed validation, and the Qwen3 generalization results. In particular, we clarified the differences and complementarity between HERO-MoE and RMoE, added a shuffled-history control showing that the gain comes from historical routing information rather than router capacity, described additional history-representation variants, and reported results over 4 random seeds as well as downstream benchmarks under the Qwen3 architecture.

If you have a chance, we would greatly appreciate any further feedback on whether there are any remaining concerns or unclear points.

Thank you for your time.

Thank you for the detailed rebuttal and additional experiments. I encourage the authors to include the promised comparison in the camera-ready version.

Dear Reviewer L3Ai,

Thank you for your re-evaluation and for raising your rating of this paper. We greatly appreciate your constructive comments, which have significantly improved the quality of this work.

All supplementary experiments and manuscript revisions mentioned in our response will be fully incorporated into the revised manuscript. Meanwhile, we will conduct the comparison against RMoE promptly.

Reviewer_E3MaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes HERO-MoE, a routing framework for sparse MoE models that injects historical routing priors into expert selection. While standard routers compute logits solely from the current token representation, HERO-MoE reuses detached, dense routing distributions from preceding layers within the same pipeline-parallel stage as an additive residual branch before standard softmax and top-kk dispatch. To prevent logit explosion and layer-dependent variance drift as the visible history grows, the authors introduce a dynamic root-mean-square (RMS) scale-preserving coefficient paired with depth-aware weight initialization, ensuring uniform initial logit variance across layers without requiring auxiliary losses.

The method is validated by training an ~8B parameter Deepseek-like MoE model (0.5B active parameters) from scratch on 100B tokens. A model without the HERO MoE modifications is used as the baseline. HERO-MoE reduces the final training loss and improves average score across 13 downstream benchmarks tasks while incurring modest overhead. Detailed mathematical analysis of the initialization and recovery of the baseline router is provided in the appendix.

Critical strengths
  1. The paper is well written and easy to read. The mathematical notations are clear and explain the main idea nicely

  2. The idea of explicitly providing routing information as input to later layers is interesting (and to me seems to have conceptual similarity to attention residuals).

  3. The energy-based rescaling of token contributions and the initialization of the matrices are important contributions.

  4. The section on interpreting how HERO influences routing distribution is insightful. However, Figure 3(c) is hard to read and any particular pattern is also hard to infer.

Critical weaknesses
  1. The results have been shown on a single model (whose architecture has not been reported) on 25K steps with 100B tokens. Two questions: (i) It is difficult to extrapolate if the moderate gains in training loss is going to hold with longer training runs. (ii) In the absence of the architecture description, it is difficult to understand that the proprietary architecture of the model is not confounding with the HERO MoE changes resulting in the observed gains.

  2. The authors reported training loss on the Qwen3 model and the difference after 25K steps (1.9056 --> 1.8985) is really small. The experimental setup for the Qwen3 is not specified and neither is the impact of the loss difference on the downstream tasks.

  3. The gains again are moderate on downstream tasks and non-monotonic. The exact benefits of including the historical information is unclear. How does HERO effect load balancing?

  4. There is an overhead when using HERO which have been quantified by the authors. How does a FLOP-matched MoE compare to HERO MoE? Do the gains reduce any further?

  5. Overall, the technique is interesting but seems to be doing precious little to the model performance downstream.

  6. The idea of introducing the routing distribution from prior layers explicitly to lower layers is interesting. However, the current way of including that information does not seem to be leading to significant gains.

Nit Pick: i. Please pick between MOE or MoE and use it consistently across the paper.

Questions for authors
  1. How do you show that this technique generalizes and the observed gains are not specific to the architecture used (especially, given the small gains on Qwen3).

  2. Refer to the section above for detailed comments and questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer E3Ma

We thank the reviewer for the careful review and constructive comments.

W1, W2, Q1 — Generalization and extrapolation

Architecture (W1-ii). We apologize for the unclear description. Our model uses a standard, general-purpose architecture, and its MoE architecture is the same as the basic architecture of DeepSeek-V3. Section 4.1 provides a complete description of the configuration: 30 layers, hidden size 1024, MLA with 16 heads and query LoRA rank 512, 128 experts with intermediate size 640, and top-4 routing. HERO-MoE is added directly on top of this baseline, and every added component is described in Section 3. Apart from these components, the baseline and HERO-MoE share the same architecture, data, and hyperparameters, so no hidden architectural change contributes to the gain.

Longer training (W1-i). Fig. 2(b) shows that the advantage of HERO-MoE keeps increasing in the late stage of training and has the steepest slope among all variants, so the gain is likely to persist with longer training. To verify this, we trained the same model on 200B tokens:

Training tokensBaselineHERO-MoEΔloss
100B (paper)1.63931.6184−0.0209
200B1.58241.5571−0.0253

The gain grows with longer training.

Qwen3 (W2, Q1). The small gain in Appendix C came from that experiment's use of a lower-quality open-source dataset. When we retrained Qwen3 on the same data as the main experiment, HERO-MoE reduced the loss from 1.6307 to 1.6191 (Δ = 0.0116, above 1e-2), confirming that the method is effective.

The gain is still smaller than on the DeepSeek-style architecture. We attribute this to QK-Norm in Qwen3, which smooths token features and makes the model fluctuate more in early training. Restricting the history window to the most recent 5 layers, which Fig. 2 shows already captures most of the benefit, reduces the noise introduced by distant layers and lowers the loss further to 1.6157. Downstream results under the Qwen3 architecture:

ModelHumanEvalARC-CMATHAvg.(3)
Qwen3 baseline (1.6307)16.4636.188.1020.25
Qwen3 + HERO-MoE (1.6157)19.5137.979.4822.32
Δ+3.05+1.79+1.38+2.07

HERO-MoE also brings clear gains on reasoning benchmarks with the Qwen3 architecture. Together with the main model, this shows that the gain is not specific to one architecture.

Because of limited compute, experiments on larger models will be added in future work.

W3, W5, W6 — Magnitude of the gains

  1. A loss reduction of ~0.02 at 100B training tokens is substantial at this scale, and it grows to 0.025 at 200B tokens. The downstream gains are also considerable for this training scale: +0.85 on average, with improvements on 9 of 13 tasks.
  2. The gains are non-monotonic because different benchmarks emphasize different abilities. With the same training data and length, scores on knowledge-oriented benchmarks (e.g., MMLU, CMMLU) stay close between the two models, which dilutes the average. The gains concentrate on reasoning, math, and code: HumanEval +3.66, ARC-C +2.31, MATH +1.46, HellaSwag +1.39, and GSM8K +1.22. This indicates that HERO-MoE improves the model's reasoning ability.
  3. Introducing routing history enables explicit cross-layer expert interaction, which improves model capability. In addition, the learned residual matrix W_R explicitly reveals cooperative and competitive relations between experts (Fig. 3), providing a basis for future research on expert-selection prediction and interpretable MoE.

W3 — Load balancing

  1. We apologize for not stating this clearly. The baseline and HERO-MoE use the same and only load-balancing strategy: auxiliary-loss-free bias balancing (Wang et al., 2024), as adopted in DeepSeek-V3.
  2. As shown in Appendix Fig. 5, adding HERO-MoE has almost no impact on load balancing.

W4 — FLOP-matched comparison

HERO-MoE adds only 0.64% FLOPs, which is equivalent to about 150 training steps out of 25K. Results for a FLOP-matched HERO-MoE (Qwen3, same data as the main experiment):

ModelStepsFinal loss
Baseline25,0001.6307
HERO-MoE25,0001.6191
HERO-MoE (FLOP-matched)24,8501.6193

The 0.0002 difference is within random fluctuation, so the gain is essentially unchanged under FLOP matching.

Nit pick

Thank you for pointing this out. We will use "MoE" consistently in the revised version.

Dear Reviewer E3Ma,

Thank you again for your constructive feedback. We have posted responses addressing your concerns about generalization and extrapolation, the architecture description, the Qwen3 results, the magnitude of the gains, load balancing, and the FLOP-matched comparison. In particular, we added a 200B-token training run showing that the gain grows with longer training, Qwen3 results retrained on the main-experiment data together with downstream benchmarks, a clarification of the architecture and load-balancing setup, and a FLOP-matched comparison showing that the gain is essentially unchanged.

If you have a chance, we would greatly appreciate any further feedback on whether there are any remaining concerns or unclear points.

Thank you for your time.

Thank you for the detailed responses. I am happy to see that after training for 200B tokens, we still see an improvement in loss. The FLOP matched run is also a useful data point. I am still not fully convinced about how much gain will this technique yield on large scale training runs. But, based on the responses, I will increase my score.

Dear Reviewer E3Ma,

Thank you for your re-evaluation and for raising your rating of this paper. We greatly appreciate your constructive comments, which have significantly improved the quality of this work.

All supplementary experiments and manuscript revisions mentioned in our response will be fully incorporated into the revised manuscript. Meanwhile, we will evaluate the performance gains of HERO-MoE on larger-scale models as soon as possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.