- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HERO-MoE augments the current-token router with detached, dense routing distributions from preceding MoE layers within the same pipeline-parallel stage. It projects this history into residual routing logits and uses RMS scaling and depth-aware initialization to control their magnitude while preserving standard sparse dispatch. On an approximately 8B-total/0.5B-active-parameter model trained on 100B tokens, the method reduces training loss from 1.6393 to 1.6184 and improves the average score across 13 benchmarks by 0.85 points, with a 3.85% increase in training time. The paper also presents history-window and scaling ablations, routing analyses, and supplementary Qwen3 results.
Critical strengths
- HERO-MoE preserves the current-token routing branch and standard sparse dispatch without introducing an additional auxiliary loss. Restricting history to the same pipeline stage avoids additional cross-stage communication.
- The 100B-token experiment shows lower training loss and improvements on 9 of 13 downstream tasks. The ablations and supplementary Qwen3 results provide supporting evidence, while the reported 3.85% training-time overhead suggests a potentially useful quality-cost tradeoff.
Critical weaknesses
- RMoE also uses cross-layer routing information, but is not evaluated. Without this comparison, the practical advantage of HERO-MoE over existing approaches remains unclear.
- The ablations do not separate the benefit of token-specific history from increased router capacity, or establish whether depth compensation improves upon simple RMS alignment.
- The average downstream improvement is 0.85 points, with four tasks declining. Without repeated-run results or uncertainty estimates, it is difficult to assess the reliability of these gains.
Questions for authors
- What evaluation harness, prompt templates, few-shot settings, and decoding configurations were used for Table 1? What task and metric does “PIPA” denote?
- What load-balancing settings were used for the baseline and HERO-MoE, and how was the RCP regularization coefficient selected?
- Which Qwen3 model and initialization were used in Appendix C, and what was the training-token budget?
3 comments
Q1 — Evaluation details
- All results in Table 1 use the default evaluation settings of lm-evaluation-harness, including prompt templates, few-shot settings, and decoding configurations. We will list the per-task settings in the revised version.
- We apologize: "PIPA" is a typo for PIQA (Bisk et al., "PIQA: Reasoning about Physical Commonsense in Natural Language," AAAI 2020). We will correct it in the revised version.
Q2 — Load balancing and the RCP coefficient
- We apologize for not stating this clearly. The baseline and HERO-MoE use the same and only load-balancing strategy: auxiliary-loss-free bias balancing (Wang et al., 2024), as adopted in DeepSeek-V3.
- As shown in Appendix Fig. 5, adding HERO-MoE has almost no impact on load balancing.
- RCP was an important reproduction target in our early experiments. We searched all coefficients from 1e+2 to 1e-4 with a factor of 0.1 between successive values, and the best result was obtained at 1e-1. HERO-MoE, by contrast, requires no such tuning.
Q3 — Qwen3 setup
- The Qwen3-architecture model is trained from scratch with random initialization; it is not continued from any released Qwen3 checkpoint.
- All models use standard architectures. The router initialization is described in Section 3.2, and the MoE configuration is given in Section 4.1.
- The training budget is 100B tokens.
Dear Reviewer fHQQ,
Thank you again for your constructive feedback. We have posted responses addressing your concerns about the comparison with RMoE, separating token-specific history from router capacity, the role of depth compensation, the reliability of the downstream gains, and the evaluation, load-balancing, RCP, and Qwen3 setup details. In particular, we added a shuffled-history control, a ρ ablation showing that depth compensation improves upon RMS alignment alone, results over 4 random seeds together with evaluation standard errors, and clarifications of the evaluation settings, the PIQA typo, the RCP coefficient search, and the Qwen3 training setup.
If you have a chance, we would greatly appreciate any further feedback on whether there are any remaining concerns or unclear points.
Thank you for your time.
Response to Reviewer fHQQ
We thank the reviewer for the detailed and constructive questions.
W1 — Comparison with RMoE
W2 — History vs. capacity, and depth compensation
Used alone, the two factors perform similarly. Combining them gives the best result: adding ρ₂ on top of RMS alignment reduces the loss by a further 0.0035, which confirms that depth compensation improves upon RMS alignment alone. The loss curves also show that ρ₁ drives the sustained late-stage gain, whereas ρ₂ mainly improves early training: over the last 15,000 steps, the gain of ρ₁-only grows by about 0.003 more than that of ρ₂-only.
W3 — Reliability of downstream gains