acceptodds
Under review as a conference paper at ICLR 2027

AhaMoE: Expert Routing Ahead of Language Model Computation

Abstract

Routing from intermediate hidden states limits expert prefetching in Mixture-of-Experts (MoE) models. Existing methods that route from input tokens freeze their routing policies, although the model continues to learn. Once a standalone router takes over expert selection, where can it obtain supervision to continue learning? We show that gates trained only to weight expert outputs can still provide this supervision. We introduce *AhaMoE*, which combines router initialization and partial parameter reuse from an early source model, and continual adaptation of the routing policy to determine expert assignments for all layers before model computation. Experiments with 2.6B-parameter target models trained on 32B tokens show that AhaMoE outperforms existing pre-routed baselines and matches or outperforms hidden-routed baselines without partial parameter reuse in language modeling and zero-shot evaluation. Controlled experiments further show that our partial-reuse strategy improves hidden-routed models as well.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, mixture of experts, expert offloading

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VPCuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Since MoE models must store all expert weights, which strains GPU memory, some experts are often placed in CPU memory or on disk and transferred as computation proceeds (expert offloading). However, in conventional MoE models, expert selection depends on each layer’s hidden states, so the model must wait for the previous layer to finish before knowing which experts to select. Predictive prefetching risks incorrect predictions and wasted bandwidth. Existing methods that route directly from input tokens (Hash Layers, StableMoE, Grouter) can determine experts in advance, but their routing policies are frozen during pre-routed training, so their performance lags significantly behind modern hidden-routed models. A key challenge is how to provide supervision for continual learning once an independent router takes over expert selection.

The authors use the gates, which are “responsible only for weighting”, as teachers. Since discrete top-k selection is not differentiable, traditional gates receive gradients only through the weights they assign to the selected experts. The authors retain these gates and use their logits (BCE plus RankNet loss) to continually distill the independent router, with gradients updating only the router. This creates a “bootstrapping” problem: the router learns from the gates, while the gates are trained under the expert assignments produced by the router. To address it, the authors first train a source model on about 13.3% of the token budget, use it to initialize the router by distillation, and calibrate the selection biases via quantile balancing. They also reuse part of the source model: only the embedding layer and the first two (hash-routed) blocks are copied, and the remaining layers are reinitialized. This reuses learned knowledge while avoiding forcing pretrained experts to adapt to a new routing policy. The router is an 8-layer, 256-wide KDA (Kimi Delta Attention) with SwiGLU and grouped readout, and its embeddings can be folded into a lookup table. KDA was chosen because it leverages context at low computational and state cost.

In the main experiments (2.6B parameters, 320M activated per token, 32B tokens), AhaMoE achieved the lowest perplexity on WikiText and LAMBADA, with an average accuracy of 54.13 and a geometric mean of 52.42. This is the best among all methods, and the best on every benchmark among pre-routed methods. The hidden-routed baseline, which does not use partial parameter reuse, achieved 53.29 and 51.57, so AhaMoE performed slightly better.

In ablations (8B tokens, 32 experts), freezing the routing policy raised validation loss by about 0.031, so continual adaptation performs significantly better. Removing offline initialization raised it by only about 0.003. Reusing the first two blocks outperforms reusing only the embeddings (loss higher by 0.031) or the entire model (higher by 0.022). A fully trained source gives only a small further gain (loss lower by 0.006). Reusing about five blocks works best, and the strategy is also effective for hidden routing, outperforming Drop-Upcycling.

For inference latency, the P95 latency of computing all routes on an 8-core CPU is lower than the P50 latency of running the first two layers on a GPU, suggesting that routing can potentially be hidden behind those layers.

Critical strengths
  1. Continual router adaptation. Unlike Hash Layers, StableMoE, and Grouter, which fix the routing policy during target training, AhaMoE keeps the original gates for expert-output weighting and continually distills their logits into the standalone router. This directly answers where a standalone router can obtain supervision, and freezing the policy increases validation loss by about 0.031.

  2. Expert assignment for all layers before computation. AhaMoE determines the actual routing decisions of all layers in advance, rather than predicting them (as in predictive prefetching) or looking only one block ahead (as in Pre-Gated MoE), which makes exact expert prefetching possible by design.

  3. Cheap early source model with partial reuse. A source model trained on about 13.3% of the token budget is used both to initialize the router and to transfer the embeddings and first two blocks. A fully trained source gives only about 0.006 further improvement, and prefix reuse alone reduces validation loss by about 0.04–0.05.

Critical weaknesses
  1. Unclear attribution of gains and weak isolation of the core mechanism. The hidden-routed baseline in the main comparison does not use the same partial prefix reuse, and Figure 3 shows standalone routing is slightly worse than hidden routing (by about 3–7×10−33\text{–}7 \times 10^{-3} in validation loss) at every positive reuse depth tested. Much of AhaMoE's advantage may therefore come from parameter reuse rather than pre-routing. In addition, the frozen-versus-adaptive comparison also changes input representations and bias updates, and the central hypothesis that weighting-only gates provide reliable supervision for expert selection is not tested in isolation.

  2. Limited system-level evidence and generalizability. Despite the offloading motivation, only component-level latency is reported, with no end-to-end offloading throughput and no comparison with Read-ME or prediction-based prefetching. No error bars or repeated runs are reported, part of the downstream gain comes from a single benchmark (WinoGrande), and all results are at one scale (2.6B parameters, 32B tokens).

Questions for authors
  1. How would Hash Layers, StableMoE, and Grouter perform with the same two-block prefix reuse? This would show how much of AhaMoE’s gain comes from the routing mechanism rather than parameter reuse.

  2. How does the top-k agreement between the standalone router and the target model’s gates evolve during training? This would directly test whether the gates remain reliable supervision once they no longer control selection.

  3. Could the authors report end-to-end offloading throughput or prefetch hit rate, ideally compared with prediction-based prefetching? This would establish the practical benefit of all-layer pre-routing.

3 comments

W1 and Q1. Separating the contributions of partial reuse and pre-routing.

Our original paper already states in the final paragraph of Section 4.5 that partial reuse “accounts for much of AhaMoE's quality advantage over hidden-routed models trained from scratch.”

We have now added comparisons applying partial reuse to hidden routing and all 3 pre-routed baselines. Hidden routing with partial reuse performs slightly better than AhaMoE, confirming that our quality advantage over hidden-routed baselines comes mainly from partial reuse. AhaMoE still outperforms Hash Layers, StableMoE, and Grouter on every reported metric when they also use partial reuse, showing that our advantage over other pre-routed methods does not depend on partial reuse. See Common Reply 1 for the results.

W1 and Q2. Router–gate alignment during training.

We recorded top-kk alignment between the standalone router and the target model's gates during training of the 2 smaller AhaMoE models. With k=4k=4, alignment is the size of their top-4 intersection divided by 4, averaged over valid tokens and layers 3–16. We report 3 cases: both the router and gate include selection biases, neither includes bias, or only the router includes bias. Alignment increases overall during training and approaches 80% by the end. Each table reports 12 recorded training steps, sampled more densely early in training.

0.8B AhaMoE: top-4 alignment (%).

Training stepBoth biasesNeither biasRouter bias only
1123.9523.9526.9526.9527.1827.18
646419.4819.4823.9223.9226.3226.32
12812840.7640.7656.6156.6146.9246.92
25625646.5946.5973.2773.2748.3248.32
51251257.5157.5176.2376.2347.5847.58
1,0241,02466.1566.1570.8370.8361.2361.23
2,0482,04870.2070.2071.6071.6067.9767.97
4,0964,09673.5373.5374.2874.2871.4671.46
8,1928,19275.6575.6576.2876.2873.9173.91
12,28812,28876.6976.6977.4477.4475.4175.41
16,38416,38477.3777.3777.9977.9976.0076.00
20,48020,48079.0979.0979.8279.8277.5877.58

1.4B AhaMoE: top-4 alignment (%).

Training stepBoth biasesNeither biasRouter bias only
1123.4023.4025.1525.1525.0725.07
12812842.9742.9754.9754.9747.9647.96
25625645.9045.9072.9672.9648.4248.42
51251259.0959.0975.1875.1845.3545.35
1,0241,02466.7166.7169.7069.7063.3063.30
2,0482,04869.8969.8970.9270.9268.5668.56
4,0964,09672.0872.0872.8772.8771.0571.05
8,1928,19274.2874.2875.0075.0073.4373.43
16,38416,38475.8275.8276.5076.5074.9374.93
24,57624,57676.1976.1976.7176.7175.2475.24
32,76832,76876.9876.9877.5177.5176.0276.02
40,96040,96078.3178.3178.8978.8977.1377.13

However, alignment alone does not explain model quality: in the trained 2.6B AhaMoE model, replacing the standalone router's selections with the target model's own gate top-4 increases held-out loss by 3.06×10−33.06\times10^{-3} without gate bias and 3.76×10−33.76\times10^{-3} with gate bias. This shows that the target model's adaptation to the standalone router during training must also be taken into account.

W2 and Q3. End-to-end inference.

We have added end-to-end inference experiments. AhaMoE with exact prefetching decodes 1.24–1.38× faster than the hidden-routed Hash-prefix baseline with predictive prefetching. At every cache capacity, this speedup exceeds the gain from predictive prefetching over naive on-demand loading on both hidden-routed models. The end-to-end latency benefit of partial reuse is also part of our contribution. See Common Reply 2 for the results.

We do not include Read-ME [1] because it would require training a separate model with a training recipe that is difficult to align with ours. We discuss this in detail in the paper's Related Work section.

References

[1] Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design. NeurIPS 2024.

Thank you for the response. The partial-reuse controls, additional scales, and revised claims address my main concern about attributing the gains. One remaining note, the 1.24–1.38$\times$ speedup is measured against Hash-prefix without partial reuse, so reporting the speedup over Hash-prefix (PR) with predictive prefetching would isolate the benefit of exact pre-routing. I maintain my score.

Thank you for recognizing our work and maintaining your positive rating. We also appreciate your constructive suggestions, which helped us clarify the respective contributions of partial reuse and pre-routing and strengthen the evaluation with matched controls.

Common Reply 2, at the top of this page, already reports the detailed latency results for both models. These results show that AhaMoE with exact prefetching achieves a 1.13–1.25× decode speedup over Hash-prefix (PR) with predictive prefetching, isolating the benefit of exact pre-routing when both models use partial reuse.

Reviewer_vF4COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

They propose a new method for training an external router for MoE models to simplify serving the model the model at inference time. The external router predicts experts from tokens so that fetching experts can be optimized upfront, unlike typical MoE models which cannot fetch until the last minute when they need the expert. They partially train a source MoE model, duplicate part of its parameters to a new model that only does routing for the first network. The original models routers still serve as a distillation loss for stability during training in addition to ranking loss.

Critical strengths

[S1] AhaMoE performs convincing well on the benchmarks in Table 1, beating the other pre-routed models and possibly even better than the hidden-routed models.

[S2] The ablation of their method in table 2 helps to justify its many components.

[S3] Decoding latency of their method provided in the appendix.

Critical weaknesses

[W1] The entire sell of the method is faster serving of the MoE model, however, there is no timing comparison against hidden-routed MoEs, or other pre-routed MoEs.

[W2] Some of the margins are quite small between methods in table 1/2, specifically hidden-routed and AhaMoE, a test for significance would strengthen the result.

[W3] I found the writing difficult to follow in many places, and it took multiple reads to infer what was trying to be communicated.

Questions for authors

None

3 comments

W1. End-to-end inference.

We have added end-to-end inference experiments. AhaMoE with exact prefetching decodes 1.24–1.38× faster than the hidden-routed Hash-prefix baseline with predictive prefetching. At every cache capacity, this speedup exceeds the gain from predictive prefetching over naive on-demand loading on both hidden-routed models. The end-to-end latency benefit of partial reuse is also part of our contribution. See Common Reply 2 at the top of this page for the results.

To clarify the summary, AhaMoE's standalone router determines the actual experts executed by the target model, rather than predicting selections made by another router.

W2. Interpreting the performance differences.

We have split the original Table 1 into 3 tables to separate the comparisons with hidden-routed models, pre-routed models across scales, and pre-routed models using partial reuse. We do not claim that AhaMoE outperforms hidden-routed baselines on these tasks. The results show at least comparable task performance, while AhaMoE's advantage lies in end-to-end inference performance.

The advantage over pre-routed methods is much clearer. AhaMoE outperforms every pre-routed baseline on every reported metric across all tested scales and settings, whether or not partial reuse is used. This consistent pattern provides strong evidence of an advantage, rather than merely comparable performance. Reporting results from a single training run per model configuration is also common practice in this field. Our survey identified 73 accepted papers at ICLR, ICML, and NeurIPS 2026 that report zero-shot reasoning results this way. The results and supporting examples are provided in Common Reply 1 at the top of this page.

For the original Table 2, now Table 4, the 2×10−2∼3×10−22\times10^{-2} \sim 3\times10^{-2} improvement is a substantial reduction in absolute language-modeling loss, not a percentage change in accuracy. For reference, the ablation study in Attention Residuals [1] reports an absolute loss gap of about 3×10−23\times10^{-2} between PreNorm and Full AttnRes.

W3. Presentation revisions.

We have revised the main text to make the contributions and experimental conclusions clearer:

  • Abstract: We distinguish the quality advantage over pre-routed methods from comparable quality and lower inference latency relative to hidden-routed models.
  • Introduction, Section 1: We explicitly state our 2 main contributions: continual standalone-router training with non-router supervision and partial parameter reuse.
  • Training supervision, Section 1: We emphasize: “During target-model training, the standalone router continues to learn at each step from the logits produced by the target model's MoE gates, which no longer perform routing.”
  • Experimental setup, Section 4.1: We extend evaluation to 3 model scales and clarify that target models share the same training stream and token budget within each scale.
  • Main results, Section 4.2 and Tables 1–3: We separate the hidden-routed, cross-scale pre-routed, and partial-reuse comparisons to make the quality claims and the role of partial reuse clearer.
  • Training-loss comparison, Figure 1: We add Hash-prefix (PR) to distinguish the benefit of partial reuse from that of additional training budget.
  • End-to-end inference, Section 4.5 and Tables 5–6: We add decode latency comparisons and a breakdown of loading and waiting, and report comparable prefill latency.

References

[1] Attention Residuals. Technical report, 2026.

[W1] Thank you for providing the timing comparison.

[W2] I appreciate you rescoping the claims to be more consistent with the results.

[W3] The revisions are an improvement over the original draft.

This largely addresses my concerns, I have raised my score accordingly.

Thank you for recognizing our work and raising your score. We also appreciate your constructive suggestions, which helped us strengthen the paper with end-to-end inference comparisons, clarify the scope of our claims, and improve the presentation of our method and results.

Reviewer_4St5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The submission introduces a method for improving offline assignment of MoE experts with a small separate router model, motivated by wanted to improve the speed of pre-fetching experts from CPU memory.

The process for learning this router is rather complex, and involves distilling a separate "source" MoE model before we even start training the target model. As I understood it, the main new idea is letting the target model decide how to gate the assigned experts, even though the routing is handled by a separate model. The paper claims this method outperforms standard MoE training.

Critical strengths

Faster pre-fetching of experts is a practically useful problem. The paper introduces a useful new idea that the gating of experts can be chosen by the main model instead of the router model, which makes sense as it allows the stronger model to correct the output of the router.

Critical weaknesses
  • The method has a rather complex multistage training, which may limit adoption
  • All evaluations appear to be on a single model scale. Evaluating a ladder of smaller models would not require significantly more compute, but would give a much better idea if the results would hold at larger scale.
  • The comparisons don't seem to be carefully controlled, I think their method uses more compute than the baseline because of the need to train a separate source model
  • The results are mostly given as point estimates on noisy end task evaluations, with no attempt to quantify the uncertainty
  • The motivation for the paper is in speeding up generation by pre-fetching experts, but I couldn't find any results demonstrating that their model is faster at generation.
Questions for authors
3 comments

Thank you for the feedback. Our presentation did not clearly convey AhaMoE's two main contributions. We have revised Section 1 to state them explicitly: continual standalone-router learning from target-model gates that no longer select experts, and partial parameter reuse from an early source model, which also benefits hidden-routed MoEs.

  1. Separating gating from expert selection is not our contribution. For one token at one MoE layer, let x\mathbf{x} be the input sequence, h\mathbf{h} the hidden state, and EjE_j the jj-th of NN experts, with j∈{1,…,N}j\in\{1,\ldots,N\}. The standalone router assigns scores rj(x)r_j(\mathbf{x}), while the main model assigns weights pj(h;S)p_j(\mathbf{h};\mathcal{S}) to the kk experts in the selected set S\mathcal{S}. Then

    \qquad \mathbf{y}=\sum_{j\in\mathcal{S}}p_j(\mathbf{h};\mathcal{S})E_j(\mathbf{h}).$$ StableMoE [1] already uses this separation in its second training stage, as shown in its Section 3.2, Eqs. 12–15. It selects one expert with the standalone router but multiplies that expert's output by a learned sigmoid gate computed from the main model's hidden state. Writing $a_j(\mathbf{h})$ for the main model's gate logit and $\sigma$ for the sigmoid function, its routed contribution is $$\hat j=\operatorname*{arg\\,max}_{j\in\\{1,\ldots,N\\}}r_j(\mathbf{x}), \qquad \mathbf{y}=\sigma\\!\left(a_{\hat j}(\mathbf{h})\right)E_{\hat j}(\mathbf{h}).$$ Thus, the separation is already present for $k=1$. Our top-$k$ formulation uses normalized sigmoid mixture weights, whereas StableMoE uses an unnormalized sigmoid weight. This difference does not make the separation itself a new contribution.
  2. AhaMoE can update its router while it selects experts; StableMoE and Grouter cannot. StableMoE trains its student while the main model still selects experts, then freezes the student when it takes over. Grouter [2] also freezes its distilled router during target-model training, as described in its Section 3.3. Both methods learn a router by distilling another router that actually performs expert selection. Once the student takes over, the target model no longer independently selects experts and therefore cannot supply the routing decisions required by this teacher–student formulation. Their standalone routers therefore cannot continue updating from the target model after takeover under this router-to-router distillation mechanism.

  3. Our first contribution is learning from non-router supervision. We establish empirically that the target model's gates can still provide useful supervision to the standalone router after they no longer perform routing. These gates continue to learn through their contribution to the language-model output. AhaMoE uses their detached scores to supervise the standalone router throughout target training, while that router controls expert selection, as detailed in Section 3.5 and Algorithm 1 of our paper, so the router can keep adapting without an online source teacher.

  4. Our second contribution is partial parameter reuse. As described in Section 3.4 of our paper, we retain an early source model's embedding and first two complete, hash-routed blocks, including their hash tables, while freshly initializing the remaining target parameters and optimization state. All target parameters remain trainable. This differs from both training the entire target from scratch and simply continuing training the entire source model. Its benefit also applies to hidden-routed MoE models, independently of standalone routing.

  5. The two contributions play different roles. Continual router adaptation improves language-modeling quality relative to other pre-routed methods, including Hash Layers, StableMoE, and Grouter. Partial reuse improves initialization for both hidden routing and pre-routing and is the main source of our quality gains over hidden-routed baselines without partial reuse. With the same partial reuse on both sides, AhaMoE still performs slightly worse than hidden routing in training loss and aggregate downstream quality, but this gap is much smaller than those of the other pre-routed baselines given the same reuse strategy. The detailed quality comparisons are provided in Common Reply 1, and the end-to-end inference results in Common Reply 2. Both replies are at the top of this page.

References

[1] StableMoE: Stable Routing Strategy for Mixture of Experts. ACL 2022.

[2] Grouter: Decoupling Routing from Representation for Accelerated MoE Training. ICML 2026.

W1. Multi-stage training.

Thank you for raising this concern. AhaMoE has 3 stages: training a standard source MoE, distilling it to initialize a standalone router, and training the target MoE and initialized router together.

  1. LLM training is already multi-stage. Standard LLM training combines pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL), even though these stages use different objectives. Each stage contributes to the final model; having multiple stages does not by itself imply instability.

  2. Stages are stable. In AhaMoE, the first 2 stages are ordinary MoE training and router distillation from a fixed teacher, a preparation step also used by Grouter [1]. Both finish before target training begins. The initialized router selects experts from the first target-training step, avoiding changes in the expert-selection mechanism during training. These preparation stages reuse the original training data and require no additional unique tokens.

  3. Any MoE training recipe can be converted to an AhaMoE recipe. Let NN be the source-training budget and LL the number of layers to reuse. Run the first NN steps of the original recipe, then distill the source model's gate logits to initialize the standalone router. Copy the source embedding and first LL layers into the target, and initialize the remaining parameters as in the original recipe. Keep the target-training budget, data, optimizer, learning-rate schedule, and task loss unchanged, with all target parameters trainable. During training, the standalone router selects experts and learns from the target model's detached gate logits.

    This requires only a MoE that provides expert-gate logits, without assumptions about its architecture or training loss. The standalone router need not share the main model's architecture, allowing AhaMoE to fit existing MoE training recipes.

W2. Results across model scales.

We added evaluations at the smaller 0.8B and 1.4B scales. AhaMoE's advantage over other pre-routed methods holds across all 3 scales, including the original 2.6B scale. See Common Reply 1 at the top of this page for the results.

W3. Controlled training budgets and partial reuse.

Our original paper already included a controlled-budget experiment for our strongest baseline in training loss: Hash-prefix (replay). Matching training budgets does not mean applying partial reuse, which is our contribution. Hash-prefix (replay) repeats the same data used for source training to match AhaMoE's combined source-and-target MoE training budget, yet its loss remains higher. We have now also evaluated hidden routing with partial reuse, which performs slightly better than AhaMoE. We emphasize that configurations using partial reuse are new method variants introduced by our work, rather than pre-existing baselines. See Common Reply 1 at the top of this page for these comparisons.

W4. Consistent improvements and single-run evaluation.

AhaMoE outperforms every pre-routed baseline on every reported metric across all tested scales and settings, whether or not partial reuse is used. This consistent pattern provides strong evidence of an advantage, rather than merely comparable performance. Reporting results from a single training run per model configuration is also common practice in this field. Our survey identified 73 accepted papers at ICLR, ICML, and NeurIPS 2026 that report zero-shot reasoning results this way. The results and supporting examples are provided in Common Reply 1 at the top of this page.

W5. End-to-end inference.

We have added end-to-end inference experiments. AhaMoE with exact prefetching decodes 1.24–1.38× faster than the hidden-routed Hash-prefix baseline with predictive prefetching. At every cache capacity, this speedup exceeds the gain from predictive prefetching over naive on-demand loading on both hidden-routed models. The end-to-end latency benefit of partial reuse is also part of our contribution. See Common Reply 2 at the top of this page for the results.

References

[1] Grouter: Decoupling Routing from Representation for Accelerated MoE Training. ICML 2026.

Thank you again for reviewing our work. Could you please let us know whether our responses and revisions have addressed your concerns? If any concerns remain, please let us know what else we should add or clarify. If your concerns have been resolved, would you consider raising your score in light of the additional experiments and revisions?

Comments on the paper

6 comments

Common Reply 1: Explanation for zero-shot reasoning tasks.

  1. Single-seed training is common practice. The zero-shot tasks we report do not require text generation or sampling. Their results are computed directly from model logits and are deterministic for a fixed checkpoint and evaluation setup. Quantifying uncertainty or statistical significance with respect to training randomness would require multiple training seeds. However, pretraining is expensive, and reporting downstream results from a single training run per model configuration is common practice in language-model architecture research, including Mamba-2 [1], Mamba-3 [4], LLaDA [2], and Gated DeltaNet [3]. Through AI-assisted search and manual verification of each paper, we identified 73 papers from ICLR, ICML, and NeurIPS 2026 that follow this practice in a single year. We list 10 ICLR 2026 examples below [4–13].

  2. AhaMoE performs significantly better than other pre-routed baselines. We acknowledge that a single training run is subject to random variation. We do not claim that AhaMoE consistently outperforms the best hidden-routed baseline on these tasks. Our claim is that it achieves better language modeling quality than the other pre-routed baselines. Across all 3 model scales, AhaMoE outperforms every pre-routed baseline on every reported metric. At the largest scale, this holds both with and without partial reuse for the baselines. Even without multiple training seeds, this consistent pattern provides strong empirical support for our claim.

  3. We have already controlled training budgets in the original paper. We originally matched target-model training steps so that losses could be compared directly throughout target training. Each pretraining step uses new tokens, and our target models train for only 1 epoch, so tokens unseen by the source model serve as fresh evaluation data when their loss is measured before the corresponding training update. To test whether our advantage comes entirely from additional training, we selected Hash-prefix, the baseline with the lowest training loss, and let it repeat the source-training tokens in the same order, since AhaMoE uses no additional unique tokens. Hash-prefix (replay) matches AhaMoE's combined source-and-target MoE training budget, yet its loss remains higher.

  4. Additional experiments further isolate contributions of different machenisms. Revised Figure 1 also directly compares its loss curve with Hash-prefix (PR), showing that the improvement over hidden routing comes primarily from partial reuse, rather than simply from additional training. We further added partial reuse to all 3 pre-routed baselines, giving them the same source-training benefit and target-training budget, but they still perform substantially worse on the downstream tasks.

We have reorganized the main results into 3 tables to make these conclusions clearer: comparison with hidden-routed models, comparison with pre-routed methods across scales, and comparison with pre-routed methods using partial reuse. The new smaller-scale experiments test whether our conclusions hold across model scales, while the new partial-reuse controls test whether AhaMoE's advantage over other pre-routed methods remains when they also use partial reuse. Below, we reproduce the complete results from Tables 1–3 in Section 4.2 of the revised paper.

Table 1. Comparison with hidden-routed models at the 2.6B scale. PR denotes partial parameter reuse. We report perplexity (ppl) on WikiText (Wiki.) and LAMBADA (LMB.) and accuracy (%) on 7 tasks; acc_n denotes length-normalized accuracy. Avg. and Geo Mean. are the arithmetic and geometric means of the 7 accuracies. Best results are bold\mathbf{bold}, and second-best are underlined‾\underline{\mathrm{underlined}}.

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. acc_n ↑Wino. acc ↑ARC-e acc ↑ARC-c acc_n ↑SIQA acc ↑Avg. ↑Geo Mean. ↑
Hidden-routed21.3421.3417.2417.2442.98\mathbf{42.98}74.5974.5956.3556.3553.9953.9969.2869.2836.1836.1839.6639.6653.2953.2951.5751.57
Hash-prefix21.2321.2318.3818.3840.8740.8775.14‾\underline{75.14}55.9355.9353.4353.4369.65‾\underline{69.65}38.14‾\underline{38.14}40.2840.2853.3553.3551.6651.66
Hash-prefix (replay)20.9820.9817.9117.9141.2241.2274.9274.9256.1756.1754.3054.3069.4069.4037.3737.3739.5139.5153.2753.2751.5351.53
AhaMoE20.68‾\underline{20.68}16.19‾\underline{16.19}42.2342.2375.95\mathbf{75.95}56.86‾\underline{56.86}56.67\mathbf{56.67}69.1169.1137.5437.5440.58‾\underline{40.58}54.13‾\underline{54.13}52.42‾\underline{52.42}
Hash-prefix (PR)20.53\mathbf{20.53}15.98\mathbf{15.98}42.97‾\underline{42.97}74.7074.7057.28\mathbf{57.28}56.59‾\underline{56.59}70.79\mathbf{70.79}39.68\mathbf{39.68}40.69\mathbf{40.69}54.67\mathbf{54.67}53.09\mathbf{53.09}

Table 2. Comparison with other pre-routed methods at 3 model scales.

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. acc_n ↑Wino. acc ↑ARC-e acc ↑ARC-c acc_n ↑SIQA acc ↑Avg. ↑Geo Mean. ↑
0.8B parameters; 10.7B target tokens
Hash Layers33.2633.2645.7845.7830.5130.5169.80‾\underline{69.80}40.1540.1551.93‾\underline{51.93}58.50‾\underline{58.50}29.0129.0138.1838.1845.4445.4443.3543.35
Grouter33.9233.9249.5249.5230.1830.1868.3468.3440.0840.0850.2050.2057.1557.1527.9927.9937.7737.7744.5344.5342.5042.50
StableMoE31.72‾\underline{31.72}43.39‾\underline{43.39}30.62‾\underline{30.62}69.1569.1541.60‾\underline{41.60}51.3851.3858.2158.2129.52‾\underline{29.52}38.28‾\underline{38.28}45.54‾\underline{45.54}43.56‾\underline{43.56}
AhaMoE28.87\mathbf{28.87}32.90\mathbf{32.90}34.29\mathbf{34.29}71.44\mathbf{71.44}44.64\mathbf{44.64}52.41\mathbf{52.41}61.20\mathbf{61.20}30.72\mathbf{30.72}39.05\mathbf{39.05}47.68\mathbf{47.68}45.77\mathbf{45.77}
1.4B parameters; 21.5B target tokens
Hash Layers26.3326.3326.5926.5936.8136.8171.4971.4947.1547.1552.7252.7263.0163.0130.6330.6338.43‾\underline{38.43}48.6148.6146.7146.71
Grouter26.7526.7530.0530.0534.6634.6672.14‾\underline{72.14}46.4446.4450.5950.5961.7461.7431.5731.5738.43‾\underline{38.43}47.9447.9446.0746.07
StableMoE25.19‾\underline{25.19}25.11‾\underline{25.11}37.69‾\underline{37.69}71.9871.9849.18‾\underline{49.18}54.14‾\underline{54.14}65.49‾\underline{65.49}32.51‾\underline{32.51}38.2338.2349.89‾\underline{49.89}48.01‾\underline{48.01}
AhaMoE22.95\mathbf{22.95}20.98\mathbf{20.98}39.26\mathbf{39.26}73.61\mathbf{73.61}52.57\mathbf{52.57}55.01\mathbf{55.01}65.61\mathbf{65.61}32.85\mathbf{32.85}40.12\mathbf{40.12}51.29\mathbf{51.29}49.45\mathbf{49.45}
2.6B parameters; 32.2B target tokens
Hash Layers23.5323.5321.1921.1938.7038.7072.3172.3150.8550.8554.14‾\underline{54.14}66.5466.5432.4232.4240.33‾\underline{40.33}50.7550.7548.9248.92
Grouter23.8723.8721.8621.8639.0639.0672.5272.5250.8050.8051.8551.8565.6665.6633.9633.9637.8737.8750.2550.2548.4948.49
StableMoE22.37‾\underline{22.37}19.00‾\underline{19.00}40.23‾\underline{40.23}73.67‾\underline{73.67}53.66‾\underline{53.66}53.6753.6767.05‾\underline{67.05}34.81‾\underline{34.81}38.9538.9551.72‾\underline{51.72}49.95‾\underline{49.95}
AhaMoE20.68\mathbf{20.68}16.19\mathbf{16.19}42.23\mathbf{42.23}75.95\mathbf{75.95}56.86\mathbf{56.86}56.67\mathbf{56.67}69.11\mathbf{69.11}37.54\mathbf{37.54}40.58\mathbf{40.58}54.13\mathbf{54.13}52.42\mathbf{52.42}

Table 3. Comparison with other pre-routed methods using partial reuse at the 2.6B scale.

ModelWiki. ppl ↓LMB. ppl ↓LMB. acc ↑PIQA acc ↑Hella. acc_n ↑Wino. acc ↑ARC-e acc ↑ARC-c acc_n ↑SIQA acc ↑Avg. ↑Geo Mean. ↑
Hash Layers (PR)22.8022.8019.5319.5340.6640.6673.4573.4552.9152.9154.6254.6265.8765.8734.4734.4739.1039.1051.5851.5849.8649.86
Grouter (PR)23.0923.0919.7019.7040.5240.5274.32‾\underline{74.32}52.2052.2055.25‾\underline{55.25}66.41‾\underline{66.41}34.56‾\underline{34.56}40.48‾\underline{40.48}51.9651.9650.23‾\underline{50.23}
StableMoE (PR)21.66‾\underline{21.66}17.81‾\underline{17.81}41.45‾\underline{41.45}74.0574.0554.98‾\underline{54.98}54.7854.7865.9165.9133.1133.1139.7139.7152.00‾\underline{52.00}50.1850.18
AhaMoE20.68\mathbf{20.68}16.19\mathbf{16.19}42.23\mathbf{42.23}75.95\mathbf{75.95}56.86\mathbf{56.86}56.67\mathbf{56.67}69.11\mathbf{69.11}37.54\mathbf{37.54}40.58\mathbf{40.58}54.13\mathbf{54.13}52.42\mathbf{52.42}

These comparisons also separate the sources of our performance gains: any possible quality advantage over hidden-routed models comes primarily from partial reuse. With partial reuse on both sides, AhaMoE still trails hidden routing in perplexity and aggregate accuracy, but its gap is much smaller than those of the other pre-routed baselines.

References

[1] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024.

[2] Large Language Diffusion Models. NeurIPS 2025.

[3] Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.

[4] Mamba-3: Improved Sequence Modeling using State Space Principles. ICLR 2026.

[5] Multi-Head Low-Rank Attention. ICLR 2026.

[6] ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates. ICLR 2026.

[7] Selective Rotary Position Embedding. ICLR 2026.

[8] Scaling Attention via Feature Sparsity. ICLR 2026.

[9] LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. ICLR 2026.

[10] SeeDNorm: Self-Rescaled Dynamic Normalization. ICLR 2026.

[11] DirMoE: Dirichlet-Routed Mixture of Experts. ICLR 2026.

[12] Scaling Linear Attention Capacity with Sparse State Expansion. ICLR 2026.

[13] Householder-Diagonalized Linear Attention (HDLA): Utilizing Enhanced Decay Mechanism for Efficient Sequence Modeling. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.