- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Since MoE models must store all expert weights, which strains GPU memory, some experts are often placed in CPU memory or on disk and transferred as computation proceeds (expert offloading). However, in conventional MoE models, expert selection depends on each layer’s hidden states, so the model must wait for the previous layer to finish before knowing which experts to select. Predictive prefetching risks incorrect predictions and wasted bandwidth. Existing methods that route directly from input tokens (Hash Layers, StableMoE, Grouter) can determine experts in advance, but their routing policies are frozen during pre-routed training, so their performance lags significantly behind modern hidden-routed models. A key challenge is how to provide supervision for continual learning once an independent router takes over expert selection.
The authors use the gates, which are “responsible only for weighting”, as teachers. Since discrete top-k selection is not differentiable, traditional gates receive gradients only through the weights they assign to the selected experts. The authors retain these gates and use their logits (BCE plus RankNet loss) to continually distill the independent router, with gradients updating only the router. This creates a “bootstrapping” problem: the router learns from the gates, while the gates are trained under the expert assignments produced by the router. To address it, the authors first train a source model on about 13.3% of the token budget, use it to initialize the router by distillation, and calibrate the selection biases via quantile balancing. They also reuse part of the source model: only the embedding layer and the first two (hash-routed) blocks are copied, and the remaining layers are reinitialized. This reuses learned knowledge while avoiding forcing pretrained experts to adapt to a new routing policy. The router is an 8-layer, 256-wide KDA (Kimi Delta Attention) with SwiGLU and grouped readout, and its embeddings can be folded into a lookup table. KDA was chosen because it leverages context at low computational and state cost.
In the main experiments (2.6B parameters, 320M activated per token, 32B tokens), AhaMoE achieved the lowest perplexity on WikiText and LAMBADA, with an average accuracy of 54.13 and a geometric mean of 52.42. This is the best among all methods, and the best on every benchmark among pre-routed methods. The hidden-routed baseline, which does not use partial parameter reuse, achieved 53.29 and 51.57, so AhaMoE performed slightly better.
In ablations (8B tokens, 32 experts), freezing the routing policy raised validation loss by about 0.031, so continual adaptation performs significantly better. Removing offline initialization raised it by only about 0.003. Reusing the first two blocks outperforms reusing only the embeddings (loss higher by 0.031) or the entire model (higher by 0.022). A fully trained source gives only a small further gain (loss lower by 0.006). Reusing about five blocks works best, and the strategy is also effective for hidden routing, outperforming Drop-Upcycling.
For inference latency, the P95 latency of computing all routes on an 8-core CPU is lower than the P50 latency of running the first two layers on a GPU, suggesting that routing can potentially be hidden behind those layers.
Critical strengths
-
Continual router adaptation. Unlike Hash Layers, StableMoE, and Grouter, which fix the routing policy during target training, AhaMoE keeps the original gates for expert-output weighting and continually distills their logits into the standalone router. This directly answers where a standalone router can obtain supervision, and freezing the policy increases validation loss by about 0.031.
-
Expert assignment for all layers before computation. AhaMoE determines the actual routing decisions of all layers in advance, rather than predicting them (as in predictive prefetching) or looking only one block ahead (as in Pre-Gated MoE), which makes exact expert prefetching possible by design.
-
Cheap early source model with partial reuse. A source model trained on about 13.3% of the token budget is used both to initialize the router and to transfer the embeddings and first two blocks. A fully trained source gives only about 0.006 further improvement, and prefix reuse alone reduces validation loss by about 0.04–0.05.
Critical weaknesses
-
Unclear attribution of gains and weak isolation of the core mechanism. The hidden-routed baseline in the main comparison does not use the same partial prefix reuse, and Figure 3 shows standalone routing is slightly worse than hidden routing (by about in validation loss) at every positive reuse depth tested. Much of AhaMoE's advantage may therefore come from parameter reuse rather than pre-routing. In addition, the frozen-versus-adaptive comparison also changes input representations and bias updates, and the central hypothesis that weighting-only gates provide reliable supervision for expert selection is not tested in isolation.
-
Limited system-level evidence and generalizability. Despite the offloading motivation, only component-level latency is reported, with no end-to-end offloading throughput and no comparison with Read-ME or prediction-based prefetching. No error bars or repeated runs are reported, part of the downstream gain comes from a single benchmark (WinoGrande), and all results are at one scale (2.6B parameters, 32B tokens).
Questions for authors
-
How would Hash Layers, StableMoE, and Grouter perform with the same two-block prefix reuse? This would show how much of AhaMoE’s gain comes from the routing mechanism rather than parameter reuse.
-
How does the top-k agreement between the standalone router and the target model’s gates evolve during training? This would directly test whether the gates remain reliable supervision once they no longer control selection.
-
Could the authors report end-to-end offloading throughput or prefetch hit rate, ideally compared with prediction-based prefetching? This would establish the practical benefit of all-layer pre-routing.
3 comments
Thank you for the response. The partial-reuse controls, additional scales, and revised claims address my main concern about attributing the gains. One remaining note, the 1.24–1.38$\times$ speedup is measured against Hash-prefix without partial reuse, so reporting the speedup over Hash-prefix (PR) with predictive prefetching would isolate the benefit of exact pre-routing. I maintain my score.
Thank you for recognizing our work and maintaining your positive rating. We also appreciate your constructive suggestions, which helped us clarify the respective contributions of partial reuse and pre-routing and strengthen the evaluation with matched controls.
Common Reply 2, at the top of this page, already reports the detailed latency results for both models. These results show that AhaMoE with exact prefetching achieves a 1.13–1.25× decode speedup over Hash-prefix (PR) with predictive prefetching, isolating the benefit of exact pre-routing when both models use partial reuse.
W1 and Q1. Separating the contributions of partial reuse and pre-routing.
Our original paper already states in the final paragraph of Section 4.5 that partial reuse “accounts for much of AhaMoE's quality advantage over hidden-routed models trained from scratch.”
We have now added comparisons applying partial reuse to hidden routing and all 3 pre-routed baselines. Hidden routing with partial reuse performs slightly better than AhaMoE, confirming that our quality advantage over hidden-routed baselines comes mainly from partial reuse. AhaMoE still outperforms Hash Layers, StableMoE, and Grouter on every reported metric when they also use partial reuse, showing that our advantage over other pre-routed methods does not depend on partial reuse. See Common Reply 1 for the results.
W1 and Q2. Router–gate alignment during training.
We recorded top-k alignment between the standalone router and the target model's gates during training of the 2 smaller AhaMoE models. With k=4, alignment is the size of their top-4 intersection divided by 4, averaged over valid tokens and layers 3–16. We report 3 cases: both the router and gate include selection biases, neither includes bias, or only the router includes bias. Alignment increases overall during training and approaches 80% by the end. Each table reports 12 recorded training steps, sampled more densely early in training.
0.8B AhaMoE: top-4 alignment (%).
1.4B AhaMoE: top-4 alignment (%).
However, alignment alone does not explain model quality: in the trained 2.6B AhaMoE model, replacing the standalone router's selections with the target model's own gate top-4 increases held-out loss by 3.06×10−3 without gate bias and 3.76×10−3 with gate bias. This shows that the target model's adaptation to the standalone router during training must also be taken into account.
W2 and Q3. End-to-end inference.
We have added end-to-end inference experiments. AhaMoE with exact prefetching decodes 1.24–1.38× faster than the hidden-routed Hash-prefix baseline with predictive prefetching. At every cache capacity, this speedup exceeds the gain from predictive prefetching over naive on-demand loading on both hidden-routed models. The end-to-end latency benefit of partial reuse is also part of our contribution. See Common Reply 2 for the results.
We do not include Read-ME [1] because it would require training a separate model with a training recipe that is difficult to align with ours. We discuss this in detail in the paper's Related Work section.
References
[1] Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design. NeurIPS 2024.