- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MoEAtlas coordinates request placement, expert placement, and online expert migration to reduce communication in distributed MoE inference. An offline-trained Pre-Router predicts prompt-token expert selections across layers and uses those predictions to choose each request’s home GPU. Expert placement represents complete co-activation sets as hyperedges and minimizes connectivity-minus-one under load constraints When workload drift warrants an update, overlap-aware matching reduces weight movement, and parallel transfer rounds install the new layout while inference continues. Experiments on different models report prefill-time reductions of 17.00–24.69% relative to load-balanced placement with count-balanced request homes. The paper also evaluates migration mapping, transfer scheduling, and a 30-minute Code–Math–Code adaptation stream.
Critical strengths
-
The hypergraph formulation matches the stated communication semantics. Under node-coalesced dispatch, communication depends on the distinct nodes visited by the complete selected-expert set. Pairwise co-activation counts can lose this information. The paper gives concrete examples with identical pairwise statistics but different node-visit costs, providing a clear justification for the hypergraph representation.
-
The evaluation includes complete prefill timing and migration interference. Gains appear across three public models while preserving their underlying routers and expert computations. The migration study also measures foreground-service interference and final transfer drain. These choices make the evaluation more practically informative.
Critical weaknesses
-
The baseline coverage is insufficient for the complete-system claim. Disabling replication and replica-aware routing is appropriate for isolating placement, but it narrows the EPLB and GRACE-MoE comparisons. The Sem-MoE-style predictor comparison evaluates prediction recall rather than the complete scheduling system. So the reported results do not establish whether MoEAtlas improves on closely related complete systems under comparable resource budgets.
-
The evidence for general online-serving benefits remains narrow. The testbed uses one physical eight-H100 server divided into four logical nodes. Actual InfiniBand traffic is verified, but this does not reproduce all aspects of a physically distributed deployment. The principal timing experiments are also prefill-focused and use relatively short prompts. The adaptation study covers only one Code-Math-Code stream.
Questions for authors
Are the prefill gains preserved on physically separate nodes or across interconnect bandwidth and contention settings? A focused sensitivity study would help establish how dependent the results are on the current topology and communication path.
4 comments
W2 and the interconnect question — Do prefill gains persist across physical hosts or under different bandwidth and contention?
Thank you for asking whether the gains carry over to physically separate hosts. At the time of the original experiments, our group lacked a multi-host testbed, and the rental machines available to us did not provide RDMA. We therefore emulated four two-GPU nodes on one eight-H100 server and used actual InfiniBand between the logical groups. In response to your question, we obtained temporary access to four physical H100 servers through an industry partner for the deployment comparisons below.
Complete prefill. Both deployments use the same 512 OLMoE requests. Across three paired seeds, moving to four physical hosts reduced mean prefill time by 1.62% for EPLB+CB, 0.95% for HG+CB, and 0.74% for HG+PR. The gains over each deployment’s own EPLB+CB baseline differed by 0.57 and 0.73 percentage points for HG+CB and HG+PR, respectively (Appendix E.10, Table 28).
Batch-boundary layout update. This test isolates the effect of deployment by using the same layout transition, workload, and protocol on the emulated and physical setups. We pause inference at a batch boundary and resume with the first 64 requests once all weights are installed. The cap on additional PyTorch allocation is 32 GiB/GPU. For the first-batch-only baseline, the target expert layout is already in place when timing starts. We time only the processing of the same 64 requests.
With the target already selected, the layout was ready in 216.76 ms on one host and 215.47 ms on four physical hosts. The update plus the first 64 requests took 804.76 and 796.80 ms, respectively. Both timings differ by less than 1% across deployments (Appendix E.10, Table 30).
In Section 5.4, round-wise online migration runs alongside inference with an extra-memory budget of 256 MiB per GPU. The protocols answer different questions, so we compare timing and memory within each experiment.
Bidirectional InfiniBand. In a separate test with one H100 per node, we compared two logical nodes on one host with two physical hosts. With 64 and 128 MiB sent in each direction, transfer times changed by −0.07% and +0.21%, respectively (Appendix E.10, Table 29). Across the tested settings, single-host emulation closely tracks the corresponding physical deployments for prefill, batch-boundary layout updates, and bidirectional transfers.
W2 — Why does the timing evaluation focus on prefill?
Thank you for drawing attention to the scope of our timing experiments.
Prefill is typically compute-bound, so we represent each rank’s load by the total number of tokens its experts process. When MoE decode is limited by memory bandwidth, the number of distinct experts activated on each rank in a decode step is a proxy for the cost of reading expert weights. A decode extension could retain the complete expert sets in the hypergraph and use this per-step load measure.
Our timing study targets the prefill pool in a prefill–decode disaggregated deployment (Section 5.1). The two pools manage their expert layouts independently. A decode extension would model communication time at each decode step using the active batch’s remote nodes, packed traffic at each endpoint, and synchronization (Appendix E.11). Our implementation and timing evaluation focus on prefill.
W2 — Do the placement gains persist with longer prompts?
The original experiment used lengths that fit the context windows of all three models. In the revision, we evaluate natural code prefixes at 1K, 2K, 3K, 8K, 16K, and 32K tokens within each configuration’s context limit. With context limits of 4K for OLMoE and 8K for Qwen1.5-MoE, our reported tests reach 3K and 8K, respectively. The DeepSeek-V2-Lite results extend to 32K.
Table 31 in Appendix E.11 gives the complete-prefill timings. At 1K and 2K tokens, HG+CB reduces mean time by 18.27%–26.68% across the three models relative to EPLB+CB at the same length. At 32K on DeepSeek-V2-Lite, the reported means give reductions of 12.95% with HG+CB and 15.21% with HG+PR. Expert usage in longer prompts can vary with content, so load imbalance can become more pronounced.
W2 — A single Code–Math–Code trace leaves adaptation to other workload patterns unclear.
To evaluate MoEAtlas under different workload changes, we added Code/WikiText experiments covering abrupt shifts, gradual shifts, stable mixtures, and fast reversals. In each experiment, we measured the time to first token (TTFT) for every request. In Appendix E.1 of the revised manuscript, Figures 7–8 show the layout update timelines, and Table 16 summarizes the results for each experiment.
In the 30-minute cross-domain comparison, the three phases last 5, 15, and 10 minutes. MoEAtlas’s drift-triggered policy completes the two transfers at minutes 11.00 and 25.94 and breaks even 6.68 minutes after the first shift. Including search, transfer, and service on the stale layout after the return to Code, measured cumulative TTFT is 6.92 s lower than with the frozen layout and 2.77 s lower than with periodic updates.
In each short-shift or mixed-traffic control, all policies use the same requests and arrival times, at one 256-token request per second. Search requires drift above the search threshold in two consecutive windows.
-
Five minutes of WikiText, then back to Code. The change persists long enough to start one search. By validation, traffic has returned to Code, so the candidate is rejected and no weights move.
-
Gradual shift and stable mixture. Drift stays below the search threshold in both cases.
-
Switches every 60 seconds. After traffic switches back, drift falls below the threshold in the next window, so the two-window condition is never met.
Additional experiments in the revised manuscript
We also tested MoEAtlas on two newer MoE models. Compared with EPLB+CB, HG+PR shortened complete prefill by 15.95% on GLM-4.7-Flash and 12.85% on Qwen3.5-35B-A3B. On the same hypergraph layouts, enabling Pre-Router further reduces prefill time by 2.24% and 5.30%, respectively. Tables 17–18 in Appendix E.2 of the revised manuscript give the model configurations and timings.
Thanks for the author's reply. My concern has been addressed. I’ll keep my positive score.
Thank you for recognizing that the hypergraph formulation matches the communication semantics and that our evaluation accounts for complete prefill time and migration interference, and for raising the concerns we address below.
W1 — Placement and predictor controls leave performance against related systems under comparable budgets unclear.
We added two sets of comparisons.
Complete-prefill comparison with Sem-MoE and Director. We could not locate public implementations of either full system. We used the published descriptions to reconstruct Sem-MoE’s token–expert affinity, expert grouping, and whole-request assignment and to build a Director-style expert layout from observed routes. Table 4 in Section 5.3 compares prefill time, IB payload, and overall and worst-layer expert load on a common DeepSeek-V2-Lite backend. Appendix E.5.1 gives the implementation details and placement constraints. On the same 512 requests, relative to Sem-MoE with its own assignment and Director-style placement with count-balanced homes, HG+CB reduces complete prefill time by 20.91% and 24.04%, respectively. HG+PR reduces it by 22.36% and 25.43%. These results provide a reference comparison of the reconstructed strategies on the common backend.
Comparison with expert replicas. The original comparison used one copy per expert to isolate the effect of placement. Our hypergraph placement method extends directly to settings with expert replicas. We use an EPLB-style allocation rule and represent the original expert and each replica as separate vertices in the hypergraph.
Each additional replica goes to the expert with the highest load per copy at that step. Before building the hypergraph, we distribute that expert’s calls evenly across its original copy and replicas. Each hyperedge then represents the physical copies selected after this traffic split.
We compared replica-enabled EPLB, GRACE-MoE, HG+CB, and HG+PR on OLMoE, Qwen1.5-MoE, and DeepSeek-V2-Lite. All methods share a 40% cap on extra routed-expert weights. Table 5 in Section 5.3 reports the complete prefill times. HG+CB reduces time by 9.73%–25.49% versus EPLB and 1.15%–4.01% versus GRACE-MoE. The corresponding HG+PR reductions are 12.38%–26.60% and 2.31%–6.05%. Table 21 in Appendix E.5.2 breaks down the communication and load differences. HG+CB sends less data across nodes than EPLB and lowers worst-layer expert load relative to GRACE-MoE.
Figure 9 in Appendix E.5.2 reports a separate OLMoE sweep with five caps on extra expert weights. At equal actual expert-weight usage, HG+PR has 0.52–1.69% lower mean prefill time than HG+CB across the five settings. As actual extra weights increase from zero to 39.94%, HG+PR reduces mean complete prefill time from 4.788 s to 4.087 s (14.64%), and HG+CB reduces it from 4.813 s to 4.151 s (13.76%).