acceptodds
Under review as a conference paper at ICLR 2027

MoEAtlas: Orchestrating Communication in MoE Inference via Pre-Routing, Expert Placement, and Migration

Abstract

DeepSeek reports substantial communication overhead in mixture-of-experts (MoE) inference with expert parallelism across nodes. Each MoE layer sends tokens to GPUs holding their selected experts. Placing experts selected together on fewer nodes reduces transfers; choosing where each request runs makes more transfers local. We present MoEAtlas, which coordinates Pre-Router, hypergraph expert placement, and online migration to reduce communication in distributed MoE inference. The offline-trained Pre-Router predicts expert selections of prompt tokens across MoE layers and uses these predictions with the current expert layout to select a home GPU for each request. We represent each token's co-activated experts as a hyperedge and use hypergraph partitioning algorithms to produce new expert placements to reduce inter-node communication under load and GPU memory budgets. As workloads change, our migration strategy optimizes the mapping of the target expert placement to devices to reduce migration volume and time while inference continues. New hypergraph expert layouts with Pre-Router reduce complete prefill time by 19.33%, 17.00%, and 24.69% on OLMoE, Qwen1.5-MoE, and DeepSeek-V2-Lite, respectively, compared with the original layouts. On OLMoE, min-volume mapping reduces inter-node weight-transfer volume by 11.95% relative to fixed mapping; online migration reduces the total time to serve requests and complete a layout update by 6.83% relative to blocking migration.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Communication Optimization, Request Scheduling, Hypergraph Expert Placement, Online Expert Migration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ygDHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoEAtlas coordinates request placement, expert placement, and online expert migration to reduce communication in distributed MoE inference. An offline-trained Pre-Router predicts prompt-token expert selections across layers and uses those predictions to choose each request’s home GPU. Expert placement represents complete co-activation sets as hyperedges and minimizes connectivity-minus-one under load constraints When workload drift warrants an update, overlap-aware matching reduces weight movement, and parallel transfer rounds install the new layout while inference continues. Experiments on different models report prefill-time reductions of 17.00–24.69% relative to load-balanced placement with count-balanced request homes. The paper also evaluates migration mapping, transfer scheduling, and a 30-minute Code–Math–Code adaptation stream.

Critical strengths
  1. The hypergraph formulation matches the stated communication semantics. Under node-coalesced dispatch, communication depends on the distinct nodes visited by the complete selected-expert set. Pairwise co-activation counts can lose this information. The paper gives concrete examples with identical pairwise statistics but different node-visit costs, providing a clear justification for the hypergraph representation.

  2. The evaluation includes complete prefill timing and migration interference. Gains appear across three public models while preserving their underlying routers and expert computations. The migration study also measures foreground-service interference and final transfer drain. These choices make the evaluation more practically informative.

Critical weaknesses
  1. The baseline coverage is insufficient for the complete-system claim. Disabling replication and replica-aware routing is appropriate for isolating placement, but it narrows the EPLB and GRACE-MoE comparisons. The Sem-MoE-style predictor comparison evaluates prediction recall rather than the complete scheduling system. So the reported results do not establish whether MoEAtlas improves on closely related complete systems under comparable resource budgets.

  2. The evidence for general online-serving benefits remains narrow. The testbed uses one physical eight-H100 server divided into four logical nodes. Actual InfiniBand traffic is verified, but this does not reproduce all aspects of a physically distributed deployment. The principal timing experiments are also prefill-focused and use relatively short prompts. The adaptation study covers only one Code-Math-Code stream.

Questions for authors

Are the prefill gains preserved on physically separate nodes or across interconnect bandwidth and contention settings? A focused sensitivity study would help establish how dependent the results are on the current topology and communication path.

4 comments

Thank you for recognizing that the hypergraph formulation matches the communication semantics and that our evaluation accounts for complete prefill time and migration interference, and for raising the concerns we address below.

W1 — Placement and predictor controls leave performance against related systems under comparable budgets unclear.

We added two sets of comparisons.

Complete-prefill comparison with Sem-MoE and Director. We could not locate public implementations of either full system. We used the published descriptions to reconstruct Sem-MoE’s token–expert affinity, expert grouping, and whole-request assignment and to build a Director-style expert layout from observed routes. Table 4 in Section 5.3 compares prefill time, IB payload, and overall and worst-layer expert load on a common DeepSeek-V2-Lite backend. Appendix E.5.1 gives the implementation details and placement constraints. On the same 512 requests, relative to Sem-MoE with its own assignment and Director-style placement with count-balanced homes, HG+CB reduces complete prefill time by 20.91% and 24.04%, respectively. HG+PR reduces it by 22.36% and 25.43%. These results provide a reference comparison of the reconstructed strategies on the common backend.

Comparison with expert replicas. The original comparison used one copy per expert to isolate the effect of placement. Our hypergraph placement method extends directly to settings with expert replicas. We use an EPLB-style allocation rule and represent the original expert and each replica as separate vertices in the hypergraph.

Each additional replica goes to the expert with the highest load per copy at that step. Before building the hypergraph, we distribute that expert’s calls evenly across its original copy and replicas. Each hyperedge then represents the physical copies selected after this traffic split.

We compared replica-enabled EPLB, GRACE-MoE, HG+CB, and HG+PR on OLMoE, Qwen1.5-MoE, and DeepSeek-V2-Lite. All methods share a 40% cap on extra routed-expert weights. Table 5 in Section 5.3 reports the complete prefill times. HG+CB reduces time by 9.73%–25.49% versus EPLB and 1.15%–4.01% versus GRACE-MoE. The corresponding HG+PR reductions are 12.38%–26.60% and 2.31%–6.05%. Table 21 in Appendix E.5.2 breaks down the communication and load differences. HG+CB sends less data across nodes than EPLB and lowers worst-layer expert load relative to GRACE-MoE.

Figure 9 in Appendix E.5.2 reports a separate OLMoE sweep with five caps on extra expert weights. At equal actual expert-weight usage, HG+PR has 0.52–1.69% lower mean prefill time than HG+CB across the five settings. As actual extra weights increase from zero to 39.94%, HG+PR reduces mean complete prefill time from 4.788 s to 4.087 s (14.64%), and HG+CB reduces it from 4.813 s to 4.151 s (13.76%).

W2 and the interconnect question — Do prefill gains persist across physical hosts or under different bandwidth and contention?

Thank you for asking whether the gains carry over to physically separate hosts. At the time of the original experiments, our group lacked a multi-host testbed, and the rental machines available to us did not provide RDMA. We therefore emulated four two-GPU nodes on one eight-H100 server and used actual InfiniBand between the logical groups. In response to your question, we obtained temporary access to four physical H100 servers through an industry partner for the deployment comparisons below.

Complete prefill. Both deployments use the same 512 OLMoE requests. Across three paired seeds, moving to four physical hosts reduced mean prefill time by 1.62% for EPLB+CB, 0.95% for HG+CB, and 0.74% for HG+PR. The gains over each deployment’s own EPLB+CB baseline differed by 0.57 and 0.73 percentage points for HG+CB and HG+PR, respectively (Appendix E.10, Table 28).

Batch-boundary layout update. This test isolates the effect of deployment by using the same layout transition, workload, and protocol on the emulated and physical setups. We pause inference at a batch boundary and resume with the first 64 requests once all weights are installed. The cap on additional PyTorch allocation is 32 GiB/GPU. For the first-batch-only baseline, the target expert layout is already in place when timing starts. We time only the processing of the same 64 requests.

With the target already selected, the layout was ready in 216.76 ms on one host and 215.47 ms on four physical hosts. The update plus the first 64 requests took 804.76 and 796.80 ms, respectively. Both timings differ by less than 1% across deployments (Appendix E.10, Table 30).

In Section 5.4, round-wise online migration runs alongside inference with an extra-memory budget of 256 MiB per GPU. The protocols answer different questions, so we compare timing and memory within each experiment.

Bidirectional InfiniBand. In a separate test with one H100 per node, we compared two logical nodes on one host with two physical hosts. With 64 and 128 MiB sent in each direction, transfer times changed by −0.07% and +0.21%, respectively (Appendix E.10, Table 29). Across the tested settings, single-host emulation closely tracks the corresponding physical deployments for prefill, batch-boundary layout updates, and bidirectional transfers.

W2 — Why does the timing evaluation focus on prefill?

Thank you for drawing attention to the scope of our timing experiments.

Prefill is typically compute-bound, so we represent each rank’s load by the total number of tokens its experts process. When MoE decode is limited by memory bandwidth, the number of distinct experts activated on each rank in a decode step is a proxy for the cost of reading expert weights. A decode extension could retain the complete expert sets in the hypergraph and use this per-step load measure.

Our timing study targets the prefill pool in a prefill–decode disaggregated deployment (Section 5.1). The two pools manage their expert layouts independently. A decode extension would model communication time at each decode step using the active batch’s remote nodes, packed traffic at each endpoint, and synchronization (Appendix E.11). Our implementation and timing evaluation focus on prefill.

W2 — Do the placement gains persist with longer prompts?

The original experiment used lengths that fit the context windows of all three models. In the revision, we evaluate natural code prefixes at 1K, 2K, 3K, 8K, 16K, and 32K tokens within each configuration’s context limit. With context limits of 4K for OLMoE and 8K for Qwen1.5-MoE, our reported tests reach 3K and 8K, respectively. The DeepSeek-V2-Lite results extend to 32K.

Table 31 in Appendix E.11 gives the complete-prefill timings. At 1K and 2K tokens, HG+CB reduces mean time by 18.27%–26.68% across the three models relative to EPLB+CB at the same length. At 32K on DeepSeek-V2-Lite, the reported means give reductions of 12.95% with HG+CB and 15.21% with HG+PR. Expert usage in longer prompts can vary with content, so load imbalance can become more pronounced.

W2 — A single Code–Math–Code trace leaves adaptation to other workload patterns unclear.

To evaluate MoEAtlas under different workload changes, we added Code/WikiText experiments covering abrupt shifts, gradual shifts, stable mixtures, and fast reversals. In each experiment, we measured the time to first token (TTFT) for every request. In Appendix E.1 of the revised manuscript, Figures 7–8 show the layout update timelines, and Table 16 summarizes the results for each experiment.

In the 30-minute cross-domain comparison, the three phases last 5, 15, and 10 minutes. MoEAtlas’s drift-triggered policy completes the two transfers at minutes 11.00 and 25.94 and breaks even 6.68 minutes after the first shift. Including search, transfer, and service on the stale layout after the return to Code, measured cumulative TTFT is 6.92 s lower than with the frozen layout and 2.77 s lower than with periodic updates.

In each short-shift or mixed-traffic control, all policies use the same requests and arrival times, at one 256-token request per second. Search requires drift above the search threshold in two consecutive windows.

  • Five minutes of WikiText, then back to Code. The change persists long enough to start one search. By validation, traffic has returned to Code, so the candidate is rejected and no weights move.

  • Gradual shift and stable mixture. Drift stays below the search threshold in both cases.

  • Switches every 60 seconds. After traffic switches back, drift falls below the threshold in the next window, so the two-window condition is never met.

Additional experiments in the revised manuscript

We also tested MoEAtlas on two newer MoE models. Compared with EPLB+CB, HG+PR shortened complete prefill by 15.95% on GLM-4.7-Flash and 12.85% on Qwen3.5-35B-A3B. On the same hypergraph layouts, enabling Pre-Router further reduces prefill time by 2.24% and 5.30%, respectively. Tables 17–18 in Appendix E.2 of the revised manuscript give the model configurations and timings.

Thanks for the author's reply. My concern has been addressed. I’ll keep my positive score.

Reviewer_jNpcOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MoEAtlas reduces distributed MoE communication by coordinating request-home selection, expert placement, and online migration while preserving the target model’s logical routing decisions and expert computations. An offline-trained Pre-Router uses frozen early model features and parallel prediction heads to predict prompt-token expert selections across layers. These predictions guide request-home selection under the current layout. Complete co-activation sets are represented as weighted hyperedges, and constrained partitioning produces candidate expert layouts. Persistent workload drift triggers layout search, followed by overlap-aware device matching and parallel transfer rounds that install the selected layout while inference continues.

The evaluation uses OLMoE-1B-7B, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite on eight H100 GPUs organized as four logical nodes with real InfiniBand communication. The combined placement and Pre-Router configuration reduces complete prefill time by 17.00–24.69% relative to load-only placement with count-balanced homes. On OLMoE, online migration reduces the combined time for serving 512 prefills and completing a layout update by 6.83% relative to blocking migration, while increasing foreground batch intervals during transfer.

Critical strengths

S1. A well-motivated communication representation that exposes a genuine limitation of pairwise statistics

Section 3 separates remote communication into an expert-grouping term and a request-home term, clarifying why placement and request scheduling are coupled. Figure 2(a) and Appendix B.1 then construct routing records with identical expert loads and pairwise co-activation counts but different node-visit costs under the same layout. This establishes a concrete limitation of pairwise aggregation and motivates retaining complete Top-K sets.

The resulting weighted KM1 objective corresponds to the grouping component of node-coalesced communication, while load limits and device-memory checks prevent unrestricted communication minimization from simply concentrating work. The paper also distinguishes this objective from the execution-time estimator used to choose homes and layouts. This modeling clarity is consequential to my positive assessment: the work identifies which information an optimizer needs and connects that information to an implemented placement mechanism.

S2. Meaningful implementation evidence, with careful timing boundaries and appropriately scoped claims The complete-prefill evaluation measures more than communication volume or isolated kernels. It includes prediction, home selection, input movement, execution, and drain, using paired configurations, repeated timings, and rotated execution order. Table 3 separates the effects of hypergraph placement and Pre-Router, showing that placement provides most of the improvement and that prediction adds a further 2.31–2.57%. This is useful evidence of net benefit after the scheduling overhead is included.

The migration evaluation likewise distinguishes transfer time, foreground interference, and total completion time. The matching and round-count guarantees are explicitly limited to a fixed target layout and specified endpoint constraints, rather than presented as global serving optimality. Numerical-agreement and transferred-weight checks provide additional support for the intended semantics-preserving execution, although they are not a comprehensive downstream-quality evaluation.

Critical weaknesses

W1. The experiments do not sufficiently isolate the contribution of complete-set modeling or establish the advantage over full related systems The hypergraph and pairwise controls differ in more than the information they retain. Appendix D.3.1 uses balanced Kernighan–Lin bisection, group-size-preserving refinement, and count-capped packing for the pairwise control. MoEAtlas instead uses explicit load limits, KaHyPar, a slack sweep, and more flexible per-layer expert counts with cumulative storage balancing. Consequently, the observed advantage may combine better representation, different feasible layouts, stronger load control, and a different search procedure.

Table 10 illustrates why this distinction matters. On DeepSeek-V2-Lite, pairwise and hypergraph placement have the same rounded test KM1 of 1.404, while the normalized node peak improves from 1.311 to 1.097. This supports the effectiveness of the complete optimization procedure, but does not independently establish a communication benefit from higher-order information. The constructed counterexamples establish that pairwise statistics can be insufficient; they do not quantify how much that insufficiency matters in the measured workloads.

The scope of the named baselines also needs care. Disabling replicas in EPLB and GRACE-MoE is a legitimate placement-only control, but it does not compare against their complete capabilities. GRACE-MoE specifically uses replication to correct grouping-induced load skew. Likewise, Sem-MoE already couples model placement and request/token scheduling, and Director combines predictive placement with live migration. These systems are not identical to MoEAtlas, but they remain relevant to its complete-system contribution. A stronger comparison under a common total resource budget, or a narrower system-level claim, is needed.

W2. The practical operating range and generalization of the complete adaptive pipeline remain insufficiently characterized

The online experiment demonstrates that the pipeline can adapt, but its responsiveness is limited by detection and search. Appendix D.5.3 reports 208- and 222-second searches; including detection, the new layouts become available roughly six minutes after the corresponding workload changes. This may be appropriate for slowly varying traffic, but the paper does not establish the workload-stability horizon over which the update becomes worthwhile after accounting for search resources, stale-layout execution, and migration interference. Moreover, Figure 4’s cumulative savings integrate window-level P95 differences; they are not the sum of actual per-request latency savings.

The main inference evidence is also concentrated on cache-miss prefills with relatively short prompts. Pre-Router’s additional complete-prefill benefit beyond hypergraph placement is approximately 2.3–2.6%, so the case for the full pipeline would benefit from clearer evidence of when this incremental gain and online adaptation justify their added machinery. This is not a claim that the measured gain is negligible; rather, its useful operating range is not yet sufficiently established.

Finally, the predictor capacity and home-decision controls use source-file-disjoint splits, whereas the main system runs use a request-level split that may contain fragments from the same source across partitions. This does not establish duplicate-request leakage, which the authors explicitly rule out, but it leaves cross-source generalization of the end-to-end result less well supported than the predictor study.

Questions for authors

Q1. About contribution of the hypergraph representation

Can the pairwise and hypergraph approaches be compared under matched load and memory constraints, comparable per-layer placement flexibility, and a comparable search budget? Please report communication cost, load balance, and complete-prefill timing on the same held-out requests.

This is the most important additional evidence for my assessment: it would distinguish the value of complete-set information from improvements attributable to constraints and optimization. Separately, can a representative end-to-end comparison retain the relevant replica-aware mechanisms of a strong baseline under a common total memory budget, or can the claimed comparison scope be explicitly restricted to replica-free placement?

Q2. About net value and timescale of online adaptation

What workload-stability duration or amount of subsequent traffic is needed for an accepted update to repay its costs? Please distinguish search-resource consumption, latency while the old layout remains active, migration interference, and the subsequent steady-state benefit.

The existing 30-minute trace could support a useful analysis without requiring a new deployment: please report actual request-level latency outcomes over the full trace, alongside the windowed P95 results, and explain how the update-admission rule accounts for—or intentionally omits—the transition cost. This would clarify when the online mechanism improves the service as a whole rather than only producing a better eventual layout.

Q3. About generalization

Can the principal complete-prefill comparison be repeated using source-file-disjoint training, profiling, validation, and test requests, rather than only request-level separation?

Please separate the gain from hypergraph placement from the additional gain due to Pre-Router on that split. Since the paper already provides source-file-separated predictor controls, extending a representative system comparison would directly address whether the measured end-to-end benefit transfers to new source documents.

Q4. About fairness

Appendix D.5.1 states that Greedy admits up to three compatible remote moves per round, while the four-node setup permits up to four ports to send concurrently. Is the three-move limit imposed by a resource constraint shared with Matching, or is it an additional restriction on the baseline?

If it is not a shared constraint, how much of Matching’s improvement remains against a greedy scheduler that fills all available endpoint-compatible transfers? This would clarify the independent contribution of the transfer-round construction.

3 comments

Thank you for recognizing our formulation of node-coalesced communication, particularly how complete expert sets preserve information lost by pairwise aggregation, and for noting the care taken in timing complete prefill and online migration. We address each of your concerns below.

W1 and Q1 — How much does complete-set modeling help under matched constraints and comparable search budgets?

As the reviewer points out, Table 12 in Appendix D.3.2 of the revised manuscript (Table 10 in the original) compares two expert layouts for DeepSeek, one generated by the pairwise method and the other by the hypergraph method. When evaluated with the same KM1 metric, both layouts score 1.404 rounded to three decimal places, and the hypergraph layout has a lower peak node load. To see whether retaining complete expert sets changes how layouts are ranked, we score the same candidate layouts with the pairwise and complete-set objectives and compare the resulting rankings. We compare DeepSeek's 19 candidate layouts two at a time. The objectives rank the layouts in opposite order in 18 of the 171 comparisons, including 5 of the 34 comparisons between layouts with similar loads. For each model, both objectives select the same best layout from the fixed candidate pool. See Table 19 in Appendix E.4.

On OLMoE, we use the pairwise and complete-set methods to obtain one expert layout each under the same constraints. Both runs use the same KaHyPar preset, per-layer load and memory limits, and a 60 s per-layer search allowance. On the same 512 held-out requests with fixed home GPUs, complete sets lower KM1 from 2.183 to 2.096 and reduce inter-node sends by 2.67%, with peak node load near 1.10. The measured complete-prefill time decreases from 4.316 s with pairwise counts to 4.148 s with complete sets, a reduction of 168 ms (about 3.89%). Table 20 in Appendix E.4 gives the matched constraints and communication and load metrics.

W1 and Q1 — Do the gains hold against related baselines when replica mechanisms and resource budgets are accounted for?

We added two sets of comparisons.

Complete-prefill comparison with Sem-MoE and Director. We could not locate public implementations of either full system. We used the published descriptions to reconstruct Sem-MoE's token–expert affinity, expert grouping, and whole-request assignment and to build a Director-style expert layout from observed routes. Table 4 in Section 5.3 compares prefill time, IB payload, and overall and worst-layer expert load on a common DeepSeek-V2-Lite backend. Appendix E.5.1 gives the implementation details and placement constraints. On the same 512 requests, relative to Sem-MoE with its own assignment and Director-style placement with count-balanced homes, HG+CB reduces complete prefill time by 20.91% and 24.04%, respectively. HG+PR reduces it by 22.36% and 25.43%. These results provide a reference comparison of the reconstructed strategies on the common backend.

Comparison with expert replicas. The original comparison used one copy per expert to isolate the effect of placement. Our hypergraph placement method extends directly to settings with expert replicas. We use an EPLB-style allocation rule and represent the original expert and each replica as separate vertices in the hypergraph.

Each additional replica goes to the expert with the highest load per copy at that step. Before building the hypergraph, we distribute that expert's calls evenly across its original copy and replicas. Each hyperedge then represents the physical copies selected after this traffic split.

We compared replica-enabled EPLB, GRACE-MoE, HG+CB, and HG+PR on OLMoE, Qwen1.5-MoE, and DeepSeek-V2-Lite. All methods share a 40% cap on extra routed-expert weights. Table 5 in Section 5.3 reports the complete prefill times. HG+CB reduces time by 9.73%–25.49% versus EPLB and 1.15%–4.01% versus GRACE-MoE. The corresponding HG+PR reductions are 12.38%–26.60% and 2.31%–6.05%. Table 21 in Appendix E.5.2 breaks down the communication and load differences. HG+CB sends less data across nodes than EPLB and lowers worst-layer expert load relative to GRACE-MoE.

Figure 9 in Appendix E.5.2 reports a separate OLMoE sweep with five caps on extra expert weights. At equal actual expert-weight usage, HG+PR has 0.52–1.69% lower mean prefill time than HG+CB across the five settings. As actual extra weights increase from zero to 39.94%, HG+PR reduces mean complete prefill time from 4.788 s to 4.087 s (14.64%), and HG+CB reduces it from 4.813 s to 4.151 s (13.76%).

W2 and Q2 — How long must a workload stay stable for per-request savings to offset update costs?

We measured per-request TTFT over the full 30-minute Code–Math–Code trace. We compared a fixed layout with MoEAtlas's drift-triggered update policy using the same requests and arrival times. Across 1 742 paired requests, mean TTFT fell from 138.97 to 133.83 ms, and trace-wide P95 changed from 170.29 to 170.31 ms. Table 22 and Figure 10 in Appendix E.6 summarize the trace-wide metrics and the cumulative TTFT comparison.

We calculate each update's break-even time against a baseline that keeps the old layout. In the paired measurements, the Code-to-Math update breaks even 6.17 minutes after the workload shift, after serving 10 requests on the new layout. The corresponding Math-to-Code values are 8.41 minutes and 136 requests. The admission check uses newly arrived requests to assess candidate feasibility and the savings after activation. The break-even calculation includes the additional waiting caused by search and migration. Table 23 in Appendix E.6 gives the cost breakdown for both shifts and reports the CPU resources used for search.

W2 — When do Pre-Router and online migration provide a net benefit?

We assess Pre-Router by the net time saved over complete prefill and online migration by how long it takes subsequent requests to offset the update cost.

Pre-Router. For the 512 OLMoE requests, the pipeline breakdown in Table 24 of Appendix E.7 attributes 104.4 ms of savings to better home assignment. The first batch spends 4.9 ms on prediction and preparation, 1.5 ms exchanging predicted token counts across GPUs, and 2.0 ms selecting home GPUs. In our measurements, each later batch was ready before the preceding batch's prefill finished, so only the first batch's 8.4 ms of startup overhead added to the total time. The net saving is 96 ms, or 2.31% of the 4.148 s baseline. Table 3 in Section 5.3 reports incremental gains of 2.31%–2.57% across the three models.

Online migration. The new workload needs to persist long enough for the new layout's savings to cover search and transfer costs. In the per-request measurements, using the old layout at each shift as the baseline, Code to Math pays back in about 6.17 minutes and Math to Code in about 8.41 minutes. Both intervals start at the workload shift. See Table 23 in Appendix E.6.

W2 and Q3 — Do placement and Pre-Router still help when all four splits use disjoint source files?

The request-level split used in Table 3 of Section 5.3 allowed different fragments of a source file to appear in different groups. To test OLMoE on unseen source files, we now assign each file and all requests derived from it to exactly one group, used for training, layout profiling, validation, or testing. We verified zero overlap in source files and requests across the groups. Table 25 in Appendix E.8 gives the size and purpose of each group.

We retrained the predictor on the training split and built the layouts using the independent profiling split. All three configurations used the same 512 test requests. On this source-file-disjoint split, hypergraph placement reduced measured complete prefill time from 5.060 to 4.180 s (17.4%) with count-balanced home assignment fixed. On the same hypergraph layout, enabling Pre-Router further reduced time to 4.084 s (2.3%), including prediction and scheduling overhead. Table 26 in Appendix E.8 gives the three configurations and timing boundary.

Q4 — How much does Matching save when Greedy can fill all endpoint-compatible slots in a round?

Thank you for pointing out the three-move cap. It was an additional restriction on Greedy in the original comparison. We removed the cap and reran the comparison with both schedulers subject to the same endpoint and receive-buffer constraints. On four nodes, these constraints allow up to four compatible transfers per remote round. The revised Greedy scans pending moves in fixed order and fills available compatible slots.

With min-volume mapping and the same expert weights fixed, we timed the complete transfer for both schedulers on the same 21 layout pairs under these shared constraints. Matching reduced transfer time by 12.46% on average. Appendix E.9 gives the shared scheduling rules and reports the timing comparison in Table 27.

Additional experiments in the revised manuscript

We also tested MoEAtlas on two newer MoE models. Compared with EPLB+CB, HG+PR shortened complete prefill by 15.95% on GLM-4.7-Flash and 12.85% on Qwen3.5-35B-A3B. On the same hypergraph layouts, enabling Pre-Router further reduces prefill time by 2.24% and 5.30%, respectively. Tables 17–18 in Appendix E.2 of the revised manuscript give the model configurations and timings.

Reviewer_jNpcOpenReview ↗

Official Comment

The authors have addressed parts of my concerns. I have no more questions. My rating would be preserved.

Reviewer_gZJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MoEAtlas, a system for reducing communication overhead in distributed MoE inference by jointly coordinating three mechanisms: (1) a learned Pre-Router that predicts per-token expert selections before actual routing and uses these predictions to select a request's home GPU, (2) hypergraph-based expert placement that groups complete top-k co-activation sets, and (3) online expert migration that updates the expert layout in the background while serving continues.

Critical strengths

The paper identifies an important coupling between request placement and expert placement. The system is designed to preserve model semantics. The actual model router and expert computation remain unchanged. The Pre-Router only influences request placement. The evaluation contains useful component-level analyses. The paper separately evaluates Pre-Router prediction quality, home selection, hypergraph placement, migration mapping. The evaluation is sound and convincable.

Critical weaknesses
  1. The proposed online expert migration may be too slow to respond to realistic workload changes. In the Code–Math–Code experiment, the workload switches every several minutes, but practically the task types of workloads may change much faster, where the effectiveness could be marginal. More importantly, the layout searches take hundreds of seconds, which might be harmful in the case that the task type frequently changes.
  2. The evaluation relies on relatively old MoE models (e.g. Qwen1.5-MoE-A2.7B, and DeepSeek-V2) and does not sufficiently demonstrate relevance to current large-scale MoE serving. The effectiveness when the parameter size of experts becomes larger is unclear.
  3. The presentation makes the core contribution hard to follow. The paper introduces several cost functions, constraints and formulations, but the motivation behind these formulations is not always sufficiently clear before the mathematical definitions are introduced.
Questions for authors
  1. How does MoEAtlas perform when workload type changes on a much shorter timescale? Will the effectiveness remain?
  2. Can the authors evaluate MoEAtlas on more recent and substantially larger-scale MoE models? Will the overhead of migration become unacceptable in this case?
5 comments

Thank you for recognizing the coupling between request and expert placement, the preservation of model semantics, and the component-level evaluation, and for raising the questions we address below.

W1 and Q1 — What happens when workloads change faster than a layout search can finish?

MoEAtlas updates expert layouts in response to sustained workload drift. During rapid reversals, the controller limits unnecessary search and weight transfers. It starts a search only after detecting persistent drift, continues serving on the current layout during search, and checks the candidate on newly arrived requests before moving weights.

Section 5.5 describes two new Code/WikiText experiments after Figure 4. Each experiment ran for 15 minutes at one request per second, and we recorded the time to first token (TTFT) for every request. In Appendix E.1, Table 16 summarizes both controls and the left column of Figure 8(d) shows the search and rejection timeline for the five-minute shift.

  • New control — 60-second alternation. After five minutes of Code traffic, Code and WikiText alternate every minute. No two consecutive windows meet the search condition, so the system starts no search and moves no weights. The update mechanism yields zero TTFT gain or overhead relative to the fixed layout in this test.

  • New control — five-minute WikiText shift. The Code, WikiText, and return-to-Code phases each last five minutes. The shift triggers one search, but the candidate is rejected after traffic returns to Code, so no weights move. Including search interference, summed TTFT is about 0.12% higher than with the fixed layout.

  • Original experiment — 30-minute Code–Math–Code stream. The new layout starts serving about 6.03 minutes after the Code-to-Math shift. Detection and background search account for most of this interval, and weight transfer takes 2.54 s (Appendix E.6, Table 23). Relative to keeping the old layout, the update breaks even 6.17 minutes after the shift, after serving 10 requests on the new layout. Figure 11 shows the payback sensitivity.

W2 and Q2 — Do the gains carry over to newer, larger MoE models, and how costly is migration?

We evaluated GLM-4.7-Flash (46 MoE layers, 64 routed experts per layer) and Qwen3.5-35B-A3B (40 MoE layers, 256 routed experts per layer). For each model, we measured complete prefill on the same 512 requests with EPLB and hypergraph placement using count-balanced homes (EPLB+CB and HG+CB), and with Pre-Router on the same hypergraph layout (HG+PR). Appendix E.2 of the revised manuscript gives the model configurations in Table 17 and the timings in Table 18. Relative to EPLB+CB, HG+CB reduces complete prefill time by 14.01% on GLM and 7.95% on Qwen3.5. Adding Pre-Router further reduces the reported times by 2.24% and 5.30%, respectively, bringing the total reductions relative to EPLB+CB to 15.95% and 12.85%.

The two new evaluations show that the prefill gains from hypergraph placement and Pre-Router carry over to GLM-4.7-Flash and Qwen3.5. Both models can use the online migration scheme described in Section 4.3 of the paper. The scheme schedules transfers of expert weights between the old and new layouts, and migration time depends primarily on the weight bytes actually transferred and the bandwidth of the bottleneck endpoint. We therefore focused the new experiments on validating prefill gains and did not repeat the online migration experiments on these two models.

Migration scaling on four nodes. Let SS be the total routed-expert weight with one copy per expert. Averaging over all mappings of target groups to nodes retains S/4S/4 on average, so maximum-overlap matching retains at least S/4S/4 and bounds cross-node transfer volume by Vmig≤3S/4V_{\mathrm{mig}}\leq 3S/4. If both layouts store equal weight bytes on each of the four nodes, each node sends or receives at most S/4S/4 in either direction. With the expert sizes and precisions in Table 17, GLM-4.7-Flash (BF16) and Qwen3.5-35B-A3B (FP32) have 51.75 and 120 GiB of routed-expert weights. The corresponding cross-node volume bounds are about 38.81 and 90 GiB, and the per-node bounds in each direction are about 12.94 and 30 GiB.

For expert size ss and Δ\Delta parallel transfer rounds, let bmin⁡b_{\min} be the effective bandwidth floor per active send or receive direction, δmax⁡\delta_{\max} the per-round bound on scheduling and safe handoff, and T0T_0 the bound on local transfers and fixed overhead. If the full-duplex schedule fits in memory, then

Tmig≤Δ(sbmin⁡+δmax⁡)+T0\displaystyle T_{\mathrm{mig}}\leq\Delta\left(\frac{s}{b_{\min}}+\delta_{\max}\right)+T_0.

Under the balance assumption, the schedule in Section 4.3 gives Δs≤S/4\Delta s\leq S/4. At an assumed bmin⁡=10b_{\min}=10 GB/s, the bandwidth term is bounded by about 1.39 s for GLM-4.7-Flash and 3.22 s for Qwen3.5-35B-A3B. The remaining terms account for per-round handoff and local transfers.

In the existing OLMoE drift experiment, Table 23 reports about 2.54 s for weight transfer, 150 s for detection, and 208 s for background search. Detection and search dominate the time to layout activation in that trace. Future work could prioritize foreground communication, use finer-grained weight transfers, and allocate bandwidth dynamically to interleave migration with dispatch/combine and KV-cache transfers for context parallelism and prefill–decode disaggregation. This would use spare RDMA capacity and overlap transfers with computation to balance foreground latency with update time.

W3 — The design choices need motivation before the cost functions and constraints are introduced.

We agree with this concern. In several places, the original manuscript introduced equations before clearly explaining the system problems they address, so readers had to infer the motivation from later text. The revised Sections 3 and 4.1–4.3 explain what each design choice is meant to achieve before presenting the corresponding equations.

  • Section 3 — Cost model. An example shows how two experts on the same remote node share a token dispatch. This introduces the node visits used to generate placement candidates and explains why home selection uses communication time, while layout selection also accounts for expert computation.

  • Section 4.1 — Pre-Router. Exact deep-layer routes require executing the preceding layers. This motivates sharing shallow features across parallel prediction heads and leads into the frontend and head equations.

  • Section 4.2, Figure 2(a) — Hypergraph placement. The example shows why identical pairwise counts can yield different numbers of remote sends, motivating the use of complete expert sets. Before defining the placement objective, we explain how the load cap limits hot spots and how the load anchor and slack τ\tau guide the search for candidates with fewer node visits.

  • Section 4.3 — Online migration. A two-group example shows how assigning each group to a node already holding some of its weights reduces transfers. This introduces two objectives, the total bytes moved and the migration-time lower bound set by the bottleneck endpoint.

Additional experiments in the revised manuscript

We also added three sets of experiments to evaluate MoEAtlas against related baselines, with expert replicas, and on physically separate hosts.

  • System-level comparisons with Sem-MoE and Director. Building on the component-level comparisons in the original manuscript, we followed the published descriptions to reproduce Sem-MoE's expert grouping and request assignment and Director's expert placement on a common inference backend. We then measured complete prefill time. On DeepSeek-V2-Lite, MoEAtlas (HG+PR) reduces complete prefill time by 22.36% and 25.43%, respectively, relative to these two reproduced baselines. See Section 5.3, Table 4, and Appendix E.5.1 of the revised manuscript.

  • Comparison with expert replicas enabled. We extended hypergraph placement to support replicas by treating the original expert and each replica as separate vertices and distributing expert calls among them before building the hypergraph. Across three models, MoEAtlas (HG+PR) reduces complete prefill time by 12.38%–26.60% relative to replica-enabled EPLB and by 2.31%–6.05% relative to replica-enabled GRACE-MoE. See Section 5.3, Table 5, and Appendix E.5.2.

  • Validation on four physical hosts. We measured OLMoE complete prefill on four physical H100 hosts. MoEAtlas (HG+PR) reduces complete prefill time by 17.07% relative to EPLB+CB on those hosts. In a separate test that pauses inference at a batch boundary, layout-ready time differs by less than 1% between single-host emulation and four physical hosts under the same update protocol. See Appendix E.10, Tables 28 and 30.

Comments on the paper

1 comment

We thank the area chair and reviewers for their feedback. MoEAtlas targets distributed MoE prefill. It uses each token’s complete set of co-activated experts to guide expert placement, uses Pre-Router to select a home GPU for each request, and updates the expert layout in response to sustained workload changes. Sections 3 and 4.1–4.3 of the revision explain the motivation for these choices. We also expanded the evaluation in four areas.

  • Isolating the effect of complete expert sets and testing end-to-end generalization. In the OLMoE comparison with matched load and memory constraints, the same search budget, and fixed request homes, complete-set modeling reduces communication and shortens complete prefill by 3.89% relative to pairwise counts (Appendix E.4, Table 20). With source files disjoint across training, layout profiling, validation, and testing, we retrained the predictor and rebuilt the layouts. Hypergraph placement shortens complete prefill by 17.4%, and Pre-Router gives a further 2.3% net saving on the same layout (Appendix E.8, Tables 25–26).

  • Comparisons with Sem-MoE, Director, and replica-enabled baselines. On a common inference backend, we reconstructed Sem-MoE’s expert grouping and request assignment and Director-style expert placement from their published descriptions. On DeepSeek-V2-Lite, MoEAtlas shortens complete prefill by 22.36% and 25.43%, respectively, relative to configurations using these strategies (Table 4, Appendix E.5.1). We also enabled expert replicas and replica-aware dispatch and matched actual resident expert-weight costs. Across three models, MoEAtlas shortens complete prefill by 12.38%–26.60% versus replica-enabled EPLB and 2.31%–6.05% versus replica-enabled GRACE-MoE (Table 5, Appendix E.5.2). These comparisons concern the strategies evaluated on the common backend.

  • New models, four physical hosts, and longer inputs. Compared with EPLB+CB, MoEAtlas shortens complete prefill by 15.95% on GLM-4.7-Flash and 12.85% on Qwen3.5-35B-A3B (Appendix E.2). On four physical H100 hosts, MoEAtlas shortens OLMoE prefill by 17.07% against the same deployment’s EPLB+CB baseline (Appendix E.10). Long-input experiments extend the DeepSeek-V2-Lite evaluation to 32K tokens (Appendix E.11).

  • Net savings and payback time for online updates. In the paired per-request analysis of the original 30-minute Code–Math–Code stream, mean TTFT falls by 3.7% and trace-wide P95 is nearly unchanged. The two updates break even about 6.17 and 8.41 minutes after their respective workload shifts (Appendix E.6). The new short-shift controls show how the system avoids unnecessary migration and further clarify the timescale over which workload drift must persist (Appendix E.1). With Greedy’s extra cap on transfers per round removed, Matching shortens average transfer time by 12.46% under the same endpoint and receive-buffer limits (Appendix E.9).

The responses below and the revised manuscript give the settings, scope, and results of each experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.