acceptodds
Under review as a conference paper at ICLR 2027

Albus: Network-Aware Expert Localization and Dynamic Balancing with Unified Scheduling for MoE Serving

Abstract

Mixture-of-experts (MoE) models expand the capacity of large language models by increasing the number of expert parameters while activating only a small subset for each token. This sparse activation allows model capacity to grow without a proportional increase in per-token computation. The resulting growth in model size increasingly requires serving systems to distribute experts across multiple nodes, where fast intra-node links and slower inter-node connections form a hierarchical network. Compared with single-node serving, this deployment amplifies cross-node communication costs and GPU load imbalance, limiting the inference efficiency of large MoE models. Existing load-balancing methods alleviate imbalance through redundant expert placement but do not sufficiently integrate hierarchical network topology into their load-balancing optimization. We present Albus, a system that jointly optimizes communication and load balancing for MoE inference on hierarchical networks. Albus introduces expert-locality enhancement and communication-aware load balancing to optimize routing and redundant expert placement. It combines these innovations with predictive scheduling and expert-replica prefetching adapted to hierarchical networks, forming a complete execution pipeline for MoE serving. Evaluations on GLM-4.7 and DeepSeek-R1 show that Albus maintains comparable accuracy to native inference across six benchmarks covering knowledge, reasoning, language understanding, and truthfulness. Compared with native SGLang inference, Albus achieves a 1.71× prefill speedup and 1.56× decode throughput.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts (MoE), LLM Inference, Expert Parallelism, Cross-Node Communication, Load Balancing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_srGAOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Albus is a MoE serving system for hierarchical multi-node networks, where reducing cross-node traffic and balancing expert computation can conflict. It combines locality-biased expert selection, one-layer-ahead prediction of source–expert demand, communication-aware replica placement, and double-buffered expert preparation. The planner preserves source-node service when an expert copy is available locally, while the runtime overlaps prediction, planning, and weight transfers with model execution. Locality enhancement changes logical expert selection; for some normalized-score models, the system additionally filters low-weight assignments subject to a minimum retained-expert count. The principal GLM-4.7 and DeepSeek-R1 experiments retain the original Top-K cardinality.

The evaluation covers five MoE models, including GLM-4.7 and DeepSeek-R1 at EP32 across four nodes. The main prefill experiments report average speedups of 1.71× over native SGLang, 1.33× over EPLB, and 1.10× over Libra++. Additional experiments examine model quality, serving TTFT, decoding, prediction error, overlap behavior, and memory overhead.

Critical strengths

S1. A coherent design that addresses communication and load balance together, supported by mechanism-level evidence

The planner does not treat all replica destinations as equivalent: it uses source–expert demand and restricts assignments to the source node when that node already hosts the expert. The runtime then makes these decisions executable through predictive preparation and reusable execution buffers. This connects the placement policy to the actual constraints of hierarchical networks, rather than assuming that better invocation-count balance necessarily produces lower latency.

The evidence is consistent with this motivation. In Table 2, Albus and Libra++ have similar expert-computation times, while Albus reduces dispatch and combine time. Appendix G.2 also isolates the source-node-service constraint and reports a 1.12–1.17× prefill speedup from enabling it. These measurements directly support an important design choice and are consequential to my positive assessment.

S2. The experiments establish a systems benefit beyond approximate routing alone

The main prefill comparisons against EPLB and Libra++ apply the same locality multiplier to all three methods, and Libra++ uses Albus’s runtime organization. Figure 5 additionally includes both locality enhancement alone and Albus with locality enhancement disabled. The latter retains native expert selection while achieving 1.22× native SGLang’s throughput. Thus, the paper does more than compare a modified model against an unmodified baseline: it provides evidence that the scheduling and placement mechanisms themselves are useful.

The double-buffered implementation also addresses the cost of making dynamic placement practical. The paper accounts for temporary execution-buffer storage and measures foreground interference and compute–transfer overlap, rather than treating replica creation as free. This increases confidence that the reported benefit reflects an implemented execution path.

Critical weaknesses

W1. The quality evidence does not yet adequately characterize the approximate routing used in the performance experiments

Locality enhancement changes which logical experts execute; retaining original scores for output weighting does not preserve the original model function. Moreover, Equation (6) boosts all eligible local home experts rather than explicitly restricting replacements to low-margin tail choices. Consequently, the tail-localization motivation is not itself a guarantee that only negligible contributions change.

The paper does provide meaningful quality evaluation and calibration, so my concern is not their absence. Table 1 reports comparable task averages at the selected settings, and Appendix F calibrates routing parameters on WikiText-103 using Qwen3.6-35B-A3B at EP4. However, average scores can conceal task-specific regressions, uncertainty is not adequately reported, and the calibration does not by itself establish transfer across models and deployment topologies. The reported PPL increase at the selected locality multiplier also confirms that the method should be evaluated as a quality–efficiency trade-off, not as an exact systems transformation.

This matters because the main gains over native inference combine approximate routing with systems optimization. I would like a clearer account of task-level uncertainty and the quality budget for the exact configurations used in the serving results. Exact equivalence is not required, but the acceptable trade-off should be quantified rather than inferred from similar averages.

W2. The nearest-method comparison and operating conditions are not fully characterized The broad combination of prediction, replication, locality, and overlap has close precedents. Libra already predicts future expert activations and hides balancing overhead through execution restructuring. GRACE-MoE—Han et al., Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference, arXiv:2509.25041—describes a lossless multi-node combination of grouping, replication, and locality-aware replica selection. These do not subsume Albus’s specific planner and preparation pipeline, but they mean that the incremental contribution must be articulated more precisely than “jointly optimizing locality and load balance.”

The shared-runtime Libra++ comparison and communication-constraint ablation are valuable. Nevertheless, the native-routing ablation establishes a gain over native SGLang, not over a similarly configured strong balancing baseline. It remains unclear how much advantage the planner retains over a simpler locality-first replica policy, especially when logical expert selection is unchanged.

Reproducibility and scope also limit confidence. Section 5.1 specifies the interconnect types but not the GPU model or exact NIC configuration. Appendix B relies on a resident CPU model copy for some replica transfers, yet its host-memory and NUMA requirements are not fully accounted for. Independent repetition counts and variability are insufficiently specified for the main comparisons. Finally, Figure 6’s decode evidence concerns GLM-4.7 at 64–160 requests per GPU and compares against native SGLang, leaving the advantage over stronger balancing methods and at smaller effective batches unresolved.

Questions for authors

Q1. About expriments configurations How were the routing parameters selected for each model and deployment, beyond the Qwen3.6-35B-A3B EP4 calibration in Appendix F? Were the final settings fixed independently of the task-evaluation results?

For the settings used in the main serving experiments, please provide the sample counts, generation/scoring configurations, and paired task-level uncertainty or equivalent existing evidence. Because the locality multiplier applies to all local home experts, what evidence supports the interpretation that replacements predominantly affect low-cost tail choices? A focused routing-change analysis would help connect the motivation to the observed quality trade-off.

Q2. Detailed ablation comparison Figure 4 appropriately matches the locality multiplier across Albus, EPLB, and Libra++. How much of the advantage remains when all methods use native logical routing, with comparable replica capacity and execution conditions?

A focused comparison against a locality-first replica-assignment control would also help distinguish the proposed source-conditioned planning from an existing local-copy preference. Please explain the difference from GRACE-MoE’s replica-selection policy. A comparison of the relevant mechanism on a supported configuration would be sufficient; I am not requesting a complete reimplementation of every prior system.

Q3. Reproducibility and overlap conditions Please specify the GPU model, NIC count and bandwidth, inter-node topology, relevant precision/backend settings, and the footprint and placement of the resident CPU model copy. Which preparation costs, including readiness waits, are included in the measured serving times?

For the key comparisons, how many independent runs were performed, and what variability was observed? Please also clarify the operating range of the decode claim: either a matched EPLB/Libra++ result at representative batch sizes, or an explicit restriction of the claim to the measured native-SGLang comparison, would help establish how broadly the demonstrated advantage applies.

4 comments

Thank you for your constructive feedback. We address your concerns and questions below.

1. Model Quality, Parameter Selection, and Evaluation Reliability

Evaluation settings. C-Eval, MMLU, MMLU-Pro, and LIMO use greedy decoding (argmax, temperature=0). RTE and TruthfulQA MC2 use likelihood-based scoring. Table 1 summarizes the sample counts and generation/scoring configurations.

Table 1. Sample counts and evaluation protocols.

BenchmarkSamplesGeneration and scoring
C-Eval12,3420-shot CoT; max 8,192 tokens; answer-letter accuracy
MMLU14,0420-shot CoT; max 8,192 tokens; answer-letter accuracy
MMLU-Pro12,0320-shot CoT; max 8,192 tokens; answer-letter accuracy
LIMO8170-shot CoT; max 32,768 tokens; boxed-answer exact match
RTE277Candidate log-likelihood accuracy
TruthfulQA MC2817Normalized probability mass assigned to correct answers

Evaluation variability and quality budget. Our evaluation does not introduce randomness from token sampling. Existing repeated evaluations show limited run-to-run variability, with observed standard deviations below 0.5 percentage points across the six benchmarks. These results provide empirical evidence of evaluation stability. We set an empirical quality-loss target of one percentage point per dataset. The quality results reported in Table 1 of the paper were obtained from a single evaluation per configuration using the protocols in Table 1.When selecting hyperparameters, we assess degradation on every dataset rather than only the average score.

Paired MMLU-Pro results. Table 2 shows question-level correctness changes for DeepSeek-R1 at α=2 on the same 12,032 MMLU-Pro questions.False → True denotes an incorrect baseline answer becoming correct under Albus; True → False denotes the reverse.

Table 2. Paired accuracy changes for DeepSeek-R1 on MMLU-Pro (α=2).

SamplesBaselineAlbus (α=2)False → TrueTrue → False
12,0329,936 (82.58%)9,895 (82.24%)60 (0.50%)101 (0.84%)

Parameter selection. The Qwen3.6-35B-A3B calibration in Appendix F helped narrow the candidate parameter range. For deployment, we used the task-level quality and speedup results in Table 1 of the paper to select α=2 for GLM-4.7 and DeepSeek-R1. We then fixed this setting for the serving experiments. Thus, the final choice used the task-evaluation results.

Quality across deployment configurations. We added quality evaluations of DeepSeek-R1 under two deployment configurations using the same α=2 (Table 3).

Table 3. DeepSeek-R1 quality across deployment configurations (scores, %).

EPNodesMethodC-EvalMMLUMMLU-ProLIMORTETruthfulQAAvg
——Baseline90.290.782.691.274.058.181.1
324Albus, α=289.890.482.290.874.458.581.0
648Albus, α=289.890.282.591.275.158.081.1

Using the same α=2, DeepSeek-R1 maintains comparable quality at EP32 and EP64, supporting the transferability of the selected parameter across these deployment sizes.

2. Routing Rule and Tail Localization

Routing rule and quality control. Sparse MoE execution evaluates only selected experts, so not all expert outputs are available for comparison. Our motivation is therefore to use router weights as a readily available signal of expert importance to guide locality-aware replacement.

Albus applies α only to local expert scores, leaving remote scores unchanged. For a fixed candidate pool, Top-K therefore preserves the remote ordering and displaces lower-scoring remote experts first. Thus, α controls the strength of the locality bias . We select α=2 through task-level quality evaluation to limit quality degradation.

3. Contributions and Comparative Experiments

Contributions beyond Libra. We reuse next-layer prediction and double buffering from Libra. Albus adds locality-aware Top-K and a planner that uses source-node demand to jointly consider communication and load balance. Our runtime splits expert transfers across the current MoE computation and the next-layer attention. This extends the overlap window without requiring full-token AllGather or a separate local-MoE phase. GPU-side statistics and planning reduce CPU–GPU synchronization and better support decode CUDA Graphs.

Native logical routing. We added a comparison on PG-19 at 8K tokens/rank, using α=1 for all methods (Table 4). For GLM/R1, Albus achieves speedups of 1.54×/1.74× over Baseline, 1.21×/1.47× over EPLB, and 1.04×/1.05× over Libra++. This shows that Albus retains additional systems benefits when locality enhancement is disabled.

Table 4. Prefill latency on PG-19 at 8K tokens/rank with native routing (α=1; ms).

ModelBaselineEPLBLibra++Albus
GLM3842.033026.322605.642498.65
R12710.422294.481643.001562.13

Locality-first replica placement. We added two simple controls: Top2, where each rank replicates its two most frequently accessed experts not homed on that rank; and Top2 + fixed-layout LB, which fixes the Top2 layout before running Albus's load-assignment algorithm. Top2, Top2 + fixed-layout LB, and Albus all use α=2, with the native Baseline (α=1) included for reference(Table 5). Lower imbalance I(D;A)I(D;A) and fewer remote nodes per token C(D)C(D) are better.

Table 5. Load imbalance and remote nodes per token for locality-first controls.

MethodGLM I(D;A)I(D;A)GLM C(D)C(D)R1 I(D;A)I(D;A)R1 C(D)C(D)
Native Baseline, α=13.372.713.452.34
Top22.011.651.931.47
Top2 + fixed-layout LB1.641.651.691.47
Albus1.271.571.281.40

Compared with Top2 + fixed-layout LB, Albus further reduces imbalance by 22.63%/24.25% and remote nodes per token by 4.95%/4.52% for GLM/R1. We observed that independent hotspot selection duplicates expert coverage within a node, covering fewer distinct experts than Albus and limiting avoidable cross-node accesses. Fixing the layout before load balancing prevents further coordination of replica positions. These results support jointly coordinating replica placement and load assignment using source-node demand.

Comparison with GRACE-MoE. GRACE-MoE first uses offline profiling to select the busiest expert groups and replicates hot experts onto lightly loaded GPUs in a single placement pass, then applies strict locality-first replica selection online. This locality priority can restrict load balancing, whereas Albus's locality constraint allows continued optimization within the topology constraints. Albus jointly optimizes locality and load balance iteratively: using predicted demand, each iteration identifies the busiest rank, adjusts replica placement and token assignment under locality and replica-capacity constraints, and updates the load until the balance target is met or no feasible move remains.

4. Hardware Configuration, Timing, and Reproducibility

Hardware configuration. Our main multi-node experiments used the following EP32 setup:

  • GPU/CPU: 4 nodes × 8 NVIDIA 800 GPUs, with 141 GB HBM3e per GPU (platform queue label: H800-141G). Each node has two Xeon Platinum 8558 CPUs (48 cores each) and 1.95 TB of host memory.
  • Interconnect: Intra-node NVLink has theoretical/measured one-way bandwidths of 450/395.4 GB/s; nodes communicate through RoCE RDMA. Each node has 8 NIC ports, one per GPU, with nominal/measured bandwidths of 50/46.29 GB/s per port.
  • NUMA/PCIe: GPUs 0–3 and 4–7 connect to NUMA0 and NUMA1, respectively. CPU–GPU links use PCIe 5.0 ×16, with theoretical/measured bandwidths of 63/55.3 GB/s.
  • Precision/backends: GLM-4.7 uses BF16 weights, activations, and KV cache. DeepSeek-R1 uses FP8 E4M3 weights, BF16 backbone activations and KV cache, and dynamically quantized FP8 GEMM inputs. The attention and MoE backends are FlashAttention-3 and DeepGEMM, respectively.
  • CPU replicas: Expert weights belonging to other nodes are stored in host memory on a single NUMA node, requiring approximately 498.28 GB for GLM-4.7 and 482.09 GB for DeepSeek-R1 per node.

Preparation costs included. We measure model forward time. Albus's preparation work overlaps with the main execution path, and any expert-readiness stalls are included. Serving throughput additionally includes CPU batch scheduling overhead.

Repetitions and variability. Each configuration is measured 100 times after warm-up. We report the p50 latency of these 100 measurements. Table 6 reports representative 8K results; SD is the sample standard deviation across these measurements.

Table 6. Prefill latency and variability at 8K tokens/rank (α=2 for EPLB, Libra++, and Albus).

ModelMethodMean ± SD (ms)p50 (ms)
GLM-4.7SGLang3795.19 ± 99.593791.71
GLM-4.7EPLB2795.85 ± 70.742785.64
GLM-4.7Libra++2479.57 ± 90.552469.90
GLM-4.7Albus2251.36 ± 53.252228.94
DeepSeek-R1SGLang2692.98 ± 110.452662.48
DeepSeek-R1EPLB2149.78 ± 98.502139.24
DeepSeek-R1Libra++1579.61 ± 45.531562.42
DeepSeek-R1Albus1440.96 ± 18.711438.77

5. Decode Baseline and Evaluation Scope

EPLB comparison. We added an EPLB comparison on GLM-4.7 using PG-19 at BS=128 per rank, with the same locality multiplier α=2 for EPLB and Albus (Table 7).

Table 7. GLM-4.7 decode performance on PG-19 at BS=128/rank.

MethodTPOT (ms)Throughput (tokens/s)
Native SGLang179.5721,412.95
EPLB (α=2)118.6732,225.70
Albus (α=2)103.4639,442.18

At this operating point, Albus achieves 1.22× the throughput of EPLB, providing evidence of benefits beyond the native-SGLang comparison.

Evaluation scope. Our demonstrated decode benefits apply to relatively large batch sizes, and we will emphasize this operating range in the revised manuscript. The decode speedups reported in the paper are already relative to native SGLang.

Reviewer_srGAOpenReview ↗

Official Comments

The authors have addressed parts of my concerns. I have no more questions. My rating would be preserved.

Reviewer_2ouoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Albus that targets MoE inference on hierarchiccal multi-node GPU clusters, where load imbalance and inter-node communication jointly affect performance. It combines locality-biased expert selection with communication-aware replica placement, using next-layer demand prediction to prepare experts ahead of time. A double-buffered runtime overlaps planning and weight preparation with inference, using local GPU copies, intra-node NVLink transfers, and PCIe transfers from a resident CPU model copy. The evaluation examines model quality across 5 MoE models, multi-node prefil performance on GLM-4.7 and DeepSeek-R1 and decode performance on GLM-4.7.

Critical strengths
  1. This work has strong motivation with a clear focus on multi-node MoE inference. Albus addresses the interaction between hierarchical communication costs and rank-level load imbalance, providng a coherent system design for this practically important setting.

  2. Careful management of load-balancing ohead with fine-grained coordination of scheduling, preparation, and forground execution. The runtime coordinates HBM copies for home experts, NVLink transfers for intra-node replicas, and PCIe fetches from a resident CPU mdoel copy for experts on other nodes. The fine-grained scheduling of these operations alongside foreground execution limits resource contention and exposed preparation overhead.

  3. The paper comprehensively evaluates model quality across five MoE models and multiple benchmarks, and complements prefill and decode performance measurements with targeted ablations, prediction0quality analysis, and overhead measurements.

Critical weaknesses
  1. The contribution is incremental and needs more precise positioning. Libra already employs next-layer gate prediction, locality-aware expert replication, and double buffering. Although Albus acknowledges Libra nd Table 8 provides useful evidence for its source-node communication constraint, the paper should distinguish inherited mechanisms from the new contributions more explicitly. The omission of LPLB [1], which combines per-batch load balancing with topology-constrained replica layouts, also leaves an important gap. A more systematic comparison is needed to establish where Albus offers a distinct advantage over existing predictive and exact-load balancers.

  2. The justtification and operating boundaries of the proposed policies are not fully characterized. Fig. 2 motivates replacing remote tail experts with similarly scored local alterantives, but siimilar router scores do not imply similar expert outputs. Moreover, Sec. 4.1 applies a uniform locality multiplier without explicitly restricting replacements to tail positions or tightly matched scores. The quality experiments support the selected configurations, but do not establish that the actual replacements follow the stated motivation. Similarly, Table 5-6 substantiate effective overlap at the mesured prefill operating point, but do not sufficienntly characterize when preparation overhead becomes exposed. Missing hardware details further limits reproducibility and assessment of whether these benefits extend to other clusters.

[1] LPLB: https://github.com/deepseek-ai/LPLB.

Questions for authors
  1. Could you explicitly distinuish the mechanisms inhereited from Libra from the new topology-aware planning and runtime contributions? Please also compare the scope of Albus, LPLB, MoonEP, and UltraEP in terms of demand-observatrion timing, replica-placement updates, topology restricitons, and whether original experrt selections are preserved. Is a topology-constrained, exact-load baseline feasible on the evaluated cluster? If so, a matched comparison would strengthen the positioning. Otherwise, plase explaine the specific limitation.

  2. Please clarify with supprting evidence for the tail-localizaiton motivation versus the implementation rule. With alpha=2, a local canddidate can displace a remote expert with roughly twice its original score, and the rule does not explicitly protect the leading experts. What are the original ranks and score ratios of the experts actually replaced across layers and models? Do these replacements predominatly involve closely scored tail experts?

  3. Please clarify the conditions for effective overlap. Tables 5-6 show that expert transfers fit within the measured prefill computation windows with little interference. How does exposed preparation latency change with token count, replica-transfer volume, and layer, particularly during decoding? Please provide representative computation-window durations and acutal waiting times to identify when the schedule remains effective.

  4. Please specify the hardware configuration, including the GPU type and memory capacity, NIC count and link rates, CPU and host-memory configuration, and relevant PCIe/NUMA topology, together with effective transfer bandwidths. These details are central to interpreting the overlap results and the system's deployment requirements.

3 comments

We thank the reviewer for the constructive feedback. We address the concerns and questions below.

1. Novelty, Related Work, and the Exact-Load Baseline

Contributions beyond Libra. We reuse Libra's next-layer prediction and double buffering. Albus adds locality-aware Top-K and a source-conditioned planner that jointly considers communication and load balancing. Our runtime spreads expert transfers across the current MoE compute and next-layer attention, extending the overlap window and reducing transfer pressure in each stage. It does not rely on full-token AllGather or a separate local-MoE phase to hide preparation overhead. GPU-side statistics and planning reduce CPU–GPU synchronization and better support decode CUDA Graphs.

Comparison with related systems. Table 1 summarizes demand-observation timing, replica updates, topology, and preservation of original expert selections.

Table 1. Comparison of Albus and related load-balancing systems.

MethodDemand observationReplica updatesTopologyPreserves original expert selections
AlbusOne-layer-ahead predictionDynamic per layer; prepared aheadIntra-node NVLink + inter-node RDMANo by default; yes when locality enhancement is disabled
LPLBLayout statistics + exact online demandTopology-constrained layout + online reassignmentSupports intra- and inter-node constraintsYes
MoonEPCurrent exact demandDynamic replication after routingNVLink-connected domainYes
UltraEPCurrent exact demandDynamic replication after routingRack-scale scale-upYes

Exact-load baseline. LPLB is feasible on our cluster, and we have added the comparison in Table 2. For GLM-4.7 prefill at 4K–16K tokens per GPU, Albus achieves a further 1.24–1.28× speedup over LPLB.

Table 2. GLM-4.7 prefill latency (ms).

Tokens per GPUBaselineLPLB (α=2)Albus (α=2)
4K1786.501523.931210.81
8K3791.712845.232228.94
12K5959.444232.853416.07
16K7987.025853.344705.75

LPLB's EPLB-based layout is constructed from historical statistics and may differ from current hotspots. Online reassignment requires collecting demand across ranks and solving after routing, adding work to the critical path. Its public LP implementation uses predefined replica connections and allows at most one redundant copy per expert, restricting the placement space.

Creating new expert replicas only after exact routing is a separate limitation: on our cluster, slow RDMA weight transfers would introduce substantial overhead on the critical path and increase end-to-end latency.

2. Tail Localization

Routing rule and quality control. Similar router scores do not guarantee similar expert outputs. However, sparse MoE execution evaluates only selected experts, so not all expert outputs are available for comparison. Router weights provide a simple signal of expert importance, which we use to guide replacement. Albus boosts local candidates by α while leaving remote candidates unchanged and retaining the model's native selection rules.this preserves the ordering of remote experts by selection score, so replacement starts with the lowest-scoring remote experts in the original Top-K, consistent with tail localization. The parameter α controls how strongly local candidates compete with remote experts; we select α=2 through task-level quality evaluation to limit quality degradation.

Replacement examples. Table 3 presents expert-replacement examples from early, middle, and late layers of both models. It reports the original ranks of removed remote experts and the routing weights of both removed and added experts before applying α. Added local experts were originally outside the Top-8. In these examples, replacements affect tail positions; gaps in the removed ranks correspond to experts that were already local and remain selected.

Table 3. Tail replacements across model depth at α=2 (original routing weights).

ModelLayerReplaced (original ranks)Removed remote weightsAdded local weights
GLM-4.733 (6, 7, 8)0.3450/0.3178/0.28340.2639/0.2232/0.2122
GLM-4.7471 (7)0.13040.0780
GLM-4.7912 (7, 8)0.3267/0.30930.2729/0.2473
DeepSeek-R132 (7, 8)0.2598/0.24800.2436/0.2379
DeepSeek-R1323 (5, 6, 8)0.1624/0.1385/0.11600.1009/0.0981/0.0967
DeepSeek-R1601 (8)0.09270.0876

3. Overlap Conditions

Prefill overlap. We split expert transfers across the current layer's MoE compute and the next layer's attention. No exposed transfer latency was observed in the ≥4K input settings evaluated in the paper. Table 6 in the paper shows that expert transfers are fully hidden at 8K, with substantial time remaining in the computation window. Effective overlap is achievable with an average of 2K tokens per GPU.

Decode overlap. Decode has shorter computation windows. Even with high-speed NVLink, weight-readiness waits remain before dispatch. Table 4 reports representative GLM-4.7 timings, providing further analysis of Figure 6 in the paper.

Table 4. GLM-4.7 decode computation, transfer, and waiting times (ms).

ComponentBS96BS128BS160
MoE compute0.170.170.19
Expert transfer0.380.380.37
Attention0.210.230.26
Weight-readiness wait before dispatch0.090.060.03
Full-layer critical-path time0.961.041.15

The measured wait decreases as batch size increases, showing that longer computation windows help hide expert preparation. Our reported decode evaluation covers batch sizes ≥64.

Replica-transfer budget. We allocate two redundant experts per GPU per layer. Each expert is approximately 45 MB in both GLM-4.7 and DeepSeek-R1. Increasing this budget does not further improve load balance in our setting, so we evaluate only the two-expert configuration.

4. Hardware Configuration and Transfer Bandwidth

Our main multi-node experiments use the following EP32 configuration:

  • GPU: 4 nodes × 8 NVIDIA H800 GPUs, with 141 GB HBM3e per GPU.
  • CPU and host memory: Two Xeon Platinum 8558 CPUs per node, with 48 cores each and 1.95 TB of host memory per node.
  • NVLink: Intra-node theoretical/measured one-way bandwidths of 450/395.4 GB/s.
  • NIC and inter-node network: RoCE RDMA with 8 NIC ports per node, one per GPU, and nominal/measured bandwidths of 50/46.29 GB/s per port.
  • NUMA and PCIe: GPUs 0–3 and 4–7 connect to NUMA0 and NUMA1, respectively. CPU–GPU links use PCIe 5.0 ×16, with theoretical/measured bandwidths of 63/55.3 GB/s.

The response addresses most of my concerns. I will keep my score: weak accept.

Reviewer_N1VyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents Albus, a cross-node MoE inference framework for hierarchical clusters. Albus introduces expert-locality enhancement and communication-aware load balancing to optimize routing and redundant expert placement. Compared with native SGLang inference, Albus achieves a 1.71×prefill speedup and 1.56×decode throughput.

Critical strengths
  • Addresses a critical challenge with serving MoE inference at scale
  • Simple and effective idea
Critical weaknesses
  • Overall novelty is limited compared to past approaches
  • The model quality/accuracy is now deployment-dependent which is concerning
Questions for authors

This is a solid piece of work. The papers combines ideas of next-expert prediction, redundant experts, and compute-communication overlap in an elegant manner. But this also adds to my concern - the overall novelty is limited. In particular, its not clear how this work compares against works like UltraEP, MoonEP. Moreover, SGLang's

Furthermore, the evaluations are not convincing enough to demonstrate the broad applicability. In particular:

  • How do outputs and quality change across different numbers of nodes and EP sizes for the same alpha?
  • The word "oracle" is a bit misleading - as it suggests a true optimal (upper bound based on actual demand) but it seems to be the imbalance on predicted demand.

My biggest concern is also that the model quality/accuracy is now deployment-dependent which is concerning. And this has not been sufficiently evaluated.

  • Some gains look like noise; for example, GLM-4.7 TruthfulQA rises from 63.9 to 68.7 at α=1.5.
  • Decode is tested only on GLM-4.7 with one dataset and batch sizes 64–160. There is no ablation, no comparison against EPLB or Libra++, and no quality evaluation under decode-time routing
  • Why was alpha calibrated only on Qwen3.6-35B perplexity and then applied to GLM and DeepSeek?
  • For LIMO, R1 falls 91.2 → 82.7 → 71.6 as α goes from 2 to 3. This suggests errors accumulate over long generations.
  • Evaluations on advanced reasoning models such as as AIME, GPQA, LiveCodeBench, are missing.
3 comments

We appreciate for your valuable feedback.We address the concerns and questions below.

1. Novelty and Comparison with Related Work

Compared with prior work, Albus introduces locality-aware Top-K and source-conditioned planning that jointly considers communication and load balancing in a single problem. It also provides an overlap pipeline tailored to cross-node MoE serving.

MoonEP and UltraEP rely on high-bandwidth GPU interconnects for expert replication and reassignment after actual routing. In hierarchical clusters with intra-node NVLink and inter-node RDMA, slower cross-node links make weight transfers harder to hide in this short window. Albus predicts demand ahead of time and spreads preparation across the current MoE compute and next-layer attention, extending the overlap window and reducing transfer pressure in each stage.

Compared with Libra, Albus uses a GPU-based planning runtime. It reduces CPU–GPU synchronization to better support decode CUDA Graphs and hides preparation overhead without full-token AllGather or a separate local-MoE phase, making it better suited to cross-node MoE serving.

We have added this clarification to the Related Work section of the revised manuscript.

2. Deployment-dependent Quality, Parameter Selection, and Reasoning Evaluation

Quality across deployment configurations. We acknowledge that model quality can depend on the deployment configuration. To evaluate this effect, we report DeepSeek-R1's output quality under two common deployment configurations with the same α=2 in Table 1. The results show that Albus maintains quality close to the baseline across both settings.

Table 1. DeepSeek-R1 quality across deployment sizes (scores, %).

EPNodesMethodC-EvalMMLUMMLU-ProLIMORTETruthfulQAAvg
——Baseline90.290.782.691.274.058.181.1
324Albus, α=289.890.482.290.874.458.581.0
648Albus, α=289.890.282.591.275.158.081.1

TruthfulQA evaluation variability. we reran both the baseline and the α=1.5 configuration multiple times. The results remained within ±0.5 percentage points of their originally reported scores. We do not claim an improvement in model accuracy; our goal is to accelerate inference while maintaining quality close to the original model.

Selection of α across models. The Qwen3.6-35B-A3B perplexity study in Appendix F was used to narrow the candidate parameter range. We then selected α=2 for GLM-4.7 and DeepSeek-R1 based on the task-level quality and speedup results in Table 1 of the paper, and fixed this setting for subsequent serving experiments. Compared with α=1.5, α=2 improves the reported speedup from 1.08× to 1.16× for GLM-4.7 and to 1.17× for DeepSeek-R1, while maintaining quality close to the baseline on the evaluated tasks.

LIMO degradation at larger α. We agree that increasing α can substantially reduce long-generation accuracy. As reported in Table 1 of the paper, DeepSeek-R1's LIMO score decreases from 90.8 at α=2 to 82.7 at α=2.5 and 71.6 at α=3, compared with 91.2 for the baseline. Our serving experiments use α=2, where the observed degradation is limited to 0.4 percentage points.

Advanced reasoning evaluation. To address the concern about missing advanced reasoning benchmarks, we additionally evaluated AIME2024, AIME2025, GPQA Diamond, and LiveCodeBench v6 using the same α=2 configuration adopted in our serving experiments (Table 2). The maximum observed accuracy drop relative to the baseline is 0.8 percentage points, indicating that Albus maintains quality close to the baseline on these challenging reasoning tasks.

Table 2. Advanced reasoning evaluation (scores, %).

ModelMethodAIME2024AIME2025GPQA DiamondLiveCodeBench v6
DeepSeek-R1Baseline72.562.171.275.4
DeepSeek-R1Albus72.561.371.274.7
GLM-4.7Baseline87.180.077.378.8
GLM-4.7Albus87.179.677.878.2

3. Definition and Naming of “Oracle”

We agree that the term “oracle” is misleading, as it refers to the result based on predicted rather than actual demand. Our intention was to quantify the planning error introduced by predicting demand from the preceding layer's hidden states and to demonstrate that Albus achieves an imbalance ratio close to 1 on real workloads. We have relabeled the corresponding result as “Albus Predicted” in the latest revision to avoid confusion.

4. Decode Ablations, Baselines, Quality

Decode-time routing quality. The quality evaluations in Table 1 of the paper enable locality-aware Top-K during both prefill and decode.(We have added this clarification to the “Environment and Settings” paragraph of Section 5.1 (Setup) in the latest revised PDF.) Therefore, our decode performance experiments use the same α=2 routing configuration, for which the evaluated model quality remains close to the baseline.

Decode ablations and EPLB comparison. To provide a more comprehensive evaluation, we added a decode ablation study and a comparison with EPLB on GLM-4.7 using PG-19 at a batch size of 128, as shown in Table 3.

Table 3. GLM-4.7 decode ablation and EPLB comparison on PG-19 (BS=128/rank).

ConfigurationTPOT (ms)Throughput (tokens/s)
Baseline179.5721,412.95
Albus w/o locality (α=1)118.5834,422.01
Locality only (α=2)128.1931,030.81
Albus (α=2)103.4639,442.18
EPLB (α=2)118.6732,225.70

These results demonstrate the effectiveness of Albus’s load balancing and expert preparation mechanisms.

Dear Reviewer N1Vy,

We hope our response has addressed your concerns. Please feel free to let us know if any further clarification is needed.

Thank you for your time.

Comments on the paper

1 comment

We have updated the manuscript PDF with the clarifications and additional results discussed in our responses. We would appreciate any further feedback from the reviewers on whether our responses and revisions address their concerns. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.