- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Albus is a MoE serving system for hierarchical multi-node networks, where reducing cross-node traffic and balancing expert computation can conflict. It combines locality-biased expert selection, one-layer-ahead prediction of source–expert demand, communication-aware replica placement, and double-buffered expert preparation. The planner preserves source-node service when an expert copy is available locally, while the runtime overlaps prediction, planning, and weight transfers with model execution. Locality enhancement changes logical expert selection; for some normalized-score models, the system additionally filters low-weight assignments subject to a minimum retained-expert count. The principal GLM-4.7 and DeepSeek-R1 experiments retain the original Top-K cardinality.
The evaluation covers five MoE models, including GLM-4.7 and DeepSeek-R1 at EP32 across four nodes. The main prefill experiments report average speedups of 1.71× over native SGLang, 1.33× over EPLB, and 1.10× over Libra++. Additional experiments examine model quality, serving TTFT, decoding, prediction error, overlap behavior, and memory overhead.
Critical strengths
S1. A coherent design that addresses communication and load balance together, supported by mechanism-level evidence
The planner does not treat all replica destinations as equivalent: it uses source–expert demand and restricts assignments to the source node when that node already hosts the expert. The runtime then makes these decisions executable through predictive preparation and reusable execution buffers. This connects the placement policy to the actual constraints of hierarchical networks, rather than assuming that better invocation-count balance necessarily produces lower latency.
The evidence is consistent with this motivation. In Table 2, Albus and Libra++ have similar expert-computation times, while Albus reduces dispatch and combine time. Appendix G.2 also isolates the source-node-service constraint and reports a 1.12–1.17× prefill speedup from enabling it. These measurements directly support an important design choice and are consequential to my positive assessment.
S2. The experiments establish a systems benefit beyond approximate routing alone
The main prefill comparisons against EPLB and Libra++ apply the same locality multiplier to all three methods, and Libra++ uses Albus’s runtime organization. Figure 5 additionally includes both locality enhancement alone and Albus with locality enhancement disabled. The latter retains native expert selection while achieving 1.22× native SGLang’s throughput. Thus, the paper does more than compare a modified model against an unmodified baseline: it provides evidence that the scheduling and placement mechanisms themselves are useful.
The double-buffered implementation also addresses the cost of making dynamic placement practical. The paper accounts for temporary execution-buffer storage and measures foreground interference and compute–transfer overlap, rather than treating replica creation as free. This increases confidence that the reported benefit reflects an implemented execution path.
Critical weaknesses
W1. The quality evidence does not yet adequately characterize the approximate routing used in the performance experiments
Locality enhancement changes which logical experts execute; retaining original scores for output weighting does not preserve the original model function. Moreover, Equation (6) boosts all eligible local home experts rather than explicitly restricting replacements to low-margin tail choices. Consequently, the tail-localization motivation is not itself a guarantee that only negligible contributions change.
The paper does provide meaningful quality evaluation and calibration, so my concern is not their absence. Table 1 reports comparable task averages at the selected settings, and Appendix F calibrates routing parameters on WikiText-103 using Qwen3.6-35B-A3B at EP4. However, average scores can conceal task-specific regressions, uncertainty is not adequately reported, and the calibration does not by itself establish transfer across models and deployment topologies. The reported PPL increase at the selected locality multiplier also confirms that the method should be evaluated as a quality–efficiency trade-off, not as an exact systems transformation.
This matters because the main gains over native inference combine approximate routing with systems optimization. I would like a clearer account of task-level uncertainty and the quality budget for the exact configurations used in the serving results. Exact equivalence is not required, but the acceptable trade-off should be quantified rather than inferred from similar averages.
W2. The nearest-method comparison and operating conditions are not fully characterized The broad combination of prediction, replication, locality, and overlap has close precedents. Libra already predicts future expert activations and hides balancing overhead through execution restructuring. GRACE-MoE—Han et al., Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference, arXiv:2509.25041—describes a lossless multi-node combination of grouping, replication, and locality-aware replica selection. These do not subsume Albus’s specific planner and preparation pipeline, but they mean that the incremental contribution must be articulated more precisely than “jointly optimizing locality and load balance.”
The shared-runtime Libra++ comparison and communication-constraint ablation are valuable. Nevertheless, the native-routing ablation establishes a gain over native SGLang, not over a similarly configured strong balancing baseline. It remains unclear how much advantage the planner retains over a simpler locality-first replica policy, especially when logical expert selection is unchanged.
Reproducibility and scope also limit confidence. Section 5.1 specifies the interconnect types but not the GPU model or exact NIC configuration. Appendix B relies on a resident CPU model copy for some replica transfers, yet its host-memory and NUMA requirements are not fully accounted for. Independent repetition counts and variability are insufficiently specified for the main comparisons. Finally, Figure 6’s decode evidence concerns GLM-4.7 at 64–160 requests per GPU and compares against native SGLang, leaving the advantage over stronger balancing methods and at smaller effective batches unresolved.
Questions for authors
Q1. About expriments configurations How were the routing parameters selected for each model and deployment, beyond the Qwen3.6-35B-A3B EP4 calibration in Appendix F? Were the final settings fixed independently of the task-evaluation results?
For the settings used in the main serving experiments, please provide the sample counts, generation/scoring configurations, and paired task-level uncertainty or equivalent existing evidence. Because the locality multiplier applies to all local home experts, what evidence supports the interpretation that replacements predominantly affect low-cost tail choices? A focused routing-change analysis would help connect the motivation to the observed quality trade-off.
Q2. Detailed ablation comparison Figure 4 appropriately matches the locality multiplier across Albus, EPLB, and Libra++. How much of the advantage remains when all methods use native logical routing, with comparable replica capacity and execution conditions?
A focused comparison against a locality-first replica-assignment control would also help distinguish the proposed source-conditioned planning from an existing local-copy preference. Please explain the difference from GRACE-MoE’s replica-selection policy. A comparison of the relevant mechanism on a supported configuration would be sufficient; I am not requesting a complete reimplementation of every prior system.
Q3. Reproducibility and overlap conditions Please specify the GPU model, NIC count and bandwidth, inter-node topology, relevant precision/backend settings, and the footprint and placement of the resident CPU model copy. Which preparation costs, including readiness waits, are included in the measured serving times?
For the key comparisons, how many independent runs were performed, and what variability was observed? Please also clarify the operating range of the decode claim: either a matched EPLB/Libra++ result at representative batch sizes, or an explicit restriction of the claim to the measured native-SGLang comparison, would help establish how broadly the demonstrated advantage applies.
4 comments
3. Contributions and Comparative Experiments
Contributions beyond Libra. We reuse next-layer prediction and double buffering from Libra. Albus adds locality-aware Top-K and a planner that uses source-node demand to jointly consider communication and load balance. Our runtime splits expert transfers across the current MoE computation and the next-layer attention. This extends the overlap window without requiring full-token AllGather or a separate local-MoE phase. GPU-side statistics and planning reduce CPU–GPU synchronization and better support decode CUDA Graphs.
Native logical routing. We added a comparison on PG-19 at 8K tokens/rank, using α=1 for all methods (Table 4). For GLM/R1, Albus achieves speedups of 1.54×/1.74× over Baseline, 1.21×/1.47× over EPLB, and 1.04×/1.05× over Libra++. This shows that Albus retains additional systems benefits when locality enhancement is disabled.
Table 4. Prefill latency on PG-19 at 8K tokens/rank with native routing (α=1; ms).
| Model | Baseline | EPLB | Libra++ | Albus |
|---|---|---|---|---|
| GLM | 3842.03 | 3026.32 | 2605.64 | 2498.65 |
| R1 | 2710.42 | 2294.48 | 1643.00 | 1562.13 |
Locality-first replica placement. We added two simple controls: Top2, where each rank replicates its two most frequently accessed experts not homed on that rank; and Top2 + fixed-layout LB, which fixes the Top2 layout before running Albus's load-assignment algorithm. Top2, Top2 + fixed-layout LB, and Albus all use α=2, with the native Baseline (α=1) included for reference(Table 5). Lower imbalance and fewer remote nodes per token are better.
Table 5. Load imbalance and remote nodes per token for locality-first controls.
| Method | GLM | GLM | R1 | R1 |
|---|---|---|---|---|
| Native Baseline, α=1 | 3.37 | 2.71 | 3.45 | 2.34 |
| Top2 | 2.01 | 1.65 | 1.93 | 1.47 |
| Top2 + fixed-layout LB | 1.64 | 1.65 | 1.69 | 1.47 |
| Albus | 1.27 | 1.57 | 1.28 | 1.40 |
Compared with Top2 + fixed-layout LB, Albus further reduces imbalance by 22.63%/24.25% and remote nodes per token by 4.95%/4.52% for GLM/R1. We observed that independent hotspot selection duplicates expert coverage within a node, covering fewer distinct experts than Albus and limiting avoidable cross-node accesses. Fixing the layout before load balancing prevents further coordination of replica positions. These results support jointly coordinating replica placement and load assignment using source-node demand.
Comparison with GRACE-MoE. GRACE-MoE first uses offline profiling to select the busiest expert groups and replicates hot experts onto lightly loaded GPUs in a single placement pass, then applies strict locality-first replica selection online. This locality priority can restrict load balancing, whereas Albus's locality constraint allows continued optimization within the topology constraints. Albus jointly optimizes locality and load balance iteratively: using predicted demand, each iteration identifies the busiest rank, adjusts replica placement and token assignment under locality and replica-capacity constraints, and updates the load until the balance target is met or no feasible move remains.
4. Hardware Configuration, Timing, and Reproducibility
Hardware configuration. Our main multi-node experiments used the following EP32 setup:
- GPU/CPU: 4 nodes × 8 NVIDIA 800 GPUs, with 141 GB HBM3e per GPU (platform queue label: H800-141G). Each node has two Xeon Platinum 8558 CPUs (48 cores each) and 1.95 TB of host memory.
- Interconnect: Intra-node NVLink has theoretical/measured one-way bandwidths of 450/395.4 GB/s; nodes communicate through RoCE RDMA. Each node has 8 NIC ports, one per GPU, with nominal/measured bandwidths of 50/46.29 GB/s per port.
- NUMA/PCIe: GPUs 0–3 and 4–7 connect to NUMA0 and NUMA1, respectively. CPU–GPU links use PCIe 5.0 ×16, with theoretical/measured bandwidths of 63/55.3 GB/s.
- Precision/backends: GLM-4.7 uses BF16 weights, activations, and KV cache. DeepSeek-R1 uses FP8 E4M3 weights, BF16 backbone activations and KV cache, and dynamically quantized FP8 GEMM inputs. The attention and MoE backends are FlashAttention-3 and DeepGEMM, respectively.
- CPU replicas: Expert weights belonging to other nodes are stored in host memory on a single NUMA node, requiring approximately 498.28 GB for GLM-4.7 and 482.09 GB for DeepSeek-R1 per node.
Preparation costs included. We measure model forward time. Albus's preparation work overlaps with the main execution path, and any expert-readiness stalls are included. Serving throughput additionally includes CPU batch scheduling overhead.
Repetitions and variability. Each configuration is measured 100 times after warm-up. We report the p50 latency of these 100 measurements. Table 6 reports representative 8K results; SD is the sample standard deviation across these measurements.
Table 6. Prefill latency and variability at 8K tokens/rank (α=2 for EPLB, Libra++, and Albus).
| Model | Method | Mean ± SD (ms) | p50 (ms) |
|---|---|---|---|
| GLM-4.7 | SGLang | 3795.19 ± 99.59 | 3791.71 |
| GLM-4.7 | EPLB | 2795.85 ± 70.74 | 2785.64 |
| GLM-4.7 | Libra++ | 2479.57 ± 90.55 | 2469.90 |
| GLM-4.7 | Albus | 2251.36 ± 53.25 | 2228.94 |
| DeepSeek-R1 | SGLang | 2692.98 ± 110.45 | 2662.48 |
| DeepSeek-R1 | EPLB | 2149.78 ± 98.50 | 2139.24 |
| DeepSeek-R1 | Libra++ | 1579.61 ± 45.53 | 1562.42 |
| DeepSeek-R1 | Albus | 1440.96 ± 18.71 | 1438.77 |
5. Decode Baseline and Evaluation Scope
EPLB comparison. We added an EPLB comparison on GLM-4.7 using PG-19 at BS=128 per rank, with the same locality multiplier α=2 for EPLB and Albus (Table 7).
Table 7. GLM-4.7 decode performance on PG-19 at BS=128/rank.
| Method | TPOT (ms) | Throughput (tokens/s) |
|---|---|---|
| Native SGLang | 179.57 | 21,412.95 |
| EPLB (α=2) | 118.67 | 32,225.70 |
| Albus (α=2) | 103.46 | 39,442.18 |
At this operating point, Albus achieves 1.22× the throughput of EPLB, providing evidence of benefits beyond the native-SGLang comparison.
Evaluation scope. Our demonstrated decode benefits apply to relatively large batch sizes, and we will emphasize this operating range in the revised manuscript. The decode speedups reported in the paper are already relative to native SGLang.
Official Comments
The authors have addressed parts of my concerns. I have no more questions. My rating would be preserved.
Thank you for your constructive feedback. We address your concerns and questions below.
1. Model Quality, Parameter Selection, and Evaluation Reliability
Evaluation settings. C-Eval, MMLU, MMLU-Pro, and LIMO use greedy decoding (argmax, temperature=0). RTE and TruthfulQA MC2 use likelihood-based scoring. Table 1 summarizes the sample counts and generation/scoring configurations.
Table 1. Sample counts and evaluation protocols.
Evaluation variability and quality budget. Our evaluation does not introduce randomness from token sampling. Existing repeated evaluations show limited run-to-run variability, with observed standard deviations below 0.5 percentage points across the six benchmarks. These results provide empirical evidence of evaluation stability. We set an empirical quality-loss target of one percentage point per dataset. The quality results reported in Table 1 of the paper were obtained from a single evaluation per configuration using the protocols in Table 1.When selecting hyperparameters, we assess degradation on every dataset rather than only the average score.
Paired MMLU-Pro results. Table 2 shows question-level correctness changes for DeepSeek-R1 at α=2 on the same 12,032 MMLU-Pro questions.False → True denotes an incorrect baseline answer becoming correct under Albus; True → False denotes the reverse.
Table 2. Paired accuracy changes for DeepSeek-R1 on MMLU-Pro (α=2).
Parameter selection. The Qwen3.6-35B-A3B calibration in Appendix F helped narrow the candidate parameter range. For deployment, we used the task-level quality and speedup results in Table 1 of the paper to select α=2 for GLM-4.7 and DeepSeek-R1. We then fixed this setting for the serving experiments. Thus, the final choice used the task-evaluation results.
Quality across deployment configurations. We added quality evaluations of DeepSeek-R1 under two deployment configurations using the same α=2 (Table 3).
Table 3. DeepSeek-R1 quality across deployment configurations (scores, %).
Using the same α=2, DeepSeek-R1 maintains comparable quality at EP32 and EP64, supporting the transferability of the selected parameter across these deployment sizes.
2. Routing Rule and Tail Localization
Routing rule and quality control. Sparse MoE execution evaluates only selected experts, so not all expert outputs are available for comparison. Our motivation is therefore to use router weights as a readily available signal of expert importance to guide locality-aware replacement.
Albus applies α only to local expert scores, leaving remote scores unchanged. For a fixed candidate pool, Top-K therefore preserves the remote ordering and displaces lower-scoring remote experts first. Thus, α controls the strength of the locality bias . We select α=2 through task-level quality evaluation to limit quality degradation.