acceptodds
Under review as a conference paper at ICLR 2027

BEYOND OVERLAP: OS-AWARE, BANDWIDTH-CENTRIC EXPERT STREAMING FOR MOE INFERENCE ON UNIFIED-MEMORY SYSTEMS

Abstract

Mixture-of-Experts (MoE) models reduce per-token computation but can exceed the memory capacity of unified-memory devices, forcing router-selected experts to be streamed from SSD. We show that the resulting bottleneck is not expert computation or unified-memory bandwidth, but the realized bandwidth of the OS-mediated SSD-to-unified-memory path. XSTREAM increases this bandwidth by issuing routed-expert requests concurrently, jointly tuning request granularity and I/O/decompression parallelism, and streaming each expert through lossless decompression, the Gate and Up projections, and the dependent Down projection without waiting for the whole expert. On an NVIDIA GB10 system, it achieves 2.07–4.65× higher throughput than ZipMoE at matched memory budgets, and 1.47–1.53× higher throughput than serial one-expert-ahead execution without pipelining. These results demonstrate that, when experts must be fetched, exposing and consuming storage parallelism is a first-order requirement for SSD-backed MoE inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, MoE Inference, Large Language Models, Unified Memory, OS-Aware Runtime, Storage-Aware Inference, NVMe SSD, Expert Offloading, Expert Streaming, I/O Concurrency, Bandwidth Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WWT3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents XSTREAM, a system for accelerating SSD-backed MoE inference on unified-memory devices. It combines concurrent expert reads, GPU-based lossless decompression, joint calibration of chunk size and I/O concurrency, and an operator-aware streaming pipeline that starts computation before an entire expert becomes available. Experiments on NVIDIA GB10/Linux and AMD Radeon/Windows demonstrate performance improvements, including higher throughput than ZipMoE under matched memory budgets.

Critical strengths
  1. The paper addresses the practical challenge of efficient MoE inference on unified-memory devices with limited memory capacity.

  2. The proposed system demonstrates promising performance improvements in the evaluated settings.

  3. The experiments cover multiple models and platforms, providing useful evidence of the approach’s effectiveness.

Critical weaknesses
  1. Concurrent I/O, GPU-based lossless decompression, and blockwise execution are well-established techniques. While integrating them effectively into MoE inference may offer system-side value, the paper may not sufficiently explain the nontrivial technical challenges involved. The authors should clarify the novelty of their approach beyond integrating these existing techniques.

  2. The evaluation primarily uses ZipMoE as the baseline. Colibrì and the current MoE-Infinity implementation with SSD offloading support are also directly relevant but are not evaluated quantitatively.

  3. All Radeon/Windows experiments use an all-miss policy. This demonstrates portability when experts must continually be fetched from SSD, but does not establish the end-to-end benefits under a mixture of cache hits and misses.

  4. The authors state that joint parameter calibration can be performed before inference. However, the reported time of approximately three minutes applies only to the single-layer loading sweep, and the paper does not fully quantify the costs of weight encoding, full-model validation, or any additional preparation required after configuration changes. The practical overhead of this one-time calibration therefore remains unclear.

  5. Figures 1 and 2 contain crowded text, annotations, and subplot layouts. In particular, the x-axis tick labels in Figure 2(c) are tightly packed, with some labels running together.

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer WWT3 (1/2)

We thank the reviewer for highlighting the need to distinguish our design from established primitives and to broaden the evaluation. Please also see our General Response.

W1. Is this more than combining concurrent I/O, decompression, and pipelining?

We agree that these primitives are individually established. The nontrivial contribution is their critical-path-aware integration for the SSD-bound unified-memory regime, with a different objective from computation-centric overlap. The design must solve three coupled problems: expose enough independent routed-expert I/O to approach SSD throughput, decode compressed weights without creating a new serial stage, and release dependent expert operators without waiting for the whole expert.

These problems are coupled. The benefit of compression depends on available I/O concurrency: without pipelining, ANS reduced end-to-end throughput (0.961× and 0.885×; Tables 14–15), while with concurrent workers it improved throughput by 1.16–1.23×. Removing the whole-expert barrier adds a further gain on top of concurrent reads with overlapped decoding (4.4–5.4% all-miss). Out-of-order completion also had to be made deterministic: expert results are reduced in a fixed expert-ID order, and in our tested run outputs were bitwise identical across schedules (App. A). We have clarified this in the contributions (§1) and the pipeline description (§4.4), and clarified that the implementation uses operator-aligned regions and independently decodable chunks, not blockwise intermediate-dimension GEMMs.

W2. Colibrì and MoE-Infinity.

We evaluated both; full details are in our response to Reviewer 6Ak5 (W2).

SystemModel and conditionXSTREAM advantage
Adapted MoE-InfinityDeepSeek, 8 GB3.57× end to end
Adapted MoE-InfinityDeepSeek, 16 GB4.05× end to end
Colibrì, best BF16Qwen3.51.86× decode TBT (5.23× end to end)
Colibrì, GPU Q8_0Qwen3.51.57× decode TBT (3.90× end to end)

We disclose that the stock MoE-Infinity release was OOM-killed under the tested caps, so we report an explicitly labeled adaptation that uses MoE-Infinity's own SSD read path, and that Colibrì's BF16 experts run on the CPU while its Q8_0 configuration uses the GPU at lower precision. The adapted MoE-Infinity realizes only 36–48% of the SSD's pure-read bandwidth, consistent with our motivation that realized storage delivery, rather than compute overlap alone, limits performance in this regime. We do not claim these systems are inefficient on the hardware they target; the comparisons show XSTREAM's advantage in the constrained-memory, SSD-backed unified-memory setting.

AuthorsOpenReview ↗

Response to Reviewer WWT3 (2/2)

W3. Radeon/Windows experiments only use all misses. Does XSTREAM help with mixed cache hits and misses?

We agree that the original all-miss results did not establish benefits in cache-enabled execution.

We therefore performed new paired cache-enabled and cold-cache measurements on Radeon 8060S / Windows across all four models, using the 8 GB cache setting.

The reported performance ratios come from actual Radeon paired runs. The absolute cache-enabled latency values are obtained by applying those measured Radeon cached/cold ratios to the published Radeon cold baselines. The listed hit-rate reference values come from the earlier GB10 cache measurements (Table 25), and we distinguish those reference rates from independently measured Radeon hit-rate statistics.

The results are:

ModelReference hit rateCached Full TBTFull vs. cached ANS Serial, decodeFull vs. cached Raw Serial, end-to-end
Qwen1.536.8%257.8 ms1.33×1.66×
DeepSeek48.6%314.9 ms1.34×1.72×
Gemma49.2%428.2 ms1.38×1.82×
Qwen3.543.1%328.4 ms1.53×1.76×

The cache-enabled end-to-end XSTREAM throughputs are 3.17, 2.61, 2.08, and 2.71 tok/s, respectively.

The results show that XSTREAM's advantage persists in cache-enabled execution, rather than depending on a forced all-miss policy.

They also demonstrate that caching and the proposed streaming execution strategy are complementary. Caching reduces the number of experts requiring SSD transfers, while XSTREAM improves the delivery and execution of those experts that still miss.

These measurements and their reconstruction methodology are now included in the Radeon evaluation (Section 5.4 and Appendix E).

W4. Full offline preprocessing and calibration costs.

We measured the full one-time preparation cost on GB10 (Table 13):

ModelFull-model encodingStore size
Qwen1.5126.7 s16.7 GB
DeepSeek-V2-Lite84.9 s19.4 GB
Gemma4289.4 s30.9 GB
Qwen3.5287.3 s44.0 GB

Encoding takes 1.4–4.8 minutes per model, once offline, and is not charged to per-token latency. The (C, N) loading sweep (Figure 2) takes about 3 minutes, excluding re-encoding; on Qwen1.5, complete-model runs confirmed the same optimal region. For the schedule comparisons, we hold one configuration fixed (C = 1 MiB, N = 8) so that all arms are compared under identical settings. Changing C requires re-encoding the model; N is a runtime setting and needs no re-encoding. The revision reports these costs separately (Appendix A.1).

W5. Figures 1 and 2 are crowded.

We agree. Figure 1 is simplified to the three execution stages (concurrent routed-expert reads, asynchronous GPU decoding, dependency-aware Gate/Up–Down execution), and annotations implying partial intermediate-dimension GEMMs are removed. Figure 2 has more spacing and simpler labels, especially the tick labels in panel (c), and its axis labels state precisely which size parameter is varied, distinguishing it from the other storage, codec and I/O granularities.

We hope these results address your concerns. The revised PDF is uploaded with changes highlighted, and we would welcome any further questions.

Thanks for the authors' response. I am satisfied with the reply to W3-W5.

However, I am still suspicious of the novelty of the work, as I did not see what significant technical contributions the authors propose compared to existing SSD-specific MoE art. Could the authors further clarify this point?

Reviewer_JSw7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance bottlenecks of SSD-backed Mixture-of-Experts (MoE) inference on unified-memory systems and proposes XStream, a runtime designed to improve the efficiency of loading and executing nonresident routed experts. The authors argue that conventional one-expert-ahead execution cannot effectively hide SSD transfer latency when expert loading takes substantially longer than expert computation. In particular, exposing only a limited number of outstanding I/O requests prevents the OS/NVMe storage path from achieving its available bandwidth, resulting in recurring I/O stalls. To address this issue, XStream combines three main techniques. First, it performs concurrent routed-expert I/O by submitting SSD read requests for all router-selected experts concurrently, increasing the I/O concurrency exposed to the storage path. Second, it employs lossless ANS compression with GPU-resident decompression to reduce SSD traffic. The system also jointly calibrates the ANS outer chunk size and I/O worker count to select an appropriate operating point for the target deployment configuration. Third, it introduces a hierarchical expert-streaming pipeline that eliminates the whole-expert materialization barrier. By partitioning the expert's intermediate dimension into blocks, the runtime initiates Gate/Up–SwiGLU computation as soon as the corresponding weights become available and performs blockwise Down projections when their dependencies are satisfied. This enables SSD reads, decompression, and expert computation to overlap at finer granularity. The authors evaluate XStream using four MoE models on NVIDIA GB10/Linux and AMD Radeon/Windows platforms. On GB10, XStream achieves 2.07–4.65× higher throughput than ZipMoE under matched memory budgets and 1.47–1.53× higher throughput than serial no-pipeline execution in the internal schedule ablation. The Radeon/Windows experiments further demonstrate improvements in time to first token (TTFT) and time between tokens (TBT) under an all-miss configuration. Overall, the paper's primary contribution is an execution strategy that jointly considers the realized bandwidth of the OS-mediated storage path and the operator-level dependencies within MoE experts, rather than relying solely on conventional computation-I/O overlap.

Critical strengths

Strength 1: A well-motivated analysis of the storage-path bottleneck A key strength of this paper is its focus on the realized I/O bandwidth of the OS/NVMe storage path as a major performance bottleneck in SSD-backed MoE inference. Conventional one-expert-ahead execution can overlap the computation of the current expert with the loading of the next expert. However, when expert computation is substantially shorter than SSD transfer time, this approach cannot effectively eliminate recurring I/O stalls. The authors address this limitation by submitting read requests for all selected routed experts concurrently, thereby exposing additional I/O concurrency to the storage path. The distinction between parallelizing reads within an individual expert and concurrently submitting requests across multiple routed experts is particularly useful. The measurements in Section 4.1 and Appendix B support the observation that increasing the amount of independent I/O work exposed to the storage device can provide benefits beyond conventional computation-I/O overlap. This insight is relevant to the design of SSD-backed MoE inference systems, where optimizing cache behavior or prefetching alone may not be sufficient to utilize the available storage bandwidth.

Strength 2: Hierarchical expert streaming with systematic experimental validation Another strength is the hierarchical expert-streaming pipeline, which exploits the dependencies among the Gate/Up projections, SwiGLU activation, and Down projection instead of treating each expert as an indivisible execution unit. By partitioning the intermediate dimension into blocks, the runtime can initiate computation on available Gate/Up blocks and subsequently compute partial Down-projection outputs as their corresponding weights become available. This design removes the whole-expert barrier and enables finer-grained overlap among SSD reads, lossless decompression, and expert computation. The experimental evaluation is also reasonably comprehensive within the targeted deployment setting. The authors report end-to-end performance improvements across four MoE models and two unified-memory platforms. Furthermore, Table 2 and the stepwise ablations in Appendix B distinguish the effects of concurrent I/O, ANS compression, chunk-level pipelining, and operator-aware scheduling. The batch-size sensitivity experiments provide additional evidence that the benefits are not restricted to batch-one execution. These results support the practical effectiveness of the proposed approach. The fact that XStream preserves the original routing decisions and model weights is also a useful property for deployment.

Critical weaknesses

Weakness 1: Insufficient clarity on the compute-side trade-offs and baseline optimization of fine-grained operator-aware streaming My main concern is that the GPU kernel implementation and compute-side execution costs are not described in sufficient detail to fully assess the benefits of fine-grained operator-aware streaming. XStream partitions each expert's intermediate dimension into blocks and initiates computation as soon as the corresponding weights become available. Although this removes the whole-expert materialization barrier and enables finer-grained overlap between I/O and computation, it may also introduce additional kernel-dispatch and synchronization overhead, as well as reduced GPU execution efficiency due to smaller computational units. This concern is particularly relevant to batch-one MoE decoding, where expert computation often resembles GEMV rather than large GEMM operations. The execution granularity and kernel organization can therefore have a substantial impact on GPU efficiency. Whole-expert execution and fine-grained blockwise execution may differ in their kernel organization and opportunities for grouping or fusion. For example, grouped or batched execution of multiple experts or ready blocks may affect launch overhead and GPU resource utilization. These optimization opportunities are not necessarily exclusive to either approach, but their actual use matters when comparing the execution schedules. In this context, it is unclear which GPU kernel configurations are used by the concurrent no-pipeline baseline in Table 2 and the full operator-aware pipeline, and whether comparable levels of compute-side optimization are applied to both. The authors do acknowledge the relevant overheads. Section 4.2 explicitly includes chunk bookkeeping, kernel launches, and queueing in the cost model, and the appendices provide chunk-size sensitivity measurements and stepwise ablations. In particular, Tables 12 and 13 in Appendix B report additional throughput improvements of approximately 2.4% and 2.8%, respectively, when operator-aware scheduling is added after ANS/chunk-level pipelining. These results provide evidence that operator-aware scheduling offers a net performance benefit in the evaluated configurations. However, the measurements do not separately characterize the additional GPU/host scheduling costs and the reduction in exposed critical-path latency resulting from fine-grained execution. The reported end-to-end results demonstrate that XStream outperforms the evaluated baselines. Nevertheless, without a clearer description of the underlying GPU kernel implementations and their optimization levels, it remains difficult to assess how much of the advantage of fine-grained operator-aware streaming would persist relative to a comparably optimized whole-expert execution baseline. This does not invalidate the reported performance improvements, but it leaves an important question regarding the effectiveness of the proposed execution schedule and the fairness of the compute-side comparison.

Questions for authors

Q1: GPU kernel implementation and optimization of the whole-expert baseline In the concurrent no-pipeline baseline reported in Table 2, how is expert computation implemented on the GPU? Specifically, are multiple experts executed using grouped/batched kernels, or is each expert executed through separate kernel launches? Are comparable compute-side optimizations, including kernel grouping or fusion where applicable, employed in both the concurrent no-pipeline baseline and the full operator-aware pipeline? I recognize that Appendix B includes batch-size sensitivity experiments. However, batching multiple tokens and grouping expert computations within GPU kernels are distinct optimization considerations. Clarifying the actual kernel configurations would help assess whether the benefits of fine-grained streaming remain significant relative to a comparably optimized whole-expert execution baseline.

Q2: Scheduling overhead and compute efficiency of operator-aware streaming Tables 12 and 13 in Appendix B report approximately 2.4% and 2.8% additional throughput improvements, respectively, when operator-aware scheduling is introduced after ANS/chunk-level pipelining. Could the authors clarify how GPU kernel dispatch, host-side scheduling, and synchronization are handled in the fine-grained blockwise execution pipeline? In particular, does each compute block require a separate GPU kernel launch, or are multiple ready blocks processed together using batched or fused kernels? It would also be helpful to clarify how ANS outer chunks map to compute blocks and how block readiness is tracked and communicated to the GPU execution schedule. If available, a comparison of kernel launch counts, GPU compute time, and exposed scheduling overhead before and after introducing operator-aware scheduling would help explain the relationship between the additional execution costs and the resulting critical-path reduction. This clarification would be useful for assessing the independent contribution of operator-aware scheduling beyond the gains already provided by concurrent I/O and chunk-level pipelining.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JSw7 (1/2)

We thank the reviewer for recognizing the importance of routed-set I/O concurrency and dependency-aware execution, and agree that the original manuscript did not describe the GPU execution granularity and its costs precisely enough.

Q1. Separate or grouped expert kernels? Are both schedules equally optimized?

Experts execute individually with the same computation operators in all audited configurations. On GB10, expert Gate/Up and Down projections use torch.nn.functional.linear through the same cuBLAS-backed path, with the same activation implementation. For Qwen1.5 this is eight expert GEMV launches and four Triton SwiGLU launches per MoE layer (four routed experts, each with Gate/Up and Down). Gemma4 uses its unfused activation consistently across schedules. There is no grouped-expert GEMM, per-block GEMM, CUDA Graph capture, or torch.compile applied to Full only.

The Table 2 concurrent no-pipeline schedule and the full schedule follow the same per-expert computation organization. To make the comparison directly auditable, our new experiment adds a Concurrent Barrier configuration that is deliberately stronger than an I/O-only baseline: it submits expert reads concurrently and overlaps GPU decompression with outstanding reads, differing from Full only by a group-completion barrier before expert computation. We counted computation and decoder invocations in both modes: dense and expert F.linear calls and activation launches match on every model; GPU decode calls differ by less than 2%, with about 1.1–1.2 decoded segments per decoder call in both.

Grouped expert kernels are a separate potential optimization; neither schedule uses them, so the comparison isolates the scheduling mechanism.

Q2. Does fine-grained execution add kernels and synchronization? What is the actual granularity?

Thank you for identifying an ambiguity in our original description. The implementation is operator-region level, not intermediate-dimension blockwise GEMM execution. We have corrected Section 4.4, Eqs. 11–13, Figure 1 and the implementation appendix.

  • Gate/Up region. Its compressed segments are decoded independently as they arrive; once all are ready, the runtime launches the complete Gate/Up linear and the activation.
  • Down region. Its segments are read and decoded concurrently with Gate/Up processing; once the Down weights and activation are ready, the runtime launches the complete Down linear.

ANS chunks and read segments are storage and decompression units, not GEMM blocks: we do not launch a GEMM per chunk or accumulate partial Down outputs. Full-size expert GEMMs therefore keep the compute granularity of the barrier baseline; the finer granularity lies in readiness tracking, asynchronous decoding, and when each operator may start. Completion counters track region readiness, and CUDA streams and events enforce dependencies, so reading and decoding continue while a ready operator executes.

AuthorsOpenReview ↗

Response to Reviewer JSw7 (2/2)

Measured overhead and benefit

The new all-miss comparison isolates the independent contribution:

ModelBarrier TBTFull TBTTime saved/tokenThroughput gain
Qwen1.5228.3 ms219.2 ms9.1 ms+4.4%
DeepSeek340.3 ms325.5 ms14.8 ms+4.8%
Gemma4395.2 ms375.7 ms19.5 ms+5.4%
Qwen3.5313.7 ms301.5 ms12.2 ms+4.5%

All runs have zero cache hits, matched expert kernels, identical workloads, and the same concurrent read/decode mechanisms, with three repetitions per cell. Full's p99 decode latency is lower on all four models, and TTFT improves by about 5–6%.

Host-side scheduling overhead. We measured it directly on GB10 in the same all-miss runs (Qwen1.5 and DeepSeek, 3 interleaved runs of 63 tokens each; the timers did not change TBT). Full issues more event records and waits, but this extra host work costs 1.3 ms/token on Qwen1.5 and 2.3 ms/token on DeepSeek, under 1% of TBT, against measured savings of 7.8 and 13.6 ms/token in the same runs. Host issue time per layer is the same in both schedules within noise, GPU kernel counts are identical (288 and 468 per token), and the readiness-polling loop is almost entirely idle waiting for SSD reads, at about 1 µs per check.

Our implementation keeps complete operator-sized GEMVs, and its added synchronization cost is outweighed by removing dependent computation from the storage critical path. For context, the stepwise ablations in Appendix B (Tables 14–15) showed an incremental operator-aware gain of about 2.4–2.8%, and the cache-enabled decomposition shows 1.8–4.8% over Concurrent Barrier; the cache-free experiment shows 4.4–5.4% when early cache-hit execution is excluded.

We hope this clarifies the implementation and the fairness of the comparison. The revised PDF is uploaded with changes highlighted, and we would welcome any further questions.

Thank you for the clarification and the additional experiments. The revised description addresses my concern about execution granularity. Gate/Up and Down are each executed as complete operators, and the Concurrent Barrier and Full configurations use the same per-expert compute kernels.

The matched-kernel, all-miss comparison addresses my main concern about the independent benefit of operator-aware scheduling. The 4.4–5.4% throughput gain across four models is consistent, and the host-side profiling on Qwen1.5 and DeepSeek shows that the added scheduling overhead is small relative to the latency savings. The comparison establishes the benefit for the evaluated per-expert kernels. How it interacts with grouped expert kernels remains an open question.

These results address my main concerns, and I am maintaining my rating of 3 (Weak Acceptance).

AuthorsOpenReview ↗

Thank you; grouped-kernel measurement

Thank you for the careful follow-up and for confirming that the matched-kernel all-miss comparison and the host-overhead measurements address your main concerns.

On grouped expert kernels, we measured how much compute grouping could save on GB10, with all K routed experts resident (real weights, pre-stacked before timing, 200 timed iterations; outputs match to BF16 rounding):

Per MoE layerQwen1.5 (K=4)DeepSeek-V2-Lite (K=6)
Per-expert kernels (as in Full)425 µs637 µs
Grouped kernels (torch.bmm)370 µs536–539 µs
Saving, as a share of SSD delivery0.7%~0.9%

Scaled across all MoE layers, this corresponds to about 1.3 ms per token on Qwen1.5 and 2.6 ms on DeepSeek-V2-Lite, compared with the 9.1 and 14.8 ms that operator-region release saves (Table 20). A grouped launch over all K experts must also wait until every routed expert is resident, which reintroduces the group barrier; grouping smaller sets of ready experts could avoid this, so the two techniques are complementary rather than competing.

Reviewer_6Ak5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

XStream accelerates SSD-backed MoE inference on unified-memory devices with limited memory budgets. It combines concurrent expert reads, lossless GPU decompression, and block-level expert computation to improve effective storage bandwidth and overlap data loading with computation, while preserving the original model weights and routing.

Critical strengths

S1: The paper provides a complete system implementation, with validation on two hardware platforms.

S2: Writing is relatively clear and easy to follow.

S3: Component ablations help separate the benefits of compression, concurrent reads, and operator-level pipelining.

Critical weaknesses

W1: Concurrent reads and kernel pipelining are established optimization techniques. The paper does not clearly establish a research contribution beyond combining these techniques.

W2: Insufficient baseline comparisons. The evaluation mainly compares against ZipMoE and internal variants. Its advantages over other relevant offloading systems remain unclear.

W3: The main baseline comparisons use older models and short, restricted workloads. These comparisons should include newer models and more representative inputs.

W4: The GB10 system has 119 GiB of unified memory, but experiments mainly impose 8–16 GB budgets. More evidence is needed that these restrictions reflect realistic deployment conditions.

Questions for authors

See my questions above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6Ak5 (1/2)

Thank you for the careful review. Please also see our General Response, which addresses novelty in detail.

W1. The contribution appears to combine known concurrent-reading and pipelining techniques.

We agree that the individual primitives are known. The key observation is that an overlap-centric strategy can leave the SSD substantially underutilized even while reads and computation are nominally overlapped: when expert computation is much shorter than expert loading, fetching one expert ahead exposes too little independent work to the storage device. XSTREAM therefore optimizes realized storage bandwidth, not nominal overlap or the number of compressed bytes. ZipMoE illustrates the difference: it reads 1.4–3.7× fewer SSD bytes than XSTREAM yet is 2.07–4.65× slower (0.46–1.09 GB/s realized against our 3.84–4.39 GB/s; Table 12).

The mechanisms are also separately measurable. Beyond concurrent I/O with overlapped decoding, operator-aware scheduling adds a measurable gain of its own in a new all-miss experiment:

ModelConcurrent BarrierXSTREAM FullAdditional throughput
Qwen1.5-MoE-A2.7B3.726 tok/s3.889 tok/s+4.4%
DeepSeek-V2-Lite2.589 tok/s2.713 tok/s+4.8%
Gemma4-26B-A4B2.254 tok/s2.375 tok/s+5.4%
Qwen3.5-35B-A3B2.523 tok/s2.637 tok/s+4.5%

Each configuration was measured three times, and every run had a 0% expert-cache hit rate. Both modes use concurrent SSD reads and overlapped GPU decompression with identical expert kernels; only operator readiness and the execution schedule differ. The improvement therefore cannot come from early execution of cached experts, different kernels, or concurrent I/O alone.

W2. Comparisons against other MoE offloading systems.

We evaluated two additional systems: MoE-Infinity and Colibrì.

MoE-Infinity

DeepSeek-V2-Lite, 64 greedy tokens, strict 8 GB and 16 GB cgroup limits (swap disabled), as in our ZipMoE comparison.

We first ran the unmodified release. Under both caps it was OOM-killed at startup, before inference: by default it preloads all expert weights into host memory, which on a unified-memory device is the same physical pool as GPU memory. To obtain a usable comparison rather than a zero, we evaluated an explicitly labeled adaptation that disables this startup preload and returns evicted experts to disk, so that misses are served through MoE-Infinity's own SSD read path under the cap. Its speculative prefetching was disabled because it deadlocked in the adapted build.

Memory capAdapted MoE-InfinityXSTREAMSpeedup
8 GB1.249 ± 0.010 tok/s4.46 tok/s3.57×
16 GB1.960 ± 0.016 tok/s7.94 tok/s4.05×

No OOM kills occurred; peak process memory was 7.60 GB and 15.76 GB. The adapted MoE-Infinity realized 3.26 GB/s and 2.49 GB/s (SSD bytes divided by run time), about 48% and 36% of the measured 6.85 GB/s pure-read bandwidth. Its measured compute is about 4.6 s of each run; holding its transfer path fixed, perfectly overlapping that compute would improve it by at most 1.10× (8 GB) and 1.16× (16 GB). Additional overlap of the measured compute alone therefore cannot close the gap.

Correctness. The adapted 16 GB run matches the Hugging Face greedy output exactly; the 8 GB run diverges at generated token 19. Unmodified MoE-Infinity with all experts resident also diverges from the HF reference (at token 38), so the divergence is not unique to the adaptation. We report the 8 GB result with this qualification.

AuthorsOpenReview ↗

Response to Reviewer 6Ak5 (2/2)

Colibrì

Qwen3.5-35B-A3B-Base, the same BF16 weights (GGUF), expert cache matched to XSTREAM's 12 GB setting, the same ShareGPT prompt and 64 greedy tokens. We did not modify Colibrì's source code; we varied only its documented runtime settings and backend, and report its best BF16 configuration (prefetch, I/O pool, O_DIRECT) and its GPU Q8_0 configuration.

ConfigurationDecode TBTEnd-to-end throughput
Colibrì, best tested BF16349 ms0.723 tok/s
Colibrì, GPU Q8_0 (lossy)294 ms0.970 tok/s
XSTREAM, BF16 lossless187.5 ms3.782 tok/s

XSTREAM's decode is 1.86× faster than Colibrì's best BF16 configuration and 1.57× faster than its GPU Q8_0 configuration, which reads half the bytes per expert. In BF16, Colibrì's experts run on the CPU because its CUDA expert tier does not accept BF16. For reference, with all experts resident in RAM (no SSD reads; single run, prefetch configuration), Colibrì's TBT was 0.18–0.19 s, versus 0.349 s when SSD-backed; these runs are not a controlled comparison, but they indicate that expert delivery, not CPU expert compute alone, accounts for much of its decode time. The larger end-to-end gap (3.90–5.23×) is dominated by Colibrì's prefill, about half of which is CPU compute even with all experts resident, so we do not attribute it to storage scheduling. Colibrì's BF16 process peaked at 16.6 GB.

W3. Older models and short, restricted workloads.

The evaluation covers four models with top-K from 4 to 8 and different expert sizes, including two recent models, Gemma4-26B-A4B and Qwen3.5-35B-A3B (Table 2); the new Colibrì comparison also uses Qwen3.5. We additionally ran a 48-run workload sensitivity study: 256- and ~2048-token ShareGPT prompts, 64 and 128 generated tokens, ANS Serial vs. XSTREAM Full, three interleaved repetitions. We acknowledge it uses the two earlier models (Qwen1.5 and DeepSeek-V2-Lite); the recent models are covered by Table 2 and the Colibrì comparison.

ModelPromptGeneratedANS SerialXSTREAM FullGain
Qwen1.5256644.695.87+25.0%
Qwen1.52561285.286.47+22.4%
Qwen1.52048644.715.83+23.8%
Qwen1.520481285.666.79+20.0%
DeepSeek256643.895.02+29.1%
DeepSeek2561284.245.42+27.8%
DeepSeek2048643.564.67+31.3%
DeepSeek20481284.135.25+27.3%

Throughput in tok/s; all 48 runs completed, per-cell standard deviation ≤ 0.05 tok/s; cache-enabled, with Full's hit rates of 44–58%. XSTREAM wins in every setting, by 20–31%. TTFT drops by 1.18–1.26 s (Qwen1.5) and 1.40–1.45 s (DeepSeek), and decode TPOT is lower in every cell. Batch sizes 1–16 are already evaluated in Appendix B, Table 18.

W4. Why 8–16 GB budgets on a 119 GiB GB10?

We read this concern as asking whether 8–16 GB budgets correspond to real deployments. GB10's 119 GiB is the capacity of our test platform, not the intended serving budget; the budgets model devices on which the routed experts cannot all stay resident. We offer three pieces of evidence.

Real devices are in this regime. The ASUS ROG Flow Z13 (2025) ships with a Ryzen AI Max+ 395 and 32 GB of unified LPDDR5X-8000, the same processor and memory type as our Radeon test system. After Windows, other applications, the KV cache and the model's dense weights, roughly half of that memory, about 16 GB, is a realistic budget for experts; our 8–16 GB range brackets this case (Appendix E).

The budget is a hard limit, not only a cache setting. The ZipMoE and MoE-Infinity comparisons cap total process memory with cgroups (swap disabled). For the schedule study, which bounds decoded-expert cache capacity, we reran Full under a hard 12 GB whole-process cap: throughput was similar to the slot-budgeted runs (Qwen1.5 5.880 vs. 5.880 tok/s; DeepSeek 4.999 vs. 5.030; Appendix A).

Larger devices reach the same regime with larger models. At 8–12 GB, Qwen1.5 keeps 18–37% of its routed experts resident (11–22 of 60 per layer); illustratively, this is roughly the resident fraction a ~235B-parameter BF16 MoE would have on the full 119 GiB GB10.

We hope these results address your concerns. The revised PDF is uploaded with changes highlighted, and we would welcome any further questions.

AuthorsOpenReview ↗

Follow-up: summary of revisions

Dear Reviewer,

As we are approaching the end of the discussion period, we wanted to kindly follow up on our rebuttal and revised manuscript. We have carefully addressed your concerns:

  • Novelty (W1): Section 2, Table 1 and Section 5.2
  • Baselines (W2): MoE-Infinity and Colibrì comparisons (Section 5.3, Appendix D.2)
  • Models and workloads (W3): Gemma4 and Qwen3.5 (Tables 2–3) and ~2,048-token prompts (Appendix B.3)
  • Memory budgets (W4): real-device motivation and a hard 12 GB cap check (Appendices A and E)

If any concerns remain, we would be happy to address them before the discussion closes.

Thank you for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): Novelty and Evidence

We thank all three reviewers for their constructive feedback and for recognizing the practical relevance of SSD-backed MoE inference on memory-constrained unified-memory devices. We have run additional controlled experiments addressing novelty, compute-side fairness, operator-aware scheduling, baseline breadth, workload generality, Radeon caching, and preprocessing cost. A revised PDF with all changes highlighted has been uploaded.

What is novel about XSTREAM?

We agree that concurrent I/O, GPU decompression, caching, and computation–I/O overlap are individually established techniques. Our contribution is not these primitives, but a different critical-path strategy for SSD-backed MoE inference on unified-memory devices.

Existing MoE offloading systems reduce the cost of expert misses by caching or predicting experts and by prefetching them so that transfers overlap with computation (e.g., MoE-Infinity, ProMoE, EdgeMoE, Mixtral-offloading). This hides transfer time only up to the length of the available computation. In SSD-backed decoding on unified memory that window is short: in our no-cache serial Qwen1.5 profile, resident-equivalent expert computation is about 10.6% of MoE time, so even perfect overlap would save at most about 11%. We observe the same on an external system: holding MoE-Infinity's measured transfer path fixed, perfectly overlapping its measured compute would improve it by at most 1.10–1.16× (response to 6Ak5, W2).

Storage delivery is therefore the dominant cost, and XSTREAM targets it directly: it shortens SSD delivery and hides GPU decompression and computation under the outstanding reads, using three coordinated mechanisms:

  1. Routed-set concurrent I/O. Once the router selects a layer's experts, XSTREAM submits the cache-missing subset concurrently instead of one expert ahead, exposing independent work to the OS and NVMe queue.
  2. Lossless compressed streaming. Experts are stored as independently decodable compressed segments; GPU decompression starts as segments arrive, so decoding does not become a new serial stage.
  3. Operator-aware execution. Each expert is split into Gate/Up and Down regions. Gate/Up and the activation start when their region is ready while Down weights are still in flight; Down runs when its weights and activation are ready. There is no whole-expert barrier.

Evidence that this is more than combining primitives

  • Fewer bytes is not less time. ZipMoE reads 1.4–3.7× fewer SSD bytes than XSTREAM yet is 2.07–4.65× slower, because it realizes 0.46–1.09 GB/s against our 3.84–4.39 GB/s (Table 12).
  • Concurrency matters more as experts shrink. Inter-expert concurrency alone adds 1.3–2.7% for K=4/6 models with ~17 MB experts, but 20.9–38.7% for K=8 models with 6–12 MB experts (Table 2), the direction recent MoEs are moving.
  • Compression depends on I/O concurrency. Without pipelining, ANS reduced end-to-end throughput (0.961× on Qwen1.5, 0.885× on DeepSeek); with concurrent I/O workers it improved throughput by 1.16–1.23× (Tables 14–15).
  • The mechanisms are separable. On GB10, XSTREAM improves throughput by 1.26–1.52× over ANS Serial (Table 2). Concurrent routed reads with overlapped GPU decompression deliver 21.4–48.9% over ANS Serial, and operator-aware scheduling adds a further 1.8–4.8% over that Concurrent Barrier configuration. A separate controlled all-miss experiment with matched kernels confirms 4.4–5.4% from operator-aware scheduling on all four models.
  • Existing systems already contain the individual techniques. Colibrì has an asynchronous I/O pool, prefetching and an O_DIRECT path; MoE-Infinity has activation-aware prefetching and caching. On the same device, MoE-Infinity realizes only 36–48% of the SSD's pure-read bandwidth, and XSTREAM is 3.6–4.1× faster than MoE-Infinity end to end and 1.6–1.9× faster than Colibrì in decode.
  • Deterministic outputs. Expert results are reduced in a fixed expert-ID order, so outputs do not depend on I/O completion order; in our tested run, serial, barrier and full schedules produced bitwise-identical logits (App. A).
AuthorsOpenReview ↗

General Response (2/2): Summary of Evidence and Revisions

Summary of new evidence

QuestionNew evidence or clarification
More than routine overlap?Storage-centric critical path; overlap bound on MoE-Infinity (1.10–1.16×); stage decomposition
Is operator-aware scheduling independently useful?+4.4–5.4%, four models, zero cache hits, matched kernels
Are GPU kernel comparisons fair?Same per-expert kernels and launch counts in Barrier and Full
Extra scheduling cost?Measured on GB10: 1.3–2.3 ms/token of extra host work (under 1% of TBT) against 7.8–13.6 ms/token savings
Longer workloads?+20–31% across eight prompt/generation configurations
Batch sizes?Already in Appendix B, Table 18 (batch 1–16)
Are 8–16 GB budgets meaningful?Edge deployment scenario; enforcement clarified; whole-process 12 GB cap check
vs. MoE-Infinity3.57–4.05× end to end (adapted build; stock release OOM-killed under the caps)
vs. Colibrì1.57–1.86× in decode TBT; 3.90–5.23× end to end
Radeon with cachingFull keeps a 1.66–1.82× end-to-end advantage over cached raw serial
Preprocessing costFull-model compression 85–289 s; runtime preparation 14–28 s
Implementation accuracyExecution is operator-region level; §4.4, Eqs. 11–13 and Figure 1 corrected

Revisions in the uploaded PDF

  • Clarified the introduction and contributions around the storage-centric critical path.
  • New section comparing against MoE-Infinity and Colibrì; full protocols, the MoE-Infinity adaptation, and all Colibrì configurations in Appendix D.
  • Corrected §4.4, Eqs. 11–13 and Figure 1 to describe operator-region execution.
  • New implementation appendix: storage layout, I/O path, GPU decoding, readiness tracking, dispatch.
  • Workload-length study, all-miss operator-aware experiment, Radeon cache-enabled results, memory-cap checks, and preprocessing costs.
  • Related work extended (LLM in a Flash, PowerInfer-2, ProMoE, Mixtral-offloading, EdgeMoE, Pulsar); Figures 1 and 2 redrawn.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.