- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents XSTREAM, a system for accelerating SSD-backed MoE inference on unified-memory devices. It combines concurrent expert reads, GPU-based lossless decompression, joint calibration of chunk size and I/O concurrency, and an operator-aware streaming pipeline that starts computation before an entire expert becomes available. Experiments on NVIDIA GB10/Linux and AMD Radeon/Windows demonstrate performance improvements, including higher throughput than ZipMoE under matched memory budgets.
Critical strengths
-
The paper addresses the practical challenge of efficient MoE inference on unified-memory devices with limited memory capacity.
-
The proposed system demonstrates promising performance improvements in the evaluated settings.
-
The experiments cover multiple models and platforms, providing useful evidence of the approach’s effectiveness.
Critical weaknesses
-
Concurrent I/O, GPU-based lossless decompression, and blockwise execution are well-established techniques. While integrating them effectively into MoE inference may offer system-side value, the paper may not sufficiently explain the nontrivial technical challenges involved. The authors should clarify the novelty of their approach beyond integrating these existing techniques.
-
The evaluation primarily uses ZipMoE as the baseline. Colibrì and the current MoE-Infinity implementation with SSD offloading support are also directly relevant but are not evaluated quantitatively.
-
All Radeon/Windows experiments use an all-miss policy. This demonstrates portability when experts must continually be fetched from SSD, but does not establish the end-to-end benefits under a mixture of cache hits and misses.
-
The authors state that joint parameter calibration can be performed before inference. However, the reported time of approximately three minutes applies only to the single-layer loading sweep, and the paper does not fully quantify the costs of weight encoding, full-model validation, or any additional preparation required after configuration changes. The practical overhead of this one-time calibration therefore remains unclear.
-
Figures 1 and 2 contain crowded text, annotations, and subplot layouts. In particular, the x-axis tick labels in Figure 2(c) are tightly packed, with some labels running together.
Questions for authors
See weakness.
7 comments
Response to Reviewer WWT3 (2/2)
W3. Radeon/Windows experiments only use all misses. Does XSTREAM help with mixed cache hits and misses?
We agree that the original all-miss results did not establish benefits in cache-enabled execution.
We therefore performed new paired cache-enabled and cold-cache measurements on Radeon 8060S / Windows across all four models, using the 8 GB cache setting.
The reported performance ratios come from actual Radeon paired runs. The absolute cache-enabled latency values are obtained by applying those measured Radeon cached/cold ratios to the published Radeon cold baselines. The listed hit-rate reference values come from the earlier GB10 cache measurements (Table 25), and we distinguish those reference rates from independently measured Radeon hit-rate statistics.
The results are:
| Model | Reference hit rate | Cached Full TBT | Full vs. cached ANS Serial, decode | Full vs. cached Raw Serial, end-to-end |
|---|---|---|---|---|
| Qwen1.5 | 36.8% | 257.8 ms | 1.33× | 1.66× |
| DeepSeek | 48.6% | 314.9 ms | 1.34× | 1.72× |
| Gemma4 | 9.2% | 428.2 ms | 1.38× | 1.82× |
| Qwen3.5 | 43.1% | 328.4 ms | 1.53× | 1.76× |
The cache-enabled end-to-end XSTREAM throughputs are 3.17, 2.61, 2.08, and 2.71 tok/s, respectively.
The results show that XSTREAM's advantage persists in cache-enabled execution, rather than depending on a forced all-miss policy.
They also demonstrate that caching and the proposed streaming execution strategy are complementary. Caching reduces the number of experts requiring SSD transfers, while XSTREAM improves the delivery and execution of those experts that still miss.
These measurements and their reconstruction methodology are now included in the Radeon evaluation (Section 5.4 and Appendix E).
W4. Full offline preprocessing and calibration costs.
We measured the full one-time preparation cost on GB10 (Table 13):
| Model | Full-model encoding | Store size |
|---|---|---|
| Qwen1.5 | 126.7 s | 16.7 GB |
| DeepSeek-V2-Lite | 84.9 s | 19.4 GB |
| Gemma4 | 289.4 s | 30.9 GB |
| Qwen3.5 | 287.3 s | 44.0 GB |
Encoding takes 1.4–4.8 minutes per model, once offline, and is not charged to per-token latency. The (C, N) loading sweep (Figure 2) takes about 3 minutes, excluding re-encoding; on Qwen1.5, complete-model runs confirmed the same optimal region. For the schedule comparisons, we hold one configuration fixed (C = 1 MiB, N = 8) so that all arms are compared under identical settings. Changing C requires re-encoding the model; N is a runtime setting and needs no re-encoding. The revision reports these costs separately (Appendix A.1).
W5. Figures 1 and 2 are crowded.
We agree. Figure 1 is simplified to the three execution stages (concurrent routed-expert reads, asynchronous GPU decoding, dependency-aware Gate/Up–Down execution), and annotations implying partial intermediate-dimension GEMMs are removed. Figure 2 has more spacing and simpler labels, especially the tick labels in panel (c), and its axis labels state precisely which size parameter is varied, distinguishing it from the other storage, codec and I/O granularities.
We hope these results address your concerns. The revised PDF is uploaded with changes highlighted, and we would welcome any further questions.
Thanks for the authors' response. I am satisfied with the reply to W3-W5.
However, I am still suspicious of the novelty of the work, as I did not see what significant technical contributions the authors propose compared to existing SSD-specific MoE art. Could the authors further clarify this point?
Response to Reviewer WWT3 (1/2)
We thank the reviewer for highlighting the need to distinguish our design from established primitives and to broaden the evaluation. Please also see our General Response.
W1. Is this more than combining concurrent I/O, decompression, and pipelining?
We agree that these primitives are individually established. The nontrivial contribution is their critical-path-aware integration for the SSD-bound unified-memory regime, with a different objective from computation-centric overlap. The design must solve three coupled problems: expose enough independent routed-expert I/O to approach SSD throughput, decode compressed weights without creating a new serial stage, and release dependent expert operators without waiting for the whole expert.
These problems are coupled. The benefit of compression depends on available I/O concurrency: without pipelining, ANS reduced end-to-end throughput (0.961× and 0.885×; Tables 14–15), while with concurrent workers it improved throughput by 1.16–1.23×. Removing the whole-expert barrier adds a further gain on top of concurrent reads with overlapped decoding (4.4–5.4% all-miss). Out-of-order completion also had to be made deterministic: expert results are reduced in a fixed expert-ID order, and in our tested run outputs were bitwise identical across schedules (App. A). We have clarified this in the contributions (§1) and the pipeline description (§4.4), and clarified that the implementation uses operator-aligned regions and independently decodable chunks, not blockwise intermediate-dimension GEMMs.
W2. Colibrì and MoE-Infinity.
We evaluated both; full details are in our response to Reviewer 6Ak5 (W2).
We disclose that the stock MoE-Infinity release was OOM-killed under the tested caps, so we report an explicitly labeled adaptation that uses MoE-Infinity's own SSD read path, and that Colibrì's BF16 experts run on the CPU while its Q8_0 configuration uses the GPU at lower precision. The adapted MoE-Infinity realizes only 36–48% of the SSD's pure-read bandwidth, consistent with our motivation that realized storage delivery, rather than compute overlap alone, limits performance in this regime. We do not claim these systems are inefficient on the hardware they target; the comparisons show XSTREAM's advantage in the constrained-memory, SSD-backed unified-memory setting.