acceptodds
Under review as a conference paper at ICLR 2027

Byte Economics: Million-Token Inference of a 2.8-Trillion-Parameter MoE on a Consumer Desktop

Abstract

Mixture-of-experts language models now exceed a trillion parameters, while the largest consumer GPU holds 32 GB. We show that serving such a model on a desktop is governed by byte economics: decode time is the bytes each token moves across the memory hierarchy divided by the bandwidth of the links, and prefill time is set by how many times the expert store is read. We build OMOE, a tiered serving system that runs the native 2.8-trillion-parameter Kimi-K3 checkpoint (1.4 TB of MXFP4 experts, unmodified) on two RTX 5090s, 192 GB of DRAM and a 3.2 TB storage tier, with outputs gated byte-exact or within of a full-VRAM reference. A bandwidth cost model with no fitted constants guided four system generations from 4.62 to 1.65 s/token at short context and places the software ceiling for this model on this machine at 0.58 s/token. A layer-major prefill schedule reads each expert at most once per prefill, so prefill throughput falls only from 69 to 47 tokens/s between 96K and one million tokens of context, where needle retrieval passes with 6.1 h to first token and 3.53 s/token decode. The instrumented deployment measures what a production MoE asks of a memory hierarchy: an LRU hit rate of 0.34 at 3% of the store, 3.0 TB written per million-token prefill, and the bytes a model designer could remove. Concurrent consumer-scale systems report contexts up to 65K; the system, telemetry and acceptance harnesses will be released.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixture-of-experts, efficient inference, LLM serving, offloading, long-context inference, memory hierarchy, consumer hardware

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K48fOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OMOE, a system for serving a very large mixture-of-experts (MoE) model whose weights exceed GPU and host memory. Expert weights are stored on high-endurance drives and cached in host and GPU memory. The authors argue that decoding is limited mainly by weight movement and model it with a cost model based on data volume, cache hits, and bandwidth. For prefill, OMOE runs all input chunks through one layer before the next, so each expert is read from storage at most once. KV data is streamed from host memory, and temporary data goes to bounded storage buffers. On the native 2.8-trillion-parameter Kimi-K3 model, using two RTX 5090 GPUs, 192 GB of DRAM, and two Optane P5800X drives (consumer GPUs with enterprise storage), short-context decoding improves from 4.62 to 1.65 seconds/token across system versions. At one million input tokens, the final system takes 6.14 hours to the first token and then decodes at 3.53 seconds/token. Validation includes numerical reference checks, needle retrieval, cache and storage measurements, and a separate GLM-5.2 experiment for the prefill schedule. The proposed future model changes are projections, not demonstrated results.

Critical strengths
  1. A meaningful single-node demonstration with a clearly defined storage requirement

The paper reports serving the native 2.8-trillion-parameter Kimi-K3 model with million-token context using two RTX 5090 GPUs, 192 GB of DRAM, and two 1.6 TB Optane P5800X drives in RAID0. This is a useful feasibility result, but it is a consumer-GPU system backed by enterprise storage, not consumer hardware alone. The paper describes these drives as discontinued and expects NAND to retain much of the throughput, while warning that write endurance limits repeated long-context use.

  1. Useful measurements of performance and storage costs

Short-context decode improves from 4.616 to 1.649 seconds/token across system versions. Table 6 includes an overlap comparison at the same cache size, while Table 7 shows storage writes falling from about 25.1 TB to 3.0 TB per million-token request. These measurements provide useful evidence about individual engineering choices and the storage costs of long-context inference.

Critical weaknesses
  1. The cost model is not validated, and the performance floor is not well supported

The paper gives only partial breakdowns. Reads account for 2.29 of 4.62 seconds in Gen 1 and 1.76 of 3.26 seconds in Gen 2. About 2.2 seconds in Gen 1 are left unexplained and are simply blamed on software. No generation has a full predicted-versus-measured comparison. The claim that each generation changes only one term is also unclear. Gen 2→3 seems to change the I/O engine (from io_uring to production preadv) and add caching at the same time. The 0.58 seconds/token floor assumes no GPU cache hits, no compute or dispatch overhead (c = 0), and no speculation (λ = 1). Yet Table 6 suggests GPU hits do exist. In the last row, the reported miss traffic of 6.459 GB/token implies H_gpu ≈ 0.246, calculated as 1 − 6.459 / [12.915 × (1 − 0.337)]. If so, a larger GPU cache could push decode below 0.58 seconds/token. This conflicts with the claim that no cache improvement can pass the floor. Because of these gaps, the model’s predictive value and its split between software limits and model-design limits are not well supported.

  1. The long-context claims need clearer correctness checks and performance comparisons

The terms “full-VRAM reference” and “512-token logits” are not defined clearly enough to tell whether the numerical checks cover full-model execution at long context or shorter, more limited tests. The final build reports needle retrieval at one depth per context length, which supports retrieval but does not by itself establish full numerical agreement. Also, the reported 12.1× layer-major prefill improvement is measured on GLM-5.2, while the main long-context results use Kimi-K3. A matched Kimi-K3 comparison would better isolate the scheduling benefit. These gaps limit confidence in the scope of the correctness claim and the source of the main prefill improvement.

Questions for authors
  1. How accurately does Equation (2) predict the measured results? Please provide its inputs and predicted latency for each row of Table 6, including GPU/DRAM cache hit rates, bandwidths, compute overhead, and overlap assumptions. Please also explain how GPU cache hits are handled when deriving the 0.58 seconds/token floor.

  2. What exactly do the numerical reference checks cover? Please define “full-VRAM reference” and “512-token logits,” including the hardware, input context length, and whether the comparison covers the full model or individual operations. Which numerical comparisons were performed at 96K, 256K, and one million tokens?

  3. Can you isolate the layer-major prefill gain on Kimi-K3? At a manageable context length, please compare execution with and without layer-major scheduling while keeping hardware, precision, and other optimizations fixed. Please report time to first token, expert-store reads, and temporary-storage traffic.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer K48f

Thank you for catching the inconsistent denominators.

H_gpu. Table 6's 6.459 GB/token is the per-rank storage traffic of the fixed replay at a steady host share of about 0.50. The 0.337 in the text includes prefill. Substituting it for the steady host share in the same equation creates an apparent device hit rate of 0.246. With the scopes aligned, the accounting is consistent with zero device hits. We measure zero across 2,663 steady intervals in the larger replay. With 92 routed layers and top-16 routing, one untruncated token touches 1,472 rank-local records; the 96 slots are reused under global LRU.

The 0.58 s/token floor. This is 12.915 GB divided by 22.4 GB/s, an idealized no-speculation floor for full routed load with all GPU misses served by pinned host memory, H_gpu = 0 and c = 0. It is not a universal lower bound for other cache policies or speculative schemes. Offline replay gives zero host hits at 4 and 8 GiB/rank under LRU on the recorded sequence; we do not present those cells as measurements of a larger GPU cache.

Predictive accuracy. Fixing the generation-4 constants gives 2.8 to 4.8% underprediction for expert load and 0.4 to 4.1% for total time across five subsequent operating points. The generation trajectory is not a single-factor ablation: generations 2 and 3 change both the I/O engine and the cache. We report inputs, predictions and timings separately.

Reference and layer-major. The 512-token, same-machine token-major reference matches layer-major logits on both ranks byte for byte. The separate MLA decode probe has rel-L2 5.6e-7, and long-context end-to-end checks are retrieval tests. The same 4,090-token prompt took 177.6 versus 142.1 s. At 96K D02, about 62 versus 24.3 minutes and 657.6 GB/rank in layer-major prefill are cross-build evidence, not a controlled scheduling-only speedup.

Reviewer_K48fOpenReview ↗

Dear authors

I thank the authors for the additional results. I think the first concern (the cost model and the performance floor) has been well addressed. But the second concern is partially addressed, and it is still unclear whether the Kimi-K3 comparison isolates the effect of the scheduling. The remaining validation gaps makes the contribution not enough for acceptance. Therefore, I'll maintain my score.

Reviewer_zscaOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is well organized, but it has some pretty obvious signs of AI-generated scientific writing. A lot of important experiments are still unfinished. Yet the paper already presents a remarkably complete story, from "Byte Economics" to four generations of system design, three "inversions of current practice," and a five-level roadmap predicting up to 20–30 tokens/s. The problem is that the story is much more complete than the evidence. Several of the strongest conclusions are still hypotheses or projections, but the writing makes them sound like established findings.

There are also some particularly strange writing choices that make the AI slop more noticeable. The paper claims its proposed model changes are evaluated at "equal intelligence" without actually demonstrating quality parity, calls a speculative performance prediction a "pre-registered test" without providing a registration, and includes odd internal-sounding phrases such as "approved two-depth partial" and "G0–G5 ignition ladder." These read more like fragments of an AI-assisted research plan or engineering progress report than a finished scientific paper. My concern is not simply that the authors used AI, but that the paper seems to have been written around a predetermined narrative, with incomplete experiments and speculative claims fitted into an unusually polished story. It is hard to tell which conclusions are actually established by the experiments and which are still part of the authors' research plan.

My main concern for the story is the novelty. A closely related work, From Tokens to Layers (MLSys 2026), already explores layer-major prefill scheduling for MoE inference, but this paper does not discuss or compare against it.already proposes a very similar idea of processing MoE prefill layer by layer to avoid repeatedly loading experts, but OMOE doesn't cite it. The real contribution may be extending this approach to models larger than DRAM, but that distinction needs to be made clear. I also have three concerns: (1) most performance comparisons are against their own implementations rather than strong controlled baselines; (2) the million-token correctness evaluation relies on very limited needle-retrieval tests; and (3) several model-design improvements are projected rather than demonstrated.

For the experiments using "unimplemented modules", how did authors evaluate the performance or measure the data?

Overall, I'd lean reject for now.

Critical strengths
  1. Strong end-to-end system implementation. The paper demonstrates running a native 2.8T MoE model on two RTX 5090s, even though the model weights exceed both GPU and host memory. Supporting million-token inference under these constraints is a substantial engineering achievement.

  2. Useful empirical measurements. The paper provides detailed measurements of expert caching, storage traffic, and performance across different system configurations. These results help understand the actual bottlenecks of large-scale MoE inference and offer useful insights for future system designs.

Critical weaknesses

My main concern is that the paper presents a much more complete scientific story than the experimental evidence supports. The system implementation is substantial, but the paper mixes measured results, unfinished experiments, analytical projections, and future research directions into a seemingly unified framework. There are also several technical inconsistencies and unusual writing artifacts that deserve closer examination.

A. Novelty and Experimental Evaluation

1. Significant overlap with existing Layered Prefill work. (§3, §7)

The central layer-major prefill idea is closely related to From Tokens to Layers (MLSys 2026), which already proposes processing MoE prefill along the layer dimension to avoid repeated expert loading. Surprisingly, this paper does not cite or discuss it. Extending layer-major execution to models larger than DRAM may be valuable, but the authors need to explain what is fundamentally new beyond the existing scheduling approach.

2. The main performance comparisons are largely against the authors' own implementations. (§2.2, Appendix E)

The four system generations provide useful engineering measurements, but they do not establish a sufficiently strong comparison with existing offloading systems. The paper compares with SSD-LLaMA and other methods under different hardware and workloads, making it difficult to attribute the reported improvements to the proposed mechanisms.

3. The four-generation comparison is not entirely single-variable controlled. (§2.2, Appendix E)

The paper claims that each generation changes one term, but the I/O engine also changes across configurations. Generation 2 uses asynchronous io_uring, whereas the production configurations use threaded preadv. Some individual ablations are controlled, but the overall four-generation trajectory cannot be treated as a clean single-variable experiment without further explanation.

4. The headline 12.1× prefill improvement is measured on GLM-5.2, not Kimi-K3. (§3)

The strongest direct scheduling speedup uses a 15,360-token GLM-5.2 workload, while the headline trillion-scale deployment uses Kimi-K3. The paper should provide a clearer controlled comparison on the primary model to establish how much of the improvement comes specifically from layer-major scheduling.

5. Independent validation of the cost model is still unfinished. (§2.2)

The authors explicitly acknowledge that the reported measurements come from their development trajectory rather than held-out predictions, and that controlled reruns on a second storage medium are still ongoing. This weakens the claim that the model has been independently validated as a general predictive tool.

B. Correctness, Hardware, and Reproducibility

6. The full-VRAM reference is insufficiently explained. (§5, Reproducibility Statement)

The paper repeatedly emphasizes byte-exact correctness against a full-VRAM reference, but the model has approximately 1.45 TB of expert weights, far exceeding the available GPU memory. It is unclear what hardware generated the reference, how the reference execution was constructed, and how its correctness was independently established.

7. The acceptance protocol appears broader than the actual evaluation. (Appendix A, Appendix D)

Appendix A states that every tier passes needle retrieval at 2%, 50%, and 98% depth. However, Appendix D shows that the initial million-token build covers only two depths, while the final build reports only one depth per tier. The authors should reconcile the acceptance protocol with the actual experiments performed.

8. The PCIe hardware configuration requires clarification. (Reproducibility Statement)

The paper reports a Ryzen 9950X with two RTX 5090 GPUs on PCIe Gen5 x16. However, the 9950X provides only 24 usable native PCIe lanes. Two independent CPU-attached x16 GPU links would therefore require additional hardware or a different topology. The authors should report the motherboard, negotiated link widths, PCIe switches if any, and actual transfer topology. This is particularly important because PCIe bandwidth is central to the proposed model.

9. The claimed storage-traffic breakdown is incomplete. (§5.3, Appendix F)

The paper reports 19.3 TB of reads and 3.0 TB of writes during million-token prefill, attributing the writes to KV spill, prefix persistence, and scratch management. However, the final-build table aggregates these under prefill (all) rather than reporting a detailed component breakdown. The evidence therefore does not fully explain where the remaining storage traffic comes from.

C. Experiments

10. The claimed 0.58 s/token software-independent floor is not fully justified. (§2.3)

Equation 2 explicitly includes the GPU cache hit fraction, H_gpu, but the stated asymptotic floor simplifies to B_tok / BW_pcie without that factor. The claim that no cache improvement can pass this floor therefore requires additional assumptions. The paper presents a conditional estimate as a more general hardware limit.

11. The expert-versus-KV traffic comparison (§2.1, §5)

The paper states that the 12.9 GB of expert traffic per token is three orders of magnitude larger than KV and activation traffic. Yet its own million-token configuration has approximately 13.5 GiB of MLA KV per rank, potentially requiring a comparable amount of KV streaming per decode step. The original statement needs to be restricted to the relevant context regime.

12. The claim that long prefill is attention-bound lacks a sufficiently detailed bottleneck analysis. (§5.3, Appendix B)

The authors argue that attention becomes the bottleneck because the storage array averages around 1 GB/s against a 14.87 GB/s peak. Low storage utilization alone does not establish that attention is the critical path. More detailed GPU profiling and execution-time decomposition would be needed to separate attention computation from other synchronization, scheduling, and memory-management overheads.

13. The five-level model-design ladder is largely speculative. (§6, Table 3)

Table 3 projects throughput improvements from 0.61 to 20–30 tokens/s through increasingly substantial changes to the model architecture. Most of these changes are not implemented or evaluated. The paper also treats the improvements as cumulative, without demonstrating that their performance benefits and model-quality assumptions can hold simultaneously.

D. Unfinished Research Presented as a Complete Framework

14. Several important experiments and capabilities are explicitly unfinished.

The manuscript itself identifies multiple missing pieces:

  • §2.2: Controlled validation on a second storage medium is still ongoing.
  • §4.2: The proposed achievable-H frontier across LRU, static-pinning, and oracle policies is described without corresponding reported results.
  • §5.3: Persistent KV checkpointing and resumption are not implemented.
  • §6: Dynamic top-k quality calibration is still ongoing.
  • §6: Kimi-K3 has no native MTP head, so the relevant speculative-decoding improvements are not demonstrated on the target checkpoint.
  • §8: Output-quality evaluation beyond retrieval is still in progress.
  • §8: Final long-context results are based on individual runs without variance estimates.

Unfinished experiments are not inherently problematic. However, several of these missing pieces directly relate to the paper's central claims about generality, correctness, performance, and model design.

15. The practical long-context serving scenario depends on an unimplemented capability. (§5.3, §6)

The million-token prefill requires 6.14 hours, but the paper treats it as a one-time indexing cost that could be amortized through persistent KV reuse. Since checkpointing and resumption are not implemented, this deployment pattern remains a proposal rather than a demonstrated system capability.

How did authors get the results?

E. AI-Assisted Writing Artifacts and Overstated Scientific Narrative

17. The narrative appears more complete than the underlying research. (Throughout)

The paper moves neatly from "Byte Economics" to four generations of system design, three major observations, three "inversions of current practice," and finally a five-level model-design roadmap. Yet many experiments needed to validate this overall picture remain unfinished. It gives the impression that the conclusions and framework were settled before the experiments supporting them were completed. This is one of the most noticeable signs of AI-assisted narrative overconstruction in the manuscript.

18. Several phrases resemble internal engineering or agent-generated progress reports. (§5.3, Appendix A)

Examples include:

  • "the approved two-depth partial"
  • "G0–G5 ignition ladder"
  • "per-tier runbooks"
  • "admitted through byte-exact ... gates"
  • "claim-to-artifact ledger"

These phrases are not inherently evidence of fabrication or even necessarily AI-generated text. However, some are undefined or unexplained, and they read more like internal development and acceptance-process notes than finished academic writing. In particular, it is unclear who approved the partial evaluation or what the G0–G5 ignition ladder formally represents.

19. A speculative prediction is called a "pre-registered test." (§6)

The paper writes, "We offer the prediction as a pre-registered test of the byte-economics framing." However, no registration record, protocol, or independent evaluation plan is provided. A falsifiable future prediction is not automatically a pre-registered study. This is an unusually strong methodological label for what is currently an analytical projection.

20. Several ordinary observations are packaged as major scientific findings. (§4, §6, Appendix B)

The manuscript uses expressions such as "Three measured facts," "Three inversions of current practice," and "The headline numbers over-constrain one another, and the redundancy is itself evidence." One of the three measured facts is explicitly described as an arithmetic fact, while the self-consistency checks largely establish that reported quantities are numerically compatible. These are useful checks and observations, but the writing repeatedly elevates them into broader scientific contributions without corresponding independent validation.

21. The paper repeatedly uses language stronger than the evidence supports. (Throughout)

Examples include:

  • "exactly equivalent to buying faster links" for a conditional cache/bandwidth relationship.
  • "each evaluated at equal intelligence" without matched-quality experiments.
  • "verifying k drafted tokens is nearly free" without expert-union measurements.
  • "software-independent floor" despite additional assumptions in the model.
  • "long context 'for free'" despite measured decode overhead.
  • "on hardware most users own" for a specialized workstation with two RTX 5090 GPUs and enterprise Optane drives.

The concern is not individual word choice, but a repeated pattern of turning conditional analyses or limited measurements into broad, confident conclusions.

22. The manuscript contains smaller consistency and presentation problems.

Several details reinforce the impression that the writing and presentation have not undergone a thorough final consistency check:

  • Conclusion vs. Reproducibility Statement: "We release" versus "will be released," even though artifacts are not yet linked.
  • Figure 1: SSD-LLaMA is plotted at the axis despite its context length being unreported, which could visually exaggerate the gap.
  • Appendix C vs. Appendix D: The reported million-token context differs slightly between 1,048,512 and 1,048,518 tokens without explanation.
  • Figure 4: Some TTFT labels are difficult to read against the plot background.
  • References: At least one long URL is poorly wrapped and extends beyond the intended text area.

These issues are individually minor, but together with the more substantial problems above, they suggest that the paper's polished narrative has received more attention than the verification and consistency of its supporting details.

Questions for authors
  1. Can the authors provide a detailed execution-time breakdown for the six-hour million-token prefill? The paper argues that attention computation becomes the bottleneck, but the evidence is mostly based on low SSD utilization. How much time is actually spent on attention kernels, expert loading, intermediate-state transfers, synchronization, and other overheads? This would help establish whether the proposed cost model captures the real bottlenecks.

  2. What happens when the same million-token context is queried a second time? Does the system need to repeat the six-hour prefill, or can it reuse the existing context within the same running session? What is the measured latency for a subsequent query? This seems important for understanding the actual usefulness of the system beyond a single demonstration.

  3. Can the authors clarify how the full-VRAM reference was generated and validated? Given the model's 1.45 TB expert store, what hardware and execution framework were used to generate the reference outputs? How were the reference logits verified independently of OMOE? Since byte-exact correctness is one of the strongest claims, understanding the reference methodology would significantly affect my assessment.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer zsca

Thank you for the detailed review.

From Tokens to Layers. We agree that prior layer-group scheduling already addresses redundant expert traffic and interleaves prefill with decode. We cite and distinguish that work. Our objective is to bound reads from the SSD-backed expert store, using a bounded scratch path for intermediates at long context.

Layer-major on K3. On the same 4,090-token prompt, chunked and layer-major builds took 177.6 and 142.1 s to first token and read 1,022.9 and 588.6 GB/rank of expert data. For the matched 96K D02 input, chunked prefill took about 62 minutes (reconstructed from logs) and layer-major took 24.3 minutes, reading 657.6 GB/rank against a 723.2 GB/rank complete store. These are cross-build comparisons. The later build changes wave admission as well as the scheduling path; the measurements do not isolate scheduling alone.

Numerical reference. The reference is token-major execution of the full converted model on the same two GPUs, not a model entirely resident in VRAM. The 512-token check compares final-position logits for the same prompt: rel-L2 is zero and the saved tensors match byte for byte on both ranks. Separately, the host-streamed MLA decode probe matches its GPU-resident attention reference to rel-L2 5.6e-7. The long-context end-to-end check is needle retrieval, not a million-token full-model logits comparison.

Hardware. The board is MAXSUN MS-iCraft B850 AIGA. The GPUs sit on separate CPU PCIe 5.0 x8 root ports, on one NUMA node, with no intervening PCIe switch or NVLink. The measured DMA rate is 22.4 GB/s per rank, not the nominal link rate. We correct the reported memory configuration to 3600 MT/s.

Second query. Prefix reuse is not implemented. A second request over the same 1M context repeats prefill. In the same service instance a repeated 4,090-token prompt returned identical text, with TTFT 2.98 and 2.95 minutes; this was not a repeated 1M measurement.

Prefill breakdown. At 96K, the existing rank-0 timing ledger records 1,387.5 s across 93 layers, including 1,107.4 s in the expert path. Combine and runner submission account for 303.6 and 101.5 s within that scope. Decode layout restore is separate, at 21.6 s. These host timers are not an exclusive kernel profile. In the final 1M run, expert reads were 1.348 TB during prefill and 0.507 TB during decode. Total array traffic averaged 1.01 GB/s during prefill; that average does not identify the bottleneck or rule out stage-local I/O stalls.

Projections and acceptance. The design ladder reports model estimates, not measured rates. We remove the wording about preregistration and clarify the retrieval protocol: 2%, 50% and 98% on earlier 96K and 256K builds; 2% and 98% on the initial 1M build; 2% per tier on the final build. The final 1M prompt contains 1,048,512 locally counted tokens and 1,048,518 at the server after templating. The experts-dominating-KV claim applies at short context; measured decode is 3.53 s/token at 1M versus 1.92 at 96K.

Reviewer_RQDFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OMOE to address the challenge of deploying extremely large MoEs on consumer GPUs by leveraging memory storage, CPU, and GPU (RTX 5090s, 192 GB of DRAM, and a 3.2 TB storage tier). Thus, it uses a hybrid, hierarchical approach for on-demand offloading and pipeline hiding for inference latency. The paper defines the term 'Byte Economics' to quantitatively analyze the effectiveness of MoE inference.

Critical strengths
  1. The paper experiments with and targets extremely large MoEs, especially GLM. Many prior works do not tackle this type of MoE.
  2. The paper proposes Layer-major prefill to handle the repeated weight load in chunk-major prefill.
Critical weaknesses
  1. The paper proposes GPU–DRAM–SSD offloading, and pipeline hiding, which is similar to prior work. This is a well-established method.
  2. Have you compared with existing vLLM and SGLang frameworks?
Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewer RQDF

Thank you for the review.

Relation to prior work. GPU, DRAM and SSD offloading with overlap is established and is not our novelty claim. The contribution is an implementation and measurements in a regime where the 1.45 TB routed expert store substantially exceeds 192 GB of DRAM. Routed MXFP4 records and routing are preserved in the main results; resident tensors use the disclosed conversion. The layer-major schedule bounds prefill expert reads to at most one store pass in the accepted configuration. At 96K, the measured read was 657.6 GB per rank against a 723.2 GB per-rank store. The general comment reports the held-out cost-model validation.

vLLM and SGLang. We extend SGLang's front end and scheduler with expert storage, caching, scheduling and kernel hooks. The documented CPU weight offloading in vLLM and pinned host backing in SGLang Paged Experts do not provide this SSD-backed expert tier. The routed payload alone exceeds the combined DRAM and VRAM here. We did not establish a stock equal-hardware, equal-fidelity baseline; this is a capacity distinction, not a measured speed comparison against those engines.

Comparison scope. SSD-LLaMA also evaluates K3 from SSD on a consumer GPU. A newer K3.1 prototype reports faster short-context decode and prefix reuse on different hardware. We distinguish those results from our measured million-token run; we do not claim the first local K3 system or a speed ranking across setups.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Post-submission measurements and corrections

Thank you for the reviews. We distinguish measurements completed before submission from the follow-up experiments completed on the same machine on September 29. The corrections below do not rely on new experiments during discussion.

The revision separates the 512-token numerical gate from long-context retrieval, labels the K3 timing comparison as cross-build, and corrects the hardware and byte-accounting descriptions. The 1M run is evidence of capacity, not interactive performance: TTFT is 6.14 h, decode takes 3.53 s/token, and a second request would repeat prefill.

The AI use statement describes assistance with implementation and writing. For the technical claims, we distinguish measured run data, offline trace replay, and conditional model projections. We remove claims that lacked a corresponding measurement.

  1. Cost model. With the generation-4 constants held fixed (22.4 and 6.2 GB/s; c = 0.31 s), Eq. 1 predicts five new operating points using their measured cache shares and bytes. It underpredicts total decode time by 0.4 to 4.1% and expert load time by 2.8 to 4.8%. The 24-prompt timing ledger shows about 85 ms/token of exposed miss-copy time, consistent with the model's simplifying overlap assumption being optimistic. We remove the claim of a second-medium rerun, which was not performed.
  2. Cache hits. The submitted 0.337 came from a three-prompt replay and includes prefill. In a 24-prompt replay with 2,663 steady decode intervals, the steady host hit share is 0.375 (0.284 to 0.432 across requests), or 0.345 over the whole replay. Mean decode is 1.894 s/token versus 1.649 s/token on the original fixed replay. The workload and build differ, so this is not a controlled speed comparison.
  3. Cache policies. We removed static pinning, which was not implemented. Offline replay reproduced measured hits and misses for all 365,239 route events. At 20 GiB/rank, the measured LRU host hit share is 0.375; simulated LFU and offline-oracle Belady give 0.224 and 0.605.
  4. Dynamic top-k. On K3, router-mass cutoffs of 0.95, 0.90 and 0.80 give 1.11x, 1.21x and 1.51x decode speedups. Across 2,487 scored tokens in 32 segments of 12 texts, teacher-forced mean NLL changes by +0.0066, +0.0101 and +0.0338 nats/token, with top-1 agreement of 93.6%, 92.9% and 88.3%. At every cutoff, all eight greedy responses differ from the untruncated outputs; the untruncated repeat matches all eight. This is not a task-accuracy evaluation.
  5. mmap. A later isolated single-rank benchmark gives 2.68 to 6.29 GB/s across cold mmap variants, including file huge pages, versus 9.46 GB/s for O_DIRECT into pinned buffers at QD16. We remove the claim about faults falling inside compute windows; the benchmark does not measure that.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.