My main concern is that the paper presents a much more complete scientific story than the experimental evidence supports. The system implementation is substantial, but the paper mixes measured results, unfinished experiments, analytical projections, and future research directions into a seemingly unified framework. There are also several technical inconsistencies and unusual writing artifacts that deserve closer examination.
A. Novelty and Experimental Evaluation
1. Significant overlap with existing Layered Prefill work. (§3, §7)
The central layer-major prefill idea is closely related to From Tokens to Layers (MLSys 2026), which already proposes processing MoE prefill along the layer dimension to avoid repeated expert loading. Surprisingly, this paper does not cite or discuss it. Extending layer-major execution to models larger than DRAM may be valuable, but the authors need to explain what is fundamentally new beyond the existing scheduling approach.
2. The main performance comparisons are largely against the authors' own implementations. (§2.2, Appendix E)
The four system generations provide useful engineering measurements, but they do not establish a sufficiently strong comparison with existing offloading systems. The paper compares with SSD-LLaMA and other methods under different hardware and workloads, making it difficult to attribute the reported improvements to the proposed mechanisms.
3. The four-generation comparison is not entirely single-variable controlled. (§2.2, Appendix E)
The paper claims that each generation changes one term, but the I/O engine also changes across configurations. Generation 2 uses asynchronous io_uring, whereas the production configurations use threaded preadv. Some individual ablations are controlled, but the overall four-generation trajectory cannot be treated as a clean single-variable experiment without further explanation.
4. The headline 12.1× prefill improvement is measured on GLM-5.2, not Kimi-K3. (§3)
The strongest direct scheduling speedup uses a 15,360-token GLM-5.2 workload, while the headline trillion-scale deployment uses Kimi-K3. The paper should provide a clearer controlled comparison on the primary model to establish how much of the improvement comes specifically from layer-major scheduling.
5. Independent validation of the cost model is still unfinished. (§2.2)
The authors explicitly acknowledge that the reported measurements come from their development trajectory rather than held-out predictions, and that controlled reruns on a second storage medium are still ongoing. This weakens the claim that the model has been independently validated as a general predictive tool.
B. Correctness, Hardware, and Reproducibility
6. The full-VRAM reference is insufficiently explained. (§5, Reproducibility Statement)
The paper repeatedly emphasizes byte-exact correctness against a full-VRAM reference, but the model has approximately 1.45 TB of expert weights, far exceeding the available GPU memory. It is unclear what hardware generated the reference, how the reference execution was constructed, and how its correctness was independently established.
7. The acceptance protocol appears broader than the actual evaluation. (Appendix A, Appendix D)
Appendix A states that every tier passes needle retrieval at 2%, 50%, and 98% depth. However, Appendix D shows that the initial million-token build covers only two depths, while the final build reports only one depth per tier. The authors should reconcile the acceptance protocol with the actual experiments performed.
8. The PCIe hardware configuration requires clarification. (Reproducibility Statement)
The paper reports a Ryzen 9950X with two RTX 5090 GPUs on PCIe Gen5 x16. However, the 9950X provides only 24 usable native PCIe lanes. Two independent CPU-attached x16 GPU links would therefore require additional hardware or a different topology. The authors should report the motherboard, negotiated link widths, PCIe switches if any, and actual transfer topology. This is particularly important because PCIe bandwidth is central to the proposed model.
9. The claimed storage-traffic breakdown is incomplete. (§5.3, Appendix F)
The paper reports 19.3 TB of reads and 3.0 TB of writes during million-token prefill, attributing the writes to KV spill, prefix persistence, and scratch management. However, the final-build table aggregates these under prefill (all) rather than reporting a detailed component breakdown. The evidence therefore does not fully explain where the remaining storage traffic comes from.
C. Experiments
10. The claimed 0.58 s/token software-independent floor is not fully justified. (§2.3)
Equation 2 explicitly includes the GPU cache hit fraction, H_gpu, but the stated asymptotic floor simplifies to B_tok / BW_pcie without that factor. The claim that no cache improvement can pass this floor therefore requires additional assumptions. The paper presents a conditional estimate as a more general hardware limit.
11. The expert-versus-KV traffic comparison (§2.1, §5)
The paper states that the 12.9 GB of expert traffic per token is three orders of magnitude larger than KV and activation traffic. Yet its own million-token configuration has approximately 13.5 GiB of MLA KV per rank, potentially requiring a comparable amount of KV streaming per decode step. The original statement needs to be restricted to the relevant context regime.
12. The claim that long prefill is attention-bound lacks a sufficiently detailed bottleneck analysis. (§5.3, Appendix B)
The authors argue that attention becomes the bottleneck because the storage array averages around 1 GB/s against a 14.87 GB/s peak. Low storage utilization alone does not establish that attention is the critical path. More detailed GPU profiling and execution-time decomposition would be needed to separate attention computation from other synchronization, scheduling, and memory-management overheads.
13. The five-level model-design ladder is largely speculative. (§6, Table 3)
Table 3 projects throughput improvements from 0.61 to 20–30 tokens/s through increasingly substantial changes to the model architecture. Most of these changes are not implemented or evaluated. The paper also treats the improvements as cumulative, without demonstrating that their performance benefits and model-quality assumptions can hold simultaneously.
D. Unfinished Research Presented as a Complete Framework
14. Several important experiments and capabilities are explicitly unfinished.
The manuscript itself identifies multiple missing pieces:
- §2.2: Controlled validation on a second storage medium is still ongoing.
- §4.2: The proposed achievable-H frontier across LRU, static-pinning, and oracle policies is described without corresponding reported results.
- §5.3: Persistent KV checkpointing and resumption are not implemented.
- §6: Dynamic top-k quality calibration is still ongoing.
- §6: Kimi-K3 has no native MTP head, so the relevant speculative-decoding improvements are not demonstrated on the target checkpoint.
- §8: Output-quality evaluation beyond retrieval is still in progress.
- §8: Final long-context results are based on individual runs without variance estimates.
Unfinished experiments are not inherently problematic. However, several of these missing pieces directly relate to the paper's central claims about generality, correctness, performance, and model design.
15. The practical long-context serving scenario depends on an unimplemented capability. (§5.3, §6)
The million-token prefill requires 6.14 hours, but the paper treats it as a one-time indexing cost that could be amortized through persistent KV reuse. Since checkpointing and resumption are not implemented, this deployment pattern remains a proposal rather than a demonstrated system capability.
How did authors get the results?
E. AI-Assisted Writing Artifacts and Overstated Scientific Narrative
17. The narrative appears more complete than the underlying research. (Throughout)
The paper moves neatly from "Byte Economics" to four generations of system design, three major observations, three "inversions of current practice," and finally a five-level model-design roadmap. Yet many experiments needed to validate this overall picture remain unfinished. It gives the impression that the conclusions and framework were settled before the experiments supporting them were completed. This is one of the most noticeable signs of AI-assisted narrative overconstruction in the manuscript.
18. Several phrases resemble internal engineering or agent-generated progress reports. (§5.3, Appendix A)
Examples include:
- "the approved two-depth partial"
- "G0–G5 ignition ladder"
- "per-tier runbooks"
- "admitted through byte-exact ... gates"
- "claim-to-artifact ledger"
These phrases are not inherently evidence of fabrication or even necessarily AI-generated text. However, some are undefined or unexplained, and they read more like internal development and acceptance-process notes than finished academic writing. In particular, it is unclear who approved the partial evaluation or what the G0–G5 ignition ladder formally represents.
19. A speculative prediction is called a "pre-registered test." (§6)
The paper writes, "We offer the prediction as a pre-registered test of the byte-economics framing." However, no registration record, protocol, or independent evaluation plan is provided. A falsifiable future prediction is not automatically a pre-registered study. This is an unusually strong methodological label for what is currently an analytical projection.
20. Several ordinary observations are packaged as major scientific findings. (§4, §6, Appendix B)
The manuscript uses expressions such as "Three measured facts," "Three inversions of current practice," and "The headline numbers over-constrain one another, and the redundancy is itself evidence." One of the three measured facts is explicitly described as an arithmetic fact, while the self-consistency checks largely establish that reported quantities are numerically compatible. These are useful checks and observations, but the writing repeatedly elevates them into broader scientific contributions without corresponding independent validation.
21. The paper repeatedly uses language stronger than the evidence supports. (Throughout)
Examples include:
- "exactly equivalent to buying faster links" for a conditional cache/bandwidth relationship.
- "each evaluated at equal intelligence" without matched-quality experiments.
- "verifying k drafted tokens is nearly free" without expert-union measurements.
- "software-independent floor" despite additional assumptions in the model.
- "long context 'for free'" despite measured decode overhead.
- "on hardware most users own" for a specialized workstation with two RTX 5090 GPUs and enterprise Optane drives.
The concern is not individual word choice, but a repeated pattern of turning conditional analyses or limited measurements into broad, confident conclusions.
22. The manuscript contains smaller consistency and presentation problems.
Several details reinforce the impression that the writing and presentation have not undergone a thorough final consistency check:
- Conclusion vs. Reproducibility Statement: "We release" versus "will be released," even though artifacts are not yet linked.
- Figure 1: SSD-LLaMA is plotted at the axis despite its context length being unreported, which could visually exaggerate the gap.
- Appendix C vs. Appendix D: The reported million-token context differs slightly between 1,048,512 and 1,048,518 tokens without explanation.
- Figure 4: Some TTFT labels are difficult to read against the plot background.
- References: At least one long URL is poorly wrapped and extends beyond the intended text area.
These issues are individually minor, but together with the more substantial problems above, they suggest that the paper's polished narrative has received more attention than the verification and consistency of its supporting details.
Reply to Reviewer K48f
Thank you for catching the inconsistent denominators.
H_gpu. Table 6's 6.459 GB/token is the per-rank storage traffic of the fixed replay at a steady host share of about 0.50. The 0.337 in the text includes prefill. Substituting it for the steady host share in the same equation creates an apparent device hit rate of 0.246. With the scopes aligned, the accounting is consistent with zero device hits. We measure zero across 2,663 steady intervals in the larger replay. With 92 routed layers and top-16 routing, one untruncated token touches 1,472 rank-local records; the 96 slots are reused under global LRU.
The 0.58 s/token floor. This is 12.915 GB divided by 22.4 GB/s, an idealized no-speculation floor for full routed load with all GPU misses served by pinned host memory, H_gpu = 0 and c = 0. It is not a universal lower bound for other cache policies or speculative schemes. Offline replay gives zero host hits at 4 and 8 GiB/rank under LRU on the recorded sequence; we do not present those cells as measurements of a larger GPU cache.
Predictive accuracy. Fixing the generation-4 constants gives 2.8 to 4.8% underprediction for expert load and 0.4 to 4.1% for total time across five subsequent operating points. The generation trajectory is not a single-factor ablation: generations 2 and 3 change both the I/O engine and the cache. We report inputs, predictions and timings separately.
Reference and layer-major. The 512-token, same-machine token-major reference matches layer-major logits on both ranks byte for byte. The separate MLA decode probe has rel-L2 5.6e-7, and long-context end-to-end checks are retrieval tests. The same 4,090-token prompt took 177.6 versus 142.1 s. At 96K D02, about 62 versus 24.3 minutes and 657.6 GB/rank in layer-major prefill are cross-build evidence, not a controlled scheduling-only speedup.