- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper trains a resident transformer with a larger pool of experts on SSD. Expert addresses are known from the input token before the forward pass, so reads can overlap earlier layers. Growing the store improves language-modeling loss across the tested training budgets, and a small backbone with stored experts reaches similar loss to a much larger resident dense model. The actual NVMe serving path also shows faster decoding with substantially less GPU memory. The paper explores early contextual routing and multi-expert combinations, then uses these results to motivate a larger deployment scenario.
Critical strengths
The architecture and serving results make a useful package. The authors train around the storage constraint and check whether the added capacity earns its cost. Retraining the dense references at each token budget helps, and the gains persist across backbone sizes and training budgets.
I appreciate the checks on the disk path. O_DIRECT bypasses the page cache, the equal-precision path has bitwise parity, and table 10 evaluates loss through the served weights. The synchronous-fetch ablation supports the benefit of overlap too. The measured speed and memory gains are enough to make this a useful small-scale result for me, without relying on the frontier projection.
Critical weaknesses
I'm not yet sure how robust the overlap is. App. E only times batch-1 generation from token id 1, and the resident/offloaded comparison in table 12 covers the smaller backbone, not the headline 220M model. A timeline over varied prompts would help show whether the fetch stays hidden when launch overhead is reduced. The excluded attempt retaining 0.802 of resident speed also needs a clearer explanation: what independently identified contention, and how often does it recur?
I'd give sec. 6 less weight. The learned router has no NVMe timing, and four-expert averaging quadruples traffic for a small loss improvement. The frontier projection further assumes that quality scaling and transfer deadlines survive a much larger architecture. The caveats help, but the abstract leans too heavily on this untested scenario.
Questions for authors
-
For the 220M configuration, how fast is NVMe execution relative to the same model with resident experts at matched precision and on identical token traces? Please include varied held-out prompts and a timeline showing exposed I/O stall, transfer, dequantization, and the first consuming layer. This would help separate offloading overhead from the comparison with dense-1B.
-
What was the numerical exclusion criterion for attempt 4, and what independently identified contention? How often does that slowdown recur, and what do the per-token latency distributions look like when it is included?
-
Is the previous-position state in eq. (2) taken before or after the expert update? How is its dependency handled during training and prefill, and what overhead does that schedule add? Decode parity alone doesn't answer this part of the contextual-routing extension.
4 comments
W5. Point: The learned router has no NVMe timing.
Answer: It now has a measured NVMe timing at 220M, with bitwise-identical logits. Using the previous step's state keeps the lead time needed to fetch blocks. It runs at 0.984 of resident speed with its stored parameters on the drive; the 220M token-id store is served in 0.61 GB of GPU memory, against 2.11 GB for dense-1B. Q3 explains how the state is carried in decode, training and prefill.
At 1B, the same router (B-prev) matches the token-id model: 57.7 on the four-task mean against 58.2, and 0.0125 and 0.0008 nats behind in held-out loss over two runs. From NVMe, it keeps 0.97 of its resident speed in 2.48 GB of GPU memory instead of 23.1 GB, with logits bitwise identical, and decodes 1.8× faster than fetching the same blocks at the layer. It reaches 0.93 of B's speed measured beside it. The difference is the router's own compute, 1.0 ms per token, the same resident and from the drive, so the drive path costs the learned address nothing extra.
Revised paper: Add the measured router timings at 220M and 1B.
W6. Point: Averaging four experts quadruples traffic for a small improvement.
Answer: The design we scale combines experts at fixed traffic. It selects four quarter-size blocks, using the bytes of one full block. This preserves the benefit of combining experts without multiplying reads. At 220M and equal stored parameters and bytes, eight layers of four quarter-size blocks reach 2.7456 at 10^9 tokens, against 2.7661 for eight layers of one full block and 2.7596 for the two-layer store: 0.02 and 0.014 nats better. Full-block averaging remains a feasibility ablation; B uses the fixed-byte design.
Revised paper: Present fixed-byte combination as the scaled design in §7.2 and separate it from the averaging ablation.
W7. Point: The frontier projection carries too much weight relative to the measured result.
Answer: The revision labels measured components and projected quality separately, and both assumptions you name are now tested at 1B: the store keeps its advantage over dense in the larger family, and the eight-layer schedule meets its transfer deadlines, serving at 0.97 of resident speed in 2.35 GB of GPU memory instead of 23 GB.
Beyond 1B, the projection rests on the paper's three requirements (R1-R3) and the serving bound they lead to, bandwidth divided by bytes per token. Total stored parameters are not in the bound, so the store grows at fixed bytes: at 27.7M, an 8× larger store at identical bytes per token raises the gain over its backbone from 0.206 to 0.325 nats. What each stored parameter is worth, its exchange rate, is set by the address (R3). With training, the learned prefetchable router's rate holds or rises while the token-id rate falls, and at 1B it matches token id in loss, which suggests it is the address to scale with, as contextual routers are in frontier models. The bytes a step can hide also grow with the backbone and the drive (R1): the scenario's 16B backbone, at 20 tokens/s (50 ms per token), can admit up to 700 MB on a 14 GB/s PCIe-5 drive, about what A reads at 1B. The budget's measured cost already falls with scale, from 0.064 nats at 27.7M to 0.025 at 1B, in different store layouts. The other changes a released mixture of experts needs cost little (table). Released models are already moving this way: a token's share of a layer's experts fell from 25% in 2023 to as little as 1.6%, totals now reach 1.6-2.8T parameters at 3-4% active, and DeepSeek-V4 addresses its first three expert layers by token id.
| change a frontier mixture of experts needs | measured on | cost |
|---|---|---|
| R1: address known before the layer that uses it | Qwen3-30B-A3B and Qwen3-Next-80B, made prefetchable post-hoc, weights frozen | within 0.001-0.009 nats; 7-task mean 73.0 vs 73.1 at 30B |
| R1: store layers that start deep enough | 27.7M and 220M placements; B starts at layer 5 of 30 | within 0.015 nats of the two-layer layout at equal bytes |
| R2: drive-sized blocks | today's experts: 3 MiB (Qwen3-Next), 4.7 MB (Qwen3-30B) | none; already there |
| R1: a byte budget under the compute window | B against A at equal total and active parameters | 0.025 nats for 12× fewer bytes; 77-87% of A's gain kept |
The table's first row tests the address only: bytes per token stay as released, so those models stay drive-limited, as expected. It shows that the rates released learned-router models reach (0.65-0.67) survive early addressing, and those rates give 235-255B dense-equivalent. Our own measured rates give 66-163B, and the body states both ends. The abstract keeps “has the potential to match a dense model of several hundred billion parameters” and now names this support.
Revised paper: State the abstract's potential at the rates released learned routers reach, with §9 bracketing it between those rates and ours. Label measured components and projected quality separately in §9.
Q1. Point: Compare NVMe with resident experts on varied prompts and show transfer, stall, dequantization and first use.
Answer: On identical token traces and matched precision, NVMe execution retains 0.989 of resident speed with bitwise-identical tokens and logits. Both modes use the same checkpoint, int8 weights, engine and held-out spans, which isolates offloading overhead. Separately from that, the store serves in 0.61 GB of GPU memory against 2.11 GB for dense-1B and decodes about 1.7× faster (1.36× in the submission, before the engine rewrite described in W3; dense-1B runs through the same engine).
The timeline explains the small overhead: blocks arrive before their consuming layers, and dequantization fits under computation. It is from the node with the longer step; W3 covers the node whose step is 1.8× shorter.
| event, median over spans | timing |
|---|---|
| request blocks for both store layers | step start |
| transfer complete | 1.41 ms after request |
| first consuming layer asks | 2.85 ms after request |
| margin (median per step) | 1.51 ms; 0 of 4,096 steps late |
| exposed stall | 0.0075 ms |
| dequantization per store layer | 0.19 ms from host memory, 0.07 ms from GPU memory, hidden |
Revised paper: Add the timeline and matched-mode protocol in §5.5 and App. E.
Q2. Point: Give the exclusion criterion, independent diagnosis, recurrence and latency distributions including the excluded attempt.
Answer: That attempt came from the submission's engine, and the revision no longer relies on it or on any exclusion of that kind. The numerical rule was the submission's acceptance bound: the synchronous ablation had to fall within [1 + 0.5(p − 1), 1.35p], where p is the latency-model prediction. The attempt measured 3.04 against p = 1.49, above the upper bound of 2.02. Contention was inferred from timing and GPU power. That harness had no disk telemetry, so it was never independently confirmed, and it was the only run of that configuration (side-stream synchronization, which the rewrite removes).
We therefore re-measured every serving number with the rewritten engine under one protocol. It brackets each timed process with resident measurements and reruns any process whose resident bracket drifts over 2%, a check that does not depend on the result being measured; no quoted row includes a discarded process. The old harness saved only block means, so a per-token distribution including that attempt does not exist. The rewritten engine's distributions at 220M, in milliseconds: resident median 11.52, p99 11.88, maximum 16.85; read-ahead 11.63, 12.03, 17.15; synchronous fetching 13.62, 14.17, 72.97. Read-ahead stays close to resident even in the slowest tokens.
Revised paper: Replace the App. E attempts with the new protocol, measurements and latency distributions.
Q3. Point: Is the previous-position state before the expert update, and how do training, prefill and overhead work?
Answer: It is before the expert update, after that layer's attention and dense feed-forward network. Training and prefill shift an already-computed tensor by one position, preserving parallel training without another forward pass. During decode, the session carries the previous step's state, so addresses are ready before the next forward pass. During prefill, each layer starts reads once its inputs across the prompt are computed.
The router adds 44M parameters at 1B, about 4% of active. As a correctness check in full precision, the decode path matches full-sequence evaluation within 5×10^-5 at 220M. Measured scheduling overhead is 0.18 ms per token, versus 0.10-0.15 ms for token-id stores. At 1B the router's own compute adds 1.0 ms per token, about 7% of the step. This measures the dependency overhead beyond parity alone.
Revised paper: Define the state and all three schedules in §3.
With the overlap, the router and the frontier components now measured, we hope you will consider raising your score.
Thank you for the additional experiments. The NVMe/resident comparisons and shorter-step tests address my concerns about hiding I/O overhead. The larger-model results and learned-router experiments also strengthen my positive view of the paper. I will maintain my current score. Please include these results and explanations in the final version.
Names follow the table in our general response: B is the new store model (23B total, 1B active, experts on NVMe); A is a full-depth mixture of experts with the same total and active parameters; dense-1B and dense-3B are dense models. Serving results are on one L40S GPU.
W1. Point: Timing generation from token id 1 alone does not establish robust overlap.
Answer: The broader protocol now checks held-out prompts from both the training distribution and other domains. It uses 16 spans, repeated timings and identical token traces across serving modes. The broader measurement improves on the submission: the 220M store retains 0.989 of resident speed with bitwise-identical tokens and logits. It serves in 0.61 GB of GPU memory, against 2.11 GB for dense-1B. Q1 gives the transfer timeline, so the result can be checked against when each layer needs its blocks.
Revised paper: Replace the narrow App. E protocol with the broader prompt and timing protocol.
W2. Point: The resident-versus-offloaded comparison only covers the smaller backbone.
Answer: The comparison now covers both 220M and 1B, with bit-exact output through the drive at both scales. Reading ahead recovers nearly all resident speed, showing that the benefit survives the larger model's distributed store. B retains 0.97 of resident speed in 2.35 GB instead of 23 GB. Reading ahead makes it 1.9× faster than fetching the same blocks when their layer needs them. From NVMe it also decodes 1.37× faster than A held entirely in GPU memory, served by the same engine and kernels (A runs 390 expert blocks per token across 30 layers, B 32 across 8). With both read from the drive, B is 12× faster.
Revised paper: Extend Table 12 with the larger models and their synchronous-fetch controls.
W3. Point: Does overlap survive when launch overhead is reduced?
Answer: Yes, a rewritten engine tests that case, and a shorter-step node also preserves overlap. The submitted engine dequantized fetched blocks on a separate GPU stream, then handed them to each layer through an event. That required several small launches and copies. The rewritten engine reads into pinned host memory, which the GPU can access directly. One fused kernel per store layer gathers and dequantizes the blocks in place, eliminating copies and the separate stream.
The shorter-step test addresses the concern directly: with the 220M step 1.8× shorter, serving still retains 0.99 of resident speed. The larger-model result is given in W2.
Revised paper: Explain the engine change and add the shorter-step test in §3.
W4. Point: What caused the excluded attempt, and how often did it recur?
Answer: That attempt (the one that retained 0.802 of resident speed) came from the submission's engine, and no result rests on it anymore. It was the only run of its configuration (side-stream synchronization on a faster node), so we cannot say how often it would recur, and the rewritten engine has no separate stream (W3). We could not confirm its cause independently, so instead of relying on an exclusion, we removed the side stream in the rewrite and re-measured every serving number under one protocol. Q2 gives the old rule and the limits of its diagnosis.
Revised paper: Replace the App. E attempts with the new protocol and measurements.