acceptodds
Under review as a conference paper at ICLR 2027

The Prefetchable Store: Decoupling Model Capacity from Fast Memory

13 views

Abstract

The largest model a machine can run at full speed is the largest one its fast memory holds. We argue that this is an addressing limit rather than a storage limit: a transformer chooses the parameters a token needs in the middle of its forward pass, so all of them must be resident. The prefetchable store is a storage-native architecture in which most of a model's trained parameters live on a commodity SSD as expert blocks whose addresses are known from the token id before any layer runs, so they stream under the model's own early layers. We state three requirements under which storage can hold trained capacity: the address must be known early enough to hide the transfer, the transfer must be shaped for the storage tier, and every address must receive enough training data to learn. We build a serving system that meets them and reproduces the resident model exactly through the disk path at equal precision, and we train models under the constraint from the first step against a dense frontier retrained at every budget under one recipe. Up to 8×10^9 training tokens, across three backbones and four store sizes, stored capacity lowers loss at every budget: a 220M backbone with a 3B-parameter store matches a fully resident 1B dense model under the same training budget while, served from NVMe, decoding faster in a third of the GPU memory. The price is total parameters, comparable to the overhead sparse models already pay, and it is paid on a drive rather than in fast memory. We further show that two ingredients behind the efficiency of frontier mixtures of experts, learned contextual selection and the combination of several fetched experts, work inside the prefetchable constraint. Extrapolating the measured scaling, a 16B resident backbone on a 24 GB consumer GPU with a 1T-parameter store on one 1 TB drive has the potential to match a dense model of several hundred billion parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, parameter offloading, memory-efficient inference, hash routing, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BTevOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper trains a resident transformer with a larger pool of experts on SSD. Expert addresses are known from the input token before the forward pass, so reads can overlap earlier layers. Growing the store improves language-modeling loss across the tested training budgets, and a small backbone with stored experts reaches similar loss to a much larger resident dense model. The actual NVMe serving path also shows faster decoding with substantially less GPU memory. The paper explores early contextual routing and multi-expert combinations, then uses these results to motivate a larger deployment scenario.

Critical strengths

The architecture and serving results make a useful package. The authors train around the storage constraint and check whether the added capacity earns its cost. Retraining the dense references at each token budget helps, and the gains persist across backbone sizes and training budgets.

I appreciate the checks on the disk path. O_DIRECT bypasses the page cache, the equal-precision path has bitwise parity, and table 10 evaluates loss through the served weights. The synchronous-fetch ablation supports the benefit of overlap too. The measured speed and memory gains are enough to make this a useful small-scale result for me, without relying on the frontier projection.

Critical weaknesses

I'm not yet sure how robust the overlap is. App. E only times batch-1 generation from token id 1, and the resident/offloaded comparison in table 12 covers the smaller backbone, not the headline 220M model. A timeline over varied prompts would help show whether the fetch stays hidden when launch overhead is reduced. The excluded attempt retaining 0.802 of resident speed also needs a clearer explanation: what independently identified contention, and how often does it recur?

I'd give sec. 6 less weight. The learned router has no NVMe timing, and four-expert averaging quadruples traffic for a small loss improvement. The frontier projection further assumes that quality scaling and transfer deadlines survive a much larger architecture. The caveats help, but the abstract leans too heavily on this untested scenario.

Questions for authors
  1. For the 220M configuration, how fast is NVMe execution relative to the same model with resident experts at matched precision and on identical token traces? Please include varied held-out prompts and a timeline showing exposed I/O stall, transfer, dequantization, and the first consuming layer. This would help separate offloading overhead from the comparison with dense-1B.

  2. What was the numerical exclusion criterion for attempt 4, and what independently identified contention? How often does that slowdown recur, and what do the per-token latency distributions look like when it is included?

  3. Is the previous-position state in eq. (2) taken before or after the expert update? How is its dependency handled during training and prefill, and what overhead does that schedule add? Decode parity alone doesn't answer this part of the contextual-routing extension.

4 comments

Names follow the table in our general response: B is the new store model (23B total, 1B active, experts on NVMe); A is a full-depth mixture of experts with the same total and active parameters; dense-1B and dense-3B are dense models. Serving results are on one L40S GPU.

W1. Point: Timing generation from token id 1 alone does not establish robust overlap.

Answer: The broader protocol now checks held-out prompts from both the training distribution and other domains. It uses 16 spans, repeated timings and identical token traces across serving modes. The broader measurement improves on the submission: the 220M store retains 0.989 of resident speed with bitwise-identical tokens and logits. It serves in 0.61 GB of GPU memory, against 2.11 GB for dense-1B. Q1 gives the transfer timeline, so the result can be checked against when each layer needs its blocks.

Revised paper: Replace the narrow App. E protocol with the broader prompt and timing protocol.

W2. Point: The resident-versus-offloaded comparison only covers the smaller backbone.

Answer: The comparison now covers both 220M and 1B, with bit-exact output through the drive at both scales. Reading ahead recovers nearly all resident speed, showing that the benefit survives the larger model's distributed store. B retains 0.97 of resident speed in 2.35 GB instead of 23 GB. Reading ahead makes it 1.9× faster than fetching the same blocks when their layer needs them. From NVMe it also decodes 1.37× faster than A held entirely in GPU memory, served by the same engine and kernels (A runs 390 expert blocks per token across 30 layers, B 32 across 8). With both read from the drive, B is 12× faster.

modelwhere the experts liveGPU memorytokens/sagainst resident
220M storeNVMe, read ahead0.61 GB–0.989
220M storeNVMe, fetched at the layer––0.845
BNVMe, read ahead2.35 GB75.00.97
Ball in GPU memory23 GB76.91
BNVMe, fetched at the layer2.35 GB40.10.52
dense-3BGPU memory5.80 GB70.7B from NVMe is 1.06× faster
Aall in GPU memory–54.7B from NVMe is 1.37× faster
ANVMe, fetched at the layer–6.1B from NVMe is 12× faster

Revised paper: Extend Table 12 with the larger models and their synchronous-fetch controls.

W3. Point: Does overlap survive when launch overhead is reduced?

Answer: Yes, a rewritten engine tests that case, and a shorter-step node also preserves overlap. The submitted engine dequantized fetched blocks on a separate GPU stream, then handed them to each layer through an event. That required several small launches and copies. The rewritten engine reads into pinned host memory, which the GPU can access directly. One fused kernel per store layer gathers and dequantizes the blocks in place, eliminating copies and the separate stream.

The shorter-step test addresses the concern directly: with the 220M step 1.8× shorter, serving still retains 0.99 of resident speed. The larger-model result is given in W2.

Revised paper: Explain the engine change and add the shorter-step test in §3.

W4. Point: What caused the excluded attempt, and how often did it recur?

Answer: That attempt (the one that retained 0.802 of resident speed) came from the submission's engine, and no result rests on it anymore. It was the only run of its configuration (side-stream synchronization on a faster node), so we cannot say how often it would recur, and the rewritten engine has no separate stream (W3). We could not confirm its cause independently, so instead of relying on an exclusion, we removed the side stream in the rewrite and re-measured every serving number under one protocol. Q2 gives the old rule and the limits of its diagnosis.

Revised paper: Replace the App. E attempts with the new protocol and measurements.

W5. Point: The learned router has no NVMe timing.

Answer: It now has a measured NVMe timing at 220M, with bitwise-identical logits. Using the previous step's state keeps the lead time needed to fetch blocks. It runs at 0.984 of resident speed with its stored parameters on the drive; the 220M token-id store is served in 0.61 GB of GPU memory, against 2.11 GB for dense-1B. Q3 explains how the state is carried in decode, training and prefill.

At 1B, the same router (B-prev) matches the token-id model: 57.7 on the four-task mean against 58.2, and 0.0125 and 0.0008 nats behind in held-out loss over two runs. From NVMe, it keeps 0.97 of its resident speed in 2.48 GB of GPU memory instead of 23.1 GB, with logits bitwise identical, and decodes 1.8× faster than fetching the same blocks at the layer. It reaches 0.93 of B's speed measured beside it. The difference is the router's own compute, 1.0 ms per token, the same resident and from the drive, so the drive path costs the learned address nothing extra.

Revised paper: Add the measured router timings at 220M and 1B.

W6. Point: Averaging four experts quadruples traffic for a small improvement.

Answer: The design we scale combines experts at fixed traffic. It selects four quarter-size blocks, using the bytes of one full block. This preserves the benefit of combining experts without multiplying reads. At 220M and equal stored parameters and bytes, eight layers of four quarter-size blocks reach 2.7456 at 10^9 tokens, against 2.7661 for eight layers of one full block and 2.7596 for the two-layer store: 0.02 and 0.014 nats better. Full-block averaging remains a feasibility ablation; B uses the fixed-byte design.

Revised paper: Present fixed-byte combination as the scaled design in §7.2 and separate it from the averaging ablation.

W7. Point: The frontier projection carries too much weight relative to the measured result.

Answer: The revision labels measured components and projected quality separately, and both assumptions you name are now tested at 1B: the store keeps its advantage over dense in the larger family, and the eight-layer schedule meets its transfer deadlines, serving at 0.97 of resident speed in 2.35 GB of GPU memory instead of 23 GB.

Beyond 1B, the projection rests on the paper's three requirements (R1-R3) and the serving bound they lead to, bandwidth divided by bytes per token. Total stored parameters are not in the bound, so the store grows at fixed bytes: at 27.7M, an 8× larger store at identical bytes per token raises the gain over its backbone from 0.206 to 0.325 nats. What each stored parameter is worth, its exchange rate, is set by the address (R3). With training, the learned prefetchable router's rate holds or rises while the token-id rate falls, and at 1B it matches token id in loss, which suggests it is the address to scale with, as contextual routers are in frontier models. The bytes a step can hide also grow with the backbone and the drive (R1): the scenario's 16B backbone, at 20 tokens/s (50 ms per token), can admit up to 700 MB on a 14 GB/s PCIe-5 drive, about what A reads at 1B. The budget's measured cost already falls with scale, from 0.064 nats at 27.7M to 0.025 at 1B, in different store layouts. The other changes a released mixture of experts needs cost little (table). Released models are already moving this way: a token's share of a layer's experts fell from 25% in 2023 to as little as 1.6%, totals now reach 1.6-2.8T parameters at 3-4% active, and DeepSeek-V4 addresses its first three expert layers by token id.

change a frontier mixture of experts needsmeasured oncost
R1: address known before the layer that uses itQwen3-30B-A3B and Qwen3-Next-80B, made prefetchable post-hoc, weights frozenwithin 0.001-0.009 nats; 7-task mean 73.0 vs 73.1 at 30B
R1: store layers that start deep enough27.7M and 220M placements; B starts at layer 5 of 30within 0.015 nats of the two-layer layout at equal bytes
R2: drive-sized blockstoday's experts: 3 MiB (Qwen3-Next), 4.7 MB (Qwen3-30B)none; already there
R1: a byte budget under the compute windowB against A at equal total and active parameters0.025 nats for 12× fewer bytes; 77-87% of A's gain kept

The table's first row tests the address only: bytes per token stay as released, so those models stay drive-limited, as expected. It shows that the rates released learned-router models reach (0.65-0.67) survive early addressing, and those rates give 235-255B dense-equivalent. Our own measured rates give 66-163B, and the body states both ends. The abstract keeps “has the potential to match a dense model of several hundred billion parameters” and now names this support.

Revised paper: State the abstract's potential at the rates released learned routers reach, with §9 bracketing it between those rates and ours. Label measured components and projected quality separately in §9.

Q1. Point: Compare NVMe with resident experts on varied prompts and show transfer, stall, dequantization and first use.

Answer: On identical token traces and matched precision, NVMe execution retains 0.989 of resident speed with bitwise-identical tokens and logits. Both modes use the same checkpoint, int8 weights, engine and held-out spans, which isolates offloading overhead. Separately from that, the store serves in 0.61 GB of GPU memory against 2.11 GB for dense-1B and decodes about 1.7× faster (1.36× in the submission, before the engine rewrite described in W3; dense-1B runs through the same engine).

The timeline explains the small overhead: blocks arrive before their consuming layers, and dequantization fits under computation. It is from the node with the longer step; W3 covers the node whose step is 1.8× shorter.

event, median over spanstiming
request blocks for both store layersstep start
transfer complete1.41 ms after request
first consuming layer asks2.85 ms after request
margin (median per step)1.51 ms; 0 of 4,096 steps late
exposed stall0.0075 ms
dequantization per store layer0.19 ms from host memory, 0.07 ms from GPU memory, hidden

Revised paper: Add the timeline and matched-mode protocol in §5.5 and App. E.

Q2. Point: Give the exclusion criterion, independent diagnosis, recurrence and latency distributions including the excluded attempt.

Answer: That attempt came from the submission's engine, and the revision no longer relies on it or on any exclusion of that kind. The numerical rule was the submission's acceptance bound: the synchronous ablation had to fall within [1 + 0.5(p − 1), 1.35p], where p is the latency-model prediction. The attempt measured 3.04 against p = 1.49, above the upper bound of 2.02. Contention was inferred from timing and GPU power. That harness had no disk telemetry, so it was never independently confirmed, and it was the only run of that configuration (side-stream synchronization, which the rewrite removes).

We therefore re-measured every serving number with the rewritten engine under one protocol. It brackets each timed process with resident measurements and reruns any process whose resident bracket drifts over 2%, a check that does not depend on the result being measured; no quoted row includes a discarded process. The old harness saved only block means, so a per-token distribution including that attempt does not exist. The rewritten engine's distributions at 220M, in milliseconds: resident median 11.52, p99 11.88, maximum 16.85; read-ahead 11.63, 12.03, 17.15; synchronous fetching 13.62, 14.17, 72.97. Read-ahead stays close to resident even in the slowest tokens.

Revised paper: Replace the App. E attempts with the new protocol, measurements and latency distributions.

Q3. Point: Is the previous-position state before the expert update, and how do training, prefill and overhead work?

Answer: It is before the expert update, after that layer's attention and dense feed-forward network. Training and prefill shift an already-computed tensor by one position, preserving parallel training without another forward pass. During decode, the session carries the previous step's state, so addresses are ready before the next forward pass. During prefill, each layer starts reads once its inputs across the prompt are computed.

The router adds 44M parameters at 1B, about 4% of active. As a correctness check in full precision, the decode path matches full-sequence evaluation within 5×10^-5 at 220M. Measured scheduling overhead is 0.18 ms per token, versus 0.10-0.15 ms for token-id stores. At 1B the router's own compute adds 1.0 ms per token, about 7% of the step. This measures the dependency overhead beyond parity alone.

Revised paper: Define the state and all three schedules in §3.

With the overlap, the router and the frontier components now measured, we hope you will consider raising your score.

Thank you for the additional experiments. The NVMe/resident comparisons and shorter-step tests address my concerns about hiding I/O overhead. The larger-model results and learned-router experiments also strengthen my positive view of the paper. I will maintain my current score. Please include these results and explanations in the final version.

Reviewer_yDWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discussed an optimization algorithm for serving system. The key idea is that it stores most of model parameters on SSD instead of GPU memory. It uses expert blocks addressed by token ID before model runs, so that the address are known and enable It enables compute and memory traffic by reading from NVMe overlap with previous layers.

They train backbones up to 220M parameters with stores totaling 3.24B parameters on up to 8B tokens, and compare against dense models trained the same way.

A 220M backbone with a store beats a dense 1B on loss at 1B tokens and matches it at 8B tokens. The 1B-token checkpoint decodes 1.36x faster than dense 1B and uses 0.61 GB of GPU memory instead of 2.11 GB.

Critical strengths
  • The experiments are well designed and comparison is real: "models share the same recipe and batch order, isolating architectural". A same-model test on the 27.7M backbone compares resident, prefetched and synchronous loading (Table 12), and "prefetching recovers 71% of the synchronous-fetch penalty" (Sec. 5.2).

  • The problem solved is useful in real world and the insights are meaningful. This matters for local and low-memory serving. At 1B tokens the 220M store has lower loss than dense 1B, "2.7596 vs. 2.7754". Served on NVMe, it uses 0.61 GB of peak GPU memory versus 2.11 GB.

Critical weaknesses
  • The paper only discussed on a small scale and quality analysis limits to loss only. Training is up to 3.24B total parameters and 8B tokens, and the 8B token comparison uses a single seed. On tasks, the 220M store dense-1B on LAMBADA dropped by 1.6 and 2.3% at 4B and 8B tokens

  • Some of the numbers are not measured but from projection only. The learned router is checked for "parity but not benchmarked from NVMe". For the trillion-parameter scenario, store "and eight-layer placement remain untested"

Questions for authors
  • Appendix D only used 1 seed for the 220M store or for dense 1B at 8B tokens. Is it possible to test on more seeds?

  • Appendix G shows that the 220M store trails dense1B on LAMBADA by 2.3 points at 8B tokens. But on another task, it is only 1.3 points. Could you explain the difference?

3 comments

Model names (B, A, B-prev, B-lrn, dense-1B, dense-3B) follow the table in our general response. Serving results are on one L40S GPU.

W1. Point: The quality study only reaches small models and short training budgets.

Answer: The larger study confirms the store's advantage with substantially more capacity and training. All models share a recipe and token order, so the comparison remains architectural. The study reaches 23B total parameters and 32×10^9 training tokens. B, the store model at that size (1.04B active parameters, experts on NVMe), improves on dense-1B by 0.163 nats and matches dense-3B, which activates three times as many parameters. Its advantage over dense-1B also holds on language and code outside the training stream (0.37 and 0.18 nats). At 220M, the store keeps a 0.193-nat lead over dense-220M at 32×10^9 tokens. This extends the result beyond the original scale and evaluation split.

Revised paper: Add the larger family and out-of-stream evaluations in §6.

W2. Point: Quality analysis is limited to loss.

Answer: The larger family now has zero-shot results on LAMBADA, PIQA, HellaSwag and ARC-e. B leads dense-1B on every task and matches dense-3B on the mean. These tasks test whether the loss improvement translates into useful predictions. Switching off the store also tests where the capability resides: the same checkpoint's mean falls from 58.2 to 32.5. With the store active, it leads dense-1B by 5 points. The backbone alone therefore does not carry these scores.

modelLAMBADAPIQAHellaSwagARC-emean
dense-3B47.571.843.571.358.5
B46.271.342.772.758.2
B-prev (learned prefetchable router)45.870.943.070.957.7
B-lrn (conventional router)44.170.642.168.656.3
dense-1B41.268.938.664.153.2
B, store off13.754.928.333.232.5

Revised paper: Add the full task table and store-off comparison in §6.1.

W3. Point: The longest original comparison uses only one seed.

Answer: A second seed now confirms the store-versus-dense result at every original training budget. Neither seed puts the store behind, and the two-seed mean favours it throughout. At the longest budget, the first seed's tie becomes a 0.011-nat lead in the second. Across budgets, the mean advantage is 0.004-0.012 nats. The original “match” therefore survives replication and even suggests a small gain.

training tokensstore − dense-1B, seed 0 / seed 1 (nats; negative = store ahead)two-seed mean
10^9−0.0158 / −0.0077−0.0118
2×10^9−0.0135 / −0.0067−0.0101
4×10^9−0.0013 / −0.0068−0.0040
8×10^9−0.0001 / −0.0106−0.0054

Revised paper: Report both seeds in §5.2 and the seed appendix.

W4. Point: The smaller store trails dense-1B on LAMBADA.

Answer: The LAMBADA gap shrinks at the larger scale, and at equal resident size stored capacity improves LAMBADA at both scales. LAMBADA tests predicting a passage's final word from its context. The original comparison gives the dense reference a backbone 4.5× larger, and this is the one task where that mismatch shows clearly at small scale. At equal resident size, the store gains 6.5 points over dense-220M and 5 points over dense-1B. Against the larger dense model of matching loss, the deficit falls from 2.3 and 1.7 points in the original comparison (two seeds) to 1.3 points for B against dense-3B, whose task mean B matches. This is a small gap on one task that shrinks as the models grow, not a general failure of stored capacity on contextual prediction.

Revised paper: Explain the comparison sizes and add the larger model's task results to App. G.

W5. Point: The learned router has parity checks but no NVMe benchmark.

Answer: The learned previous-step router now has a measured NVMe serving result at 220M. It chooses experts from the previous step's state, so its address is ready as early as a token id is. It runs at 0.984 of the same model fully resident, with bitwise-identical logits. Its stored parameters stay on the drive; the 220M token-id store is served in 0.61 GB of GPU memory, against 2.11 GB for dense-1B. This shows that contextual selection can preserve the serving benefit of early addressing.

At 1B, the same router (B-prev) matches the token-id model: 57.7 on the four-task mean against 58.2, and 0.0125 and 0.0008 nats behind in held-out loss over two runs. From NVMe, it keeps 0.97 of its resident speed in 2.48 GB of GPU memory instead of 23.1 GB, with logits bitwise identical, and decodes 1.8× faster than fetching the same blocks at the layer. It reaches 0.93 of B's speed measured beside it. The difference is the router's own compute, 1.0 ms per token, the same resident and from the drive, so the drive path costs the learned address nothing extra.

Revised paper: Add the learned-router serving results at 220M and 1B.

W6. Point: The trillion-parameter scenario assumes an untested store and eight-layer placement.

Answer: The eight-layer serving schedule now runs at 1B, with new evidence supporting the trillion-parameter quality projection. B runs at 0.97 of resident speed in 2.35 GB instead of 23 GB.

The scenario is a budget we can check: a 16B backbone with a 1T store reads 122 MB per token, 2.4 GB/s at 20 tokens/s, in a 19.0 GB resident budget inside a 24 GB card. Speed is bounded by bandwidth divided by bytes per token, and total stored parameters are not in that bound, so quality can grow through the store at fixed bytes: at 27.7M, an 8× larger store at identical bytes per token raises the gain over its backbone from 0.206 to 0.325 nats. What each stored parameter is worth, its exchange rate (dense capacity gained per stored parameter, on a log scale), is set by the address (R3, training viability per address). With training, our learned prefetchable router's rate holds or rises while the token-id rate falls, which suggests it is the address to scale with, as contextual routers are in frontier models. At 1B it matches token id in loss, and released learned-router models reach 0.65-0.67. Reaching the scenario from a released mixture of experts means meeting R1 (address lead time) and R2 (transfer granularity): four changes, of which three cost little as measured (table). The byte budget is the one measured cost, and it loosens with scale: longer backbone steps and faster drives admit more bytes per token, and the measured cost falls from 0.064 nats at 27.7M to 0.025 at 1B, in different store layouts.

change a frontier mixture of experts needsmeasured oncost
R1: address known before the layer that uses itQwen3-30B-A3B and Qwen3-Next-80B, made prefetchable post-hoc, weights frozenwithin 0.001-0.009 nats; 7-task mean 73.0 vs 73.1 at 30B
R1: store layers that start deep enough27.7M and 220M placements; B starts at layer 5 of 30within 0.015 nats of the two-layer layout at equal bytes
R2: drive-sized blockstoday's experts: 3 MiB (Qwen3-Next), 4.7 MB (Qwen3-30B)none; already there
R1: a byte budget under the compute windowB against A (full-depth mixture of experts, equal total and active parameters)0.025 nats for 12× fewer bytes; 77-87% of A's gain kept

Released models are moving this way on their own: a token now reads as little as 1.6% of a layer's experts, down from 25% in 2023, and DeepSeek-V4 addresses its first three expert layers by token id. The post-hoc result in the table's first row keeps released learned-router quality under an early address. It tests the address only: bytes per token stay as released, so those models stay drive-limited, as expected. Together these support the abstract's stated potential, while the body reports both projections: 235-255B dense-equivalent at released learned-router rates and 66-163B at our measured rates.

Revised paper: Keep the abstract's stated potential and name its support. Proposed ending: “A 16B resident backbone with a 1T-parameter store fits a 24 GB GPU and one 1 TB drive within the bandwidth we measure. Released learned-router mixtures of experts reach exchange rates that survive an early address in our post-hoc experiments. At those rates it has the potential to match a dense model of several hundred billion parameters; we bracket this projection between those rates and our own measured ones.”

Q1. Point: Can the original store and dense reference be tested with more seeds?

Answer: Yes, the second seed is complete for both models at every original budget. At the longest budget, the two-seed loss mean favours the store by 0.0054 nats. The two-seed task means are within half a point. W3 describes the result across budgets.

Revised paper: Include both seeds with task results in §5.2 and App. G.

Q2. Point: Why is the LAMBADA deficit larger than the reported 1.3-point difference on another task?

Answer: The 1.3-point ARC-e difference is a lead, so LAMBADA is the exceptional deficit. ARC-e favours the store at both seeds at that budget (4×10^9 tokens). At the longest original budget, PIQA, HellaSwag and ARC-e are within half a point at the first seed and ahead at the second.

The LAMBADA deficit is 2.3 points at the first seed and 1.7 at the second. As W4 explains, this comparison uses a much smaller resident backbone for the store. At equal resident size, stored capacity improves LAMBADA at both scales. At the larger scale, B matches dense-3B on the task mean and is within 1.3 points of it on LAMBADA. The deficit is therefore an effect of that small-scale comparison: it shrinks with scale and does not reflect a general contextual-prediction weakness.

Revised paper: Clarify the sign of the ARC-e difference and report both seeds alongside the larger-scale comparison in App. G.

With these results measured, we hope you will consider raising your score.

Thank you for the response and providing new experiement. This resolved most of my questions. Please make sure to include them in the final version.

Reviewer_HNjGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Paper proposes a pre-fetchable store in which a model's parameters live on disk, rather than on HBM. The addresses on memory are known from the token ID alone, so when a token passes through the network, the model can use a static table to look up which expert block belongs to the token ID. This allows for more experts without being bottlenecked by the communication costs of maintaining all of the experts on HBM.

Critical strengths

The idea of storing experts on disk rather than HBM is an interesting systems-architecture co-design, as it could potentially ameliorate the costs of all-to-all communication and routing if staged correctly, and that the latency for prefetching is less than the compute-bound steps.

Critical weaknesses
  • The main weakness is that one really needs to be able to hide the latency here i.e. the GPU's compute time for preceeding layers needs to be longer than the time it takes for the SSD to stream the block. This imposes some severe limitations that are not adequately explored. For example, it prevents placing experts in the earliest layers of the networks since there's no compute to overlap. The authors should quantify exactly when we can expect this to be true during inference (eg, as a function of the training precision, sparsity, etc.). Again I think the core idea is interesting, so the value here is trying to figure out when it doesn't completely throttle you during inference.
  • the baselines are weak. Choosing MoE layers at 20% and 60% of the depth of the model feels very arbitrary and isn't adequately justified. Standard MoEs typically have expert layers at every layer for the FFN blocks. Even if the prefetchable store cannot match a standard full-depth MoE, I think it would be better to present it, so readers can see the quality cost of the 2-layer prefetch constraint.
  • Dependence on the batch size should be studied more carefully. It looks like the authors consider only bsz=1 for serving evaluation, but larger batch sizes amortize weight loading from HBM across all tokens in the batch. I suspect that scaling the batch size multiplies the SSD bandwidth linearly, which could quickly saturate.
Questions for authors

The prefetchable method seems to perform worse than token-id hash for larger amounts of train tokens in Table 7 -- eg, 2.6174 vs. 2.6129 at 8B tokens. Is this correct?

3 comments

Model names follow the table in our general response. Serving results are on a single L40S GPU unless marked.

W1. Point: Hiding the fetch imposes limits. When does it work, including changes in precision and sparsity?

Answer: It works when each layer's selected bytes can arrive within the computation preceding their use. The submitted paper states the governing requirements and validates an overlap model against synchronous fetching, predicting retention above 0.95 for backbone steps of at least 8.4 ms. Its frontier section also budgets traffic against bandwidth and layer deadlines (122 MB per token, 2.4 GB/s at 20 tokens/s). We now consolidate those building blocks into an explicit law that we will highlight in our revised paper:

Throughput bound = drive bandwidth / bytes per token.

Bytes per token = store layers × blocks per layer × block size × bytes per parameter.

R1, address lead time, determines whether reads finish before use. R2, transfer granularity, requires blocks large enough to reach drive bandwidth rather than the limit on individual read operations. R3, training viability per address, sets what each stored parameter is worth: its exchange rate, which belongs to the address. Precision and sparsity act through the byte product: at 16 bits B's 58 MB per token would be 116 MB, 17.6 ms of reading at 6.6 GB/s against a 12.5-13 ms step; at int8 it fits. Total stored parameters are thus not a factor.

Where the bytes fit, the fetch is hidden and the model runs within 1-3% of resident speed. B does this inside a 12.5-13 ms step, in 2.35 GB of GPU memory instead of 23 GB.

modelbytes per tokentime to read themmargin at the first store layerspeed from NVMe ÷ fully resident
220M store (2 store layers)5.9 MB1.41 ms1.51 ms (layer 3); 0 of 4,096 steps late0.989
B (8 store layers × 4 blocks)58 MB8.5-8.8 ms for all eight layers1.2 ms (layer 5 of 30)0.97
A (experts in every layer)708 MBabout 107 msnone: layer 0 has expertsover budget: the law allows at most 9.3 tokens/s; 6.1 measured

The same law explains designs that exceed the budget. A's 708 MB per token takes about 107 ms to read, six times its 18 ms step, so the read cannot be hidden: the law caps A at 9.3 tokens/s from this drive, and, fetched at the layer, it measures 6.1. On four drive-bound systems that read ahead (our 220M and 1B stores on datacentre NVMe and two released models made prefetchable post-hoc, on a laptop drive) throughput reaches 0.82-0.92 of the bound, and 0.97-1.00 with two micro-batches alternating, so the limit is predictable rather than a cliff.

Revised paper: Highlight the law beside R1-R3, connect it to the existing overlap model, and carry the same budget into the scaling discussion.

W2. Point: Experts cannot occupy the earliest layers because there is no computation to overlap.

Answer: Frontier models already treat their earliest layers differently, and our store follows that practice. DeepSeek-V3 leaves its first 3 of 61 layers dense. DeepSeek-V4 addresses its first three expert layers by a hash of the token id, the address our main method uses. Our first store layer is layer 3 in the smaller model and layer 5 in B. Every layer also retains its dense feed-forward network, serving the role of shared experts in Qwen3-Next and DeepSeek-V3. This gives the first fetch time to finish while retaining computation throughout the network. A few layers of lead are enough: on released Qwen3-30B-A3B, an address computed four layers ahead recovers 88.6% of the released model's expert choices. The placement experiments below show that the result changes little with how stored capacity is spread over depth.

Revised paper: Explain this placement rule and its connection to frontier architectures in §3.

W3. Point: Placing store layers at 20% and 60% of depth seems arbitrary.

Answer: That was the simplest placement meeting the lead-time rule for our initial small-scale experiments. B instead uses a distributed layout, spreading the store across eight layers with four quarter-size blocks per layer. Equal-byte layouts stay within 0.008 nats of each other at the smallest scale. At 220M every layout is within 0.015 nats of the submission's, eight single-block layers are still within 0.015 at 8×10^9 tokens, and the distributed, multi-block layout B uses is the best, 0.014 ahead. A, the every-layer baseline you asked for, is covered under W4 in our next post.

placement (equal stored parameters and bytes; 10^9 tokens)27.7M220M
2 store layers (the submission)2.91702.7596
4 store layers2.9158–
8 store layers2.92342.7661
8 store layers × 4 quarter-size blocks (the layout B uses)2.91792.7456

Revised paper: Add the placement sweep and explain why the larger model uses the distributed layout.

W4. Point: Show a standard full-depth mixture of experts, even if the store cannot match it.

Answer: We did just that in our new experiments. A is a full-depth mixture of experts (experts in all 30 layers, no dense feed-forward network, FFN) matched to B in total parameters, active parameters and training data. A's 0.025-nat advantage is the measured cost of B's smaller byte budget: B keeps 87% of A's gain over dense-1B in loss (77% on tasks) while reading 12× fewer bytes, and still matches dense-3B. A conventional router in B's own layout performs worse, separating this budget cost from early addressing. Furthermore, the cost falls with scale: a full-depth mixture of experts leads the submission's two-layer store by 0.064 nats at 27.7M, and B's eight-layer store by 0.025 at 1B. The law explains why. A larger backbone takes longer per token, so more bytes can be read under each step: the 1B step hides about 85 MB at 6.6 GB/s, while the scenario's 16B backbone, at 20 tokens/s (50 ms per token), would hide up to 700 MB on a 14 GB/s PCIe-5 drive, about what A reads per token at 1B. The budget loosens as the model grows.

modellayoutread per tokenheld-out loss vs B4-task meanserved
Aexperts in all 30 layers, no dense FFN708 MB−0.02559.7resident: 54.7 tokens/s; from the drive: 6.1
Bdense FFN everywhere + 8 store layers58 MB058.2from NVMe: 75.0 tokens/s in 2.35 GB, 0.97 of resident
B-lrnB's layout, conventional at-the-layer routernot prefetchable+0.05356.3–
dense-3Bdense–+0.03058.5resident, 5.80 GB
dense-1Bdense–+0.16353.2resident

The practical return is capacity that can be served from a drive. From the same drive, B decodes at 75.0 tokens/s against A's 6.1, and B from NVMe is also 1.37× faster than A fully resident, with both served by the same engine and kernels: A runs 390 expert blocks per token across 30 layers, B 32 across 8.

Revised paper: Add A in §6.3, explain the byte-budget comparison, and distinguish measured quality from the scaling argument.

W5. Point: Larger batches may multiply drive traffic and saturate bandwidth.

Answer: Yes, larger batches eventually reach the drive's bandwidth limit, and the new measurements locate that limit. The store targets serving where GPU memory is the constraint: from one drive its advantage is largest at small batch, and at large batch it needs more drive bandwidth. The useful distinction is what must scale. A resident model holds all parameters in GPU memory at every batch size. The store keeps GPU memory near the backbone's needs and increases throughput by adding drive bandwidth. The resource that limits serving therefore moves from GPU memory to drive bandwidth, which is inexpensive and can be added in parallel.

The engine already uses that bandwidth effectively. Two micro-batches alternate computation and reads, approaching the drive's bound. Blocks shared within a batch are read once, so traffic per token falls as batching grows. The table shows the transition from a compute-limited step to a bandwidth-limited one with the rewritten engine (fewer copies and kernel launches than the submission's; our reply to BTev, W3, gives details).

tokens in flightstore from one drive, tokens/sshare of drive boundfrom one drive / dense-1Bstore resident / dense-1B
1 (one stream, compute-bound)150-1550.141.65-1.731.74-1.75
4 (one stream, compute-bound)402-4900.37-0.451.12-1.671.70
16 (two micro-batches of 8)1,074-1,0980.97-0.990.78-0.821.56-1.63
32 (2×16)1,123-1,1350.99-1.000.40-0.411.70-1.72
64 (2×32)1,185-1,1901.000.25-0.261.67
128 (2×64)1,266-1,2711.000.211.56

At small batch, the 220M store from one drive beats resident dense-1B: 1.65-1.73× at batch 1 and 1.12-1.67× at batch 4, in 0.61 GB of GPU memory against dense-1B's 2.11 GB. The submission's 1.36× came from the earlier engine; dense-1B runs through the same rewritten engine. At larger batch, the crossover becomes a bandwidth requirement: matching dense-1B needs 8.0 GB/s at 16 tokens in flight, within one PCIe-5 drive. At 128 in flight, it needs 30 GB/s: five drives of the kind we measured, or three PCIe-5 drives. The drive is the only part that falls behind: the same checkpoint held in GPU memory stays 1.56-1.75× faster than dense-1B at every batch size (last column), and that is the level added drive bandwidth moves toward.

The larger model follows the same rule: B becomes drive-bound from batch 4 and reaches 0.89-0.92 of the drive's bound at batch 4 and 16 (A100 80 GB nodes), so its throughput at large batch is also set by drive bandwidth.

Revised paper: Add the measured batching regimes, bandwidth crossovers and resident control in §5.5 and §6.4.

Q1. Point: Is the learned prefetchable router worse than token-id hashing at the largest Table 7 budget?

Answer: Yes at that budget, by 0.005 nats, comparable to these runs' seed spreads, but in our new 1B experiments, the two now match. Both addresses are prefetchable: token id is the main method, and the previous-step router is its learned contextual variant. A gap that closes with training is what a contextual router should show: it spreads each token's training over more addresses than a fixed address does (R3), so it needs more training before it pays off.

This gap closes with training and with scale, which is what matters for a growing store. From 10^9 to 8×10^9 tokens it falls from 0.025 to 0.005 nats at 27.7M. At 220M it falls from 0.038 (10^9) to 0.021 (8×10^9) and 0.010 in the new 32×10^9-token run. At 1B, in B's layout, the learned router is 0.0125 and 0.0008 nats behind token id in two runs of one configuration, a mean of 0.007 that sits inside the 0.0117 by which the two runs differ. It scores 57.7 on the four-task mean against B's 58.2. It beats the conventional at-the-layer router there by 0.0405 and 0.0523 nats.

The exchange rates explain the trend. Total stored parameters are not in the serving bound, so the store grows at fixed bytes per token, and what each added parameter is worth is its exchange rate, the dense capacity gained per stored parameter on a log scale. The address sets it: the learned router's rate holds or rises with training (0.471, 0.496, 0.486 at 220M) while the token-id rate falls (0.616, 0.557, 0.512). The learned router's rate is the one that holds as training grows, so we expect it to be the address to scale with, as contextual routers are in frontier models. Released learned-router mixtures of experts reach 0.65-0.67. The store is not tied to token-id addressing as it grows: we measure the learned early address in two ways. Trained from scratch, it closes on token id as we scale and matches it at 1B. Applied post-hoc to released models (weights frozen, experts moved to the drive, a small custom router fitted to choose them early), it keeps the quality of Qwen3-30B-A3B and Qwen3-Next-80B within 0.001-0.009 nats. This tests the address only, not speed: bytes per token stay as released, so those models stay limited by the drive, as expected (W1).

backbonetraining tokenslearned router's gap to token id (nats)exchange rate, learned / token id
27.7M10^90.025–
2×10^90.021–
4×10^90.011–
8×10^90.005–
220M10^90.0380.471 / 0.616
8×10^90.0210.496 / 0.557
32×10^90.0100.486 / 0.512
1B (B's layout)32×10^90.0125, 0.0008 (two runs)0.40-0.43 / 0.43

Revised paper: Extend Table 7 with the learned router at longer training and larger backbones, and explain the training-dependent exchange rates.

Given this new evidence, we hope you will reconsider your soundness score and rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their careful, specific reviews. Four concerns recur: the quality evidence stops at small models, short training and one seed; there is no standard full-depth mixture-of-experts baseline; the serving evidence is narrow, with no learned router served from a drive; and the frontier scenario rests on untested components. We ran new experiments for each.

New models at a 1B backbone. We trained six models with the same recipe and token order on 32×10^9 tokens, up to 23B total parameters. The replies use these names.

modeltotal / activeread from the drive per tokenheld-out loss vs B4-task meanGPU memory when served
B (token-id address, experts on NVMe)23.18B / 1.04B58 MB058.22.35 GB
B-prev (learned previous-step router, prefetchable)23.22B / 1.08B58 MB+0.0125, +0.0008 (two runs)57.72.48 GB
B-lrn (conventional at-the-layer router)23.20B / 1.06Bnot prefetchable+0.05356.3–
A (full-depth mixture of experts)23.10B / 1.04B708 MB−0.02559.7all experts in GPU memory
dense-3B2.97B / 2.97B–+0.03058.55.80 GB
dense-1B0.98B / 0.98B–+0.16353.2–
  1. Does stored capacity improve quality at equal active compute? Yes. B beats dense-1B by 0.163 nats of held-out loss and by 5 points across four zero-shot tasks. It matches dense-3B, which has three times the active parameters: 0.030 nats ahead in loss and level on the task mean. From NVMe, B decodes 1.06× faster than dense-3B in 2.35 GB of GPU memory against 5.80 GB. A second seed confirms the submission's store-versus-dense result at every tested budget.
  2. Does knowing the address early sacrifice quality? Not in the matched layout. A conventional router that chooses experts at their layer is 0.053 nats worse than B's token-id address. The same router recipe reading the previous step's state, which makes it prefetchable, matches token id (0.0125 and 0.0008 nats behind in two runs).
  3. What does the drive constraint cost? A smaller byte budget retains most of the full-depth model's gain. The full-depth mixture of experts, A, has equal total and active parameters. B keeps 87% of A's improvement over dense-1B in loss, and 77% on the task mean, while reading 12× fewer bytes per token. A's remaining advantage, 0.025 nats, is the measured cost of the tighter budget.

Serving. On one L40S, B decodes from NVMe at 0.97 of its fully resident speed in 2.35 GB of GPU memory instead of 23 GB, with bit-exact output. From the same drive it is 12× faster than A, and 1.37× faster than A held entirely in GPU memory, in the same engine. Measurements now use varied held-out prompts at 220M and 1B, with fetch-at-the-layer controls, a transfer timeline and batches up to 128 tokens in flight. They make the limit explicit: speed from a drive is bounded by bandwidth divided by bytes read per token, and total stored parameters aren't a factor. Throughput reaches 0.82-0.92 of that bound on four systems where the drive sets the pace, and 0.97-1.00 with two micro-batches alternating.

The frontier scenario. Its eight-layer serving schedule now runs at 1B. Because total stored parameters are not in the bound, the store can grow at fixed bytes per token: at 27.7M, an 8× larger store raises the gain over its backbone from 0.206 to 0.325 nats. The store is not tied to token-id addressing: the learned prefetchable router's gap to token id shrinks with training and scale and closes at 1B, suggesting it is the address to scale with, as contextual routers are in frontier models. The byte budget, the one measured cost, shrinks at the larger scale: 0.064 nats at 27.7M and 0.025 at 1B, in different store layouts. The other changes a released mixture of experts would need cost little. On Qwen3-30B-A3B & Qwen3-Next-80B, with every weight frozen and only a small early router fitted, moving the address earlier keeps quality within 0.001-0.009 nats. Note this tests the address only: bytes per token stay as released, so they stay drive-bound and, as expected, do not reach our stores' near-resident speed. The scenario's final quality remains a projection, and the revision labels measured and projected parts separately (replies to yDWm W6 and BTev W7).

Revisions. The revised paper will (section numbers are the revision's):

  • State the throughput bound beside the three requirements.
  • Add the 1B family, with the full-depth baseline and both routers, the zero-shot tasks, the second seeds and out-of-stream evaluation (§5.2, §6).
  • Use the broader serving protocol and add the 1B measurements, controls, timeline and batching regimes (§5.5, §6.4, App. E).
  • Add the placement sweep, the learned router at longer training and larger scale with its serving results, and the post-hoc prefetchable routers on released models.
  • Separate measured components from projected quality in the frontier section (§9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.