The Prefetchable Store: Decoupling Model Capacity from Fast Memory

The largest model a machine can run at full speed is the largest one its fast memory holds. We argue that this is an addressing limit rather than a storage limit: a transformer chooses the parameters a token needs in the middle of its forward pass, so all of them must be resident. The prefetchable store is a storage-native architecture in which most of a model's trained parameters live on a commodity SSD as expert blocks whose addresses are known from the token id before any layer runs, so they stream under the model's own early layers. We state three requirements under which storage can hold trained capacity: the address must be known early enough to hide the transfer, the transfer must be shaped for the storage tier, and every address must receive enough training data to learn. We build a serving system that meets them and reproduces the resident model exactly through the disk path at equal precision, and we train models under the constraint from the first step against a dense frontier retrained at every budget under one recipe. Up to 8×10^9 training tokens, across three backbones and four store sizes, stored capacity lowers loss at every budget: a 220M backbone with a 3B-parameter store matches a fully resident 1B dense model under the same training budget while, served from NVMe, decoding faster in a third of the GPU memory. The price is total parameters, comparable to the overhead sparse models already pay, and it is paid on a drive rather than in fast memory. We further show that two ingredients behind the efficiency of frontier mixtures of experts, learned contextual selection and the combination of several fetched experts, work inside the prefetchable constraint. Extrapolating the measured scaling, a 16B resident backbone on a 24 GB consumer GPU with a 1T-parameter store on one 1 TB drive has the potential to match a dense model of several hundred billion parameters.