Predictive Memory Complexity: Persistent-State Requirements for Continual Prediction

Continual world models accumulate an ever-growing history of experience, while the memory that can be persistently retained remains fundamentally bounded. This mismatch raises a fundamental question: how much information from the past must remain encoded in persistent state to preserve future predictive accuracy? We introduce predictive memory complexity (PMC), the minimum persistent-state capacity required to attain a prescribed prediction risk. (i) Under stationary fully observed Gaussian dynamics, Theorem 1 gives the exact finite-memory law $R_B^{\star }=q+a^2{D}_{2^B}(v)$, showing that arbitrary recursive memory reduces to quantization of the current predictive state and yielding $B^{\star }(d)=\frac{1}{2}{\log }_2\frac{a^{2}v}{d-q}+O(1)$. (ii) Removing stationarity, Theorem 2 shows that memory tracks the scale of the predictive state: $B_t^{\star }(d)=\frac{1}{2}{\log }_2{v}_t+O(1)$, producing bounded, logarithmic, and linear growth for stable, critical, and unstable dynamics, respectively. (iii) Under partial observability, Theorem 3 bounds the excess prediction risk between quantization of the Bayesian predictive state and the better of two finite-memory mechanisms—history retention and recursive belief representation—thereby separating inferential from representational memory. (iv) Beyond Gaussianity, Theorem 4 preserves the exact quantization reduction while introducing the entropy-power factor $\beta =e^{-2D_{\mathrm{KL}}(P_X\Vert \mathcal{N}(\mu ,v))}$, showing that equal variance need not imply equal finite-memory difficulty. (v) Theorem 5 specializes the theory to an isotropic KV-cache model: with $R=B/(dL)$ bits per scalar, $D_L^{\star }\ge {\sigma }_{\eta }^2/(1-{\rho }^2{2}^{-2R})$, and half of the asymptotically available predictive gain occurs at $R_{\mathrm{onset}}=\frac{1}{2}{\log }_2(1/(1-{\rho }^2))$. Measurements from Transformer key representations provide an empirical scale check for the stylized cache regime. The results show that persistent memory is a property of the predictive process rather than merely of the predictor architecture: its required capacity is jointly governed by predictive-state geometry, dynamical expansion, and inferential uncertainty. This provides a principled basis for understanding memory in continual world models, including what predictive information must persist, how accurately it must be represented, and when fixed-capacity memory is fundamentally insufficient.