Predictive Statistics Shape Emergent World Representations of Grid Walkers

Next-token predictors often appear to develop internal representations of the latent world and its rules. The probabilistic nature of these models suggests a deep connection between the structure of the world and that of sequence probabilities. In order to better understand this link, we use a minimal stochastic process as a controlled setting: constrained random walks on a two-dimensional lattice that must reach a fixed endpoint after a predetermined number of steps. Their world's grid geometry is a statistic for prediction of the full future. We train decoder-only transformers on exactly sampled walks and ask which statistics are decodable and how they are used for computation. The transformer's computation factors into two stages: the first attention block extracts the world statistic from the input by path integration, and later layers transform it into the next-step predictive geometry. Across the fully observed variants the post-attention representation is universal: a shared world-state of the lattice that can be read directly as a world model, which we can trace back to the predictive structure of the data. Later layers then specialize it to each variant's next-step distribution. We also explore a variant with masked steps and latent variables, and the spatial map persists and remains causally used, even though it no longer suffices for prediction. Moreover, we train recurrent networks which solve the fully observed tasks near-optimally, but they organize the computation differently. Although demonstrated in a toy system, these results suggest that world representations emerge as intermediate computations in prediction, with a geometry shaped jointly by the predictive distribution and the architecture that computes it.