acceptodds
Under review as a conference paper at ICLR 2027

Predictive Statistics Shape Emergent World Representations of Grid Walkers

Abstract

Next-token predictors often appear to develop internal representations of the latent world and its rules. The probabilistic nature of these models suggests a deep connection between the structure of the world and that of sequence probabilities. In order to better understand this link, we use a minimal stochastic process as a controlled setting: constrained random walks on a two-dimensional lattice that must reach a fixed endpoint after a predetermined number of steps. Their world's grid geometry is a statistic for prediction of the full future. We train decoder-only transformers on exactly sampled walks and ask which statistics are decodable and how they are used for computation. The transformer's computation factors into two stages: the first attention block extracts the world statistic from the input by path integration, and later layers transform it into the next-step predictive geometry. Across the fully observed variants the post-attention representation is universal: a shared world-state of the lattice that can be read directly as a world model, which we can trace back to the predictive structure of the data. Later layers then specialize it to each variant's next-step distribution. We also explore a variant with masked steps and latent variables, and the spatial map persists and remains causally used, even though it no longer suffices for prediction. Moreover, we train recurrent networks which solve the fully observed tasks near-optimally, but they organize the computation differently. Although demonstrated in a toy system, these results suggest that world representations emerge as intermediate computations in prediction, with a geometry shaped jointly by the predictive distribution and the architecture that computes it.

Primary area:
interpretability and explainable AI
Keywords:
world models, mechanistic interpretability, next-token prediction, transformers, emergent representations, sufficient statistics, recurrent neural networks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xQRKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses constrained random walks on a two-dimensional lattice as a controlled setting to study how next-token prediction models form world representations. Because the predictive distribution can be computed exactly, the authors can directly compare the network’s internal representations with the theoretical sufficient statistic. The main finding is that, after the first attention layer, a Transformer forms a representation that encodes only position and time, and that this representation is largely consistent across variants with different rules; subsequent layers then transform it into the next-step prediction. The authors use intervention experiments to show that this representation is actually used by the network, and they provide a mechanistic explanation: near-uniform attention takes a linear average of the direction encodings of the individual steps, yielding the mean displacement and thereby implementing path integration.

Critical strengths

The paper designs a simple experiment to test whether next-token prediction models can learn a world model. The exactly computable predictive distribution provides a reliable reference for all analyses, the mechanistic explanation is clear, and the experimental setting should be of value to future research.

Critical weaknesses

The discussion of architecture is not sufficiently in-depth. The RNN experiments show that RNNs also encode the world state, but organize it quite differently from Transformers: the Transformer has a layer that holds the world state on its own and is shared across tasks, whereas the RNN’s hidden state mixes the world state with the next-step prediction, and its representational form changes with the task. This suggests that the paper’s most specific conclusions, such as the form of the coordinates, the two-stage structure, and the consistency across tasks, are likely specific to Transformers. The paper does not analyze which architectural factors give rise to these differences, so the more general claims in the title and abstract lack sufficient support.

Moreover, in this setting each token corresponds to a fixed displacement, so position is a linear function of the token counts; and the attention here is near-uniform, so what it computes is precisely a linear average of the per-step encodings. A linear average cannot merge only those states that share the same next-step distribution, so the fact that the first layer outputs the full position is largely determined jointly by the task structure and this linear aggregation, and does not necessarily show that the network is inclined to build a world model.

I suggest adding ablation experiments to clarify which architectural factors the world representation depends on:

  1. Ablate the Transformer to identify which design changes cause its world-model representation to disappear. For example, remove the positional encoding, or replace the linear averaging of attention with content-dependent nonlinear aggregation; one could also change the task to use non-additive token semantics (e.g., “forward, turn left, turn right”), so that position is no longer a linear function of the token counts. This would make it possible to determine which factors the world representation depends on.
  2. Conversely, apply corresponding changes to the RNN to identify which changes allow it to produce a Transformer-like representation, for example by adopting architectures that lie between the two, such as linear recurrences or state-space models.

Together, these two directions would help attribute the differences between the two architectures to specific factors.

Questions for authors

In a non-additive task (i.e., one in which position is no longer a linear function of the token counts), does a linear representation of the world state still emerge in the first layer?

4 comments

We thank the reviewer for the positive evaluation and constructive feedback.

While the dependence of mechanism on architecture was already an important part of our narrative, with RNNs as architectural controls, this axis was not as empirically explored as others in the paper, as the reviewer pointed out. Their suggestions in this regard have been useful. Following them, we have run new experiments that test which aspect of attention is most relevant to the mechanism's emergence. Concretely, we:

  • Replaced softmax attention with linear attention, state-space layers or nonlinear recurrences (tanh RNN, LSTM)
  • Trained linear recurrent networks read out linearly
  • Removed the positional embedding
  • Trained a walker with forward and turn tokens instead of xx/yy displacements

The results indeed help attribute the difference between architectures; we report them below. Specifically, the analyses confirm that the organization that set our transformers apart from our RNNs came from the linearity of their aggregation step.

The reviewer said:

A linear average cannot merge only those states that share the same next-step distribution, so the fact that the first layer outputs the full position is largely determined jointly by the task structure and this linear aggregation, and does not necessarily show that the network is inclined to build a world model.

We fully agree with this statement. Because a built-in inclination to model the world is lacking in principle, we are interested in better understanding why and under which conditions world models emerge in next-token predictors (indeed, they do not always develop; see Vafa et al., 2024; 2025). Our example points to one explanation for emergence, namely, an architectural 'simplicity bias' (related to Huh et al., 2024): among the statistics that help predict the next step, the network finds a simple one for attention to compute, which carries the full position.

AuthorsOpenReview ↗

Experiments

1. Transformer Ablations

Following the first suggestion, we ablated the transformer in the three ways proposed, to see which changes make its world representation disappear:

  • Positional embedding removed (50 runs: five architectures, ten seeds each). Time is no longer linearly decodable from the output of the first attention layer, while position remains so: x/tx/t and y/ty/t decode at held-out R2=0.96R^2 = 0.96–1.001.00, whereas log⁡t\log t falls from ≥0.997\geq 0.997 to ≤0.24\leq 0.24. The first MLP recovers part of the time coordinate (R2≤0.49R^2 \leq 0.49 for log⁡t\log t), and the loss is higher than with the positional embedding in every run.
  • Nonlinear, content-dependent aggregation (120 runs; results for walkers L and D). With a tanh RNN or an LSTM in place of attention, position and time remain linearly decodable (R2=0.99R^2 = 0.99–1.001.00 for xx, 0.950.95–1.001.00 for tt), but an affine function of position and time explains only 7272–9292% of the activation variance (the paper's RNNs: 7979–9595%), against over 9999% after softmax attention. The LSTM predicts about as well as attention, the tanh RNN somewhat worse (excess loss 0.800.80–1.23×1.23\times and 1.341.34–2.52×2.52\times that of softmax).
  • Non-additive tokens. We rewrote walker L's walks with three tokens: forward (F\mathtt{F}), left turns (L\mathtt{L}) and right turns (R\mathtt{R}). The underlying stochastic process is the same as walker L's (further details will be available in the revised manuscript, which we will upload in the next few days). With these tokens, a linear representation of the world state is no longer possible in the first layer, because each forward step's direction depends on earlier turns. Indeed, lattice position is not linearly decodable after the first block, and one-layer networks predict much worse than their two-layer counterparts. Two-layer networks, however, do find a different solution, slightly more complicated but interpretable. The first block encodes, at every token, the balance of left and right turns made so far. The second block's attention pools, for the present token, the balances recorded at all F\mathtt{F} steps in the past context, and the second MLP combines them with the present balance to compute the walker's egocentric position, i.e., its position relative to its own heading. This is exactly the coordinate the next-step distribution depends on, and it becomes linearly decodable at the MLP (held-out R2R^2 up to 0.960.96), whereas the lattice position never does (at most 0.150.15).

2. Architectures Between the Two

  • Linear aggregation. Linear attention or a state-space layer in place of attention gives a linear representation of the same kind, with the tokens averaged or summed depending on normalization. Normalized linear attention reproduces the softmax representation (CKA 0.9960.996–0.9970.997).
  • Readout. A linear recurrent network with a linear readout holds position and time exactly (R2≥0.98R^2 \geq 0.98) but falls far short of the next-step distribution (excess loss 4343–156156 times that of a GRU).

We will upload the revised manuscript with these results in the next few days.

We remain at the reviewer's disposal to answer any further questions.

We thank the authors for these revisions and additional experiments. The new results substantially improve the rigor and persuasiveness of the paper, and I have decided to raise my score from 3 to 4.

We sincerely thank the reviewer for engaging with our new experiments and for raising their score!

Reviewer_KsV5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the representations that a standard transformer neural network learns in service of predicting left-right-up-down token sequences consistent with a nonstationary random walk on a 2-d lattice. The authors observe that the first attention layer [block?] effectively extracts the current node of the graph via path integration (somewhat nontrivially: vector sum of the token embeddings divided by time-elapsed t, together with a dimension for the logarithm of t). Subsequent layers determine the next-token distribution, supposedly from both (i) this inferred position and (ii) the remaining steps (indicated by the positional embedding?). The representation differs from that learned in an RNN trained on the same task. This paper builds naturally off of other recent work on predictive geometry and makes a nice contribution towards understanding the geometric representation of a class of world model that arises over training in a neural network. Experiments, controls, and analysis seem to be performed competently. The writing is mostly clear. This work implicitly suggests further directions for fruitful scientific inquiry, which I judge as a success.

Critical strengths
  1. This paper does a good job bringing together both belief geometry and world models, in a setting that several communities can engage with and build useful conversations and insights around.

  2. Controls and baselines are used well throughout this work.

Critical weaknesses
  1. There is an opportunity for more mechanistic insight. See my questions about OV vs. token embedding in the next section. However, I think the paper has established enough, and I recognize that it is good that the current work identifies and stimulates a research direction rich in directions that would be exciting to explore further.

  2. While the paper is overall quite good, and scientifically sound, I find some opportunities for greater clarity in some of the writing. For example, “The probabilistic nature of these models suggests a deep connection between the structure of the world and that of sequence probabilities.” What does this mean? How would the probabilistic nature of models suggest a deep connection between any two things? I suggest that sentences like these should either be (i) left out or (ii) scrutinized and adapted—either way leading to a more concise crisp paper.

Questions for authors
  1. Mechanistically, uniform attention can be achieved by the query and key matrices both being zero. At small initialization, they’re already near zero, and so attention should be close to uniform over all past tokens. This almost already gives the x/t and y/t from the causal masking. The only thing needed is for \pm x to be learned as anti-parallel token embeddings, and for \pm y to also be learned as anti-parallel token embeddings. It would be very helpful to see how these token embeddings evolve over training (they should almost always initialize to all be nearly orthogonal by default), and whether the Q-K matrices change appreciably at all. Alternatively, the OV-matrices could implement the effective antiparallel contributions even if the token embeddings all remain orthogonal. It would be interesting to see whether the token embedding or the OV solution dominates, or if some other solution is found by the network.

  2. Is there a good reason to expect the log(t) representation? How is it used mechanistically?

  3. I’m envisioning the nonstationary random-walk task as a collection of distinct dynamical systems, depending on which endpoint is selected from the endpoint set. If the each of these dynamical systems were stationary, then it would be a collection of ergodic components, and collections of ergodic components have been observed recently to leave specific representational structures in transformers. Might this perspective add anything useful to the analysis? (If it does not seem like a useful perspective, then that is totally fair. I’m only bringing this up in case it inspires new insights.)

1 comment

We are thankful to the reviewer for their positive and constructive review.

The reviewer's questions about mechanism are indeed important, and understanding how it evolves enriches our analysis.

Question 1: Mechanistic Interpretability

The storyline appears to be the following: First, the antiparallel encoding is implemented by the OV circuit. The angle between +x+x and −x-x (and ±y\pm y) value-write tokens gradually opens (centered cosine −0.33→−0.96-0.33 \to -0.96 by step 42k of training, median of 150 models), while embeddings remain pretty much randomly oriented over the same initial training phase (−0.37→−0.40-0.37 \to -0.40). Attention scores flatten over the same phase, and attention is nearly uniform once the angle between value-write embeddings of opposite steps gets close to π\pi (by step 42k the KL to uniform attention is down to 10% of its initial value). The QK matrix restricted to the subspace spanned by the inputs is responsible for the flattening: WQW_Q and WKW_K first re-orient against each other, then shrink (the RMS of their product there drops to 0.43 of its initial value), whereas the norm of the full matrix barely changes (0.90). Only then does the tetrahedron formed by the four token embeddings partly flatten, but never fully. The end state recurs in all runs (value-write cosine ≤−0.94\le -0.94 in all 150 models, embeddings never below −0.83-0.83). This shows that the step-averaging mechanism is not initially present, but gradually develops during training reproducibly across runs.

The initial uniformity of attention depends on the initialization scale. At our default scale, attention scores become more uniform during training. If we decrease the initialization scale by a factor of 4, however, the initial score SD is even lower than the post-training value at the default scale. It then rises and settles at about the level of the ×1 and ×0.5 scales. Interestingly, in the one-head walker L and D models, the trained attention slightly upweights the first key and downweights the last one. The explanation is that positional encodings also matter for the OV circuit, and their scales need compensation in the QK

Question 2: log⁡t\log t form

While log⁡t\log t fits walker L's time code hth_t well at context 8, hth_t becomes nearly linear when we train with context 16. So we see log⁡t\log t as a good empirical description at the trained context. The reason why log⁡t\log t fits at shorter contexts is the following: attention averages the position codes pip_i (Sec. 4.3), so δht+1=(pt+1−ht)/(t+1)\delta h_{t+1}=(p_{t+1}-h_t)/(t+1), and if ptp_t grows at the same rate as hth_t, then δht∼1/t\delta h_t \sim 1/t, and a sum of 1/t1/t terms approximates the logarithm. It turns out that, at context 8, ptp_t does increase at this rate over the first 3-4 positions, which are the most significant. At context 16, however, walker L’s ptp_t grows too fast, which makes hth_t linear. The importance of encoding time as a separate dimension, regardless of the form, is that the MLP needs it to un-normalize (xt/t,yt/t)(x_t/t, y_t/t) and compute the remaining time τ=T−t\tau=T-t. Moreover, we performed new architectural ablations, and one shows that the model's performance drastically decreases after removing the positional encoding.

Question 3: Nonergodic Components

We thank the reviewer for this interesting suggestion. Walker D does fit the setting formally if one loops the walker into a stationary process. However, walker D does not lend itself nicely to the belief-state perspective because there is no uncertainty about (t,xt)(t, x_t), only the endpoint. For each component, the belief jumps from vertex to vertex of the simplex, so recovering it linearly from the activations would require the state representations to span as many dimensions as there are states, which is not what it does. Instead, it uses a geometry of spatiotemporal coordinates, which is the same for either endpoint.


With respect to writing, we concede that there is room to make some statements in the abstract and the introduction more concrete. The provided quote is a clear example, and we appreciate the reviewer for pointing this out. We are revising these passages and adding the training-dynamics analysis from Q1. We will upload the revised manuscript in the next few days.

We remain at the reviewer's disposal to answer any further questions.

Reviewer_XsSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how internal world representations emerge in transformers trained on constrained random walks. Using a setting where the exact predictive distribution is known, the authors show that the first attention block develops a representation closely related to spatial position and time. They investigate this mechanism through probing, causal interventions, generalization tests, and comparisons with recurrent networks.

Critical strengths

The mechanistic analysis is particularly convincing. The authors go beyond showing that spatial coordinates can be decoded. Their steering and ablation experiments demonstrate that these representations are actually used to generate predictions. This is an important distinction that many interpretability studies do not establish.

The experimental design is rigorous. The availability of exact predictive distributions, evaluations across architectures and random seeds, and the partially observed setting make the findings substantially more informative than a typical representation-visualization study.

Critical weaknesses

The main limitation is the generality of the findings. The experiments rely on relatively simple, additive movements on a fixed lattice, where spatial position can be recovered through averaging displacement tokens. While this provides a clean setting for understanding the learned representations, it is unclear whether the same mechanism would emerge in environments with nonlinear dynamics, longer temporal dependencies, or more complex observations. The masked-walker experiments partially address this concern but still retain the same underlying motion rules.

Another limitation is the sensitivity to initialization. The authors report that models with larger initialization scales can achieve near-optimal prediction accuracy while developing weaker spatial representations. This suggests that the proposed representation is not necessarily required for successful prediction. Although these limitations do not undermine the main findings, they raise questions about how broadly the conclusions extend beyond the controlled experimental setting.

Questions for authors

The experiments with larger initialization scales are interesting. When models achieve near-optimal prediction without developing the same strong spatial representation, what alternative computation do they use? Understanding these cases would help clarify how universal the proposed mechanism really is.

1 comment

We sincerely thank the reviewer for their positive and useful review.

We appreciate the reviewer's question on initialization. It turns out that there is a close link between representational fidelity and performance. Indeed, the larger-initialization-scale cases do not perform as well on the prediction objective: at x4 scale, the excess loss over the exact optimum is 14.8 times that of the default scale. Moreover:

  1. The spatiotemporal representation is also present when the initialization scale is four times larger, albeit less cleanly. The median in-sample R2R^2 of the affine map from the state coordinates to post-attention activations is 0.81 at x4 and 0.94 at the default scale.
  2. Using a more suitable learning rate increases both the fit to the spatiotemporal coordinates and the performance. With a 4x higher learning rate, the x4 runs come within 1.27x of the default-scale excess loss and match its R2R^2.
  3. Training five times longer at the original learning rate also improves both metrics. The excess loss reaches 2.62x that of the default scale runs trained equally long, and was still decreasing when training ended.

Thus, we do not find a fundamentally different mechanism when we change the initialization scale. We will add these analyses to the revised manuscript, which we will upload in the next few days.

We remain at the reviewer's disposal to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in revised version

We thank the reviewers again. We have just uploaded a revised version of the manuscript, which we believe is an improvement of our work. The main changes in the revision are the following:

  • Initialization scale (XsSH): Appendix G.2, Table 11. With a four times larger initialization, a four times larger learning rate gives a similar representation and loss.
  • Training dynamics (KsV5, Q1): Section 4.3 and Appendix C.1, Figures 5–6.
  • Time code (KsV5, Q2): Appendix C.2.
  • Architectural ablations (xQRK): Section 4.6 and Appendix E, Tables 7–9.
  • Turn tokens (xQRK): Section 4.7 and Appendix F, Table 10.
  • Writing (KsV5): we made the vaguer statements more concrete.
  • Supplementary material: updated anonymized code.

We have also changed the title to better reflect the paper's content, purpose and scope. We are at the reviewer's disposal for any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.