- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper uses constrained random walks on a two-dimensional lattice as a controlled setting to study how next-token prediction models form world representations. Because the predictive distribution can be computed exactly, the authors can directly compare the network’s internal representations with the theoretical sufficient statistic. The main finding is that, after the first attention layer, a Transformer forms a representation that encodes only position and time, and that this representation is largely consistent across variants with different rules; subsequent layers then transform it into the next-step prediction. The authors use intervention experiments to show that this representation is actually used by the network, and they provide a mechanistic explanation: near-uniform attention takes a linear average of the direction encodings of the individual steps, yielding the mean displacement and thereby implementing path integration.
Critical strengths
The paper designs a simple experiment to test whether next-token prediction models can learn a world model. The exactly computable predictive distribution provides a reliable reference for all analyses, the mechanistic explanation is clear, and the experimental setting should be of value to future research.
Critical weaknesses
The discussion of architecture is not sufficiently in-depth. The RNN experiments show that RNNs also encode the world state, but organize it quite differently from Transformers: the Transformer has a layer that holds the world state on its own and is shared across tasks, whereas the RNN’s hidden state mixes the world state with the next-step prediction, and its representational form changes with the task. This suggests that the paper’s most specific conclusions, such as the form of the coordinates, the two-stage structure, and the consistency across tasks, are likely specific to Transformers. The paper does not analyze which architectural factors give rise to these differences, so the more general claims in the title and abstract lack sufficient support.
Moreover, in this setting each token corresponds to a fixed displacement, so position is a linear function of the token counts; and the attention here is near-uniform, so what it computes is precisely a linear average of the per-step encodings. A linear average cannot merge only those states that share the same next-step distribution, so the fact that the first layer outputs the full position is largely determined jointly by the task structure and this linear aggregation, and does not necessarily show that the network is inclined to build a world model.
I suggest adding ablation experiments to clarify which architectural factors the world representation depends on:
- Ablate the Transformer to identify which design changes cause its world-model representation to disappear. For example, remove the positional encoding, or replace the linear averaging of attention with content-dependent nonlinear aggregation; one could also change the task to use non-additive token semantics (e.g., “forward, turn left, turn right”), so that position is no longer a linear function of the token counts. This would make it possible to determine which factors the world representation depends on.
- Conversely, apply corresponding changes to the RNN to identify which changes allow it to produce a Transformer-like representation, for example by adopting architectures that lie between the two, such as linear recurrences or state-space models.
Together, these two directions would help attribute the differences between the two architectures to specific factors.
Questions for authors
In a non-additive task (i.e., one in which position is no longer a linear function of the token counts), does a linear representation of the world state still emerge in the first layer?
4 comments
Experiments
1. Transformer Ablations
Following the first suggestion, we ablated the transformer in the three ways proposed, to see which changes make its world representation disappear:
- Positional embedding removed (50 runs: five architectures, ten seeds each). Time is no longer linearly decodable from the output of the first attention layer, while position remains so: and decode at held-out –, whereas falls from to . The first MLP recovers part of the time coordinate ( for ), and the loss is higher than with the positional embedding in every run.
- Nonlinear, content-dependent aggregation (120 runs; results for walkers L and D). With a tanh RNN or an LSTM in place of attention, position and time remain linearly decodable (– for , – for ), but an affine function of position and time explains only –% of the activation variance (the paper's RNNs: –%), against over % after softmax attention. The LSTM predicts about as well as attention, the tanh RNN somewhat worse (excess loss – and – that of softmax).
- Non-additive tokens. We rewrote walker L's walks with three tokens: forward (), left turns () and right turns (). The underlying stochastic process is the same as walker L's (further details will be available in the revised manuscript, which we will upload in the next few days). With these tokens, a linear representation of the world state is no longer possible in the first layer, because each forward step's direction depends on earlier turns. Indeed, lattice position is not linearly decodable after the first block, and one-layer networks predict much worse than their two-layer counterparts. Two-layer networks, however, do find a different solution, slightly more complicated but interpretable. The first block encodes, at every token, the balance of left and right turns made so far. The second block's attention pools, for the present token, the balances recorded at all steps in the past context, and the second MLP combines them with the present balance to compute the walker's egocentric position, i.e., its position relative to its own heading. This is exactly the coordinate the next-step distribution depends on, and it becomes linearly decodable at the MLP (held-out up to ), whereas the lattice position never does (at most ).
2. Architectures Between the Two
- Linear aggregation. Linear attention or a state-space layer in place of attention gives a linear representation of the same kind, with the tokens averaged or summed depending on normalization. Normalized linear attention reproduces the softmax representation (CKA –).
- Readout. A linear recurrent network with a linear readout holds position and time exactly () but falls far short of the next-step distribution (excess loss – times that of a GRU).
We will upload the revised manuscript with these results in the next few days.
We remain at the reviewer's disposal to answer any further questions.
We thank the authors for these revisions and additional experiments. The new results substantially improve the rigor and persuasiveness of the paper, and I have decided to raise my score from 3 to 4.
We sincerely thank the reviewer for engaging with our new experiments and for raising their score!
We thank the reviewer for the positive evaluation and constructive feedback.
While the dependence of mechanism on architecture was already an important part of our narrative, with RNNs as architectural controls, this axis was not as empirically explored as others in the paper, as the reviewer pointed out. Their suggestions in this regard have been useful. Following them, we have run new experiments that test which aspect of attention is most relevant to the mechanism's emergence. Concretely, we:
The results indeed help attribute the difference between architectures; we report them below. Specifically, the analyses confirm that the organization that set our transformers apart from our RNNs came from the linearity of their aggregation step.
The reviewer said:
We fully agree with this statement. Because a built-in inclination to model the world is lacking in principle, we are interested in better understanding why and under which conditions world models emerge in next-token predictors (indeed, they do not always develop; see Vafa et al., 2024; 2025). Our example points to one explanation for emergence, namely, an architectural 'simplicity bias' (related to Huh et al., 2024): among the statistics that help predict the next step, the network finds a simple one for attention to compute, which carries the full position.