acceptodds
Under review as a conference paper at ICLR 2027

World Modeling in Transformers

Abstract

Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.

Primary area:
interpretability and explainable AI
Keywords:
World models, world modeling, mechanistic interpretability, transformers, superposition, state tracking, affordance packing, mechanistic indicators.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t6E6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper revisits TaxiGPT, a transformer trained on taxi trajectories through Manhattan, whose behavioral failures were previously interpreted as evidence of an incoherent internal world model. The authors use probing and causal interventions to show that the model encodes intersections, street connectivity, and goal direction, and uses these representations for localization and navigation. They further attribute many navigation failures to interference between superposed intersection representations, and identify an affordance-based organization of these representations that can mitigate the consequences of localization errors. Finally, the paper proposes mechanistic indicators for comparing world-modeling capabilities across model configurations and studying their emergence during training. The central argument is that behavioral failures need not imply the absence of a faithful world representation, since representing and reliably using the environment are distinct capabilities.

<<< UPDATES >>>

i would in general agree that the problem is interesting -- if modes have an internal world representation, and if models can use it for actions / planning / execution. what i liked about this work is that it really went beyond simple linear probing with teleportation / steering / local interventions, which made the conclusions much more convincing to me. while the mechanism is quite interesting, i wanted more evidence that it generalizes beyond manhattan navigation and affordance packing causally protects against local errors (more than correlating it with robustness).

the rebuttal is quite successful imo. authors did provide a new 10x10 and 20x20 maze experiments, and suggested that (some levels of) superposed affordance-packed representations also emerge in another navigation setting. i am still concerned that these mazes are deterministic / discrete navigation environments but this is good generalization evidence.

i would maintain my positive evaluation of this work.

Critical strengths
  1. Strong causal evidence beyond representation probing. I appreciate that the authors do not rely solely on linear decodability to argue for the existence of an internal map. The teleportation experiments, street-transition analyses, goal-compass interventions, and localization experiments provide complementary evidence that the learned representations are functionally involved in navigation. This makes the mechanistic interpretation substantially more convincing than conclusions based on probing accuracy alone.

  2. An informative explanation of behavioral failures. The distinction between a faithful representation and the mechanisms that recruit it for action is an important contribution. The analysis of superposition interference offers a concrete account of why a model with highly accurate spatial representations can nevertheless make illegal moves. The interventions that strengthen the correct position signal or suppress interference provide useful causal support for this explanation. I also find the comparison of world-modeling capabilities across training stages interesting, particularly the observation that goal-direction and legal-move representations emerge before precise localization.

Critical weaknesses
  1. Insufficient positioning against prior work and unclear novelty. A substantial body of literature has already studied emergent world representations in transformers, including spatial representations, state tracking, causal interventions, and the distinction between learned representations and their behavioral use. While the paper discusses some relevant studies, the coverage is incomplete, and the relationship between these earlier findings and the proposed mechanistic account is not sufficiently clear. For example, Gurnee and Tegmark (2024) and Rohekar et al. (2025) are relevant but not discussed. Several of the paper's broader conclusions, particularly that behavioral failures do not necessarily imply the absence of an internal world model, are closely related to existing findings. The authors should more clearly distinguish established observations from their new contributions. As it stands, I see the detailed explanation of TaxiGPT's localization failures as the strongest novelty, rather than the broader conceptual framing.

  2. Limited generality and incomplete causal validation of the proposed mechanisms. The analysis primarily focuses on a single, deterministic navigation environment with explicitly identifiable states and transitions. While the causal interventions provide convincing evidence that the model uses internal spatial representations, it is less clear whether the identified mechanisms generalize beyond TaxiGPT. In particular, the proposed protective role of affordance packing is supported largely by representational similarity and observed error patterns, rather than interventions directly manipulating the packing structure. Stronger evidence would help establish whether affordance packing is itself responsible for improved robustness or simply correlates with other properties of the learned representations.

https://arxiv.org/abs/2310.02207

https://arxiv.org/abs/2412.07446

Questions for authors
  1. I'd like to see stronger causal evidence that affordance packing is what drives the robustness gains. One possible test would be to perturb the representation geometry so that within-affordance alignment is reduced while intersection decodability stays intact. Does this lead to more illegal moves when localization errors occur?

  2. The paper shows that intersection identity and street transitions can be decoded from internal activations, and that steering the position representation changes navigation behavior. What I'm less sure about is whether this reflects a coherent map the model can use globally, or a collection of state and transition features that are only useful locally. Is there evidence that the model uses multi-step connectivity, beyond local transition prediction and goal-direction heuristics?

  3. The cross-model comparisons vary training data, architecture, and objectives, sometimes all at once, which makes it hard to know how robust the proposed indicators are to these differences. Could the authors clarify which conclusions about training objectives or model scale rest on controlled comparisons, and which come from observational differences across configurations?

Reviewer_bFfEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that behavioral failure does not imply the absence of a world model. Using a released TaxiGPT checkpoint (GPT-2 XL, 48L×1600d) trained on Manhattan random walks, the authors extract intersection features by difference-in-means, validate them causally with a "teleportation" edit, recover street connectivity by steering and probing, and identify a running position estimate and a circular goal compass. They then re-diagnose Vafa et al.'s stress, detour and compression failures as out-of-distribution localization slips produced by superposition interference, argue that affordance packing (intersections with identical legal moves have aligned features) limits the damage, and reuse the probes as "mechanistic indicators" to compare five models and twelve training checkpoints.

Critical strengths

S1. The causal evidence is genuinely interventional, not probing-only. Teleportation (App. A.2, Table 3) succeeds on ≥1 scene for 99.3% of 4,202 intersections and on >50% of scenes for 94.0%, against 0.0% without an edit and 8.0% with random features (Table 2). Compass ablation keeps moves legal (≥99.5%) while far-bin goal-reaching falls 94%→23% against a norm-matched random-plane control at baseline (Table 8); look-back ablation drops position decode 99.8%→14.8% while a matched random-subspace edit leaves it unchanged (App. C.1). Matched controls make this an intervention, not a decoding inference.

S2. The benchmark re-reading is falsifiable and consequential. Both compression prefixes decode to their shared intersection in all 146 trials, including all 94 failing pairs (App. G, Table 11); fixing the routes and moving only the goal drops compression 0.983→0.167 as illegal moves rise (Table 12, r=−0.957). If this holds, behavioral map-reconstruction benchmarks measure localization robustness, not map fidelity.

Critical weaknesses

W1. A single toy environment carries the whole claim, including the title. Every model in Tables 1 and 13 is trained on the same 4,516-intersection Manhattan grid with deterministic transitions; the proposed mechanism — superposition → weak position write → slip → affordance protection — is never tested on a second environment, domain, or stochastic transition structure, and Section 6 concedes generality is future work. The generic title and §4's claim that the decode indicator is usable "in other world-modeling tasks" therefore outrun the evidence, which is one case study of one checkpoint family. This caps the verdict: excellent existence proof, not a general account of transformers.

W2. Affordance packing is never intervened on, and the paper's own training sweep contradicts it. The evidence is cosine alignment (Table 5: 91.4% nearest own-affordance mean; AMI .574) and the shared-affordance/illegal-move correlation in Fig. 3c — no manipulation of packing. Worse, Table 16 shows affordance alignment peaking at 98.3% (step 10k) then declining monotonically to 89.0%, while stress on-graph rises 48.5→95.0 and detour 29.0→77.0. A quantity that anti-correlates with behavior over most of training cannot support the abstract's claim that packing "helps limit the consequences" of slips. Packing is a headline mechanism; the story loses a leg.

Questions for authors
  1. Out-of-sample prediction. Table 10's taxonomy is fitted and scored on the same 1,681 illegal moves, so "93.5% accounted for" is a redescription. Do write/noise measured before a failure predict it on held-out rides (report AUC)?
  2. Intervene on packing. If packing is causal, an auxiliary loss discouraging same-affordance alignment, or steering a node's feature toward another affordance group's mean, should raise illegal moves at fixed localization error. Is that observed?
  3. Budget confound. Small RW models receive 28.41B tokens vs 4.74B for large RW (Table 14), a gap App. H.1 concedes. Do "smaller models navigate better with comparable maps" and the L11–18 vs L27–40 layer shift survive compute-matched training?
Reviewer_Ha1hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits TaxiGPT, a transformer trained on Manhattan taxi trajectories, and argues that poor navigation does not imply the absence of a learned map. It extracts intersection and street representations, studies localization and goal-direction signals, and uses activation interventions to explain illegal moves. The authors propose that superposed intersection features interfere with localization and that grouping intersections by legal moves reduces the consequences of some errors. They also compare mechanistic indicators across training stages and models.

The paper is organized and explains the main experimental claims, although the interpretation of several success statistics requires close attention to the appendices. The mechanistic decomposition is informative, but the conclusions remain specific to a constrained navigation model. The experiments are substantive and include causal controls. The principal concerns are the strength of causal attribution and how some headline metrics are interpreted.

Critical strengths

The work usefully separates map representation, localization, and goal-conditioned action selection. This is a more precise account than equating poor behavior with an absent world model. The combination of decoding, street-steering tests, and targeted interventions is a meaningful empirical strength, although the evidence is not equally strong for all components

Critical weaknesses

First, the evidence for a highly faithful operational map is less uniform than the framing suggests. Intersection decoding is strong, but direct street steering ranks the correct successor first on only 76.6% of tested streets, while a separate trained probe reaches 89.2%. These measurements address different questions and should not be combined as if they establish uniformly reliable transition computation. The reported teleportation coverage of 99.3% is based on at least one successful eligible scene per testable node, not a 99.3% success rate over attempts. This is an important distinction for interpreting causal reliability.

Second, the claim that superposition interference explains the navigation failures is not completely isolated from the stress-test distribution shift. The manuscript reports substantially longer origin–goal distances in the stress test than in training-distribution rides. Longer trajectories, weakened localization, and accumulated errors may interact with the proposed mechanism. The targeted interventions support a causal role for feature interference, but do not by themselves establish that it is the dominant cause. Similarly, clustering by legal-move affordances may result from shared training signals; its proposed protective role would be more convincing with an intervention that selectively disrupts clustering while preserving decoding.

Questions for authors

What is the per-attempt teleportation success rate, and how does it vary across eligible scenes per node? Can the authors report the full distribution rather than the at-least-one-success statistic? Can the causal contribution of feature interference be measured under matched goal distances and trajectory lengths? Can affordance clustering be disrupted independently of intersection decoding to test whether it specifically improves recovery from localization errors?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to AC

Dear Area Chair,

Below we list the concerns shared across reviews, then reviewer's remaining points. All changes are in blue in the uploaded manuscript.

Shared concerns

  • Causal evidence for the failure mechanism (Ha1h, bFfE). The failures come from a weak position write and noise in the position code. Setting these two levers induces the failures in correctly localized states (Fig. 3b). Resetting them on the stress-test failures removes most of the illegal probability, against none for random edits of the same size (Fig. 5). The two levers also predict illegal moves on held-out rides (AUC 0.94, new, App. E.3). Section 3 now brings this evidence together more clearly.

  • Packing is not intervened on (Ha1h, bFfE, t6E6). We now test it causally. Steering toward an equally close intersection with the same legal moves makes no move illegal, against 41% for one with different legal moves. Shuffling the legal-move sets across intersections while keeping the features raises illegal moves after a slip from 16% to 65% (App. A.4).

  • Single environment (bFfE, t6E6). In 10×10 and 20×20 mazes, the location features are again superposed and packed by affordance. We also ran the other indicators on the 10×10 maze, and they carry over (§4, App. I).

Other remaining points

  • Ha1h. Teleportation succeeds in 87.4% of attempts, and the full distribution is in App. A.2. Section 2 now states that the street evidence is less strong than for intersections. On the distribution shift, the repairs act on the failing states themselves, so goal distance and ride length are matched, and they work as well in hard rides as in easy ones.

  • bFfE. We added a compute-matched run (Table 13, App. H.1) and an analysis of packing late in training (App. H.3).

  • t6E6. Related work now discusses more papers. The caption of Table 1 states which model comparisons are controlled.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.