- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper revisits TaxiGPT, a transformer trained on taxi trajectories through Manhattan, whose behavioral failures were previously interpreted as evidence of an incoherent internal world model. The authors use probing and causal interventions to show that the model encodes intersections, street connectivity, and goal direction, and uses these representations for localization and navigation. They further attribute many navigation failures to interference between superposed intersection representations, and identify an affordance-based organization of these representations that can mitigate the consequences of localization errors. Finally, the paper proposes mechanistic indicators for comparing world-modeling capabilities across model configurations and studying their emergence during training. The central argument is that behavioral failures need not imply the absence of a faithful world representation, since representing and reliably using the environment are distinct capabilities.
<<< UPDATES >>>
i would in general agree that the problem is interesting -- if modes have an internal world representation, and if models can use it for actions / planning / execution. what i liked about this work is that it really went beyond simple linear probing with teleportation / steering / local interventions, which made the conclusions much more convincing to me. while the mechanism is quite interesting, i wanted more evidence that it generalizes beyond manhattan navigation and affordance packing causally protects against local errors (more than correlating it with robustness).
the rebuttal is quite successful imo. authors did provide a new 10x10 and 20x20 maze experiments, and suggested that (some levels of) superposed affordance-packed representations also emerge in another navigation setting. i am still concerned that these mazes are deterministic / discrete navigation environments but this is good generalization evidence.
i would maintain my positive evaluation of this work.
Critical strengths
-
Strong causal evidence beyond representation probing. I appreciate that the authors do not rely solely on linear decodability to argue for the existence of an internal map. The teleportation experiments, street-transition analyses, goal-compass interventions, and localization experiments provide complementary evidence that the learned representations are functionally involved in navigation. This makes the mechanistic interpretation substantially more convincing than conclusions based on probing accuracy alone.
-
An informative explanation of behavioral failures. The distinction between a faithful representation and the mechanisms that recruit it for action is an important contribution. The analysis of superposition interference offers a concrete account of why a model with highly accurate spatial representations can nevertheless make illegal moves. The interventions that strengthen the correct position signal or suppress interference provide useful causal support for this explanation. I also find the comparison of world-modeling capabilities across training stages interesting, particularly the observation that goal-direction and legal-move representations emerge before precise localization.
Critical weaknesses
-
Insufficient positioning against prior work and unclear novelty. A substantial body of literature has already studied emergent world representations in transformers, including spatial representations, state tracking, causal interventions, and the distinction between learned representations and their behavioral use. While the paper discusses some relevant studies, the coverage is incomplete, and the relationship between these earlier findings and the proposed mechanistic account is not sufficiently clear. For example, Gurnee and Tegmark (2024) and Rohekar et al. (2025) are relevant but not discussed. Several of the paper's broader conclusions, particularly that behavioral failures do not necessarily imply the absence of an internal world model, are closely related to existing findings. The authors should more clearly distinguish established observations from their new contributions. As it stands, I see the detailed explanation of TaxiGPT's localization failures as the strongest novelty, rather than the broader conceptual framing.
-
Limited generality and incomplete causal validation of the proposed mechanisms. The analysis primarily focuses on a single, deterministic navigation environment with explicitly identifiable states and transitions. While the causal interventions provide convincing evidence that the model uses internal spatial representations, it is less clear whether the identified mechanisms generalize beyond TaxiGPT. In particular, the proposed protective role of affordance packing is supported largely by representational similarity and observed error patterns, rather than interventions directly manipulating the packing structure. Stronger evidence would help establish whether affordance packing is itself responsible for improved robustness or simply correlates with other properties of the learned representations.
Questions for authors
-
I'd like to see stronger causal evidence that affordance packing is what drives the robustness gains. One possible test would be to perturb the representation geometry so that within-affordance alignment is reduced while intersection decodability stays intact. Does this lead to more illegal moves when localization errors occur?
-
The paper shows that intersection identity and street transitions can be decoded from internal activations, and that steering the position representation changes navigation behavior. What I'm less sure about is whether this reflects a coherent map the model can use globally, or a collection of state and transition features that are only useful locally. Is there evidence that the model uses multi-step connectivity, beyond local transition prediction and goal-direction heuristics?
-
The cross-model comparisons vary training data, architecture, and objectives, sometimes all at once, which makes it hard to know how robust the proposed indicators are to these differences. Could the authors clarify which conclusions about training objectives or model scale rest on controlled comparisons, and which come from observational differences across configurations?
Message to AC
Dear Area Chair,
Below we list the concerns shared across reviews, then reviewer's remaining points. All changes are in blue in the uploaded manuscript.
Shared concerns
Causal evidence for the failure mechanism (Ha1h, bFfE). The failures come from a weak position write and noise in the position code. Setting these two levers induces the failures in correctly localized states (Fig. 3b). Resetting them on the stress-test failures removes most of the illegal probability, against none for random edits of the same size (Fig. 5). The two levers also predict illegal moves on held-out rides (AUC 0.94, new, App. E.3). Section 3 now brings this evidence together more clearly.
Packing is not intervened on (Ha1h, bFfE, t6E6). We now test it causally. Steering toward an equally close intersection with the same legal moves makes no move illegal, against 41% for one with different legal moves. Shuffling the legal-move sets across intersections while keeping the features raises illegal moves after a slip from 16% to 65% (App. A.4).
Single environment (bFfE, t6E6). In 10×10 and 20×20 mazes, the location features are again superposed and packed by affordance. We also ran the other indicators on the 10×10 maze, and they carry over (§4, App. I).
Other remaining points
Ha1h. Teleportation succeeds in 87.4% of attempts, and the full distribution is in App. A.2. Section 2 now states that the street evidence is less strong than for intersections. On the distribution shift, the repairs act on the failing states themselves, so goal distance and ride length are matched, and they work as well in hard rides as in easy ones.
bFfE. We added a compute-matched run (Table 13, App. H.1) and an analysis of packing late in training (App. H.3).
t6E6. Related work now discusses more papers. The caption of Table 1 states which model comparisons are controlled.
Thank you for your time.