acceptodds
Under review as a conference paper at ICLR 2027

Knowing What Will Matter Narrows World-State Coverage in Transformers

Abstract

Accurately predicting one object's state does not reveal how much of an evolving world remains accessible inside a Transformer. We test whether advance knowledge of the evaluated object changes this breadth of accessible state. In a controlled multi-object state-tracking task, paired two-layer Transformers receive identical trajectories and one selected-object final-state label per example. Announced models know the selected identity before the trajectory, whereas Delayed models never receive it during the forward pass. Across the 14 of 20 paired initializations meeting a development-only success criterion in both conditions, selected-state accuracy is closely matched ( vs. ). Yet Delayed models achieve percentage points higher non-target accuracy through native readouts and points higher accuracy with independently fitted linear probes. Counterfactual interventions further show that second-layer attention updates can substantially redirect selected-state predictions while preserving the recipient residual. Development-ranked head sets outperform matched alternatives on held-out interventions, while complementary reversion tests show larger effects on average for the same ranked contributions. Together, these results separate representational breadth from selected-state computation: even among models that answer the evaluated question accurately, knowing what will matter can narrow what remains accessible.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, state tracking, latent representations, world-state representations, activation patching, probing, causal interventions, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c82FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks: if a model knows in advance which object it will be asked about, does it keep less information about the other objects? The authors test this on a small synthetic task where three objects move between three boxes. Two-layer Transformers are trained two ways: the input names the scored object (Announced), or it does not (Delayed). On the 14 of 20 seed pairs where both models train well, accuracy on the scored object is similar. On the other objects, Delayed is far better. The paper also swaps internal activations between stories and finds that final-layer attention heads share the work unevenly. They conclude that knowing what will matter can narrow what a model keeps, in this setting.

Critical strengths
  1. The paper gives an interesting observation: equal accuracy on the asked question can hide large differences in what else can be read from the model. Section 8 also shows that probe directions transfer little of the model’s output. This is a good reminder for interpretability work.
  2. The experiments are easy to understand. The two models are paired. The 14-pair gate and the head ranking are fixed before testing. The paper also explains clearly what each check can and cannot show.
Critical weaknesses
  1. The model is too small. It has two layers and is trained from scratch. With only three objects, it can easily store everything. It is unclear what this says about larger models._

  2. There is no test of generalization. The paper tests one task and one change. It does not vary the number of objects, the model size, the task, or the story length. The title says “in Transformers,” but the evidence covers one small setup.

  3. The dataset is not representative. The data are synthetic and simple: three objects, fixed symbolic operations, and 8–9 steps. The results come from one generated dataset, so it is unclear how far they apply to real tasks.

  4. Training from scratch is not justified. The title sounds like an effect at inference time: does a model keep less when it knows the question first? This could be tested on a pretrained model by placing the question before or after the text. The paper does not do this or explain why.

  5. "Knowing what will matter" is unclear in a small model without pretraining. Here, “knowing” is only a target token given to the model. In Delayed, every object’s output is trained, so it must track all objects. Announced is not trained with loss on all objects, so these two causes are not separated.

Questions for authors
  1. Does the result hold in larger and deeper models?

  2. Does the result hold with more objects, other tasks, and longer stories?

  3. Can you show the result on a more natural task, or with different generated data?

  4. Why not test a pretrained model with the question placed before or after the text?

  5. If Announced is also trained with loss on all objects, does the gap remain?

Reviewer_EzapOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether knowing the target in advance makes other objects' final state harder to recover from the Transformer's final representation. In a synthetic tracking task, the authors compare Announced models, which are told the target before reading the trajectory, with Delayed models, which don't receive the target identity during the forward pass. Among successfully trained pairs, both models predict the selected object's final state with similar accuracy, but Delayed models are much more accurate on the other non-target final state. Activation patching also shows that some attention heads have a larger effect than others on predicting the donor's selected-object state.

Critical strengths
  1. Figures are easy to follow. In particular, Figure 2 clearly displays the difference in non-target final state accuracy. Figure 3 brings the main activation-patching results together and makes the effects of the different intervention points and attention heads easy to compare.
  2. The experimental setup and design are clear. In Figure 1, it includes the difference between Announced and Delayed conditions, the elements held fixed, and how the native state readout and selected-object loss are applied.
Critical weaknesses
  1. The paper shows the difference in world-state coverage between two models, but it does not fully explain why this question matters and how this finding applies to realistic tasks or model behaviour. A stronger motivation would clarify when advance knowledge of the target is useful and when broader world-state coverage is expected.
  2. The activation-patching experiments examine the attention heads and MLP updates, but they focus on transferring predictions for the selected object. They do not identify which components support non-target state coverage. An analysis of which components preserve non-target information would help explain the difference between Announced and Delayed models.
Questions for authors
  1. What real-world situations does the Delayed condition represent?
  2. When would the Delayed condition be preferable?
  3. Which internal components account for the difference in non-target state coverage between the Announced and Delayed models?
Reviewer_B38eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate how target conditioning affects the coverage of a transformer's predictions in a state-tracking task. Specifically, given a sequence of actions involving a set of objects, a two-layer transformer is trained to predict the final state of a queried object after all actions have been executed, under two settings: with or without the target object's identity provided as input. At test time, the model is evaluated on predicting the states of all objects. The main finding is that providing the target object as input narrows the model's prediction coverage, leading to poorer performance on objects that were not explicitly queried during training. The authors further conduct intervention experiments by transplanting intermediate features from different components of the model trained without target conditioning into the target-conditioned model, thereby revealing the internal state-selection mechanism.

Critical strengths

Although the experiments are conducted in a synthetic setting, the experimental design and findings are interesting and novel. In particular, the results suggest that models tend to learn compact representations that capture only the information necessary for the specified task, with potentially broader implications for understanding transformers' inductive biases in general learning problems. The intervention experiments are also interesting, providing mechanistic insights into the roles of different components within the architecture.

Critical weaknesses
  • The writing would benefit from a higher-level formulation, as the current presentation is too specific to the multi-object state-tracking task. I recommend introducing a general theoretical framework that formally defines the underlying world, states, and learning objective. The multi-object state-tracking task could then be presented as a controlled synthetic environment for investigating this broader problem.

  • The practical implications of the findings remain unclear. In many real-world applications, models are conditioned on a target during both training and inference. It would be helpful to identify concrete scenarios in which a model trained with target conditioning must generalize to unqueried states or targets. One possible connection is to self-supervised representation learning and downstream adaptation. However, the paper should provide a concrete motivating example in which the observed phenomenon has practical impact.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.