- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks: if a model knows in advance which object it will be asked about, does it keep less information about the other objects? The authors test this on a small synthetic task where three objects move between three boxes. Two-layer Transformers are trained two ways: the input names the scored object (Announced), or it does not (Delayed). On the 14 of 20 seed pairs where both models train well, accuracy on the scored object is similar. On the other objects, Delayed is far better. The paper also swaps internal activations between stories and finds that final-layer attention heads share the work unevenly. They conclude that knowing what will matter can narrow what a model keeps, in this setting.
Critical strengths
- The paper gives an interesting observation: equal accuracy on the asked question can hide large differences in what else can be read from the model. Section 8 also shows that probe directions transfer little of the model’s output. This is a good reminder for interpretability work.
- The experiments are easy to understand. The two models are paired. The 14-pair gate and the head ranking are fixed before testing. The paper also explains clearly what each check can and cannot show.
Critical weaknesses
-
The model is too small. It has two layers and is trained from scratch. With only three objects, it can easily store everything. It is unclear what this says about larger models._
-
There is no test of generalization. The paper tests one task and one change. It does not vary the number of objects, the model size, the task, or the story length. The title says “in Transformers,” but the evidence covers one small setup.
-
The dataset is not representative. The data are synthetic and simple: three objects, fixed symbolic operations, and 8–9 steps. The results come from one generated dataset, so it is unclear how far they apply to real tasks.
-
Training from scratch is not justified. The title sounds like an effect at inference time: does a model keep less when it knows the question first? This could be tested on a pretrained model by placing the question before or after the text. The paper does not do this or explain why.
-
"Knowing what will matter" is unclear in a small model without pretraining. Here, “knowing” is only a target token given to the model. In Delayed, every object’s output is trained, so it must track all objects. Announced is not trained with loss on all objects, so these two causes are not separated.
Questions for authors
-
Does the result hold in larger and deeper models?
-
Does the result hold with more objects, other tasks, and longer stories?
-
Can you show the result on a more natural task, or with different generated data?
-
Why not test a pretrained model with the question placed before or after the text?
-
If Announced is also trained with loss on all objects, does the gap remain?