- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the effect of repetitive task-irrelevant filler tokens. With the assumption of such fillers degrades model's retrieval capabilities, the authors generates false-belief-based ToM tasks using the CausalToM templates. The authors tested three models from the Llama3.1 and Qwen2.5 model families on the ToM task. Results show that fillers such as whitespace and dots leads to more severe performance degradation than other fillers such as irrelevant or shuffled text. Further investigation show that such fillers capture disproportionate attention weights, thereby diluting the useful information. The authors show that causal attention intervention effectively mitigate the effect of fillers and trace the root cause to abnormal computations in early layers. Further, neuron-level patching demonstrate the the effect of fillers can be largely mitigated by applying patching to a small number of neurons (1 or 4 depending on the model).
Critical strengths
- The phenomenon of distorted attention pattern (activation value) introduced by certain fillers, if proven to be generalizable to a wide range of tasks and models, is interesting and would spark more research efforts in understanding and mitigating the phenomenon.
- Activation patching studies are conducted to probe the causal relationship between the observed phenomenon and certain components/aspects of the models.
- The authors included a range of fillers, covering various aspects such as whitespace, punctuation, and random words.
- The paper is well-written and follows a good logical flow. Starting from the phenomenon, the authors then march into causal attention intervention, critical neuron localization and patching.
Critical weaknesses
- My biggest concern is the limited scope of this study.
- The tested models are very out-dated (e.g., Llama-3.1 and Qwen 2.5), leading to limited insights to current models. Will we observe a similar phenomenon in more recent models, particularly the large reasoning models?
- Only one Theory of Mind task is used for analysis. Would similar phenomenon be observed in other tasks? While the authors mentioned that ToM task is suitable as it demands contextual information retrieval, many other tasks are also applicable including prevalent ones such as GSM8K, which has a shorter context, or LOCOMO which support long context. Even within ToM, data such as FANToM and OpenToM can also be used to validate the generalizability of the identified phenomenon.
- With limited model tested, the trend of the phenomenon is unclear. For instance, Table 1 shows that Qwen2.5-14B is completely prone to the whitespace attack. Does this suggest that only smaller model is affected by repeated whitespaces?
- While the authors have pointed many findings and phenomenon, no root cause or in-depth analysis was given, making the overall analysis superficial and insights limited. For instance, why would fillers affect specific MLP neurons in the shallow layers?
Questions for authors
- Does the stated phenomenon generalize to ToM tasks other than CausalToM or tasks outside the domain of ToM? Is the stated phenomenon observed in more recent models. It would also be interesting to see the behavior of Large Reasoning Models with fillers.
- Is there ablation study that test placing the fillers at a different location other than the given
[Instruction] [Story] [Filler] [Question] [Answer]template? - If chosen examples where the model get wrong at first, would there be cases where adding fillers drive the model to give accurate answer? If so, what would be the explanation for such phenomenon?
- For the remaining erroneous cases after module intervention or neuron patching, what are the (potential) causes of their persistent errors?
5 comments
Response For Q2
Q2: Is there ablation study that test placing the fillers at a different location other than the given [Instruction] [Story] [Filler] [Question] [Answer] template?
Thank you for the question. In our experiments the filler is fixed between the story and the question, and we did not vary its position. However, prior work shows that the phenomenon does not depend on this particular position. Du et al. [1] tested the effect of filler position directly. In addition to the [Evidence] [Whitespace] [Question] layout, which is the same as ours, they moved the whitespace filler before the evidence, i.e., [Whitespace] [Evidence] [Question], so that the evidence is adjacent to the question. Model performance on math, question answering, and coding tasks still dropped under this layout.
- [1] Du et al. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. Findings of EMNLP 2025.
Response For Q3
Q3: If chosen examples where the model get wrong at first, would there be cases where adding fillers drive the model to give accurate answer? If so, what would be the explanation for such phenomenon?
Our evaluation set contains only prompts that all three models answer correctly without filler, so that every model starts from the same baseline. It therefore does not include prompts that are answered incorrectly without filler. We elaborate below.
We do not expect the harmful fillers in our work to correct errors systematically. These fillers draw the attention of post-filler positions away from the story. For example, under dot filler in Llama-3.1-8B, the attention from the readout position to the correct answer token falls to 1% of its no-filler value (Section 3). For a sample that is already answered incorrectly without filler, the model would likewise read less information from the story once the filler is added. A few samples might land on the gold answer by coincidence because the output distribution shifts, but this would be chance, not an improvement in belief tracking.
Prior work has reported two cases in which fillers help, and both rely on mechanisms different from ours. First, for non-repeated irrelevant text, Cuconasu et al. [1] found that adding random documents to the prompt can improve question-answering accuracy by up to 35%. Second, on whether repeated symbols can provide extra computation, Pfau et al. [2] showed that models can be trained to use repeated dots for hidden computation, but that learning this requires specific, dense supervision. They also note, citing earlier results, that off-the-shelf models do not benefit from filler tokens. All models in our work are off-the-shelf models without such training.
- [1] Cuconasu et al. The Power of Noise: Redefining Retrieval for RAG Systems. SIGIR 2024.
- [2] Pfau et al. Let's Think Dot by Dot: Hidden Computation in Transformer Language Models. COLM 2024.
Response For Q1
Q1: Does the stated phenomenon generalize to ToM tasks other than CausalToM or tasks outside the domain of ToM? Is the stated phenomenon observed in more recent models. It would also be interesting to see the behavior of Large Reasoning Models with fillers.
Thank you for raising these points, which also provide clear directions for our future work.
For other tasks: Two sets of results in our paper show that this phenomenon does not depend on a specific prompt. In Appendix E, we replaced the residual stream at the filler positions in the original prompt ([Story][Filler][Question][Answer]) with that from a prompt in which the same filler follows an irrelevant natural context ([Geology][Filler]). Accuracy still dropped (Table 7), indicating that the erroneous representations arise from the model's processing of the filler itself rather than from the preceding context. We also tested factual questions (e.g., "What does the mug contain?") that do not require tracking an agent's belief, and observed the same pattern (Appendix B, Table 4). In addition, Du et al. [1] found that whitespace filler hurts model performance on math, question answering, and coding tasks.
We chose CausalToM [2] because its internal mechanism is already well characterized (Section 1), all three models reach 100% accuracy without filler, and the answer is a single token, which makes the masking and patching experiments straightforward. In contrast, FANToM, OpenToM, GSM8K, and LoCoMo involve multi-step reasoning, which makes it difficult to align a single readout position for neuron-level localization. They are well suited to behavioral evaluation, which we leave to future work. We will revise the manuscript and expand this discussion in the future work section.
For more recent models and reasoning models: Activation patching requires reading and modifying a model's internal states, so it can only be done on open-weight models. Llama-3.1 and Qwen2.5 are common choices in mechanistic interpretability; the original CausalToM study [2] used Llama-3/3.1 and Qwen2.5. Because of limited computational resources, we did not test more recent or larger models, or reasoning models. However, existing evidence suggests that the mechanisms underlying this phenomenon are shared across models: massive activations [3] and the mechanism that forms attention sinks [4] have been found across model families and sizes. Our own results also show that Qwen2.5-7B and Qwen2.5-14B are sensitive to different fillers, which suggests that this sensitivity is learned during training. We therefore expect similar neurons to exist in newer models, although the token that triggers them may differ.
For reasoning models, under causal attention the activations at the filler positions depend only on the preceding input and are not affected by any token generated afterwards. The abnormal activations we observe will therefore form regardless of whether the model later performs explicit reasoning such as CoT. NoLiMa [5] also reports that models with reasoning capabilities struggle to maintain performance in long contexts. How multi-step reasoning interacts with these early representations remains to be tested directly. We will discuss this in Section 7 and list it as future work.