- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study false witness, where a true but irrelevant fact about another entity makes a Transformer more likely to follow an incorrect suggestion. To investigate the problem, the authors introduce an equal-function test that changes whether such an irrelevant match is present while keeping all correct two-step answers unchanged. Through experimental results, they show that the false witnesses can substantially increase suggestion-following behavior. The authors further use activation patching to analyze how attention-based record selection contributes to this effect, and find that query/key interactions determine which record is selected, while value vectors carry answer information from the selected record. Lastly, they study how this behavior changes during training and shows that improving information access can substantially increase robustness to misleading suggestions.
Critical strengths
- The equal-function test provides a clean and convincing control. By changing whether an irrelevant fact matches the incorrect suggestion while keeping all correct two-step answers unchanged, the experiment isolates the false-witness effect clearly. This makes the main phenomenon easy to interpret and rules out several simpler confounding explanations.
- The mechanistic analysis is novel and interesting. The paper goes beyond behavioral observations and further uses activation patching to conduct intervention-based analyses from the perspective of Transformer Q/K/V components. This analysis is both interesting and relatively novel, and it also provides convincing evidence supporting the findings.
- The analysis is thorough and well developed. The paper studies the phenomenon from different aspects, including behavioral effects, mechanistic interventions, learning dynamics, and information-access interventions. The progression across different experiments makes the overall analysis feel complete rather than relying on a single diagnostic or experiment.
Critical weaknesses
- The broader significance of the study appears limited. The paper motivates and analyzes the problem primarily in a controlled Transformer setting, but provides little evidence or discussion about whether the same false-witness behavior occurs in modern large language models. This gap makes it difficult to assess the practical importance of the phenomenon. If the issue is largely specific to small Transformer and does not persist in contemporary LLMs, then the overall significance of the study would be substantially limited.
- The paper is somewhat difficult to read and follow. The writing is not always sufficiently clear, and the logics between sentences and ideas could be improved in many places. For example, the second paragraph of the Introduction directly introduces the two random functions and , which makes it difficult for readers to quickly grasp the core problem on a first reading. Providing a simple real-world analogy or concrete example before the formal definition would make both the motivation and the false-witness phenomenon much easier to understand.
Questions for authors
- All experiments in the paper are conducted in a controlled synthetic Transformer setting. Do the authors have any evidence that the false-witness effect also occurs in pretrained LLMs or natural-language reasoning tasks?
- Can the authors provide some mitigation strategies for the false-witness effect? The information-access intervention in Section 6 assumes that the positions and roles of the - and -table entries are known in advance, allowing the corresponding attention connections to be manually controlled. In realistic settings, however, relevant pieces of information may be interleaved and their roles may not be known a priori, making this method unpractical.
- Do the authors expect the Q/K/V-based mechanistic analysis to extend to deeper Transformers? The current mechanistic experiments mainly focus on two-layer models. Would similar false-witness effects and mechanisms be expected in deeper models?
- Does the false-witness effect represent a stable failure mode, or is it mainly a temporary shortcut that appears before the model has fully learned the correct retrieval strategy? Since in many training process, some models become more accurate and the false-witness effect decreases substantially. Could the authors clarify whether this effect persists once the underlying two-step task has been fully learned?
- How common is the false-witness effect under more natural inputs? The experiments clearly demonstrate this effect in a controlled and highly structured setting, where the input is explicitly organized into , , and the query. In natural-language inputs, however, relevant and irrelevant information is usually interleaved rather than presented in such a fixed structure. Could the authors provide further discussion or analysis of whether the false-witness effect is expected to persist under more natural and less structured inputs?
4 comments
Response to Reviewer v2BJ (2/2): the introduction now opens with a concrete example; fine-tuning removes the effect, prompting does not; deeper models are misled too; the effect also appears in unstructured text
3. The introduction now starts from a concrete example (W2)
W2: "The paper is somewhat difficult to read and follow … the second paragraph of the Introduction directly introduces the two random functions f and g … Providing a simple real-world analogy or concrete example before the formal definition would make both the motivation and the false-witness phenomenon much easier to understand."
Thank you for this concrete suggestion; we adopted it.
- The revised introduction gives the example from Part 1 before any symbol (L46–49), then introduces (person → company) and (company → country) as its abstraction. The example previously appeared only in Sec. 4 (now L231–243, Table 2) and App. B, too late for a first reading.
- We rewrote the densest passages, the definition of the equal-function (EF) test in Sec. 2 (L150–160) and the patching overview in Sec. 4 (L244–264), in shorter sentences tied to the example.
Although the other two reviewers found the presentation clear, a first reading should be easy for every reader, and your comments showed us where it was not; we hope the revised passages now read more easily.
Why the analysis relies heavily on symbols. The EF test must change a record that is certainly irrelevant while keeping every answer fixed. Random functions make this exact, because the tables alone define what is relevant. Natural language is harder to control this tightly: names and companies carry prior associations, and a person may plausibly work at two places. The symbolic task is therefore the minimal form of the two-step reasoning in the example, with symbols in place of its people, companies and countries; App. B develops the full database analogy (see Table 4). The symbols do not create the effect: the new LLM experiments show it in natural language too.
4. Fine-tuning removes the effect without knowing positions; prompting does not (Q2)
Q2: "Can the authors provide some mitigation strategies for the false-witness effect? … In realistic settings … their roles may not be known a priori, making this method unpractical."
Indeed, Sec. 6 was not meant as a practical mitigation, but as a causal test of the mechanism, and the revision now says so explicitly (L425–426). Two strategies need no knowledge of positions:
- Training on fallible hints removes the effect. Qwen3-4B drops from 50.0 to 0.2 percentage points (pp) within 25 updates of LoRA (low-rank) fine-tuning, tested on held-out names (App. I.5, Table 12).
- Asking the model to state the queried person's employer first (a prompting idea from our Discussion) lowers the effect in 7 of 9 prompts (8 models plus 4-shot Qwen3-4B), e.g. Qwen3-4B 49.6 → 3.4 pp, mostly because the model then follows the wrong hint more often without the irrelevant record, and raises it in the other 2 (L506–508). It is not a mitigation.
The EF test itself is a black-box check that needs no access to internals or positions (L530–532).
5. The analysis extends to deeper models and LLMs (Q3)
Q3: "Do the authors expect the Q/K/V-based mechanistic analysis to extend to deeper Transformers?"
Yes.
- 4- and 6-layer models trained on our task show the effect in every architecture. Replacing the final-position query still repairs the answer, but the repair is spread over layers: 28.5–76.2 pp from the best single layer (layer 2, 3 or 4), vs 74.3–88.8 pp in 2-layer models (App. J.3).
- LLMs (32–36 layers). The hint enters through the queries after the hint, mainly in one band of four late layers. Replacing them removes most of the effect (49.0 → 7.6 pp; 44.3 → 1.6 pp; Table 10). Replacing only the irrelevant record's keys with those from the unchanged prompt removes it as well.
6. The effect persists in unstructured text (Q5)
Q5: "How common is the false-witness effect under more natural inputs?"
All LLM experiments use natural sentences. We also removed the headers and mixed all 16 records with unrelated sentences ("Paul enjoys hiking.") in one paragraph (design: App. I.3, item (iii), L1635–1636). The effect remains in all 9 prompts tested (8 models plus 4-shot Qwen3-4B), by 3.3–17.3 pp (Table 7, row "Unstructured text").
We hope these additions address your concerns about significance and readability, and we would be grateful if you would reconsider your score. If any concern remains, please let us know, and we will be glad to address it before the discussion closes.
Thank you for your feedback. My concerns have been addressed, and I have raised my score. Good luck!
Thank you for your follow-up and for raising your score. We are glad that our responses addressed your concerns, and we appreciate your suggestions, which have improved the paper! If anything else could further improve your assessment, please let us know, and we will do our best to address it.
Response to Reviewer v2BJ (1/2): current LLMs are misled by an irrelevant record that matches a wrong hint; a model that answers almost every question correctly can still be misled
Thank you for the detailed review. We are glad the equal-function (EF) test "makes the main phenomenon easy to interpret" and that the mechanistic analysis is "novel and interesting". Line numbers refer to the revised PDF.
1. The effect persists in contemporary LLMs (W1, Q1)
Yes, it persists in contemporary LLMs. In plain terms:
The test. We gave LLMs the paper's tables as sentences, for example:
The correct answer is Japan; the wrong hint points to France. Now change one irrelevant record: "Bob works at Maple." → "Bob works at Cedar."
This is the EF test, in natural language.
What happens (Tables 3 and 8):
We tested 12 LLMs from 6 developers (1.5B–14B parameters, up to 48 layers). The effect is significant in all 20 settings of our main test (8 models and 2 prompt variants, each with the hint before and after the question; new Sec. 7 and App. I), and in 15 of 16 settings of a separate check on the other 4 (two larger, 14B, and two from other developers; Table 8).
We repeated our paper's patching and found the same roles (App. I.4, Table 10):
2. Whether the effect persists depends on the model and the task; high accuracy alone does not remove it (Q4)
It depends on the model and the task.