acceptodds
Under review as a conference paper at ICLR 2027

False Witnesses: When Irrelevant Facts Change Transformer Answers

Abstract

We identify and study a failure mode in which a true relation about the wrong entity (*irrelevant fact*) steers a transformer toward a wrong answer. Specifically, we study a controlled two-step reasoning task that composes independent random functions: the first maps the queried entity to an intermediate value, and the second maps this value to the final answer. This defines *relevance* exactly, avoiding the interpretive ambiguity of natural-language contexts. An optional suggestion in the input prompt proposes the intermediate value. We call a *true* first-step relation about *another* entity that matches an *incorrect suggestion* a **false witness**. Our **equal-function test** changes whether this relation is present while preserving every entity's correct final answer, isolating its effect on suggestion following. Across two-layer transformers trained from different seeds, adding a false witness doubles predictions following the incorrect suggestion, from 27.4% to 54.8%. Causal interventions via activation patching identify the underlying attention mechanism: query–key matching () favors the irrelevant entity and its value vector () supplies the answer. Replacing alone can recover the queried entity's answer. Training dynamics reveal a gap between solving the task and resisting misleading suggestions: models first become accurate only without suggestions, while resistance to incorrect suggestions develops later. This gap closes when each entity can look up its own final answer in the first attention layer; accuracy under incorrect suggestions then rises from 38.3% to 98.9%, nearly matching accuracy obtained without suggestions. In sum, we establish *false witnesses* as a failure of *relevance*, identify the attention mechanism responsible, and show how training and information access shape resistance to false witnesses.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, entity binding, relational reasoning, causal interventions, shortcut learning, transformer circuits

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v2BJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study false witness, where a true but irrelevant fact about another entity makes a Transformer more likely to follow an incorrect suggestion. To investigate the problem, the authors introduce an equal-function test that changes whether such an irrelevant match is present while keeping all correct two-step answers unchanged. Through experimental results, they show that the false witnesses can substantially increase suggestion-following behavior. The authors further use activation patching to analyze how attention-based record selection contributes to this effect, and find that query/key interactions determine which record is selected, while value vectors carry answer information from the selected record. Lastly, they study how this behavior changes during training and shows that improving information access can substantially increase robustness to misleading suggestions.

Critical strengths
  1. The equal-function test provides a clean and convincing control. By changing whether an irrelevant fact matches the incorrect suggestion while keeping all correct two-step answers unchanged, the experiment isolates the false-witness effect clearly. This makes the main phenomenon easy to interpret and rules out several simpler confounding explanations.
  2. The mechanistic analysis is novel and interesting. The paper goes beyond behavioral observations and further uses activation patching to conduct intervention-based analyses from the perspective of Transformer Q/K/V components. This analysis is both interesting and relatively novel, and it also provides convincing evidence supporting the findings.
  3. The analysis is thorough and well developed. The paper studies the phenomenon from different aspects, including behavioral effects, mechanistic interventions, learning dynamics, and information-access interventions. The progression across different experiments makes the overall analysis feel complete rather than relying on a single diagnostic or experiment.
Critical weaknesses
  1. The broader significance of the study appears limited. The paper motivates and analyzes the problem primarily in a controlled Transformer setting, but provides little evidence or discussion about whether the same false-witness behavior occurs in modern large language models. This gap makes it difficult to assess the practical importance of the phenomenon. If the issue is largely specific to small Transformer and does not persist in contemporary LLMs, then the overall significance of the study would be substantially limited.
  2. The paper is somewhat difficult to read and follow. The writing is not always sufficiently clear, and the logics between sentences and ideas could be improved in many places. For example, the second paragraph of the Introduction directly introduces the two random functions ff and gg, which makes it difficult for readers to quickly grasp the core problem on a first reading. Providing a simple real-world analogy or concrete example before the formal definition would make both the motivation and the false-witness phenomenon much easier to understand.
Questions for authors
  1. All experiments in the paper are conducted in a controlled synthetic Transformer setting. Do the authors have any evidence that the false-witness effect also occurs in pretrained LLMs or natural-language reasoning tasks?
  2. Can the authors provide some mitigation strategies for the false-witness effect? The information-access intervention in Section 6 assumes that the positions and roles of the ff- and gg-table entries are known in advance, allowing the corresponding attention connections to be manually controlled. In realistic settings, however, relevant pieces of information may be interleaved and their roles may not be known a priori, making this method unpractical.
  3. Do the authors expect the Q/K/V-based mechanistic analysis to extend to deeper Transformers? The current mechanistic experiments mainly focus on two-layer models. Would similar false-witness effects and mechanisms be expected in deeper models?
  4. Does the false-witness effect represent a stable failure mode, or is it mainly a temporary shortcut that appears before the model has fully learned the correct retrieval strategy? Since in many training process, some models become more accurate and the false-witness effect decreases substantially. Could the authors clarify whether this effect persists once the underlying two-step task has been fully learned?
  5. How common is the false-witness effect under more natural inputs? The experiments clearly demonstrate this effect in a controlled and highly structured setting, where the input is explicitly organized into ff, gg, and the query. In natural-language inputs, however, relevant and irrelevant information is usually interleaved rather than presented in such a fixed structure. Could the authors provide further discussion or analysis of whether the false-witness effect is expected to persist under more natural and less structured inputs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer v2BJ (1/2): current LLMs are misled by an irrelevant record that matches a wrong hint; a model that answers almost every question correctly can still be misled

Thank you for the detailed review. We are glad the equal-function (EF) test "makes the main phenomenon easy to interpret" and that the mechanistic analysis is "novel and interesting". Line numbers refer to the revised PDF.

Your concernShort answer
Does the effect occur in pretrained LLMs? (W1, Q1)yes: in all 20 settings of our main test across 8 LLMs; e.g., in Qwen3-14B, one irrelevant record raises wrong-hint following from 57.1% to 91.3%
Stable or temporary? (Q4)it depends on the model and the task: training on the task removes it, but off-the-shelf LLMs show it at 99.5% accuracy
Readability (W2)the introduction now starts from a concrete example (Part 2)
Mitigation without known positions (Q2)fine-tuning on fallible hints removes the effect within 25 updates; prompting does not (Part 2)
Deeper Transformers (Q3)yes: it extends to 4–6-layer models and two LLMs (Part 2)
More natural inputs (Q5)the effect persists in unstructured text: with the headers removed and the records mixed with unrelated sentences in one paragraph, all 9 prompts tested still show it, by 3.3–17.3 percentage points (pp; Part 2)

1. The effect persists in contemporary LLMs (W1, Q1)

W1/Q1: "If the issue is largely specific to small Transformer and does not persist in contemporary LLMs, then the overall significance of the study would be substantially limited." "Do the authors have any evidence that the false-witness effect also occurs in pretrained LLMs or natural-language reasoning tasks?"

Yes, it persists in contemporary LLMs. In plain terms:

The test. We gave LLMs the paper's tables as sentences, for example:

Directory: Alice works at Harbor. Bob works at Maple. ...
Locations: Harbor is located in Japan. Maple is located in France. Cedar is located in France. ...
Question: In which country is Alice's employer located?
Hint: Alice works at Cedar.

The correct answer is Japan; the wrong hint points to France. Now change one irrelevant record: "Bob works at Maple." → "Bob works at Cedar."

  • Maple and Cedar are both in France, so every person's correct answer stays the same.
  • The only change is that a true record now matches the wrong hint.

This is the EF test, in natural language.

What happens (Tables 3 and 8):

ModelCorrect without a hintFollows the wrong hint: without → with the irrelevant record
Qwen3-14B99.5%57.1% → 91.3%
Qwen3-4B94.9%21.7% → 69.0%
Mistral-7B94.8%37.9% → 74.6%

We tested 12 LLMs from 6 developers (1.5B–14B parameters, up to 48 layers). The effect is significant in all 20 settings of our main test (8 models and 2 prompt variants, each with the hint before and after the question; new Sec. 7 and App. I), and in 15 of 16 settings of a separate check on the other 4 (two larger, 14B, and two from other developers; Table 8).

We repeated our paper's patching and found the same roles (App. I.4, Table 10):

  • Giving the irrelevant record the attention keys it has in the unchanged prompt removes the effect.
  • Swapping the attention queries after the hint for those of a question about another person makes 87–93% of answers switch to that person's answer.

2. Whether the effect persists depends on the model and the task; high accuracy alone does not remove it (Q4)

Q4: "Does the false-witness effect represent a stable failure mode, or is it mainly a temporary shortcut that appears before the model has fully learned the correct retrieval strategy?"

It depends on the model and the task.

  • Models trained on this exact task (our toy models; low-rank (LoRA) fine-tuning of an LLM): the effect is a learning stage. It must disappear once the model rejects wrong hints: following the hint and answering correctly exclude each other, so on the prompt with the matching irrelevant record, following≤1−accuracy\text{following} \le 1 - \text{accuracy}.
  • Answering hint-free questions correctly is not enough to remove it.
    • Our main models answer 99.8–99.9% correctly without a hint and still show effects of 27–38 pp (Table 1).
    • Fresh 2-layer models with the gg table first reach 99.8% no-hint accuracy at 1,000 updates, with a 24.7 pp effect (App. J.2).
    • Ordinary accuracy does not reveal the problem; the EF test does.
  • Off-the-shelf LLMs show the effect at 99.5% accuracy without any training on our task. It is absent in LLMs too weak for the task: those that answer at most 30.3% correctly show at most 3.8 pp (App. I.2). All our tests read the answer directly, with thinking modes disabled. Test-time reasoning, as in the thinking modes of current frontier models, may weaken it, and heavily quantized models may be too weak for the task.
AuthorsOpenReview ↗

Response to Reviewer v2BJ (2/2): the introduction now opens with a concrete example; fine-tuning removes the effect, prompting does not; deeper models are misled too; the effect also appears in unstructured text

3. The introduction now starts from a concrete example (W2)

W2: "The paper is somewhat difficult to read and follow … the second paragraph of the Introduction directly introduces the two random functions f and g … Providing a simple real-world analogy or concrete example before the formal definition would make both the motivation and the false-witness phenomenon much easier to understand."

Thank you for this concrete suggestion; we adopted it.

  • The revised introduction gives the example from Part 1 before any symbol (L46–49), then introduces ff (person → company) and gg (company → country) as its abstraction. The example previously appeared only in Sec. 4 (now L231–243, Table 2) and App. B, too late for a first reading.
  • We rewrote the densest passages, the definition of the equal-function (EF) test in Sec. 2 (L150–160) and the patching overview in Sec. 4 (L244–264), in shorter sentences tied to the example.

Although the other two reviewers found the presentation clear, a first reading should be easy for every reader, and your comments showed us where it was not; we hope the revised passages now read more easily.

Why the analysis relies heavily on symbols. The EF test must change a record that is certainly irrelevant while keeping every answer fixed. Random functions make this exact, because the tables alone define what is relevant. Natural language is harder to control this tightly: names and companies carry prior associations, and a person may plausibly work at two places. The symbolic task is therefore the minimal form of the two-step reasoning in the example, with symbols in place of its people, companies and countries; App. B develops the full database analogy (see Table 4). The symbols do not create the effect: the new LLM experiments show it in natural language too.


4. Fine-tuning removes the effect without knowing positions; prompting does not (Q2)

Q2: "Can the authors provide some mitigation strategies for the false-witness effect? … In realistic settings … their roles may not be known a priori, making this method unpractical."

Indeed, Sec. 6 was not meant as a practical mitigation, but as a causal test of the mechanism, and the revision now says so explicitly (L425–426). Two strategies need no knowledge of positions:

  • Training on fallible hints removes the effect. Qwen3-4B drops from 50.0 to ≤\le 0.2 percentage points (pp) within 25 updates of LoRA (low-rank) fine-tuning, tested on held-out names (App. I.5, Table 12).
  • Asking the model to state the queried person's employer first (a prompting idea from our Discussion) lowers the effect in 7 of 9 prompts (8 models plus 4-shot Qwen3-4B), e.g. Qwen3-4B 49.6 → 3.4 pp, mostly because the model then follows the wrong hint more often without the irrelevant record, and raises it in the other 2 (L506–508). It is not a mitigation.

The EF test itself is a black-box check that needs no access to internals or positions (L530–532).


5. The analysis extends to deeper models and LLMs (Q3)

Q3: "Do the authors expect the Q/K/V-based mechanistic analysis to extend to deeper Transformers?"

Yes.

  • 4- and 6-layer models trained on our task show the effect in every architecture. Replacing the final-position query still repairs the answer, but the repair is spread over layers: 28.5–76.2 pp from the best single layer (layer 2, 3 or 4), vs 74.3–88.8 pp in 2-layer models (App. J.3).
  • LLMs (32–36 layers). The hint enters through the queries after the hint, mainly in one band of four late layers. Replacing them removes most of the effect (49.0 → 7.6 pp; 44.3 → 1.6 pp; Table 10). Replacing only the irrelevant record's keys with those from the unchanged prompt removes it as well.

6. The effect persists in unstructured text (Q5)

Q5: "How common is the false-witness effect under more natural inputs?"

All LLM experiments use natural sentences. We also removed the headers and mixed all 16 records with unrelated sentences ("Paul enjoys hiking.") in one paragraph (design: App. I.3, item (iii), L1635–1636). The effect remains in all 9 prompts tested (8 models plus 4-shot Qwen3-4B), by 3.3–17.3 pp (Table 7, row "Unstructured text").


We hope these additions address your concerns about significance and readability, and we would be grateful if you would reconsider your score. If any concern remains, please let us know, and we will be glad to address it before the discussion closes.

Thank you for your feedback. My concerns have been addressed, and I have raised my score. Good luck!

Thank you for your follow-up and for raising your score. We are glad that our responses addressed your concerns, and we appreciate your suggestions, which have improved the paper! If anything else could further improve your assessment, please let us know, and we will do our best to address it.

Reviewer_SToROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the false witness phenomenon, where irrelevant facts reinforce incorrect suggestions and steer Transformers toward wrong answers. The authors propose an equal-function test that preserves all correct answers while modifying an irrelevant fact, showing that incorrect suggestion following increases from 27.4% to 54.8%. Through activation patching, they identify how Q/K-based record selection and V-based information retrieval contribute to this behavior. They further examine how training dynamics and attention-based information access affect this vulnerability.

Critical strengths
  • The equal-function test isolates the effect of false witnesses by changing an irrelevant fact while preserving the correct answers for all entities. This provides a clean and convincing way to measure how irrelevant information influences model predictions.

  • The authors use individual and combined Q/K/V activation patching to distinguish incorrect record selection from answer retrieval. The attention-access interventions further provide causal evidence for how information availability affects the learned behavior.

Critical weaknesses
  • All experiments are conducted on small synthetic tasks, primarily using two-layer Transformers. It remains unclear whether the same false witness phenomenon and Q/K/V mechanisms arise in pretrained LLMs. This limits the broader significance and applicability of the findings.

  • Appendix H shows that query-last models achieve 100% accuracy across all conditions without exhibiting false witness effects. Meanwhile, fixed-answer-token models that retain large effects after extended training achieve only around 41% accuracy without suggestions. These results raise questions about whether false witnesses represent a general Transformer vulnerability or a shortcut specific to certain input layouts and incompletely learned models.

Questions for authors
  • Can the authors demonstrate the false witness effect in pretrained LLMs or natural-language multi-hop QA? At minimum, testing whether the same mechanism persists in larger and deeper Transformers would strengthen the generality of the findings.

  • What determines whether the false witness effect emerges or disappears? In particular, can the authors better explain why query-last models are fully robust while fixed-answer-token models remain vulnerable even after extended training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SToR (1/2): 12 off-the-shelf LLMs are misled by an irrelevant record that matches a wrong hint, through the same attention keys and queries as our two-layer models

Thank you for finding the equal-function (EF) test "clean and convincing" and for seeing the attention-access interventions as "causal evidence". Below, we first summarize our answers and then give the details (line numbers refer to the revised PDF):

Concern (abridged)Answer
W1Only small synthetic two-layer models; do the effect and the Q/K/V mechanism arise in pretrained LLMs?Both arise. 12 pretrained LLMs (28–48 layers) show the effect; it is significant in all 20 settings of our main test (8 models and 2 prompt variants, each with the hint before and after the question), up to 49.6 percentage points (pp), and in 15 of 16 settings of a separate check on the other 4 (two larger, 14B, and two from other developers). Giving the witness record (the irrelevant record that matches the hint) its keys from the control prompt removes the effect, and the attention queries after the hint carry the hint and decide whose record is read.
Q1LLMs or natural-language multi-hop QA; at minimum larger and deeper modelsAll done. Natural-language two-hop QA in 12 LLMs; 4- and 6-layer and width-128 models trained on our task show the effect in every architecture (Part 2).
W2A shortcut of specific layouts and incompletely learned models?Not just incomplete learning: it depends on the model and the task. It appears in models that answer almost every question correctly: Qwen3-14B has 99.5% no-hint accuracy, yet its wrong-hint following rises 57.1% → 91.3%. It disappears once a model learns to reject wrong hints, and is absent in LLMs too weak for the task (Part 2). LLMs answer at a fixed position and show the effect with the question in either order. Already in our paper: 2-layer models with ≥\ge 99.95% no-hint accuracy show 45.7–56.3 pp (10 entities per table; L1516–1518).
Q2What determines emergence? Why is query-last robust but the fixed answer token vulnerable?Two variables: (i) whether the attention query that selects a record already holds the queried entity: in our two-layer models with the query last it does, so the effect is zero, while a fixed answer token must fetch the entity by attention and absorbs the hint; (ii) whether the model has learned to reject wrong hints (Part 2).

1. The effect and its Q/K/V roles appear in pretrained LLMs (W1, Q1)

We wrote our paper's task as sentences (new Sec. 7 and App. I):

Our paper (symbols)LLM prompt (sentences)
table ff: person → company8 records "PP works at CC."
table gg: company → country8 records "CC is located in KK."
query xx"In which country is xx's employer located?"
wrong suggestion hh"Hint: xx works at hh."
equal-function (EF) edit: one unqueried row jj changes from rr to hh, with g(r)=g(h)g(r)=g(h)"Hugo works at Birch." → "Hugo works at Harbor."; both companies are in India
model outputthe country with the highest log-probability

Every person's correct answer is the same in the two prompts of an EF pair. Δ\Delta is the rise in wrong-hint following from the control prompt to the witness prompt, which contains the irrelevant record that matches the hint (2,000 pairs per setting; every effect is statistically significant with p<10−10p<10^{-10}). For example, Qwen3-14B answers 99.5% correctly without a hint, yet its following of the wrong hint rises from 57.1% to 91.3% (Table 8); Qwen2.5-7B shows 12.9 pp with the hint after the question and 32.9 pp with it before (Table 3).

We repeated our key and query replacements in Phi-3.5-mini and Qwen3-4B, on 512 fresh pairs whose two prompts differ only in the company token(s) of the witness record (App. I.4, Table 10):

ReplacementTrained 2-layer models (our paper)Phi-3.5-miniQwen3-4B
None—Δ\Delta = 49.0 ppΔ\Delta = 44.3 pp
Witness record's keys ← control'sthe first-step records' keys reproduce most of the effect (Fig. 2)Δ\Delta → 0.0Δ\Delta → 5.5
Queries at the positions after the hint ← same prompt without the hintaccuracy 54.7% → 92.5% (Fig. 3c)Δ\Delta → 7.6 (layers 21–24)Δ\Delta → 1.6 (layers 25–28)
Queries after the hint ← a question about another person90.8% switch to that person's answer87% switch93% switch

In LLMs these roles are spread over depth: the witness's keys matter in earlier layers, and the hint enters mainly through the queries of one late four-layer window. Blocking attention to the witness record also removes the effect (Phi-3.5-mini 46.7 → −1.8 pp; Qwen3-4B 44.5 → −2.0; Mistral-7B 39.1 → −0.6; Table 9).

AuthorsOpenReview ↗

Response to Reviewer SToR (2/2): deeper models are misled too; the effect disappears when the model knows exactly whom it is asked about or has learned to reject wrong hints

2. Deeper and wider models trained on our task show the effect in every architecture (Q1)

We trained models with 2–6 layers and widths 64–128, with either table first, 8 fresh seeds each (App. J.2). Width 128 used learning rate 10−310^{-3}, at which it learns the task.

  • The effect appears during learning in all 10 cells (5 depth–width pairs × 2 table orders); the 8-seed mean peaks at 16.8–30.5 percentage points (pp).
  • Replacing the attention query at the last position in one layer, with that of the same prompt without the hint, restores 74.3–88.8 pp of accuracy in 2-layer models, and 28.5–76.2 pp in 4–6-layer models (best single layer: 2, 3 or 4; App. J.3).

3. Two variables decide when the effect appears (Q2, W2)

In LLMs, both presuppose a model that can do the task: those that answer at most 30.3% of the questions correctly (at most 1.1B parameters, or Mistral-7B and OLMo-2-7B on a harder symbolic version) show at most 3.8 pp (App. I.2).

  1. Information at the record-selecting query. By Theorem G.1 (L1396–1404), sensitivity to a witness (an irrelevant record that matches the hint) is zero exactly when the query identifies the queried entity with certainty (a point mass).
    • Two-layer query-last: the queried entity sits at the position that produces the answer, so the selecting query has it. The effect is zero (L1477–1479), as with a direct connection from the query's embedding (L1508–1509).
    • Fixed answer token: the query must be fetched by attention and mixes in the suggestion. The effect stays at 70.8–77.8 pp after 16,000 updates (L410–412). You are right that these models never learn the lookup (about 41% hint-free accuracy), so they are the partly learned case. That the effect is more than incomplete learning is shown by models that do solve the hint-free task: the N = 10 models (≥\ge 99.95%, 45.7–56.3 pp; L1516–1518) and the LLMs (Part 1).
    • New: deeper models show that the variable is the query, not the layout. 6-layer query-last models with the ff table first show a transient 14–32 pp effect mid-training (5 checkpoints, 4 of 8 seeds). In these states, the final position attends to the suggestion in layer 1 (18–69% of attention, vs 2–8% in matched states without the effect). Replacing only the layer-2 query at the last position with the one from the same prompt without the hint removes the effect (→ 0–4.5 pp; App. J.4, Table 16).
    • LLMs answer at a fixed position, like our fixed answer token. The hint acts through the attention queries after it, mainly in one window of four late layers, and replacing them with queries from the same prompt without the hint removes the effect (49.0 → 7.6 pp; 44.3 → 1.6 pp; Table 10).
  2. Learning stage. Following the hint and answering correctly exclude each other, so the effect Δ\Delta (the rise in wrong-hint following caused by the witness) is at most 1−accuracy(witness)1 - \text{accuracy(witness)}.
    • The effect must vanish once wrong hints are rejected. It need not vanish when a model merely answers hint-free questions correctly: our models reach 99.8–99.9% no-hint accuracy with 27–38 pp effects (Table 1).
    • Evaluated every 250 updates, each of 32 two-layer models (16 main, 16 replication) peaks at 15.1–67.0 pp while answering ≥\ge 98.7% of hint-free questions correctly (App. J.1).
    • Fine-tuning Qwen3-4B with LoRA (low-rank adapters) on fallible hints removes the effect within 25 updates (App. I.5, Table 12).

So incomplete learning is not necessary for the effect: in pretrained LLMs it coexists with 99.5% accuracy. A new Scope paragraph states these two variables (L517–528).


We hope these results address your concerns about generality and about when the effect appears, and we would be grateful if you would reconsider your score. If any additional result could further improve your assessment, please let us know, and we will be glad to provide it before the discussion closes.

Dear Reviewer SToR,

Thank you for the time and effort you have devoted to reviewing our paper, and for your service to the community. We understand this is a busy period, but as the author–reviewer discussion closes on November 18, we would be grateful to know whether our responses have addressed your concerns. In brief, off-the-shelf LLMs and our deeper and wider synthetic models also show the effect, even a model that answers 99.5% of the questions correctly; and the same roles of attention keys and queries appear in LLMs as in our two-layer models (see the new Sec. 7 and Appendices I and J). If any concern remains, we would be happy to address it before the discussion closes.

Dear Reviewer SToR,

Thank you again for your time, and sorry to write once more on the last day of the discussion. Reviewers v2BJ and LCuo, who raised the same concern about pretrained LLMs and deeper models, have since raised their scores. If any concern remains, we would be grateful if you could tell us which one, so that we can answer it before the discussion closes.

Reviewer_LCuoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper constructs an artificial two-hop lookup table: x -> f(x) -> g(f(x)). All function inputs and outputs are integers 0 to 7 (N in total), except in some ablations which modify this range slightly.

Each prompt to the model gives the full random lookup tables f and g, and the query x. It sometimes gives a so-called "witness" h, which is usually f(x) but sometimes not.

The paper trains toy 2-layer Transformers on these prompts, with a simple N-way head to predict the final g(f(x)). From what I understood, the training loss is only on this simple classification head at the last position, and there is no next token prediction loss.

The main effect claimed is that if there is a j != x present that has f(j) = h when h != f(x) and g(h) != g(f(x)), the model outputs the wrong answer g(h) much more often.

The paper also investigates how this false witness actually gets used and how different prompt template layouts affect results.

Critical strengths

The paper's main research question, how irrelevant facts can affect decision making, is important and mirrors research in behavioral economics on human decision making.

The paper's experiments and ablations are thorough.

The authors present results that contradict their desired claim, which is a plus in my opinion.

The paper's deep dive into how the false witness gets used is sound.

The paper's artificial setting allows it to do controlled experiments well.

Critical weaknesses

The paper's artificial setting is far removed from real language models.

The training loss is not next token prediction.

The toy models are very small.

The toy models lack the actual learned capabilities that result from large pretraining plus posttraining in real models.

Just sharing the underlying big-picture Transformer architecture with language models is not enough to make this good evidence about behavior in actual language models, especially given how small the chosen toy architecture is.

The paper's main claimed result is fragile and subject to high variance according to their own reporting. Changing the prompt template layouts or the training update count changes the main claimed false-witness effect by a lot. E.g., see line 1395 (Appendix H, opening paragraph) where it says a query-last prompt layout shows zero false witness following. The different seeds show very high variance (e.g., see Figure 1d).

The paper's mechanism deep dive focuses on a particular setting ("We focus on the eight g-first, suggestion-last models at 2,000 updates."). While this is interesting, the mechanism being so dependent on such hyperparameters and not being generic is a weak point.

The limitations do not summarize clearly enough where the claim is and isn't valid.

Questions for authors
  • Some plots (e.g., Figure 1d, Figure 6) show curved lines between two points. Please clarify what the curve is measuring and why it is not a straight line.

  • Please explain the same-row control (line 184, Appendix D.1) more clearly. Specifically, "the control sets that same entry to a value different from both h and its previous value" is unclear. Isn't the previous value random (but not equal to h)? If so, how is setting it to a new random value meaningful for the model? The model never sees the original lookup tables, so replacing one random value with another random value shouldn't make a difference.

  • Is there any analysis of why the different seeds show such high variance in the false-witness effect?

  • Please explain how the hyperparameters such as 4000 and 2000, 2 layers, etc., were chosen.

  • New experiment that would raise my assessment: What I would like to see is a repeat of the main experiment on real models, with the same lookup table design, using few-shot in-context learning (no training) and fine-tuning settings. If these models do actual reasoning, that'd be a big plus.

7 comments
AuthorsOpenReview ↗

Response to Reviewer LCuo (1/4): all the real-model experiments you requested are done; in each, LLMs are misled by an irrelevant record that matches a wrong hint, even at 99.5% accuracy

Thank you for your service to the community and for the precise review. We answer the requested experiment first, then every other point (line numbers refer to the revised PDF):

Your concernShort answer
Real models, few-shot and fine-tuning (Q5)all done: the effect appears in all 20 settings of our main test (8 pretrained LLMs and 2 prompt variants, each with the hint before and after the question), up to 49.6 percentage points (pp)
Toy setting, loss and model size (W1–W5)LLMs trained with next-token prediction that answer nearly every question correctly are still misled, through the same roles of attention queries and keys (Q/K) (Parts 2 and 4)
Mechanism tied to one setting; how settings were chosen (W7, Q4)no: each setting has a reason (now in App. C); App. E repeats the analysis in both table orders and at both checkpoints, and the core mechanism (the answer position's query picks the wrong record) also holds in 4–6-layer models and two LLMs (Part 2)
Where the claim holds (W8)a new Scope paragraph states where the effect holds and where it does not (Part 3)
Variance across seeds and layouts (W6, Q3)not noise: with the hint last, every model's effect rises and then falls during training (all 32 tracked models peak at 15.1–67.0 pp), and seeds differ only in timing, so one update count catches them at different stages; with the query last, Theorem G.1 predicts the zero effect (Part 3)
Curved lines in the plots (Q1)only the endpoints are data; the curvature is only an aesthetic choice that pulls each line toward the means; box plots now replace the curves (Part 3)
Same-row control (Q2)clarified, both inputs are written out (Part 3)

1. Every requested real-model experiment shows the effect (Q5)

"New experiment that would raise my assessment: … a repeat of the main experiment on real models, with the same lookup table design, using few-shot in-context learning (no training) and fine-tuning settings. If these models do actual reasoning, that'd be a big plus."

We ran every part of this request (details are in the new Sec. 7 and App. I). We wrote our paper's lookup tables as sentences, with fictional people and companies and real countries:

Our paper (symbols)LLM prompt (sentences)
table ff: person → company8 records "PP works at CC."
table gg: company → country8 records "CC is located in KK."
query xx"In which country is xx's employer located?"
wrong suggestion hh"Hint: xx works at hh."
equal-function (EF) edit: one unqueried row jj changes from rr to hh, with g(r)=g(h)g(r)=g(h)"Hugo works at Birch." → "Hugo works at Harbor."; both companies are in India
model outputthe country with the highest log-probability

Every person's correct answer is the same in the two prompts of an EF pair. Below, the witness is the irrelevant record that matches the wrong hint, and Δ\Delta is the false-witness effect, i.e. how much it raises the share of answers that follow the hint (2,000 pairs per setting; every effect is statistically significant with p<10−10p<10^{-10}).

RequestedWhat we ranResult
Real models, same lookup-table design, no training12 off-the-shelf LLMs (1.5B–14B, 28–48 layers), no task trainingSignificant in all 20 natural-language settings of the main test (8 models), up to 49.6 pp (Table 3); with the paper's symbolic tables, Qwen2.5-7B shows the same layout pattern as our two-layer models (13.6 pp with the hint last, −0.4 pp with the question last)
Few-shot in-context learning, no trainingprompts with 0 or 4 solved examples4-shot Qwen3-4B: wrong-hint following 41.7% → 85.1% (Δ\Delta = 43.4 pp; Table 3)
Fine-tuningLoRA, i.e. low-rank adapters (Qwen3-4B); full fine-tuning (SmolLM2-135M, Qwen2.5-0.5B); next-token lossThe learning-stage pattern of our two-layer models. Small LMs show a transient effect (SmolLM2-135M peaks at 6.2–8.7 pp). In Qwen3-4B, Δ\Delta falls from 50.0 to ≤\le 0.2 pp once wrong-hint accuracy saturates (≥\ge 98.9%; App. I.5)
Do models "actually reason"?Accuracy without a hint99.5% (Phi-3.5-mini), 99.3% (Qwen3-4B, 4-shot), 99.7% (Qwen2.5-14B)

The clearest cases.

  • Qwen3-14B answers the two-hop question 99.5% correctly. Yet one true record about another person raises its following of a wrong hint from 57.1% to 91.3% (Table 8).
  • Qwen2.5-7B is told that the records are correct and the hint may be mistaken. It still follows the wrong hint 28.0% of the time without the witness and 75.7% with it (App. I.3).
AuthorsOpenReview ↗

Response to Reviewer LCuo (2/4): LLMs trained with next-token prediction are misled through the same attention mechanism; each training setting has a reason, and neither the effect nor its mechanism depends on it

2. Next-token-trained LLMs that answer nearly every question correctly are still misled, through the same Q/K roles (W1–W5)

W1–W5: "artificial setting … far removed from real language models"; "training loss is not next token prediction"; "toy models are very small"; "lack the actual learned capabilities that result from large pretraining plus posttraining"; sharing the architecture "is not enough".

These LLMs are pretrained with next-token prediction, post-trained, 28–48 layers deep, and answer nearly every question correctly; they are still misled.

The evidence also no longer rests on a shared architecture: the roles of attention queries and keys (Q/K) in our two-layer models appear inside the LLMs (Sec. 7, App. I.4).

  • Blocking attention to the witness record (the irrelevant record that matches the hint) removes the effect in three models, e.g., 46.7 → −1.8 percentage points (pp) in Phi-3.5-mini.
  • Giving the witness record its keys from the control prompt, where it does not match the hint, removes it (49.0 → 0.0 pp in Phi-3.5-mini; 44.3 → 5.5 pp in Qwen3-4B).
  • Swapping the attention queries after the hint for those of a question about another person makes 87–93% of answers switch to that person's answer.

3. Each setting has a reason, and neither the effect nor its mechanism depends on it (W7, Q4)

W7: "…the mechanism being so dependent on such hyperparameters and not being generic is a weak point."

Q4: "Please explain how the hyperparameters such as 4000 and 2000, 2 layers, etc., were chosen."

Thank you for these questions; the paper did not explain the choices clearly. Each has a reason, now stated in App. C (L1050–1074):

  • 4,000 updates is enough training: by 1,000 updates the models answer 98.7–99.9% of hint-free questions correctly on average, and by 4,000, 10 of the 16 main models with the hint last also reject over 90% of wrong hints (L346–356). Training longer, up to 16,000 updates, changes no conclusion (App. H.1, J.2).
  • 2,000 updates, the midpoint, represents the stage the paper is about: the model already answers hint-free questions correctly but does not yet reject wrong hints. At 2,000, all 16 answer ≥\ge 99.4% of hint-free questions, yet 13 still fail most wrong-hint questions. All 32 models we evaluated every 250 updates pass through this stage, only at different times (Part 3). The effect is tied to this stage, not to the update count: 2,000 is simply a checkpoint at which most models are in it.
  • The main results compare every model at these same two checkpoints, never at a checkpoint chosen per model.
  • Two layers is the smallest depth for two reads in sequence, which a two-hop question needs (Sec. 4).
  • Tables of N=8N=8 rows put random-guess acc. at 1/8 = 12.5%, which is low enough; larger tables would only raise the training cost.
  • Depth, width, table size and learning rate are also ablated in App. H.2.

The mechanism analysis covers more than the quoted setting. App. E repeats it in both table orders and at both checkpoints (32 models × 2 checkpoints):

  • Replacing only the attention query at the last position with a donor query for the same entity raises accuracy from 18.98% to 90.65% (ff first) and from 54.67% to 92.55% (gg first); a donor query for another entity redirects 87.94% / 90.82% of answers to that entity's answer (L1278–1280).
  • Table order changes which keys and values carry the effect (L1224–1227), but replacing the last-position query repairs both orders.

Prompted by your concern and the other reviewers' requests, we also ran a systematic sweep (App. J.2): 2, 4 or 6 layers at width 64 and 2 or 4 at width 128 (learning rate 0.001), both table orders, 8 new seeds each. It shows:

  • the effect appears in all 10 settings with the hint last: mean peaks are 16.8–30.5 percentage points (pp), and in each, at least 5 of 8 models exceed 10 pp (L1879–1882);
  • depth and width shift only when it peaks (1,000–4,000 updates; Table 14);
  • it ends as models learn to reject wrong hints: every setting ends within 1.1 pp of zero;
  • the same query replacement works in deeper models: the best single layer restores 28.5–76.2 pp of accuracy in 4–6-layer models (App. J.3), and in two LLMs the queries after the hint carry the effect.
AuthorsOpenReview ↗

Response to Reviewer LCuo (3/4): the revision now states when the effect appears; the spread across seeds comes from models learning at different speeds; figures revised; the control input is explained

4. A new Scope paragraph states where the effect holds and where it does not (W8)

W8: "The limitations do not summarize clearly enough where the claim is and isn't valid."

The new Scope paragraph (L517–528) states when the effect appears: (i) the attention query that selects a record must be gathered by attention, so it can absorb the hint, as in LLMs with the hint before or after the question; (ii) the model has not yet learned to reject wrong hints, since the effect cannot exceed its error rate on the witness prompt; (iii) an LLM must be able to do the task. High hint-free accuracy does not protect a model: the effect persists at 99.5–99.9% in our main models and in LLMs.

The effect is absent when any of these fails: (i) the attention query already has complete information about the queried entity (Theorem G.1), as in our two-layer query-last models and with a direct query-embedding path; (ii) the model rejects wrong hints, as at the end of training in every architecture we trained (2–6 layers, width 64–128) and in an LLM after 25 fine-tuning updates; (iii) the LLM is too weak for the task, e.g., one with at most 1.1B parameters, or Mistral-7B and OLMo-2-7B on a harder, symbolic version (App. I.2).


5. The variation across seeds and layouts follows our account (W6, Q3)

W6/Q3: "fragile and subject to high variance … a query-last prompt layout shows zero false witness following … different seeds show very high variance (e.g., see Figure 1d)"

The variation follows the paper's account; it is not noise.

  • Seeds and training time. The effect Δ\Delta is the rise in wrong-hint following caused by the witness record. Following the hint and answering correctly exclude each other, so Δ≤1−accuracy(witness)\Delta \le 1 - \text{accuracy(witness)}: once a model rejects wrong hints, the effect must vanish.
    • We replayed the 16 main models with the hint last (bit-identical at 2,000/4,000 updates), trained 16 more with new seeds, and evaluated all 32 every 250 updates (App. J.1).
    • All follow one course: Δ\Delta rises while a model learns to use the tables and falls as wrong-hint accuracy rises. Every one of the 32 models peaks at 15.1–67.0 pp while answering ≥\ge 98.7% of hint-free questions correctly.
    • With the ff table first, binned means are 29.7 / 13.3 / 6.1 / 2.6 pp at wrong-hint accuracy < 30 / 30–70 / 70–90 / ≥\ge 90% (see the new Fig. 10). At a fixed update count, seeds sit at different points of this trajectory. That is the spread in Fig. 1d.
  • Layout. The query-last zero you cite (L1477–1479) is predicted by our account.
    • By Theorem G.1 (L1396–1404), sensitivity to a witness is zero exactly when the information about the queried entity is complete. With the query last, the queried entity sits at the position that produces the answer.
    • A direct connection from the query's embedding to that position also gives zero (L1508–1509).
    • A fixed answer token (the same final token in every prompt), which must fetch the query by attention, keeps 70.8–77.8 pp after 16,000 updates (L410–412).
    • Pretrained LLMs, which answer at a fixed position, show the effect with the question in either order.

6. The curves carry no information; the revision uses box plots (Q1)

Only the endpoints are data. Each line is drawn bent toward the line joining the two group means, more strongly the farther the pair lies from it, and the bend carries no information. The revision replaces these curves with box plots, with individual models as dots (Figs. 1, 3, 4 and 6).


7. The same-row control differs from the witness input only in the match (Q2)

You are right that the model cannot tell a new random value from the old one; that is exactly why it is the control.

  • The two compared inputs differ only at row jj (another person) of the person → company table ff: f(j)=hf(j)=h, the wrongly suggested company, in the witness input, vs f(j)=cf(j)=c, another company, in the control.
  • "Different from its previous value" only makes both inputs edits of the same base table. It plays no role for the model.
  • Both inputs keep the queried person xx, f(x)f(x), the company → country table gg, hh and the correct answer fixed. Unlike in the equal-function (EF) test, g(c)g(c) need not equal g(h)g(h).
  • Following of the wrong hint rises by 51.9 / 41.1 percentage points (pp; ff / gg table first, Table 1), so the match, not the edit, drives the effect.

The revision writes out both inputs in App. D.1 to make the control clear (L1120–1129).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: a true but irrelevant record that matches a wrong hint also misleads 12 off-the-shelf LLMs

We sincerely thank the reviewers for their probing questions, which helped us strengthen the paper. Although toy models on controlled tasks are standard in mechanistic interpretability, all three reviewers asked whether the effect reaches real language models and where it holds. During the discussion period we extended the study to off-the-shelf LLMs, with findings we are excited to share.

The false-witness effect is robust in current LLMs. We wrote the paper's two tables as sentences ("Alice works at Harbor.", "Harbor is located in Japan.") and asked for the country of a person's employer, with a wrong hint. One true but irrelevant record is then changed to match the hint, while every person's correct answer stays the same. This misleads every LLM of our main test, in all 20 settings, and larger LLMs too. E.g., Qwen3-14B answers 99.5% of the questions correctly, yet one irrelevant record raises its following of the wrong hint from 57.1% to 91.3%, i.e. a false-witness effect of 34.2 percentage points (pp). Our patching experiments inside two LLMs find the same mechanism as in our two-layer models.

QuestionAsked byShort answer
Does the effect occur in real LLMs?LCuo (W1–W5, Q5), SToR (W1, Q1), v2BJ (W1, Q1)Yes. In all 20 settings of our main test (8 LLMs and 2 prompt variants, each with the hint before and after the question), by up to 49.6 pp, each with p<10−10p<10^{-10} (Sec. 7, Table 3). A separate check on 4 more LLMs, two larger (14B) and two from other developers, finds it in 15 of 16 settings (both hint wordings and positions; Table 8). Neither involves any training on our task.
Does the same mechanism operate inside LLMs?LCuo (W5, W7), SToR (W1), v2BJ (Q3)Yes. Repeating our patching experiments in Phi-3.5-mini and Qwen3-4B, we find, as in our two-layer models, that the model is misled because it attends to the irrelevant record; blocking that attention removes the effect (Sec. 7).
When does the effect appear, and when does it disappear?LCuo (W6, W8), SToR (W2, Q2), v2BJ (Q4)It depends on the model and the task, not only on training time. (1) It appears in models that answer almost every question correctly (Qwen3-14B: 34.2 pp at 99.5% accuracy) (2) It disappears once a model is trained to reject wrong hints (3) It is absent in LLMs too weak for the task (rules of thumb below).
Is it fragile or specific to one layout?LCuo (W6, W7, W8, Q3, Q4), SToR (W2, Q2), v2BJ (Q5)No. (1) In LLMs it appears with the hint before or after the question, under free generation, with a warning that the hint may be wrong, and in unstructured text (Table 7); (2) in our two-layer models, each setting has a stated reason, and neither the effect nor its mechanism depends on it (App. C).
Does it persist in larger, deeper models?LCuo (W3), SToR (Q1), v2BJ (Q3)Yes. During learning, in all 5 trained architectures we tested (2–6 layers, width 64–128; App. J.2), and in LLMs with 28–48 layers.

What the new experiments taught us: rules of thumb for when the effect holds

  • An LLM must be capable enough to do the task. Small LLMs (up to 1.1B parameters) answer at most 30.3% of the questions correctly and show at most 3.8 pp. Larger 7B models face the same issue when the task is changed to a harder, symbolic version: Mistral-7B and OLMo-2-7B are misled in the natural-language test (14.2–36.8 pp), but on the symbolic version they answer at most 23.8% correctly and show no effect (App. I.2).
  • Training on the task removes it; prompting does not. Fine-tuning Qwen3-4B with hints that are wrong 25% of the time removes the effect within 25 updates (App. I.5). Asking the model to first write out the person's employer lowers it in 7 of 9 prompts, mostly because the model then follows the wrong hint more often without the irrelevant record, and raises it in the other 2 (Sec. 7).
  • Conjectured conditions that may weaken it. (i) All our tests read the answer directly, with thinking modes disabled; test-time reasoning, as in the thinking modes of current frontier models, may let a model check the hint against the queried person's record. (ii) Heavy quantization weakens a model and may leave it unable to do the task (first point above).

Per the reviewers' suggestion, we add a Scope paragraph in the Discussion that states where the effect holds and where it does not (L517–528).

These results are added to our revised manuscript (changes in blue), and all line numbers in our replies refer to the revised PDF. Each reviewer's reply gives the details for that reviewer's questions. We thank the reviewers again for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.