acceptodds
Under review as a conference paper at ICLR 2027

The Circuit Basis of Metacognitive Failure in Language Models: Conflict and Hallucination

Abstract

Language models often answer with high confidence, as measured by the entropy of the output distribution, even when they are wrong. We study two situations that lead to such errors, *knowledge conflict*, where the context contradicts a fact stored in the weights, and *missing knowledge*, where the queried fact was never learned and the model *hallucinates*. To control exactly what the model has learned, we fine-tune it on synthetic facts that map arbitrary entity identifiers to random codes, installed through low-rank adapters (LoRA) on the multi-layer perceptron (MLP), query-key (QK), or value-output (VO) components. Our analysis suggests that MLP is the main substrate for storing facts in *basins*, regions of representation space into which a fact's representations converge, while QK steers generation toward a basin or the context, and VO writes the retrieved content into the residual stream. *Under knowledge conflict*, the arbitration between the stored fact and the context fails in a way that depends on how the fact is stored. A fact learned from a single phrasing yields to the context, but errors accumulate in the later digits, while a fact learned from many phrasings overrides the context, and the answer drifts into one that matches neither source. *Under missing knowledge*, the hidden state reaches no basin, yet every adapter placement produces confident hallucinations, most of all through VO. In both situations, the output entropy reflects how sharply the model commits rather than whether the right fact was retrieved, even though this information remains present in its representations. Across 12 pretrained models, the overall hallucination rate falls with scale, but a growing fraction of the remaining hallucinations are delivered with high confidence. Our mechanistic insights thus pinpoint a systemic metacognitive gap that does not close with scale in the models we study.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, hallucination, knowledge conflict, factual recall, metacognition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vb9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • This paper investigates knowledge conflict and hallucination in LLMs using a synthetic dataset of entity–code associations.
  • The authors examine how different components (QK, VO, and MLP) affect knowledge storage, retrieval, and confidence through LoRA fine-tuning.
  • They analyze hidden-state geometry to explain why models can produce confident but incorrect answers.
Critical strengths
  • The synthetic dataset is an interesting way to control what the model has learned and isolate different failure cases.
  • The experiments are carefully conducted, covering different adapter placements, training settings, and failure modes.
  • The analysis provides interesting observations on the relationship between knowledge storage, context utilization, and hallucination.
Critical weaknesses
  • Several related findings have already been explored in prior work, which somewhat limits the novelty.
    • [1] Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer (Dong et al., 2025).
  • [2] Taming Knowledge Conflicts in Language Models (Li et al., ICML 2025).
  • [3] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (Orgad et al., ICLR 2025).
  • [4] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (Gekhman et al., EMNLP 2024).
  • The main mechanistic experiments are conducted on a single model, making it difficult to assess how well the findings generalize.
  • The reliance on synthetic entity–code associations also leaves open whether similar behaviors occur with natural-language knowledge.
Questions for authors
  • Do the findings hold for other model families and sizes?
  • Would similar patterns emerge when using natural-language facts rather than synthetic codes?
  • How sensitive are the results to the choice of LoRA configuration?
5 comments

We thank the reviewer for the detailed and constructive review.

Summary of the response. The reviewer raises three concerns, namely the overlap with prior work, the single model and the synthetic facts. The revision addresses all three.

  • Novelty (W1). None of the four papers has a setting in which what the model learned is known exactly, which is what lets us show that the same fact yields to the context or overrides it depending only on how it was trained.
  • Beyond one model (W2). The fine-tuning results repeat on Falcon3-3B-Instruct.
  • Real-world facts (W3). The base model shows the same conflict behavior with real-world facts it already knows.
  • LoRA rank (Q3). The same failures appear at rank 8, 32 and 64.

Together, these experiments substantially strengthen the thesis of the paper, as the findings now hold on a second model family, with real-world facts and across LoRA ranks. They are therefore not tied to one model, one rank or synthetic facts.

Below, "brittle" facts are those trained on a single phrasing and "robust" facts those trained on 53 (ln. 137–139). Locations given as "rev." refer to the revised manuscript.

W1: Relation to prior work.

"Several related findings have already been explored in prior work, which somewhat limits the novelty."

Thank you for pointing to these works. The results are consistent with all four papers, and what this paper adds is a setting in which what the model learned is known exactly. The revision now cites [1] and [2].

WorkEstablishesThis paper adds
Dong et al. 2025 [1]In models trained from scratch, attention handles in-context reasoning and the MLP handles storageThe roles of QK, VO and MLP inside a pretrained model, and the distinct error each one produces when it stores the fact
Li et al., ICML 2025 [2]Attention heads mix memory and context, and can be steered toward either at test timeThe same fact yields to the context or overrides it depending only on how many phrasings it was trained on
Orgad et al., ICLR 2025 [3]Internal states carry a hallucination signalExact ground truth for what the model learned, and an explanation of why output entropy misses the signal
Gekhman et al., EMNLP 2024 [4]Fine-tuning on new knowledge encourages hallucinationThe same effect under controlled conditions, broken down by adapter placement

W2: A single model.

"The main mechanistic experiments are conducted on a single model, making it difficult to assess how well the findings generalize."

Following the reviewer's suggestion, the fine-tuning experiments were repeated on a second model, Falcon3-3B-Instruct, and the same behavior is observed (rev. App. G, Table 15).

Qwen2.5-3BFalcon3-3B
Base model copies a code from the prompt91%95%
Robust MLP adapter copies a code from the prompt0%14%
Robust MLP adapter recalls its facts on unseen phrasings100%100%
Brittle facts override a contradicting prompt0%0%
Hallucinated answers that are another entity's trained code, for QK / VO / MLP / Full adapters4 / 8 / 28 / 32%3 / 4 / 9 / 12%

On both models, robust storage keeps recall intact and removes most or all of the copying from the prompt, brittle facts never override the prompt, and hallucinated answers reuse trained codes most often with MLP and Full adapters. One difference is that brittle adapters on Falcon leave copying intact for all placements.

In addition, the VO signature replicates across five model families (ln. 194–195, App. H), and the scaling results cover 12 models from four families.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W3: Synthetic codes.

"The reliance on synthetic entity–code associations also leaves open whether similar behaviors occur with natural-language knowledge."

The conflict behavior also occurs with real-world facts that the base model already knows (rev. App. K). Note also that the scaling analysis in the paper uses natural-language benchmarks (TriviaQA, NQ-Open).

In the paper, a robust fact overrides a contradicting line in the prompt. We tested whether the base model treats its own pretrained knowledge in the same way. We took 192 facts that it answers correctly (capitals, languages, continents) and placed a line stating a wrong answer before the question. The model kept its own answer in 73% of trials and took the answer from the prompt in 25%. For our synthetic entities, about which it knows nothing, it copies the code from the prompt in 92–100% of trials. Hence the base model copies from the prompt when it has no knowledge of its own and keeps its own answer when it does, which is the behavior of our robust adapters.

We used synthetic codes as natural language does not tell us what a model has learned.

Q1: Other model families and sizes.

"Do the findings hold for other model families and sizes?"

Yes, the findings hold on a second model family (see W2). Regarding model size, the scaling result holds across 12 models from four families, and the hallucination rate across 17 models from six.

Q2: Natural-language facts.

"Would similar patterns emerge when using natural-language facts rather than synthetic codes?"

Yes, please see the response to W3.

Q3: Sensitivity to the LoRA configuration.

"How sensitive are the results to the choice of LoRA configuration?"

The same failures are observed at rank 8, 32 and 64, with no sign that they weaken at higher rank (rev. App. G, Table 16).

Fig. 7b shows that learning speed changes by at most about 15% across ranks (ln. 877–881). We have now also evaluated the behavior of the adapters from that sweep (Full placement, brittle facts).

RankRecalls its factsCopies a code from the promptHallucinated answers that are another entity's trained code
8100%31%32%
32100%43%73%
64100%4%85%

At every rank the adapter recalls its facts, copies from the prompt less often than the base model (91%), and reuses trained codes when it has no answer. Learning rate and LoRA scale are the same for all placements, so placement is the only variable in the main comparison.

With the results on a second model family, on real-world facts and across LoRA ranks, the findings are now shown to generalize, and the paper is better as a result. We thank the reviewer for suggesting these experiments! We hope we have sufficiently addressed all of the reviewer's concerns.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

Reviewer_7gXsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study the circuit mechanisms underlying metacognitive failure in language models, specifically, why models generate incorrect answers with high confidence (low output entropy) under knowledge conflict (context contradicts parametric memory) and missing knowledge (unlearned entities leading to hallucination).

Critical strengths

A central obstacle in studying LLM metacognition and calibration is distinguishing genuine parametric recall from lucky guessing or in-context inference. By designing an exact synthetic entity-code task and decoupling what is stored, how robustly it is learned, and where it is localized via modular LoRA targets, the authors achieve exceptional causal control. Connecting this framework to exact sublayer Jacobians (contracting symmetric vs. rotational antisymmetric components) provides an elegant, mathematically sound mechanistic account of the functional division between QK, VO, and MLP.

Critical weaknesses
  • The paper's strongest mechanistic conclusions (e.g., discrete basin formation, context deafness, and margin separations) rely entirely on synthetic 5-digit numeric codes learned via LoRA on a single 3B base model. As candidly acknowledged in Appendix K, an earlier attempt to extend the geometric margin metric to natural-language knowledge (e.g., capitals, languages, TruthfulQA) collapsed due to surface-form and base-rate confounds. Natural language facts are semantic, polysemous, and distributed across overlapping subspaces; it remains unproven whether dense pretraining weights organize real-world knowledge into the isolated "basins" observed in this low-rank synthetic setting.
  • While the paper diagnoses why post-hoc calibration fails, its internal detection and mitigation findings highlight severe architectural constraints that are not fully resolved. Appendix I shows that end-to-end training of a metacognitive head distorts the underlying basin geometry (r=0.30r = 0.30), while Appendix I.2 demonstrates that basin separation emerges only late in the network (peaking at layers 28–30). Because QK routing and arbitration occur in earlier layers, single-pass feedforward architectures are fundamentally incapable of conditioning source selection on their own epistemic state. The paper clearly identifies this "timing constraint" but offers limited actionable remedies beyond multi-query paraphrase consistency.
Questions for authors
  1. Could you elaborate on why the geometric margin failed when applied to natural-language facts in the base model? Specifically, did natural-language queries fail to form contracting clusters, or did lexical variations overpower the distance metric in hidden state space? Does this imply that the "basin" geometry is an artifact of rote-memorizing arbitrary codes via LoRA?

  2. In Appendix H, you show that the high mid-layer VO symmetry (ϕ≈0.64\phi \approx 0.64) is dynamically driven by attention sinks driving at,t≈0a_{t,t} \approx 0 for non-informative heads, preserving heads with symmetric WOWVW_O W_V. If evaluated on architectures trained with sink-free mechanisms (e.g., gated attention or explicit sink tokens), does this VO symmetry peak disappear, and would that alter the observed error patterns of VO adapters?

  3. In Section 4.1 and Figure 4c, under robust storage conflict, the generation begins with the stored digit and subsequently drifts into an answer matching neither source. During this drift (digits 2 to 5), does the hidden trajectory escape into an unpopulated region of representation space, or does it get captured by a neighboring memorized basin belonging to a different entity?

3 comments

We thank the reviewer for the detailed and constructive review.

Summary of the response. The reviewer raises two concerns, that the conclusions rest on synthetic codes in a single model and that the timing constraint leaves few actionable remedies. The revision addresses both, and we answer the three questions with new experiments.

  • Real-world facts (W1, Q1). The base model shows the same conflict behavior with real-world facts it already knows.
  • Second model (W1). The fine-tuning results repeat on Falcon3-3B-Instruct.
  • Remedies (W2). The results give three concrete remedies, including detection of a missing fact in a single forward pass.
  • Sink-free models (Q2). The VO symmetry peak weakens when the attention sink is removed, as App. H predicts.
  • Drift (Q3). The hidden state ends far from every stored fact, so it is not captured by another entity's fact.

Together, these experiments substantially strengthen the thesis of the paper. The behavior we isolate with synthetic codes is one that pretrained knowledge already shows, as the base model keeps its own real-world facts against a contradicting prompt, and the fine-tuning results repeat on a second model family. Hence the synthetic setting does not create the behavior. Rather, it lets us say where the behavior comes from and what can be done about it.

Below, "brittle" facts are those trained on a single phrasing and "robust" facts those trained on 53 (ln. 137–139). Locations given as "rev." refer to the revised manuscript.

W1: Synthetic codes on a single model.

"rely entirely on synthetic 5-digit numeric codes learned via LoRA on a single 3B base model. [...] it remains unproven whether dense pretraining weights organize real-world knowledge into the isolated "basins" observed in this low-rank synthetic setting."

The same conflict behavior appears in the base model for real-world facts it already knows, and the fine-tuning results hold on a second model family (rev. App. K and G).

Real-world facts. In the paper, a robust fact overrides a contradicting line in the prompt. We tested whether the base model treats its own pretrained knowledge in the same way. We took 192 facts that it answers correctly (capitals, languages, continents) and placed a line stating a wrong answer before the question. The model kept its own answer in 73% of trials and took the answer from the prompt in 25%. For our synthetic entities, about which it knows nothing, it copies the code from the prompt in 92–100% of trials. Hence the base model copies from the prompt when it has no knowledge of its own and keeps its own answer when it does, which is the behavior of our robust adapters.

Second model family. We repeated the fine-tuning on Falcon3-3B-Instruct. The robust MLP adapter copies a code from the prompt in only 14% of trials, compared with 95% for the base model, while it recalls its stored facts on unseen phrasings in 100% of trials. The full comparison is in our reply to Reviewer Vb9f (W2).

We chose synthetic codes as they give the causal control the reviewer mentions, whereas with real-world facts we cannot know exactly what the model has learned. For basins in real-world facts, see Q1.

W2: The timing constraint.

"single-pass feedforward architectures are fundamentally incapable of conditioning source selection on their own epistemic state. The paper clearly identifies this "timing constraint" but offers limited actionable remedies beyond multi-query paraphrase consistency."

The results do suggest three remedies beyond paraphrase consistency, and the revision adds a paragraph that discusses them (rev. App. I.2).

The timing constraint says that the signal telling the model whether it knows the answer appears too late in the forward pass to steer that same pass. None of the remedies below needs the signal to arrive earlier.

  • Detect after the pass, then abstain or retrieve. The distance from a query's hidden state to the nearest stored fact (the margin, Eq. 3) can be read at the end of a single forward pass, with no paraphrase queries. It is about 22 for recalled facts and 133 for never-learned queries, and none of a hundred never-learned queries falls in the range of recalled facts (ln. 351–354).
  • A learned detector. A small network predicts this distance from the hidden state alone (App. I.1). It catches every hallucination while keeping 58.5% of correct answers (ln. 1301–1312).
  • Choose how the fact is trained. Whether a stored fact yields to the context is decided at training time. A fact trained on up to 5 phrasings yields to the context in over 90% of trials, and a fact trained on 20 or more in at most 2% (new, reply to Reviewer vqKT, W3b).

The two-pass design at ln. 1364–1365, in which the model first assesses whether it knows the answer and then processes the input again, remains the direct way around the constraint.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

Q1: Natural-language margin, and rote memorization.

"did natural-language queries fail to form contracting clusters, or did lexical variations overpower the distance metric in hidden state space? Does this imply that the "basin" geometry is an artifact of rote-memorizing arbitrary codes via LoRA?"

Natural-language queries do form clusters, but the distance is dominated by the type of question. The basin is unlikely to be an artifact of rote memorization, since an adapter that memorizes each code from a single phrasing has no basin.

Clusters. The margin comparison was inconclusive because it needs wrong answers to set against right ones, and the model answers 185 of the 196 facts correctly. We therefore ran a test that does not need errors (rev. App. K). For each fact we average its hidden state over several phrasings, and then ask whether a further phrasing of the same fact lies nearest to its own average. This is the case in 15.8% of trials, about 30 times above chance (0.5%), and in 97% of cases the nearest average belongs to a fact of the same type (for example another capital). Hence the clusters form but the type of question dominates the distance, which is the second of the reviewer's two alternatives.

Rote memorization. The brittle adapter is the rote-memorization case, as it learns each code from a single phrasing and recalls it in 100% of trials. It nevertheless has no basin, as never-trained entities end just as close to the memorized states as recalled facts do, whereas with the robust adapter they end 97 farther away. Hence the basin comes from training on varied phrasings rather than from memorizing arbitrary codes.

Q2: Sink-free architectures.

"If evaluated on architectures trained with sink-free mechanisms (e.g., gated attention or explicit sink tokens), does this VO symmetry peak disappear, and would that alter the observed error patterns of VO adapters?"

In models trained without the sink, the peak becomes weaker, as App. H predicts, but it does not disappear (Table 17 in rev. App. H).

App. H attributes the VO symmetry peak to heads that are switched off by attending to the first token (the attention sink). We measured VO symmetry on three 1B models from Qiu et al. (NeurIPS 2025) that were trained identically apart from the attention mechanism. One is a standard softmax model, and the other two use a gate to switch heads off, so they no longer need the sink.

ModelAttention on the first tokenVO symmetry, middle layersVO symmetry, peak
Softmax baseline0.590.400.60
Head-wise gate0.240.250.46
Element-wise gate0.170.100.44

As attention on the first token falls, VO symmetry in the middle layers falls with it. The peak falls less, because the gate now does what the sink did. On the second part of the question, the error patterns of VO adapters are measured directly from the model's answers, hence they do not rely on the symmetry.

Q3: Drift: escape or capture.

"does the hidden trajectory escape into an unpopulated region of representation space, or does it get captured by a neighboring memorized basin belonging to a different entity?"

The trajectory escapes into an unpopulated region. In 200 conflict trials, the hidden state never comes close enough to any memorized state to recall it, at any of the five digits (rev. App. F).

The drift refers to Fig. 4c, where the context line gives the stored code with only its first digit changed, and the model gets the later digits wrong more and more often. We measured the distance from the hidden state to the nearest memorized state while each digit is generated. Recall requires a distance of at most about 50.

Distance to the nearest memorized stateFirst digitLast digit
Recall (correct answer)3630
Conflict (drift)107129
Never-trained entity (hallucination)134117

Under conflict, the state is as far from every memorized state as it is for an entity the model never learned. Hence it is not captured by a neighboring fact. The answers nevertheless look like stored facts, as in 66 of the 200 trials the answer is the complete code of a different trained entity, about 19 times chance. Conflict and missing knowledge therefore end in the same failure, where the internal state is far from every stored fact and the output still looks like one.

The experiments with real-world facts, a second model family and sink-free models extend the conclusions beyond the original setting, and the paper is better as a result. We thank the reviewer for the thoughtful questions that led to them! We hope we have sufficiently addressed all of the reviewer's concerns.

Reference. Qiu, Z., et al. (2025). Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. NeurIPS 2025.

I think the authors solved my concerns, so I decided to raise my score to 3, but still with low confidence, as I am sure I am not quite familiar with this domain.

Reviewer_vqKTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why language models produce wrong answers with high confidence, in two situations: knowledge conflicts (where the queried facts in context contradict the parametric knowledge), and missing knowledge (where the queried facts are never learnt). The authors fine-tuned a Qwen2.5-3B-Instruct on a synthetic dataset of random mappings between entity identifiers and five digit codes, varying in both how robust the knowledge is stored and where it is installed (LoRA adapters on MLP/QK/VO). The authors infer distinct roles of LM components with Jacobian decomposition and support with observed varying LoRA behaviors. They further argue that a robustly stored fact forms a "basin" in representation space that overrides the context, so that under conflict the answer starts on the stored code and then drifts into one matching neither source; and that after fine-tuning, the model answers unseen entities as confidently as learned ones, even though its hidden states reach no basins.

Critical strengths
  1. The synthetic dataset cleanly controls what the model has learned, and separates four scenarios (recall, reading, knowledge conflict, and missing knowledge). The template variety and LoRA adapters enable independent manipulation of knowledge storage robustness and location.

  2. The error mode under knowledge conflict, where the answer starts on the stored code and then drifts to one matching neither source, is interesting and may offer useful insights for knowledge injection and RAG.

Critical weaknesses
  1. Key experimental details are missing or unclear, including the prompts for reading, knowledge conflict, and missing-knowledge conditions, the definition of "stability" (used as an illustrative measurement in fig 1&3), and the layer used for the memorized states. This limits both evaluation and reproducibility.

  2. The central construct, the "basin", is not well defined, yet it is used to explain the error patterns under knowledge conflict and missing knowledge. I could not find an operational definition of its depth or radius in representation space, and it is never tested by intervention. Nevertheless, the authors use depth as an explanatory variable (e.g., ln.93: "a fact learned from many phrasings has a basin deep enough that the model can no longer read the context at all"), although depth is never measured independently and is therefore inferred from the outcome it is meant to explain.

  3. The analysis of the knowledge-conflict scenario is insufficient. The paper reports which source the answer follows and explains it through "arbitration", but includes no causal experiment or internal analysis (e.g., attention pattern differences, logitlens, patching). The claim that arbitration depends on how the fact is stored rests on two conditions (1 vs. 53 templates), without characterizing how behavior varies with training diversity. "Context deafness" is measured on entities with no learned fact, and the context is not introduced during the LoRA training process, so the 0% reading accuracy of the MLP and Full adapters under varied-template training may reflect suppression of an existing copying mechanism and/or deep basins introduced by LoRA adapters. The paper does not distinguish between them. The drift in the later digits is likewise reported but neither systematically characterized nor explained.

Questions for authors
  1. What are the prompts used in the reading and knowledge conflict scenarios, and how the context is introduced?
  2. What is the base model's performance on reading/knowledge conflict setting?
  3. Under robust storage, the model follows its weights at the first digit in 77% of cases. What do the remaining answers do at the first digit (follow the context, or match neither source)? Can the first-digit outcome (stored, context, or neither) be predicted or explained, e.g., by basin-related features?
  4. When the paper calls a basin shallow, deep, wide, or narrow (e.g., Sec.1 ln.91, Fig.3, ...), which quantity is measured in each case (margin or gap in representation space, relative distance, the output logit gap, or something else), and at which layer and position?
4 comments

We thank the reviewer for the detailed and constructive review.

Summary of the response. The reviewer raises two main concerns, that the terms of the paper are not defined precisely and that its claims about the basin and the knowledge conflict are not tested causally. The revision addresses both.

  • Definitions (W1, W2). We point to where each term is defined, and added a table of all terms (rev. App. D, Table 9).
  • Testing the basin (W2). We now also measure depth and radius by adding noise to the hidden state.
  • Testing the conflict (W3). The model ignores the context because the adapter suppresses copying from the prompt in the middle layers.

Together, these experiments substantially strengthen the thesis of the paper, that the way a fact is stored decides whether the model can still use its context. For the knowledge-injection and RAG setting the reviewer mentions, this implies that making an injected fact robust in the usual way also makes the model ignore retrieved text.

Below, "brittle" facts are those trained on a single phrasing and "robust" facts those trained on 53 (ln. 137–139). Locations marked "rev." refer to the revised manuscript.

W1: Experimental details.

"Key experimental details are missing or unclear, including the prompts [...], the definition of "stability" [...], and the layer used for the memorized states."

We clarify the three items below with their line numbers, and the exact prompts are now in rev. App. A (Table 2).

Prompts. Every evaluation asks Q: What is {entity}'s code?\nA: (ln. 136). For missing knowledge, the entity was never trained. For reading and conflict, we put one line that states a code before the question, for example Note: {entity} → {code} (ln. 118–129). In reading the entity was never trained, and in conflict it was trained with a different code.

Stability. By stability we mean the amount of noise that can be added before the model's answer changes. It is the third basin property that we listed in the definition of basin (ln. 241–243; App. E, ln. 1064–1065), and we measure it by adding Gaussian noise to the input embeddings (App. F, Fig. 11). Robustness to rephrasing is a separate property (convergence, ln. 214–215).

Layer. The memorized state of a fact is the final-layer hidden state at the position where the model is about to produce the first digit, averaged over three phrasings (ln. 204–205). We chose this layer and position as the model computes the first digit directly from this state. This choice is supported by two following observations:

  • Trained facts and never-trained entities are separated most clearly at the final layer (ln. 1076–1079).
  • When we replace this state with that of another trained fact, the model outputs the other fact's first digit in 98% of trials (new, rev. App. F).

W2: The basin.

"I could not find an operational definition of its depth or radius in representation space [...] depth is never measured independently and is therefore inferred from the outcome it is meant to explain."

The basin is defined by three measured properties (ln. 211–215, 241–243, all measured during plain recall with no context in the prompt). We now also measure depth and radius directly in the hidden state (new, rev. App. E).

Definition. A fact's basin is the region around its memorized state that hidden states are pulled into and stay in (ln. 211–212). Its three properties are contraction (the MLP update pulls nearby states together), convergence (different phrasings of a fact end close to its memorized state) and stability (recall survives added noise).

New test. During plain recall, we add random noise to the hidden state at one layer (20% or 40% of its size) and count how often the first digit of the answer changes.

NoiseStorageLayer 12Layer 20Layer 28
20%Robust0%1%8%
20%Brittle2%38%28%
40%Robust0%22%64%
40%Brittle50%88%66%

Result. Robust facts change their answer less often than brittle ones at every layer and noise level, which is what we mean by a deeper basin. The answer stays correct while the final-layer state is within about 50 of the memorized state, which gives the radius.

W3(a): Internal and causal analysis of the conflict.

"includes no causal experiment or internal analysis (e.g., attention pattern differences, logitlens, patching)"

The layer-wise analysis of the conflict is in App. D.4 (Fig. 10), and we have added the three additional analyses from reviewer's suggestions (new, rev. App. F).

  • Attention. While producing the first digit, the base model puts 23% of its layer-33 attention on the code in the context line, and the robust model only 5%.
  • Logit lens. For brittle facts, where the context wins, the context code only pulls ahead in layers 31–32 of 36, i.e. at the very end of the network.
  • Patching. See W3(c).

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W3(b): Training diversity.

"rests on two conditions (1 vs. 53 templates), without characterizing how behavior varies with training diversity"

Following the reviewer's suggestion, we retrained the MLP adapter with 5, 10 and 20 phrasings per fact. We find that the behavior changes between 5 and 10 phrasings (rev. App. F, Table 11).

Training phrasingsRecall on an unseen phrasingCopies the code from the contextBasin
10%90%≈0
50%85%−5
10100%24%+80
20100%2%+97
53100%0%+97
46, no distractor text100%0%+91

The basin column is how much farther never-trained entities end from the memorized states than recalled facts do (near 0 means no basin). Recall on unseen phrasings, the basin and the loss of copying all appear together. The last row is a control that leaves out the seven phrasings with an irrelevant sentence before the question, which could have taught the model to ignore whatever precedes the question. Copying is lost all the same.

W3(c): Suppression or basins.

"may reflect suppression of an existing copying mechanism and/or deep basins introduced by LoRA adapters. The paper does not distinguish between them."

Our new interventions show that it is a suppression of the copying mechanism, located in the middle layers (rev. App. F, Tables 12–14).

The robust model fails to copy a code from the context even for entities it was never trained on (0% of trials; ln. 306–308, Fig. 4a). As nothing is stored for these entities, a basin cannot be what pulls the answer away. We then further teased out where the suppression happens in the model.

  • Adapter in the middle layers only. With the adapter active in layers 13–24 only, copying falls to 2–10%, even though the model then recalls no stored fact. With the adapter removed from those layers only, copying returns to 92–100%.
  • Weakened adapter. At 70% of the adapter's strength, copying is already at 0% while only 8% of the stored facts can be recalled. Hence copying is lost before the facts are stored.
  • Activation patching. We run the same prompt through the base model and the robust model, and at one layer we replace the robust model's hidden states with the base model's. With the replacement at layer 20, the robust model copies the code in 93% of trials (20% at layer 16 and 77% at layer 24). Hence copying is lost by layer 20, while the later layers can still copy.

Training on varied phrasings therefore forms the basin and suppresses copying at the same time, but the suppression does not require the basin. This agrees with Sec. 4.1 (ln. 310–311), and we have reworded ln. 93 accordingly.

W3(d): Drift.

"The drift in the later digits is likewise reported but neither systematically characterized nor explained."

The drift is reported digit by digit in Fig. 4c. We have now extended this analysis to 200 trials and tested what causes it (rev. App. F).

In Fig. 4c, the context line gives the stored code with only its first digit changed. The model usually outputs the stored first digit but gets the later digits wrong more and more often, even though the two sources agree on them.

  • What happens. The answer matches the stored code in 80%, 66%, 48%, 32% and 21% of trials at digits 1 to 5. In 66 of the 200 trials the answer is the complete code of another trained entity, about 19 times chance. It thus falls back on codes from training.
  • Why. We forced the first digit to be correct, by copying in the hidden state from a run without the context line. The first digit is then correct in 98% of trials, but the full code in only 23%, compared with 22% without this correction and 100% when the context line is removed. Hence the later digits fail because the context line is still in the prompt, whether or not the first digit is correct.

Q1. One line placed before the question (see W1).

Q2. The base model was never trained on our entities, so a context line conflicts with nothing and it copies the code in 92–100% of trials. To give it a real conflict, we placed a wrong answer before 192 real-world questions that it answers correctly (rev. App. K). It kept its own answer in 73% of cases, as our robust adapters do.

Q3. In the 200 trials, the first digit is the stored one in 160, the context one in 7, and neither in 33. The outcome depends mainly on how far the context line moves the hidden state from the memorized state (AUROC 0.83), and much less on the model's confidence in the fact (0.58).

Q4. Wide and narrow refer to how much input-embedding noise recall survives (App. F, ln. 1088–1095), and deep and shallow to hidden-state noise (W2).

With the causal tests the reviewer suggested, the paper now shows where and why the model stops using its context. We thank the reviewer for suggesting them! We hope we have sufficiently addressed all of the reviewer's concerns.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

Thanks for addressing my major concerns about this paper. I raised the score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.