acceptodds
Under review as a conference paper at ICLR 2027

A Mechanistic Analysis of In-Context Learning and Out-of-Context Hallucination

Abstract

Large language models can infer and follow specific rules through in-context learning (ICL), yet may fail to remain faithful to provided evidence, giving rise to out-of-context (OOC) hallucination. Existing work has largely investigated the underlying mechanisms of ICL and OOC hallucination along separate lines. However, this separation obscures the mechanistic basis for these seemingly opposing outcomes. In this work, we study this contrast through the unified view of functional roles in dynamic context. The key observation is that role information can be internally represented without reliably constraining output. We derive a first-order decomposition to characterize the role effect and identify two necessary conditions for role span contributions to become consequential for generation. Causal analysis further shows that role effects act through key token choices. These findings support a common mechanistic view of ICL and OOC hallucination: each functional role defines the pattern of output decisions over which role span contributions must remain effective, and differences in this organization account for the seemingly opposing outcomes.

Primary area:
interpretability and explainable AI
Keywords:
large language models, mechanistic interpretability, in-context learning, hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aqTjOpenReview ↗
Rating
1
Confidence
1(−2)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies out-of-context hallucinations, where the LM response do not align with the evidence provided in the context. The authors study how the functional role of different token-spans in the prompt influences this behavior. They also contrast this with in-context learning setups, where the LM does reliably follow the prompt accordingly.

Critical strengths

Not assessed yet. See the Critical Weaknesses and Questions For Authors sections

Critical weaknesses

The authors clam in the Abstract and Intro (even in Title) that they are studying out-of-context hallucinations in LMs. In their actual operationalization for most of the experiments, they force in the LM response as Cf⊕x⊕yC_f \oplus x \oplus y, where CfC_f is the context, xx is the query. They call yy the "response", but it is not LLM generated, instead it is forced in the input. So, there is really no way to study know whether the LM's response would have been faithful to the context or not. The authors are not studying hallucinations at all, but rather they are studying how the LM's representation of a forced input changes with different contexts.

Questions for authors

Please see what I flagged in the Critical Weaknesses section above. If my understanding of the operationalization is correct, then it is not even worth it to go over the other technical details of this work, unless the framing is changed significantly. If I find the authors' response to this issue satisfactory, I will be happy to re-read the paper and revise my assessment accordingly.

And, in case there is a fundamental disagreement between the authors and me on how causal analysis on autoregressive LLMs should be conducted, then I will leave it to the discretion of the AC and other reviewers to decide whether this paper is suitable for ICLR or not.

Thanks!

Reviewer_ZPbvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why LLMs can reliably follow in-context rules (ICL) but often fail to stay faithful to provided facts (OOC hallucination). The authors analyze both tasks under a unified framework by treating rules and evidence as different "functional roles" in the context. Using matched counterfactual prompts, they first show that both roles are well-represented in the residual stream at the end of the prompt. Next, they isolate the direct attention output from the role span (Bl,ifB_{l,i}^f) during decoding and use a first-order Taylor expansion to analyze how it shifts output scores. Finally, through causal ablation and amplification experiments, they show that rules require continuous support across many structural decisions, whereas evidence mainly acts on a few local factual token decisions.

Critical strengths

Comparing ICL and OOC hallucination under the same autoregressive decoding setup is a great idea. The distinction between "having the representation in the prompt" and "actually using it during generation" is very clear.

The paper tests three models (Qwen2.5-7B, Mistral-7B, Llama-3.1-8B) and includes solid extensions in the appendix, such as few-shot rule induction (SST-2, TREC) and multi-hop or conflicting evidence (MuSiQue).

Critical weaknesses

Proposition 1 simply states that cross-position information in a Transformer must first travel through Attention because LayerNorm and MLP are position-wise. This is a basic architectural property and does not need a formal proof. Similarly, Theorem 1 is essentially standard first-order Attribution Patching (Gradient ×\times Activation difference) applied to attention outputs from a specific token span. The remainder bound in Eq. (5) relies on an unknown Lipschitz constant LtL_t and an unmodeled error term Et⊥E_t^\perp, so the method still relies on empirical correlation (Table 5) to show it works.

The main text claims that evidence spans mainly act at the specific token where the factual value is generated. However, Table 12(b) shows that when the evidence pathway is removed, 84%–87% of the first token changes happen 6 to 8 tokens before the factual decision (whereas rule changes happen at offsets of 00 to −5-5). This suggests the model plans the sentence structure several tokens ahead using indirect pathways (ΔIl,i\Delta I_{l,i}), which the main paper largely ignores.

In Table 4, rule failures have a huge margin deficit (−14.6-14.6 to −19.0-19.0) because formatting rules (like JSON) change the entire syntax of the sentence, while evidence failures (−2.2-2.2 to −4.0-4.0) are simple word replacements. It is unclear if this failure to recover is truly due to "rules vs. evidence" or just "global syntax vs. single-token choice."

Questions for authors

In Section 5.3 (Tables 3 and 4), do you scale the single-input term Bl,if(P)B_{l,i}^f(P) or the counterfactual difference ΔBl,if\Delta B_{l,i}^f? Please clarify the exact intervention formula.In Table 12(b), why does removing the evidence pathway cause generation to diverge 6–8 tokens before the factual decision? Doesn't this mean indirect attention transport (ΔIl,i\Delta I_{l,i}) across generated tokens is very important for evidence grounding?If you run the amplification recovery experiment (Table 4) on failed examples from the few-shot ICL tasks (Appendix D.4, SST-2/TREC)—where the rule also decides a single label token—can the failed rule decisions be recovered?

and, I dont kown why in sec 1 having "Statement of Contribution." this sentences

It is the novel ways to show contributions?

Reviewer_f79AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors attempt to quantify the distinction between in-context learning and out-of-context hallucination, a disparity whereby LLMs may be able to follow and induces rules provided in-context but simultaneous hallucinate information or contradict provided statements. Authors investigate this using matched counterfactual prompts

Critical strengths

The key tension between felicitous ICL generalization and contradictory OOC hallucinations is a rather interesting one, and the analysis of treating both as the same underlying phenomenon parameterized by some kind of different role is prima facie appealing and, I think, an productive way of conceptualizing LLM behavior.

Critical weaknesses
  1. The presentation of main results is hard to grok. To quote from the intro & abstract:

Second, we derive a first-order decomposition and identify two necessary conditions for role span contributions to exert a causal influence on generation. Third, we show that the distinct outcomes of the two settings can be explained by different decision-level realizations of a shared role span contribution mechanism.

It's rather unclear to me what this means. What are the two necessary conditions? What are the different decision-level realizations? Even inside the main part of the paper, the claims are extremely vague:

Together, these findings support the insight that role span shapes generation through key token choices.

What does this mean? What choices are made? I trust the authors believe they have answers to this, but it's quite difficult to see what they are intending at a glance. Relatedly, I think the presentation of the experimental setup is overly technical to the detriment of understanding. Cf⊕x⊕yC_f \oplus x \oplus y is really unneccesary notation for what I think is being presented here; futhermore, the notation can be misleading: if I understand correctly, the main setup is to compare C~f\tilde{C}_f to CfC_f, where the difference is in the nature of the functional role...but ff is the thing changing here, I think; All of this conspires to make the paper less readable than it ought to be, and that makes analysis or conveyance of the core claims quite difficult. It would be really helpful even to have an table of example Cf,C~f,x,yC_f, \tilde{C}_f, x, y values to make the setup clearer.

  1. Where does yy come from? It seems like yy is chosen prior to any intervention, which suggests that authors are pre-supplying an answer (eg, some kind of teacher-forcing); this strikes me as informative for OOCH only if the authors can demonstrate that their yy are representative of likely hallucinations, rather than forced incoherent interventions, since if yy are OOD wrt to model generations it won't tell us anything about model behavior.
Questions for authors

Aside from the more technical points above, I'd want to understand how authors see their work impacting the field. Suppose this unified framework for analyzing ICL and OOC hallucinations is sound: what then? What are the main takeaways for scientists or practitioners regarding LLM behavior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.