acceptodds
Under review as a conference paper at ICLR 2027

A Narrow Residual Bottleneck from Evidence to Answer in Language Models

38 views

Abstract

When a language model answers a question about a passage, the answer hinges on a small span of the input. We ask where inside the model that information is forced to pass, and find a narrow residual bottleneck: a small set of evidence-token positions at a single residual-stream layer that carries the evidence-to-answer computation on its own. We locate it with two hidden-state interventions on a single candidate set. One copies the original evidence into a corrupted run and asks whether the correct answer returns (sufficiency); the other injects the corrupted state into a clean run and asks whether the answer breaks (necessity). Requiring the same set to pass both is strictly stronger than either direction alone. Against a size-matched permutation null, the existence of a bottleneck becomes falsifiable: a bottleneck must pass the two-sided test far more often than random sets of its own size, and it does. On the controlled benchmark, in our best-powered models the cut passes about 0.98 of the time against about 0.01 for the matched control, and is narrow (typically two to three positions, one to two on real text) throughout. It reappears across three normalization schemes on real single-hop, multi-hop, arithmetic, and long-context question answering, and at a fixed budget steers the answer far more reliably than editing a random set. Most tellingly, the bottleneck is not simply the top of an importance ranking: fixing the set size, the test, and the model, and changing only how the set is chosen, the same-size set of highest-importance attention heads fails the very test the located cut passes, a gap that holds across four model families.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Mechanistic Interpretability, Narrow Residual Bottleneck

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Us2EOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes an interventionist approach to identifying minimal sets of computation components, at the token-layer-level, which are key to an LLM’s answer output. These sets are shown to be effective at a smaller size than importance-based methods, and the methodology is shown to be general to many types of question-answering tasks.

Critical strengths
  1. The paper is exceptionally well-written and clear
  2. The experimental procedure is rigorous and reproducible
  3. The argumentation and analysis is convincing
  4. The claims made by the paper are careful and exact
Critical weaknesses
  1. Impact: results presented show that the importance baseline can pass the dual test for some (larger) position allowance. If I care only that I find some passing set and not that it is minimal (i.e. I care only about recall), then the presented method is not preferable to the baseline
  2. While in the strengths I said that the experimental procedure is rigorous (which it largely is), the importance baseline used was a simplistic one. The work’s own related-works section outlined a variety of methodologies, but only DLA was used as comparison.
Questions for authors
  1. “For real text we enlarge the null pool to include uncontaminated tokens before the answer span, since answer-span positions alone sit so close to the cut that random draws would nearly reproduce it.” I’m not sure I understood this. Real text as opposed to what? What is an answer span? If the cut is 2-3 token-positions, and the input text is at least a couple sentences, how could random 3-token subsets of the input text reproduce the chosen cuts?
  2. Is there any evidence to indicate that the representation at a given token position in a middle layer is actually associated with that position in the input? That is to say, if at layer l, if position s is found to be in the cut then does this mean that the information at position s of the input is a key piece of evidence? Since the representation at a given token position is a function of the previous representations at all token positions, I don’t think this is the case. While this may still allow for claims e.g. “the model concentrates information into a representation bottleneck”, it may not allow for explainability claims such as “we can identify which pieces of the input are key to the model’s answer”. Did you find that identified cuts overlapped with key pieces of information in your synthetic data experiments?
Reviewer_kTt3OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies small sets of residual-stream positions that both restore answers in corrupted runs and disrupt answers in clean runs. It evaluates these sets across model families and QA tasks, comparing them with random and importance-ranked alternatives.

Critical strengths
  1. Testing the same set in both patching directions provides a useful diagnostic beyond either direction alone.
  2. The experiments cover multiple architectures, normalization schemes, and task types.
  3. The depth and cross-corruption analyses provide useful robustness evidence.
Critical weaknesses
  1. The bottleneck interpretation exceeds what the test establishes. The dual test in Section 3.3 demonstrates control over the answer for a clean–corrupted pair, but does not rule out parallel evidence pathways or establish that all relevant information must pass through the identified set.
  2. The real-data random controls provide limited evidence. Table 5 draws control-only sets from pre-span tokens, whose states may be unchanged by the corruption under causal attention. Their zero-pass rate would then be expected, rather than evidence of a distinctive bottleneck; adding such positions also makes the enlarged random baseline easier to outperform.
  3. The head-level comparison is narrower than the conclusion. Section 4.3 compares against non-causal direct logit attribution, while the proposed search directly optimizes the patching criterion. This establishes a distinction from DLA rankings, but not from causal importance rankings or iterative circuit-discovery methods.
Questions for authors
  1. What evidence distinguishes a required information pathway from a site whose intervention overrides information available elsewhere?
  2. Do the pre-span controls in Table 5 have nonzero clean–corrupted activation differences?
  3. Does the head-level advantage persist against a causal importance baseline?
1 comment
Reviewer_kTt3OpenReview ↗

Thank you for your reply.

I thank the authors for a detailed and partly self-correcting response. As the authors have accepted and corrected W2, I will increase my score. However, I still have some concerns about W3, as the head-level comparison rests on DLA alone, and the causal baselines are position-level only.

Reviewer_MeLNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates input context attribution at the residual layers and checks for the presence of a narrow residual bottleneck. This bottleneck is defined as a small set of evidence-token positions at a single residual-stream layer that carries the evidence-to-answer computation on its own. The core idea involves a dual causal intervention framework that tests and selects tokens based on a necessity and sufficiency objective. The test for necessity injects the corrupted state into a clean run and asks whether the answer breaks. The test for sufficiency injects the original evidence into a corrupted run (that replaces the answer with a plausible candidate) into the residual stream and asks whether the answer reverts back to the original.The authors find that the resulting set of tokens that passes these conditions is better than a null-matched control (set of tokens selected at random) The experiments are replicated on real single-hop, multi-hop, arithmetic, and long-context question answering.

Critical strengths

S1. Novel and clear framing: The dual-nature framing of the intervention tests is interesting and is described with clarity.

S2. Comprehensive Coverage: The experiments cover seven different models and various datasets.

Critical weaknesses

W1. Unclear motivation. It is not clear as to what underlying research gap is being addressed by the method. There is no comparison with prior methods that do context-level attribution and what their limitations are. The paper states that knowing which components are important is not the same as knowing that the answer's computation runs through it. However, it is not evident why that difference matters.

W2. Lack of stronger controls and biased evaluation. The controls presented in the paper include (i) the size-matched permutation null, (ii) top-importance sets at matched edit budget, and (iii) random-head controls. The proposed method is not compared against any other causally-identified sets. Moreover, the evaluation involves assessing performance on the dual-intervention test, which is unfair because that was exactly what was used to create the proposed cut in the first place.

Questions for authors

Q1. Can the authors show examples of text instances where a dual cut was identifiable versus not?

Q2. When comparing with the most important heads, how is the importance determined? How is it relevant to the identification of the evidence tokens?

Q3. How do the evidence tokens differ from each other as the patching moves across layers? Can you measure the overlap in identified tokens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.