acceptodds
Under review as a conference paper at ICLR 2027

Quantifying the Link Between Localization and Context Interventions in Long-Context QA

Abstract

Search-augmented question answering, knowledge-base question answering, and multi-document analysis require language models to identify the right evidence among many documents and use it to produce an answer. Whether models can do this reliably determines whether longer context windows translate into stronger problem-solving capabilities. Existing methods use localization or relevance sig- nals to prioritize context and improve its utilization through attention calibration, document reordering, compression, or pruning. Although prior work reports both localization quality and downstream QA performance, it does not explicitly quan- tify how localization successes and failures jointly determine intervention out- comes. We present a systematic analysis linking localization accuracy to down- stream QA outcomes across multiple matched context interventions. In the actual locate-then-intervene workflow, we group records by whether the localizer se- lects the answer document, measuring the average gain after a hit and the average loss after a miss. A separate gold-document control applies the same interven- tion directly to the answer document to measure the intervention’s improvement potential. We further show that document-localization signals concentrate in a small number of attention heads, and select these heads using questions disjoint from evaluation to connect localization quality to intervention outcomes. Experi- ments across multiple models and datasets show that localization and intervention based on a small number of attention heads improve answer-hit accuracy by 4.46 percentage points on average and yield positive effects in the large majority of configurations. Gold-document controls further show substantial remaining im- provement potential across the tested interventions. On unseen models, using only a small labeled set, our method predicts whether an intervention will help or hurt with 78.1% accuracy, compared with 58.8% for the better constant policy, without using held-out full-QA outcomes at prediction time. These results turn localiza- tion accuracy from an isolated intermediate metric into a basis for explaining and predicting the effectiveness of context interventions, providing a unified view of how such methods can be analyzed and transferred.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model Interpretability, Long-Context Language Models, Multi-Document Question Answering, Context Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ftzxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how document-localization accuracy affects context interventions in long-context QA. It separately analyzes the gains after correct localization and the losses after incorrect localization. A gold-document control is used to measure the potential improvement. The authors use a small set of attention heads to select documents for attention biasing, document reordering, or pruning. They also use localization accuracy to predict whether an intervention will help or hurt on an unseen model. Experiments report an average improvement of 4.46 percentage points in answer-hit accuracy and 78.1% accuracy in predicting the direction of intervention effects.

Critical strengths

Useful analysis of localization and intervention effectiveness. The paper separates the gains after correct localization from the losses after incorrect localization and includes a gold-document control. This helps distinguish poor document selection from limited intervention effectiveness and provides a useful way to understand why improving localization accuracy does not always improve QA performance.

Critical weaknesses

Unclear additional value of the gate. Section 5.6 notes that the gate mainly helps by rejecting unreliable localizer families. For attention-head configurations, gating increases the mean QA gain from 4.46 to 4.62 percentage points, but the confidence interval for this difference includes zero. However, the paper does not compare the gate with a simple family-based rule, such as using each localizer family’s average QA gain on other models to decide whether to enable its interventions on a new model. Therefore, it is unclear whether collecting localization labels on a new model provides useful information beyond the localizer family itself.

Questions for authors
  1. Appendix E states that some wrong-target outcomes are imputed, and the reported simulator sign fidelity is 0.807. Could the authors explain how these outcomes were imputed and how the simulator was validated?
  2. Figure 1 is confusing and difficult to follow. The text is small, and too many boxes, arrows, equations, and numerical results are packed into a single figure. This makes it difficult for readers to identify the main message. I suggest that the authors simplify the layout and increase the font size to make the workflow clearer.
  3. The shaded region labeled “break-even” in Figure 2 is unclear. What does it represent, and how are its upper and lower bounds determined? Please explain this in the caption.
Reviewer_sWYZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper systematically quantifies how document localization accuracy determines the success of context interventions in long-context QA. Discovering that localization signals concentrate in a few attention heads, the authors propose a locate-then-intervene strategy that improves QA accuracy by 4.46%. Impressively, this approach predicts intervention effectiveness on unseen models with 78.1% accuracy, providing a unified predictive framework for long-context optimization.

Critical strengths
  1. The paper innovatively decomposes the overall effect of context intervention into gains from accurate localization and losses incurred by mislocalization, while formulating a concrete break-even accuracy metric.

  2. The study demonstrates that target document localization signals in large language models are not uniformly distributed, but are instead heavily concentrated within a sparse subset of attention heads.

Critical weaknesses
  1. Both the gain-loss decomposition and the gold-document contrastive design rely on the strong assumption that a query can be resolved using only a single target document. Consequently, this approach is confined to simple single-hop retrieval-augmented QA and fails to address more fundamental long-context challenges, such as cross-document information synthesis and multi-hop reasoning.

  2. The QA evaluation relies entirely on substring match rates. However, context interventions can substantially alter the model's output length and stylistic tendencies. Without semantic equivalence metrics or LLM-as-a-judge evaluations, the observed gains may simply be a statistical artifact resulting from shorter, more direct responses, rather than reflecting genuine improvements in comprehension.

  3. The main experiments only test relatively modest context settings with 20 to 30 candidate documents, leaving the core threshold prediction mechanism prone to failure when scaled to 120 documents. Furthermore, the paper lacks systematic comparative evaluations on standard 100k+ token long-context benchmarks such as full LongBench or L-Eval, which weakens its persuasiveness within the long-context research community.

Questions for authors

see Critical Weaknesses

1 comment

Thank you for the response, but the main concerns remain. The rebuttal attributes the MuSiQue result to a change in the target definition, where a hit is defined over a sufficient evidence set or reasoning chain, yet no results are reported under this definition. The position-level analysis of the 120-document stress test explains why the frozen threshold rejects positive configurations, but does not resolve the issue. It remains unclear whether a threshold calibrated at 20 to 30 documents can transfer to longer contexts.

Reviewer_DMCAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the accuracy of document localisation determines whether a downstream context intervention (i.e. attention biasing, reordering the selected document to the front, or pruning the context around it) helps or hurts long-context QA.

For each configuration, records are split by whether the localiser selects the answer document, and the overall QA effect is expressed as the average gain after a hit, weighted by localisation accuracy, minus the average loss after a miss, weighted by the miss rate. This gives a post-hoc break-even accuracy approximating the localisation accuracy at which the intervention's expected effect changes sign. The authors also select a few attention heads as a localiser and propose a localisation-only predictor of whether an intervention will help on an unseen model.

Experiments cover eight models (Qwen3, Llama-3/3.1 and Vicuna), four datasets (i.e. NaturalQuestions, PopQA, a hard-distractor PopQA variant and MuSiQue). Sparse-head and sparse-head/self-report ensemble interventions improve answer-hit accuracy by 4.46 pp on average (138 of 163 configurations positive), and the localisation-only predictor reaches 78.1% directional accuracy against 58.8% for the better constant policy.

Critical strengths
  • The sparse-head localisation results are solid: heads are selected on disjoint question folds, and 1-8 heads outperform late-layer all-head averaging on all seven models (Figure 2). I also appreciated that the authors report where their gate fails (Sections 5.6-5.7 and Appendix F) rather than only the favourable aggregate.
  • The matched three-condition design (i.e. baseline, localiser-selected and gold-document control) is clean. Since the last two share the same code path and give identical generations whenever the localiser hits (as specified in Appendix), their gap isolates target-selection error from intervention capacity, which explains why similar localisation accuracy can lead to opposite outcomes across operations.
Critical weaknesses
  • The held-out predictor mainly filters out unreliable localiser families. As the authors acknowledge (on lines 453-460), the gate's advantage comes primarily from rejecting FITM and raw-attention configurations. Within the attention-head family, gating moves the mean gain from 4.46 to 4.62 pp.
    • Removing FITM and keeping only NQ shrinks the directional-accuracy advantage to 0.8 pp ([−21.2,+6.1][−21.2, +6.1]). The headline 78.1% vs 58.8% therefore depends largely on the mix of families in the pool.
    • The threshold also transfers poorly across context lengths: calibrated at 20-30 documents, it rejects three positive 120-document configurations.
  • The "quantitative link" feels definitional. Eq. 6 is an exact identity obtained by conditioning on the hit indicator, and pc⋆p_c^\star needs full QA outcomes, so it serves only as a post-hoc diagnostic.
    • Appendix F gives case-by-case reasons why gcg_c and lcl_c change, such as hard distractors, pruning and dose, but the paper does not model or predict them systematically; the predictive part is the threshold gate discussed above.
  • Limited scale. Each setting uses 100 questions, with the answer document moved to five fixed positions among 20-30 candidates.
    • The 388 configurations share models, questions and candidate sets, yet the 4.46 pp average and the "138 of 163" count have no intervals
    • Some analyses are also reported without significance claims (on lines 300-302).
    • The ground-truth direction used for directional accuracy is the sign of a point estimate on 100 questions, with ties counted as non-positive (on lines 252-257). For families with near-zero effectsm part of the 78.1% accuracy may reflect agreement with noise.
Questions for authors
  • Are MuSiQue configurations pooled into the 388 configurations behind Table 1 and the 4.46 pp average? The authors note in Appendix F that multi-hop use requires an evidence set rather than a single answer-document target. It would help to report the main results with MuSiQue excluded.
  • Could you report top-1 localisation accuracy for every localiser (BM25, cross-encoder, self-report, FITM, raw attention and sparse heads) under the same models and settings? Figure 2 only compares sparse heads with all-head averaging and the position prior.
  • Could you report directional accuracy and captured gain per localiser family under the leave-one-model-out protocol, in addition to the mean QA changes in Table 4?
  • Where does the fixed threshold τ=0.454\tau = 0.454 used in the sensitivity analysis (on lines 438 and 717) come from? If it was fitted on all models, it would explain why it yields a higher directional accuracy (0.796 at b=20b = 20) than the leave-one-model-out estimate (0.781).
  • The directional accuracy of the "older" 20-record protocol is 0.765 in the text (on lines 442 and 722) but 0.764 in Table 2, and the deferral analysis starts from 0.773 (on line 729). I would suggest that the authors revisit these and explain what "older" refers to.
  • The paper's header reads "Under review as a conference paper at ICLR 2026".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.