acceptodds
Withdrawn from ICLR 2027

Differential Attention for Context Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models with long context windows can answer questions over full-length documents, but using the entire document is costly, slow, and not always more accurate. This paper studies long-context evidence selection: given a long document, a query, and a strict token budget, the goal is to expose only the most useful evidence to a downstream generator. Standard retrieval-augmented generation (RAG) methods retrieve semantically similar chunks using strong embedding models, but they can miss evidence that is sparse, structurally distributed, or weakly matched to the query. Prompt-compression methods, such as LongLLMLingua and CPC, reduce context in another way, but often do not explicitly model where query-specific evidence appears. We propose Selective Attention-Guided Extraction (SAGE), a training-free, plug-and-play context reduction framework. SAGE uses a lightweight frozen LLM during prefilling, without decoding or fine-tuning, to turn query-to-document attention into a document-level relevance heatmap. It then applies differential attention with a contrast query to suppress query-irrelevant background signals and isolate question-specific evidence. Finally, SAGE selects coherent high-scoring spans under the token budget and passes only the reduced context to a downstream generation LLM. Across QuALITY-hard, NarrativeQA, QASPER, and AIT-QA, SAGE improves the accuracy–token trade-off over strong embedding-based RAG baselines using Qwen3-Embedding and ColBERT, as well as prompt-compression baselines such as LongLLMLingua and CPC. On QuALITY-hard, SAGE achieves a top-4 leaderboard result using only a 30% context budget, showing that carefully selected attention-guided evidence can outperform much larger retrieved or compressed contexts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Differential Attention, Long-Context LLMs, Context Selection, Evidence Selection, Prompt Compression, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2nTROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SAGE uses prefill attention from a frozen selector model to construct query-dependent relevance heatmaps, subtracts attention from a contrast query, and selects coherent spans under a token budget before generation. The paper evaluates several selector sizes, budgets, document types, and retrieval or prompt-compression baselines.

Critical strengths

The proposed pipeline is simple to inspect and does not require training a compressor. The budget sweep, raw-versus-differential attention comparison, selector-size study, and table-row experiment provide useful empirical probes. Reducing context while retaining performance close to full-context generation is an important practical goal.

Critical weaknesses

The main comparisons do not provide sufficiently controlled evidence for the headline claims. The selector, generator, and evaluator are different models, including a proprietary Gemini generator and GPT-4o-mini as the correctness judge. The paper does not report confidence intervals or repeated-run variability for the main tables, so it is difficult to know whether the small QASPER gain over CPC (0.4 points) is meaningful. The FINCH comparison is explicitly made with a different generator and different budget definitions, yet it is still used in the surrounding argument about superiority.

The contrast-query setting also changes the available information: the farthest-query variant requires another question from the same document, which is not generally available at inference time. The fixed contrast prompt is not justified by a sensitivity study. Finally, the “top-4” leaderboard statement is not independently documented in the paper, and the evaluation relies heavily on LLM judging rather than task-native metrics or human verification. These are validity problems in the central evaluation, not merely missing implementation details.

The related-work coverage is incomplete. The paper should position SAGE against the following directly relevant works:

  • COMI: Coarse-to-Fine Context Compression via Marginal Information Gain — ICLR 2026
  • GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment — ACL 2026
  • Adapting Language Models to Compress Contexts — EMNLP 2023
  • Selective Context: Efficient Contextualization for Large Language Models — EMNLP 2023
  • Long-Context LLMs Meet RAG: Overcoming Challenges for Long Context Understanding — ACL 2024

This would clarify how differential-attention span selection differs from latent compression, sentence-level compression, and long-context retrieval-compression approaches.

Questions for authors

Please provide task-native evaluation and statistical uncertainty for every main result, and repeat the comparisons with one open generator and exactly matched retained tokens. How sensitive are the results to the wording of the fixed contrast query and to the availability of a same-document contrast question? Please document the leaderboard protocol and the exact source of the claimed ranking.

Reviewer_1UnZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAGE (Selective Attention-Guided Extraction), a training-free framework for budgeted context selection in long-context question answering. SAGE uses a frozen lightweight language model to extract query-to-context attention scores during prefilling, constructs a relevance heatmap, and applies differential attention by subtracting the attention induced by a contrast query. The resulting scores are used to select contiguous text spans under a predefined token budget, which are subsequently passed to a separate generation model.

The authors evaluate SAGE on QuALITY-hard, QASPER, a NarrativeQA-derived benchmark, and several document and table QA datasets. The results show improvements over embedding-based retrieval and prompt-compression baselines, particularly on QuALITY-hard, where SAGE achieves 63.1% accuracy with approximately 30% of the original context.

Critical strengths
  1. The research question is practical. Efficient context selection is important for reducing the inference costs of long-context LLM applications.

  2. The empirical results are promising.

  3. The component analyses are useful.

Critical weaknesses
  1. The framework's novelty is limited. Using attention weights to identify and retain important context tokens has been extensively explored in previous research. In particular, AttentionRAG [1], which the authors explicitly discuss and compare against, already uses attention-guided context pruning for retrieval-augmented generation. Attn-GS [2] also leverages LLM attention to select relevant contextual information.

  2. The experimental benchmark selection is insufficiently convincing for the broader claims about long-context evidence selection. For instance, a substantial portion of the additional analyses relies on the Paper and Notice datasets from ZenDB. The selected dataset has limited representativeness. Many of the questions in these datasets concern highly structured metadata, such as author names, publication information, notice dates, and addresses. Such questions often have strong positional or structural regularities, which may make the task considerably easier than general long-context reasoning or evidence retrieval.

  3. The baselines are generally weak, and the paper should provide comparison to stronger retrieval and selection approaches. In particular, the main retrieval baselines rely primarily on embedding similarity and fixed-size chunk retrieval. Modern retrieval pipelines can incorporate reranking, hybrid retrieval, and finer-grained evidence selection. It is unclear whether SAGE would retain its advantages over an embedding-based retriever followed by a strong cross-encoder or LLM-based reranker, especially when operating under comparable context and computational budgets.

Additionally, the retrieval baselines use approximately 200-token chunks with overlap, while SAGE is allowed to select more flexible contiguous spans. Consequently, part of the improvement may come from selection granularity rather than the proposed attention-based scoring mechanism.

Questions for authors
  1. What is the fundamental methodological contribution of SAGE beyond existing attention-guided selection methods such as AttentionRAG and contrastive attention scoring?

  2. How does SAGE compare against strong reranking-based retrieval pipelines when both methods use the same candidate-window granularity and context budget?

Reviewer_V4LVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SAGE, a training-free context reduction method for long-document QA. A frozen LLM scores document tokens by summing query to context attention over all layers and heads, computed chunk by chunk. The heatmap is denoised by subtracting the attention of a contrast query (a fixed prompt or the most distant question on the same document). Fixed-size windows are then greedily selected under a token budget and passed to a separate LLM generator.

Critical strengths

Simple, training-free and easy to reproduce.

The selected evidence is original text and remains inspectable.

The contrast subtraction gives a large and consistent gain over raw attention (Table 4), and the normalization ablation is clean (Table 13).

The cross-model ranking-consistency analysis (App. B.2) is a useful sanity check.

Critical weaknesses
  1. Limited novelty and missing closest prior work. Using query to context attention of an LLM as a training-free relevance signal, and calibrating it by subtracting attention from a content-free query, was proposed in ICR [1]. Other relevant works: QUITO/QUITO-X [2][3], Speculative Prefill (ICML 2025), GemFilter (the full references are provided below). None of these are cited or compared.

  2. The paper claims general long-context evidence selection, but evaluates only single-document, mostly single-hop QA on relatively short documents. The method's design may struggle in the settings that matter most:

2.1 Multi-hop datasets (HotpotQA, 2WikiMQA and MuSiQue) are absent.

2.2 Temporal/ordered reasoning and very long inputs such as BABILong [7].

  1. Weak and incomplete baselines. RAG baselines are single-step dense retrievers, with no cross-encoder reranker. There are no multi-step RL baselines (e.g., Q-RAG [6]).

References

[1] ICR - Chen et al., Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers, ICLR 2025. https://arxiv.org/abs/2410.02642

[2] QUITO - Wang et al., 2024. https://arxiv.org/abs/2408.00274

[3] QUITO-X - https://arxiv.org/abs/2408.10497

[4] Speculative Prefill - Liu et al., ICML 2025. https://arxiv.org/abs/2502.02789

[5] GemFilter - Shi et al. https://arxiv.org/abs/2409.17422

[6] Q-RAG - Sorokin et al., ICLR 2026. https://arxiv.org/abs/2511.07328

[7] BABILong - Kuratov et al., NeurIPS 2024. https://arxiv.org/abs/2406.10149

Questions for authors
  1. How does SAGE compare with ICR's calibration, QUITO and SpecPrefill at matched budgets?

  2. Can SAGE be made iterative (append selected context to the query, re-score using the cached KV, with the previous step as contrast query)? How does it perform on HotpotQA, MuSiQue and BABILong (128K+) against Q-RAG?

  3. How does performance change with an absolute token budget instead of a fractional one, and what is the selector cost at 128K+ tokens?

  4. Please add a cross-encoder reranker baseline over the same chunks (dense top-k over the same chunks, reranked with a cross-encoder, filling the same token budget). The natural choice is Qwen3-Reranker (0.6B/8B).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.