acceptodds
Under review as a conference paper at ICLR 2027

Language Models Can Control Their Own Attention

Abstract

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach pre-selects relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? We introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (45.8%, 19.6%) with modest accuracy drops (1.40pp, 2.10pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods. Anonymized code is available at https://anonymous.4open.science/r/declarative-attention-4D7F/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long Context, Efficient Inference, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FPZDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Declarative Attention (DA) which is a prompt protocol that allows a vanilla LLM to generate declarations of global, focus and local scopes while generating. The runtime can then use these declarations to mask out the KV blocks being consumed by the global attention layers and leave the rest of the KV cache intact for future reference. Using a curated set of 15 long context tasks they were able to reduce the number of tokens attended to by 45.8% on Gemma-4-31B and 19.6% on Qwen-3.6-27B with a drop in mean accuracy of 1.40 and 2.10 percentage points respectively. They also show results for our unmasked baseline as well as some scaling results. Timing of decode per step and roofline estimate for full response decode are also provided.

Critical strengths
  1. Good evidence for zero training transfer. It would be nice to see that existing models can learn to follow an attention control protocol on various long context tasks. This is helpful to know empirically. Also the DA no mask ablation helps show that it's due to the mask and not from generating less tokens.

  2. They provide a concrete method to implement this with vLLM. And include time in their ablation when comparing the reversible mask. They also report many failures which I think is good to see. As well as validating their judge with 98.5% agreement rate and kappa score of .94.

Critical weaknesses
  • The methodological advance over existing span selection is limited, and its comparative value is not established. SSAS [1] already has the model output span references that determine hard attention masks while retaining skipped context. DA’s main advance is zero-training elicitation at larger scale and broader coverage. However, the experiments do not establish what the three-mode protocol adds over a simpler span-reference or one-shot selection protocol on the same backbone. Comparisons against vanilla and DA-no-mask establish savings relative to full attention, but not a better accuracy-cost trade-off than existing sparse-attention methods such as TidalDecode [2]. Complementarity does not remove the need to establish comparative benefit.

  • The evaluation does not establish broadly useful full-response efficiency. Appendix F.3 explicitly selects the headline suite based on DA preserving accuracy and reducing attended tokens. The excluded sources include accuracy losses of 29.7pp on cwe and 21.0pp on structured_data, and five of six increase total attended tokens. This limits the generality of the headline trade-off. Moreover, the measured speedups are per-step results with responses truncated and lengths matched; the full-response comparison is a roofline estimate. DA generates 38% and 32% more steps on the main models, and its DSA composition increases total indexer-scanned tokens to 128% of the baseline. The current evidence therefore does not establish a net serving advantage over complete responses or competitive baselines.

[1] Tian Jin etal. Self-Selected Attention Span for Accelerating Large Language Model Inference.

[2] Lijie Yang etal. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025

Questions for authors
  • On the same backbone and task set, how does DA compare with a simpler SSAS-style span-reference prompt or one-shot chunk selection, and with a sparse-attention baseline at comparable accuracy? These comparisons would determine whether the contribution extends beyond zero-training transfer of the existing mechanism.

  • What were the source-selection criteria, and were they fixed before observing DA’s results? Can you report matched-run aggregate results over all evaluated sources, including the six in Appendix F.3, for both main models?

  • What are the measured complete-response decode latency and throughput under a fixed continuous-batching workload with natural termination, charging all extra steps and runtime overhead? How do the conclusions change when prefill is included?

3 comments

We thank the reviewer for the comments and are encouraged that the reviewer valued the evidence for zero-training transfer and the concrete vLLM implementation.

.

W1-1 / Q1. What DA adds over SSAS and over a simpler span-reference or one-shot selection protocol

DA responses use two capabilities beyond the simpler protocols: selecting again as the response goes on, which a one-shot selection lacks, and local mode, which neither has. SSAS [1] shares DA's premise that a model can select its own attention span (Appendix A.4). However, SSAS demonstrates this premise only in a task-specific setting. It fine-tunes a 7B model separately for each of two tasks, on contexts within 2K tokens, with a partition of the context designed by hand for each task. DA's focus mode plays the role of the SSAS span reference in a task-agnostic protocol, elicited zero-shot with one prompt across 15 tasks.

We have not run the two suggested protocols as separate prompts. Instead, we measured how often the stored DA responses use these two capabilities.

  • DA can select again as the response goes on, which a one-shot selection cannot do. 29% (Gemma) and 25% (Qwen) of responses make more than one focus declaration, and in 95% of these a later declaration names a segment outside the first selection. In the example below (Qwen, singledoc_qa), the first focused segment only points to another section, so the model returns to global mode to locate it and then focuses on it:

    <focus magic_chunks="2">Section 111(a) states that activities shall be continued "through the date specified in section 106(3)."</focus>
    <global>[...] I need to find Section 106(3) [...] Magic Chunk 6 contains Section 106. [...]</global>
    <focus magic_chunks="6">December 20, 2024</focus>
    
  • Local mode reduces attended tokens further, and SSAS has no counterpart to it. In local mode, the entire document is masked out while the model answers from its own notes. Replaying the stored responses with local mode replaced by focus mode on the segments already focused raises attended tokens per response by 13.7% (Gemma) and 12.9% (Qwen).

.

W1-2 / W2-3. Comparison with sparse attention methods such as TidalDecode, and the DA+DSA composition

DA is complementary to index-based selection by construction. TidalDecode [2] and DeepSeek Sparse Attention (DSA) [3] select tokens again at every decode step. Each selection has a cost that grows with the context. For example, the indexer reads 3% of the bytes of the full KV cache on Qwen3.8-Flash-Next [4] and 13% on DeepSeek-V4 [5] (Appendix D.6). DA's selection instead stays fixed for 95 (Gemma) and 181 (Qwen) decode steps on average. The two act at different points. DA masks out segments before the index runs, so in focus mode and local mode the index scans only the segments that remain visible. In global mode, where DA attends to the whole context, the index reduces the attention reads.

DA empirically reduces the per-step cost of DSA. On DeepSeek-V4-Flash [5], DA with DSA scans less than half as many index tokens per step as DSA alone, and is 3.1 pp more accurate (Appendix I). Over complete responses, the total scan is 28% higher than with DSA alone, because DA generates more decode steps. On Gemma and Qwen, most extra decode steps are redundant text that the protocol does not require, which we expect post-training to remove (see W2-2 / Q3 below).

We expect the same synergy with training-free methods. We have not run DA with a training-free method, or a comparison at matched accuracy, on Gemma and Qwen. TidalDecode also selects at every step. DA masks segments before this selection takes place, so we expect the same reduction of selection cost as with DSA.

.

W2-1. Accuracy loss on the excluded sources (cwe, structured_data)

Our additional analysis of every incorrect response shows that the accuracy loss mainly comes from simple errors of protocol use, which we expect post-training to correct (see details in Additional Analysis A of the general response). It covers the vanilla model and DA on the tasks with the largest accuracy drops, including these two. Comprehension or reasoning errors do not become more frequent with DA: they occur in 7% of DA responses against 13% of vanilla responses on Gemma, and in about 1% of both on Qwen.

  • Indeed, our additional experiment shows that only a minor revision of the zero-shot prompt recovers about half of the cwe loss. Adding three short rules raises Gemma's accuracy with DA from 69.8% to 84.0% on the 96 held-out items (see details in Additional Experiment B of the general response). Note: the paper's 70.3% is the same DA run over all 128 items.

Q2. Source-selection criteria and aggregate results over all sources

Selection criteria: the submission states that the suite is selected and gives the criteria qualitatively (Section 4, Appendix F.3), and the revision adds the exact thresholds and that they were set after observing results (now Appendix F.5). A source is outside the suite if, on Gemma-4-31B, DA's accuracy falls by more than 10 pp or its attended tokens per response exceed those of the vanilla model.

Aggregate results: DA's efficiency gain holds over all 21 sources. With the six outside sources included, DA still attends to fewer tokens than the vanilla model on both models, per decode step and per response.

  • Per decode step, attended tokens fall by 50% (Gemma) and 23% (Qwen).
  • Per response, they still fall by 12% (Gemma) and 15% (Qwen), with all extra decode steps included.
  • On Qwen-3.6-27B, adding the six outside sources actually improves the accuracy result, from −2.1 pp on the main suite to 0.0 pp over all 21 sources. The six sources are affected there by vanilla responses that reach the output limit. Where neither reaches the limit, the accuracy difference on them is −2.7 pp.

DA relative to the vanilla model. Main suite: three-run average. Other six: one run (Table 13 of the revision).

GemmaQwen
SourcesAccuracyStepsAttended per stepAttended per responseAccuracyStepsAttended per stepAttended per response
Main suite (15)−1.4 pp1.38×0.39×0.54×−2.1 pp1.32×0.61×0.80×
Outside the suite (6)−9.0 pp2.18×0.55×1.19×+5.3 pp0.98×0.89×0.87×
All (21)−3.6 pp1.75×0.50×0.88×0.0 pp1.10×0.77×0.85×

.

W2-2 / Q3. Complete-response decode time under continuous batching, and the effect of prefill

Our additional experiment shows that DA does lower complete-response decode time, by 35% (Qwen) and 4% (Gemma) against the vanilla model, on tasks without extra decode steps (see details in Additional Experiment C of the general response). For each model, we submitted all requests of one task at once to vLLM on a single B200. The requests were batched continuously and ended naturally. Prompt processing is slower with DA, by 5% (Qwen) and 12% (Gemma), because of the extra input tokens. With prompt processing included, the total time falls by about 8% on Qwen and rises by 9% on Gemma.

The remaining overhead is extra tokens that we expect post-training to remove. The extra input tokens are prompt scaffolding, and most of the extra decode steps are redundant text that the protocol does not require (see details in Additional Analysis D and Additional Analysis E of the general response).

  • Indeed, our additional experiment shows that adding one rule to the zero-shot prompt already removes more than half of the extra decode steps on Qwen niah_multikey_3, with accuracy unchanged (see details in Additional Experiment F of the general response).

Roofline estimate in the target setting of large-batch serving at long contexts. Using our roofline-based cost model, we identify that KV cache reads account for 94% of decode time at a 1M-token context on Kimi-K3 [6] and 83% on DeepSeek-V4-Flash [5] (Appendix D.7). Under the same model, on the 15 main tasks, complete-response decode time falls by 24% (Gemma) and 13% (Qwen), with all extra decode steps included (Section 5.2, Appendix F.4).

.

References

[1] Jin et al. Self-Selected Attention Span for Accelerating Large Language Model Inference. arXiv:2404.09336, 2024.

[2] Yang et al. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025.

[3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.

[4] Qwen Team. Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency. Qwen blog, 2026.

[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.

[6] Kimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026.

We thank you again for your constructive feedback. As the discussion deadline is approaching, please let us know if you have any remaining concerns or questions. We would be very happy to promptly address them in the remaining time.

As summarized in our general response, our paper shows that DA's key mechanism already works zero-shot on off-the-shelf models. We believe DA has significant potential in production, where dedicated post-training can teach the protocol directly. In this light, our additional analyses and experiments support that the weaknesses raised are limitations of zero-shot elicitation, which we expect post-training to remove.

Our rebuttal addresses your main concerns, including the following:

  • What DA adds over SSAS and simpler protocols (W1-1 / Q1): SSAS fine-tunes a separate model for each task, while DA is a single task-agnostic protocol that off-the-shelf models follow zero-shot across 15 tasks. DA can also select again during a response and answer from its own notes in local mode, which we consider fundamental for broad applicability and robustness.
  • Accuracy loss on the excluded sources (W2-1): our additional analysis shows that the loss mainly comes from simple errors of protocol use. Adding only three rules to the zero-shot prompt raises accuracy on Gemma cwe from 69.8% to 84.0%.
  • Source selection (Q2): the six outside sources were selected as failure cases on Gemma. Over all 21 sources, DA still attends to 12% (Gemma) and 15% (Qwen) fewer tokens per response than the vanilla model, and on Qwen its accuracy matches the vanilla model (0.0 pp).
  • Complete-response decode time (W2-2 / Q3): our additional experiment shows that, on tasks without extra decode steps, DA lowers the measured wall time of decoding complete responses by 35% (Qwen) and 4% (Gemma) against the vanilla model, under continuous batching in vLLM. Most extra decode steps are redundant text, and adding one rule to the zero-shot prompt removes more than half of them on one task.
Reviewer_fxzEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Declarative Attention (DA), a prompting protocol that enables language models to control their attention scope through generated text tags. Models alternate between global, focus, and local modes, which the inference engine implements by modifying KV-cache block tables without changing attention kernels. On 15 selected long-context tasks, DA reduces total attended tokens by 45.8% and 19.6% on Gemma and Qwen, respectively, with average accuracy drops of 1.40 and 2.10 percentage points. The paper also presents masking ablations, model-scaling analyses, and per-step latency measurements to characterize its benefits and limitations.

Critical strengths

An interesting approach to model-controlled computation. The paper shows that off-the-shelf models can use prompted textual declarations to dynamically control their attention scope without additional training.

Useful ablations and implementation details. The no-mask baseline helps isolate the effect of attention masking from changes induced by prompting, while the vLLM integration and per-step timing measurements demonstrate the mechanism’s feasibility.

Critical weaknesses
  1. Missing comparisons with existing sparse attention methods. DA selects which KV blocks to read through model-generated declarations, addressing the same computational problem as query-dependent sparse attention, albeit with a different selection mechanism. However, the main experiments compare only against full attention and a no-mask ablation. Matched-backbone comparisons with representative sparse attention methods are needed to establish DA’s accuracy–efficiency trade-off. The supplementary DA+DSA experiment demonstrates compatibility, but increases total indexer-scanned tokens by 28%, rather than establishing an efficiency advantage.

  2. Substantial task-dependent accuracy degradation raises concerns about practical reliability. Although the headline average accuracy loss is modest, Appendix F.3 reports a drop from 100% to 70.3% on RULER/cwe and from 68.4% to 47.4% on LongBench-v2 structured_data. The headline suite was explicitly selected based on favorable accuracy and efficiency behavior, with six unsuccessful sources reported separately. Consequently, the main averages provide an optimistic picture of applicability, and the paper does not establish how to identify suitable requests before applying DA.

  3. The performance impact of fragmented KV access is insufficiently characterized. DA implements sparse attention by modifying KV-cache block tables while reusing existing attention kernels. Although the reported per-step measurements demonstrate speedups at high masking ratios, they do not isolate how the number and distribution of selected chunks, or heterogeneous masks across batched requests, affect performance. Evaluating these factors at a fixed retained-token budget, together with full-response latency and serving throughput, would clarify whether the KV-read savings translate into robust practical acceleration.

Questions for authors

Please refer to the weaknesses.

3 comments

We thank the reviewer for the comments and are encouraged that the reviewer found DA an interesting approach to model-controlled computation.

.

W1. Missing comparisons with existing sparse attention methods, and the DA+DSA experiment

DA is complementary to index-based sparse attention by construction. Quest [1], TidalDecode [2] and DeepSeek Sparse Attention (DSA) [3] select entries again at every decode step. This selection still costs O(N) memory reads per step. An indexer reads 3% of the bytes of the full KV cache on Qwen3.8-Flash-Next [4] and 13% on DeepSeek-V4 [5] (Appendix D.6). DA instead keeps its selection fixed for 95 (Gemma) and 181 (Qwen) decode steps on average, with no selection cost. DA masks out segments before the index runs, so in focus mode and local mode the index scans only the segments that remain visible. In global mode, where DA attends to the whole context, the index reduces the attention reads.

DA empirically reduces the per-step cost of DSA. On DeepSeek-V4-Flash, DA on top of DSA cuts the tokens scanned by the indexer at each decode step by 54%, and is 3.1 pp more accurate (Appendix I). Over a full response, the indexer still scans 28% more tokens with DA, because DA generates more decode steps. Our additional analysis and experiment on Gemma and Qwen show that most extra decode steps are redundant text, which we expect post-training to remove, and that one rule added to the zero-shot prompt already removes more than half of them on one task (see details in Additional Analysis E and Additional Experiment F of the general response).

We expect the same synergy with training-free methods. We used DSA as the representative method and have not run a matched-backbone comparison with Quest or TidalDecode. DA masks out segments before their selection runs, so we expect the same reduction of selection cost as with DSA.

.

W2. Task-dependent accuracy degradation and practical reliability

Our additional analysis of every incorrect response shows that the accuracy loss mainly comes from simple errors of protocol use, which we expect post-training to correct (see details in Additional Analysis A of the general response). It covers the vanilla model and DA on the tasks with the largest accuracy drops, including cwe and structured_data. Comprehension or reasoning errors do not become more frequent with DA.

  • Indeed, our additional experiment shows that only a minor revision of the zero-shot prompt corrects many of these errors. Adding three short rules raises Gemma cwe accuracy from 69.8% to 84.0% (see details in Additional Experiment B of the general response). This supports our expectation that post-training can correct these errors. Note: the paper reports 70.3% over all 128 items, while 69.8% is the same run on the 96 held-out items.

The accuracy gap narrows as the model grows, which also supports the post-training argument. Larger models follow the protocol better: DA's accuracy relative to the vanilla model rises from 29% (Gemma-4-E4B) to 98% (Gemma-4-31B), and from 64% (Qwen-3.5-4B) to 98% (Qwen-3.6-27B) (Section 5).

.

W2 (continued). Selection of the suite, and results over all 21 sources

Selection criteria: the submission states that the suite is selected and gives the criteria qualitatively (Section 4, Appendix F.3), and the revision adds the exact thresholds and that they were set after observing results (now Appendix F.5). A source is outside the suite if, on Gemma-4-31B, DA's accuracy falls by more than 10 pp or its attended tokens per response exceed those of the vanilla model.

Aggregate results: DA's efficiency gain holds over all 21 sources. With the six outside sources included, DA still attends to fewer tokens than the vanilla model on both models, per decode step and per response.

  • Per decode step, attended tokens fall by 50% (Gemma) and 23% (Qwen).
  • Per response, they still fall by 12% (Gemma) and 15% (Qwen), with all extra decode steps included.
  • On Qwen-3.6-27B, adding the six outside sources actually improves the accuracy result, from −2.1 pp on the main suite to 0.0 pp over all 21 sources. The six sources are affected there by vanilla responses that reach the output limit. Where neither reaches the limit, the accuracy difference on them is −2.7 pp.

DA relative to the vanilla model. Main suite: three-run average. Other six: one run (Table 13 of the revision).

GemmaQwen
SourcesAccuracyStepsAttended per stepAttended per responseAccuracyStepsAttended per stepAttended per response
Main suite (15)−1.4 pp1.38×0.39×0.54×−2.1 pp1.32×0.61×0.80×
Outside the suite (6)−9.0 pp2.18×0.55×1.19×+5.3 pp0.98×0.89×0.87×
All (21)−3.6 pp1.75×0.50×0.88×0.0 pp1.10×0.77×0.85×

W3. Impact of fragmented KV access and heterogeneous masks

Our additional experiment shows that per-step decode time does not depend on the sparsity pattern, within 0.3% (see details in Additional Experiment G right below). At a fixed number of attended tokens, we varied the number and placement of the attended segments, and whether masks differ across batched requests. The pattern does not matter because the attention kernel reads the KV cache through a block table. It gathers keys and values block by block and does not rely on contiguity across blocks, even under vanilla decoding, so the read cost depends on how many blocks are kept and not on where they are (Section 2.3). The one exception is marked with † below.

Additional Experiment G. Does per-step decode time depend on the sparsity pattern?

Table 11 of the revised paper. Per-step decode time relative to one contiguous attended segment (1.001 means 0.1% slower), at a fixed number of attended tokens. Single B200 GPU. Columns give the number of sequences × prompt tokens per sequence.

PatternGemma, 14 × 64KGemma, 7 × 128KQwen, 23 × 64K
Three segments1.0011.0011.001
Scattered segments1.0011.0011.001
Different mask per sequence1.0011.0021.002
Mask updated at intervals1.0021.0011.003
Different attended count per sequence †1.1941.0371.005
Reference repeated (noise)1.0001.0011.000

† The exception (19% slower on Gemma with 14 sequences) reflects an inefficiency of vLLM's Triton attention kernel, whose time in large batches is set by the sequence that keeps the most tokens (revised Appendix F.1). Qwen's FlashInfer kernel splits each sequence's KV cache across thread blocks, and the same row costs 0.5% there. We therefore expect Gemma's savings in our initial per-step decode measurements (Section 5.2), which include this effect, to improve further with such a kernel.

.

W3 (continued). Full-response latency

Our additional experiment shows that DA does lower end-to-end decode time, by 35% (Qwen) and 4% (Gemma), on tasks without extra decode steps (see details in Additional Experiment C of the general response). We measured one task per model where DA and the vanilla model generate a similar number of decode steps, with all requests submitted at once on a single B200. With prompt processing included, which is slower with DA because of its extra input tokens, the total time falls by about 8% on Qwen and rises by 9% on Gemma. We expect post-training to remove the remaining overhead of extra input tokens and decode steps (see details in Additional Analysis D to Additional Experiment F of the general response).

Roofline estimate in the target setting. Under our roofline model of large-batch inference, full-response decode time falls by 24% (Gemma) and 13% (Qwen), with all extra decode steps included (Section 5.2, Appendix F.4).

.

References

[1] Tang et al. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. ICML 2024.

[2] Yang et al. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025.

[3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.

[4] Qwen Team. Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency. Qwen blog, 2026.

[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.

We thank you again for your constructive feedback. As the discussion deadline is approaching, please let us know if you have any remaining concerns or questions. We would be very happy to promptly address them in the remaining time.

As summarized in our general response, our paper shows that DA's key mechanism already works zero-shot on off-the-shelf models. We believe DA has significant potential in production, where dedicated post-training can teach the protocol directly. In this light, our additional analyses and experiments support that the weaknesses raised are limitations of zero-shot elicitation, which we expect post-training to remove.

Our rebuttal addresses each of your concerns, including the following:

  • Comparison with sparse attention methods (W1): on DeepSeek-V4-Flash, DA reduces the per-step index cost of DSA by 54%. Over full responses, this gain is offset by extra decode steps, which we address below.
  • Task-dependent accuracy loss (W2): our additional analysis shows that the loss mainly comes from simple errors of protocol use. Adding only three rules to the zero-shot prompt raises accuracy on Gemma cwe from 69.8% to 84.0%.
  • Fragmented KV access and heterogeneous masks (W3): our additional experiment shows that per-step decode time does not depend on the sparsity pattern, within 0.3%. The one exception reflects an inefficiency of vLLM's Triton attention kernel used for Gemma, so we expect Gemma's measured savings to improve further with an optimized kernel.
  • Full-response latency (W3): our additional experiment shows that, on tasks without extra decode steps, DA lowers the measured wall time of decoding complete responses by 35% (Qwen) and 4% (Gemma) against the vanilla model, under continuous batching in vLLM. Most extra decode steps are redundant text, and adding one rule to the zero-shot prompt removes more than half of them on one task.
Reviewer_XNHoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Declarative Attention (DA), an inference-time protocol in which a language model specifies its desired attention scope while generating a response. The input context is divided into addressable chunks, and the model alternates among <global>, <focus>, and <local> modes. The inference engine translates these declarations into restrictions on which KV-cache blocks are read by global-attention layers. The approach requires neither model fine-tuning nor changes to the attention kernel. Across 15 long-context task sources, DA reduces cumulative attended tokens by 45.8% on Gemma-4-31B and 19.6% on Qwen-3.6-27B.

Overall, the proposed method is interesting, but its practical benefits appear limited in its current form. More rigorous investigation and optimization are needed to establish its real-world value.

Critical strengths
  • Novel inference-time control mechanism. The paper turns attention selection into an explicit model output, offering an alternative to external heuristics or per-step relevance scoring.
  • Practical implementation path. DA operates on off-the-shelf models without fine-tuning and modifies vLLM’s KV block tables without changing attention kernels.
  • Clear and interpretable protocol. The global, focus, and local modes make the model’s intended information access visible in its reasoning trace, which helps inspect both successful decisions and failures.
  • Meaningful reduction in KV reads. Across the selected 15 long-context task sources, cumulative attended tokens fall by 45.8% on Gemma-4-31B and 19.6% on Qwen-3.6-27B, despite additional decoding steps.
  • Useful ablations and system analysis. The no-mask control separates the effect of the prompt from that of restricting attention. The paper also reports measured per-step timing and a decode-cost model, rather than relying solely on token-count reductions.
Critical weaknesses
  • Accuracy costs are concentrated in demanding tasks. The average declines of 1.40–2.10 percentage points obscure larger losses on multi-span reasoning: 2.48 points on Gemma-4-31B and 4.84 points on Qwen-3.6-27B. This is a consequential weakness for tasks that require evidence from several chunks.
  • End-to-end benefits remain insufficiently established. DA generates more tokens to navigate and declare attention modes. Reduced attended-token counts and favorable per-step measurements therefore do not, on their own, demonstrate lower latency for complete requests under realistic serving workloads.
  • Dynamic conversational use is underexplored. The experiments primarily treat context as a fixed, pre-segmented input. A continuously growing conversation would require robust chunk indexing and mapping as new turns arrive or old context is discarded.
  • The paper would benefit from a more detailed analysis of accuracy changes on tasks requiring cross-chunk reasoning, global aggregation, table understanding, and highly reliable question answering.

minors:

  • The method does not reduce KV-cache capacity requirements. The full context must still be stored.
  • The comparison with existing efficiency techniques is not enough.
Questions for authors
  • Does DA perform substantially worse on smaller models? How would it handle multi-turn conversations, where the context and chunk boundaries change over time?
  • For the two large models in the main experiments, changing the prompt and output format alone causes little accuracy loss; the losses of approximately 1.93 and 2.05 percentage points arise only when access to the context is restricted. Does this suggest that the model is not sufficiently reliable at identifying and attending to the relevant chunks?
  • The tool-like global and focus modes appear to impose additional reasoning and control overhead on the model. Could these declarations interfere with other tool-calling workflows?
3 comments

We thank the reviewer for the thoughtful feedback and are encouraged that the reviewer found DA to be a novel and interpretable inference-time control mechanism.

.

W1 / W4. Accuracy loss on tasks that need evidence from several chunks, and the need for a detailed error analysis

DA is able to gather evidence from several segments. One focus declaration can name up to three segments, and the model can make more than one in a response (Appendix C.2). On the 15 main tasks, 29% (Gemma) and 25% (Qwen) of DA responses make more than one focus declaration.

The multi-span drop on Qwen (4.84 pp, Table 2). On the multi-span task singledoc_qa (accuracy 95.1 → 89.1 over three runs), only 2 of DA's 15 errors in the analyzed run are comprehension or reasoning errors. Ten are simple errors of protocol use. For example, the model invents the content of a segment that is masked out. In the example below, the document states "It was a 1956 Montblanc 342". The model focuses on a segment that lacks this sentence, and then "quotes" the document in local mode, where the whole document is masked out:

<focus magic_chunks="25">1985</focus>
<local>[...] In Magic Chunk 25, within the email from Lawrence Treherne, he states: "It was a 1985 Montblanc." [...]</local>

A single rule in the prompt forbids this behavior: quote only from the segments in focus or from the model's own notes.

An expanded error analysis shows that the accuracy loss across tasks mainly comes from such simple errors of protocol use, which we expect post-training to correct (see details in Additional Analysis A of the general response). On the tasks with the largest accuracy drops, comprehension or reasoning errors do not become more frequent with DA. They occur in 7% of DA responses against 13% of vanilla responses on Gemma, and in about 1% of both on Qwen.

  • Indeed, our additional experiment shows that only a minor revision of the zero-shot prompt corrects many of these errors. Adding three short rules raises accuracy on Gemma cwe, a global aggregation task, from 69.8% to 84.0% (see details in Additional Experiment B of the general response).

.

W2. End-to-end benefits are insufficiently established because of extra decoding steps

Our additional experiment shows that DA does lower the end-to-end decode time, by 35% (Qwen) and 4% (Gemma) against the vanilla model, on tasks without extra decode steps (see details in Additional Experiment C of the general response). We timed complete responses on one task per model where DA and the vanilla model generate a similar number of decode steps. All requests were submitted at once to a single B200 GPU. Prompt processing remains slower with DA, because DA adds input tokens and applies no mask there. With prompt processing included, the total time falls by about 8% on Qwen and rises by 9% on Gemma. †

The remaining overhead is extra tokens that we expect post-training to remove. The extra input tokens are prompt scaffolding, and most of the extra decode steps are redundant text that the protocol does not require (see details in Additional Analysis D and Additional Analysis E of the general response).

  • Indeed, our additional experiment shows that adding one rule to the zero-shot prompt already removes more than half of the extra decode steps on Qwen niah_multikey_3, with accuracy unchanged (see details in Additional Experiment F of the general response).

Roofline estimate of end-to-end decode time in the target setting. DA targets large-batch serving at long contexts, where memory reads bound decode time. DA reduces the total attended tokens per response by 45.8% (Gemma) and 19.6% (Qwen). Under a roofline cost model, this lowers the decode time of a full response by 24% (Gemma) and 13% (Qwen), with all extra decode steps included (Section 5.2, Appendix F.4).

† Why the paper measures time per decode step. In the additional experiment above, with only 18 long requests on Qwen, the end-to-end time is sensitive to tail concurrency: the longest responses of each pass finish with few other requests left in the batch, so the total varies widely between repeats, as the results show. Across five repeats on Qwen, DA ranged from 63% less decode time than the vanilla model to 10% more. The paper's main measurement is therefore the time per decode step. It compares DA and the vanilla model at the same context length and batch, so the only difference is the mask. Per-step decode time falls by up to 55% (Gemma) and 31% (Qwen) (Section 5.2).

W3 / Q1-2. Dynamic and multi-turn conversations

In agentic settings, the turn boundaries can serve as the segments. Each user turn, tool call and tool response is a segment whose id is assigned once when it arrives and never changes, so no existing segment is re-indexed as the conversation grows. When a harness discards old context, the surviving turns keep their ids and a summary enters as a new segment. In 3,557 public trajectories from four agent benchmarks (SWE-smith, SWE-rebench, Terminal-Bench 2.0 and tau2-bench), the median trajectory has 21 to 123 such segments, depending on the benchmark. 94 to 97% of segments are under 2,048 tokens.

We plan to adapt DA to the agentic setting and to evaluate both models on agentic benchmarks.

.

Q1-1. Does DA perform substantially worse on smaller models?

The gap narrows consistently with model size (Section 5). Measured as a share of the vanilla model's accuracy, DA's accuracy rises from 29% on Gemma-4-E4B to 98% on Gemma-4-31B. On Qwen, it rises from 64% (Qwen-3.5-4B) to 98% (Qwen-3.6-27B). The gap on small models is consistent with a simple error of protocol use, invalid focus declarations: only 58% of focus declarations are valid on Gemma-4-E4B, against 98.7% on Gemma-4-31B (Appendix G.7). Following a response format is a behavior that fine-tuning on a small number of examples is known to instill [1], so we expect small models to benefit the most from post-training.

.

Q2. Does the loss under attention masking suggest that the model is unreliable at identifying the relevant chunks?

The model's selection of segments is reliable. On the five RULER needle-retrieval tasks, the first focus declaration contains the target segment in 640 of 640 (Gemma) and 637 of 638 (Qwen) responses. On the tasks with the largest accuracy drops, the outcome was not decided by the choice of segments in 561 of 570 (Gemma) and 650 of 659 (Qwen) DA responses (revised Appendix G.2, Table 14). The model can also correct a selection by returning to global mode, as the examples in Appendix G.1 and Appendix H show. The loss comes instead from simple errors of protocol use after the selection (please see W1 / W4).

.

Q3. Could the declarations interfere with other tool-calling workflows?

In our experiments the declarations are plain text tags, and the context is already delivered as calls to a declared tool in each model's native tool format (Appendix C.1). For agentic settings, we think it is more appropriate to implement the DA protocol itself as a standard tool, as we propose in Appendix B.2. DA then becomes one more tool, and we expect the model to be able to use it in tandem with its other tools. We plan to evaluate this and, if possible, will add a comment with the results.

.

Minor weaknesses

No reduction in KV cache capacity. This is by design: retaining the KV cache is what makes the mask reversible. DA targets the memory reads per decode step, which bound decode time at long contexts (Section 3).

Comparison with existing efficiency techniques. Index-based sparse attention, such as DeepSeek Sparse Attention (DSA), selects KV cache entries at every decode step with an index scan, whose cost grows with the context length. DA's selection instead stays fixed for 95 (Gemma) and 181 (Qwen) decode steps on average, and has no selection cost during those steps. The two compose: on DeepSeek-V4-Flash, DA reduces the per-step index cost of DSA by 54% (Appendix I).

.

References

[1] Zhou et al. LIMA: Less Is More for Alignment. NeurIPS 2023.

We thank you again for your constructive feedback. As the discussion deadline is approaching, please let us know if you have any remaining concerns or questions. We would be very happy to promptly address them in the remaining time.

As summarized in our general response, our paper shows that DA's key mechanism already works zero-shot on off-the-shelf models. We believe DA has significant potential in production, where dedicated post-training can teach the protocol directly. In this light, our additional analyses and experiments support that the weaknesses raised are limitations of zero-shot elicitation, which we expect post-training to remove.

Our rebuttal addresses each of your concerns, including the following:

  • Accuracy loss and error analysis (W1 / W4): our additional analysis shows that the loss mainly comes from simple errors of protocol use. Adding only three rules to the zero-shot prompt raises accuracy on Gemma cwe from 69.8% to 84.0%.
  • End-to-end benefits (W2): our additional experiment shows that, on tasks without extra decode steps, DA lowers the measured wall time of decoding complete responses by 35% (Qwen) and 4% (Gemma) against the vanilla model, under continuous batching in vLLM. Most extra decode steps are redundant text, and adding one rule to the zero-shot prompt removes more than half of them on one task.
  • Dynamic and multi-turn conversations (W3 / Q1-2): in agentic settings, each user turn, tool call and tool response can serve as a segment. Its id is assigned once when it arrives and never changes, so DA applies unchanged as the conversation grows.

Comments on the paper

3 comments

We extend our gratitude to all the reviewers for providing comprehensive and thoughtful feedback on our manuscript. We appreciate your valuable insights into the strengths and areas for improvement of our work.

We have uploaded a revised paper, with new and changed text highlighted in yellow.

.

Summary of Scope and Key Result

We ultimately view DA as a protocol to be learned in post-training (Abstract, Section 1, Appendix B.2), and this paper aims to show zero-shot that its key mechanism works on off-the-shelf models. It does: the models declare the segments that they need, and answer accurately while attending only to those segments. Across 15 tasks, attended tokens fall by 45.8% / 19.6% with accuracy within 1.4 / 2.1 pp, which the paper frames as "a floor rather than a ceiling" (Section 7).

The question for each weakness raised in the reviews is therefore whether it is inherent to attention masking, or a limitation of zero-shot elicitation that post-training can remove. Our additional analysis and experiments support the latter (Additional Analysis A to Additional Experiment F).

.

Core Contributions of Our Work

  • Novelty of approach: Declarative Attention (DA) elicits the model's own knowledge of which context is relevant to its reasoning, through a tool-call-like interface, to reduce redundant KV cache reads. It is an alternative to index-based scoring, which is widely used for sparse attention [1, 2, 3], and is orthogonal to it:
    • Difference from index-based selection: A lightweight index selects the top-k KV cache entries at every decode step [3], which still costs O(N) memory reads per step: the index amounts to 3% of the bytes of the full KV cache on Qwen3.8-Flash-Next [4] and 13% on DeepSeek-V4 [5] (Appendix D.6). In DA, the selection stays fixed during focus and local modes for 95 (Gemma) and 181 (Qwen) decode steps on average, with no selection cost in those modes.
    • Synergy with index-based selection: Applied on top of DeepSeek Sparse Attention on DeepSeek-V4-Flash [5], DA reduces per-step index costs by 54%.
    • Difference from SSAS [6]: SSAS hand-crafts segmentation and reasoning formats for two fixed-shape tasks and fine-tunes a model for each. DA is a task-agnostic protocol, validated across 15 diverse tasks on two models with one fixed zero-shot prompt.
  • Zero-shot effectiveness: Across 15 long-context tasks on two off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B), DA reduces total attended tokens during decoding by 45.8% and 19.6%, with average accuracy drops of 1.40 pp and 2.10 pp.
  • Scalability: The accuracy drop narrows as the backbone model grows from 4B to 31B.
  • Practical implementation: The selected segments are contiguous, so DA is implemented as block-aligned attention masking of the KV cache with no kernel changes. We provide an integration for vLLM.
  • Interpretability: The model declares its attention scope, which makes its intended information access interpretable.
  • Theoretical full-response decode speedup in large-batch inference: Under a roofline-based cost model for disaggregated large-batch inference, DA lowers decode wall time by 24% (Gemma) and 13% (Qwen) on average across the evaluation tasks.
  • Measured per-step speedup on a single GPU: Per-step decode wall time falls with the mask ratio, by up to 55% (Gemma) and 31% (Qwen) at mask ratios above 90%.

.

Summary of Strengths Cited by Reviewers

  • Novelty: Reviewers XNHo and fxzE found the approach interesting: an inference-time mechanism that makes attention selection an explicit model output.
  • Zero-shot effectiveness: All reviewers noted that off-the-shelf models follow the protocol without training, with meaningful reductions in KV reads (XNHo).
  • Practical implementation: All reviewers acknowledged the vLLM integration, which needs no fine-tuning.
  • Experiments: All reviewers found the no-mask ablation useful, along with the per-step timing (XNHo, fxzE), the cost model (XNHo) and the reported failures and judge validation (FPZD).
  • Interpretability: Reviewer XNHo noted that the protocol makes the model's intended information access visible.

.

References

[1] Tang et al. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. ICML 2024.

[2] Yang et al. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025.

[3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.

[4] Qwen Team. Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency. Qwen blog, 2026.

[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.

[6] Jin et al. Self-Selected Attention Span for Accelerating Large Language Model Inference. arXiv:2404.09336, 2024.

Additional Analysis and Experiments: Accuracy and End-to-End Efficiency


The reviewers raised two shared concerns. We address both with additional analysis and experiments, summarized here and detailed below.

Concern 1: accuracy falls on some tasks. The loss is not due to extra errors in comprehension or reasoning, but mainly to simple errors of protocol use that we expect post-training to correct (Additional Analysis A). In support of this expectation, adding only three rules to the zero-shot prompt raises accuracy on Gemma cwe from 69.8% to 84.0% (Additional Experiment B).

Concern 2: extra input tokens and decode steps offset the end-to-end efficiency. On tasks without extra decode steps, end-to-end decode time falls by 35% (Qwen) and 4% (Gemma) (Additional Experiment C). The remaining overhead is extra tokens that we expect post-training to remove:

  • Extra input tokens (+4.7%). They are prompt scaffolding that elicits the protocol zero-shot, which we expect a trained model not to need (Additional Analysis D).
  • Extra decode steps. The protocol requires only the mode tags, which amount to 11% / 6% of a vanilla response, and the rest is redundant text (Additional Analysis E). Adding one rule to the zero-shot prompt removes more than half of the extra decode steps on Qwen niah_multikey_3 (Additional Experiment F).

.

Concern 1. Accuracy falls on some tasks


Additional Analysis A. Where does the accuracy loss come from?

Comprehension or reasoning errors do not become more frequent with DA, and the accuracy loss mainly comes from simple errors of protocol use that we expect post-training to correct. We categorized every incorrect response of the vanilla model and of DA on the tasks with the largest accuracy drops, using an LLM annotator with a fixed guide (two independent runs, 95% agreement). The full analysis is in the revised Appendix G.2 (Table 14).

  • Comprehension or reasoning errors, in which the model attends to the needed information and still misunderstands it or reasons incorrectly, occur in 7% of DA responses against 13% of vanilla responses on Gemma, and in about 1% of both on Qwen.

  • DA adds five kinds of simple errors of protocol use. These are mistakes in how the model operates the protocol: it uses a mode for something that the mode does not allow, such as quoting a segment that is masked out, or switching to local mode before taking enough notes. Each has a direct remedy, which we expect a few training examples to instill [7]. A revised zero-shot prompt already mitigates them, as we show right below.

Additional Experiment B. Does a minor revision of the zero-shot prompt recover accuracy?

Adding three short rules to the zero-shot prompt raises accuracy on Gemma cwe from 69.8% to 84.0%. This supports our expectation that post-training can correct these errors.

Table 16 of the revised paper. Effect of a revised instruction in the zero-shot prompt on Gemma cwe. The instruction was selected on 32 development items and evaluated on the 96 held-out items. The row with the revised instruction is the mean of three seeds.

Gemma, cweAccuracyAttended tokens per response
Vanilla100%26.9M
DA69.8%26.6M
DA, revised instruction84.0%26.6M

Concern 2. Extra input tokens and decode steps offset the end-to-end efficiency


Additional Experiment C. Is DA faster end-to-end on tasks without extra decode steps?

On tasks without extra decode steps, DA lowers end-to-end decode time by 35% (Qwen) and 4% (Gemma) against the vanilla model, and by 12% on both models against DA without mask. We measured one task per model where DA and the vanilla model generate a similar number of decode steps.

  • Prompt processing remains slower with DA, by 5% (Qwen) and 12% (Gemma), since DA adds input tokens and applies no mask there.
  • With prompt processing included, the total time falls by about 8% on Qwen, where responses are long, and rises by 9% on Gemma, where responses are short.
  • The remaining question is therefore whether the extra input tokens and decode steps can be removed (Additional Analysis D to Additional Experiment F).

Table 12 of the revised paper. End-to-end decode time (the sum of all decode-step times) on one task per model. Single B200 GPU, all requests submitted at once, median of 5 repeats. Across repeats, the ratio of DA to the vanilla model ranges from 0.37 to 1.10 on Qwen and from 0.90 to 1.00 on Gemma.

End-to-end decode timeDecode steps per responseAccuracy
Qwen, longbench_v2_code_repo: vanilla91.3 s1,43064.4%
Qwen, longbench_v2_code_repo: DA59.0 s1,44772.2%
Qwen, longbench_v2_code_repo: DA without mask66.8 s1,40377.8%
Gemma, niah_multikey_3: vanilla36.4 s281100.0%
Gemma, niah_multikey_3: DA34.9 s26899.1%
Gemma, niah_multikey_3: DA without mask39.8 s27899.8%

.

Additional Analysis D. Where do the extra input tokens come from? (+4.7% on both models)

The extra input tokens are prompt scaffolding that elicits the protocol zero-shot, which we expect to be unnecessary after post-training. To find what the extra input tokens consist of, we compared the DA prompt with the vanilla prompt on the 15 main tasks and assigned each added token to its purpose. The full analysis is in the revised Appendix G.5.

  • Instruction text (2.4%) explains the protocol. We expect a model trained on the protocol not to need it.
  • Segment markup and its framing (2.3%) present segments as tool-call transcripts, so that a model that has never seen the protocol treats them as addressable units. With training, we expect a short marker per segment to suffice.

.

Additional Analysis E. Where do the extra decode steps come from?

The extra decode steps required by the protocol are minimal (11% / 6% of a vanilla response), and the rest is redundant text that we expect post-training to remove. We labeled every generated token of both models on the 15 main tasks by the kind of text it belongs to, with the same LLM annotation procedure. The full analysis is in the revised Appendix G.3 (Table 15).

  • The protocol requires only the mode tags. DA does not lengthen the reasoning itself: DA writes less evidence and reasoning text than the vanilla model, by 27% / 8% of a vanilla response.
  • The rest is redundant text, such as announcing which segment the model will read, or restating earlier notes. Response length is known to be shaped by post-training [8, 9, 10], and a revised zero-shot prompt already removes more than half of the extra decode steps on one task, as we show right below.

Additional Experiment F. Does a minor revision of the zero-shot prompt remove extra decode steps?

Adding one rule to the zero-shot prompt reduces the extra decode steps on Qwen niah_multikey_3 from +45% to +19% of a vanilla response, with accuracy unchanged. The rule tells the model not to quote segments that are masked out. This supports our expectation that post-training can remove the redundant text.

Table 17 of the revised paper. Effect of a revised instruction in the zero-shot prompt on Qwen niah_multikey_3, on the 96 held-out items. Both DA rows are measured in this experiment and are the mean of three seeds.

Qwen, niah_multikey_3AccuracyDecode steps per responseAttended tokens per response
Vanilla100%2684.31M
DA99.3%3892.88M
DA, revised instruction99.7%3202.62M

.

References

[7] Zhou et al. LIMA: Less Is More for Alignment. NeurIPS 2023.

[8] Arora and Zanette. Training Language Models to Reason Efficiently. NeurIPS 2025.

[9] Fatemi et al. Concise Reasoning via Reinforcement Learning. arXiv:2504.05185, 2025.

[10] Li et al. DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.