- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Declarative Attention (DA) which is a prompt protocol that allows a vanilla LLM to generate declarations of global, focus and local scopes while generating. The runtime can then use these declarations to mask out the KV blocks being consumed by the global attention layers and leave the rest of the KV cache intact for future reference. Using a curated set of 15 long context tasks they were able to reduce the number of tokens attended to by 45.8% on Gemma-4-31B and 19.6% on Qwen-3.6-27B with a drop in mean accuracy of 1.40 and 2.10 percentage points respectively. They also show results for our unmasked baseline as well as some scaling results. Timing of decode per step and roofline estimate for full response decode are also provided.
Critical strengths
-
Good evidence for zero training transfer. It would be nice to see that existing models can learn to follow an attention control protocol on various long context tasks. This is helpful to know empirically. Also the DA no mask ablation helps show that it's due to the mask and not from generating less tokens.
-
They provide a concrete method to implement this with vLLM. And include time in their ablation when comparing the reversible mask. They also report many failures which I think is good to see. As well as validating their judge with 98.5% agreement rate and kappa score of .94.
Critical weaknesses
-
The methodological advance over existing span selection is limited, and its comparative value is not established. SSAS [1] already has the model output span references that determine hard attention masks while retaining skipped context. DA’s main advance is zero-training elicitation at larger scale and broader coverage. However, the experiments do not establish what the three-mode protocol adds over a simpler span-reference or one-shot selection protocol on the same backbone. Comparisons against vanilla and DA-no-mask establish savings relative to full attention, but not a better accuracy-cost trade-off than existing sparse-attention methods such as TidalDecode [2]. Complementarity does not remove the need to establish comparative benefit.
-
The evaluation does not establish broadly useful full-response efficiency. Appendix F.3 explicitly selects the headline suite based on DA preserving accuracy and reducing attended tokens. The excluded sources include accuracy losses of 29.7pp on cwe and 21.0pp on structured_data, and five of six increase total attended tokens. This limits the generality of the headline trade-off. Moreover, the measured speedups are per-step results with responses truncated and lengths matched; the full-response comparison is a roofline estimate. DA generates 38% and 32% more steps on the main models, and its DSA composition increases total indexer-scanned tokens to 128% of the baseline. The current evidence therefore does not establish a net serving advantage over complete responses or competitive baselines.
[1] Tian Jin etal. Self-Selected Attention Span for Accelerating Large Language Model Inference.
[2] Lijie Yang etal. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025
Questions for authors
-
On the same backbone and task set, how does DA compare with a simpler SSAS-style span-reference prompt or one-shot chunk selection, and with a sparse-attention baseline at comparable accuracy? These comparisons would determine whether the contribution extends beyond zero-training transfer of the existing mechanism.
-
What were the source-selection criteria, and were they fixed before observing DA’s results? Can you report matched-run aggregate results over all evaluated sources, including the six in Appendix F.3, for both main models?
-
What are the measured complete-response decode latency and throughput under a fixed continuous-batching workload with natural termination, charging all extra steps and runtime overhead? How do the conclusions change when prefill is included?
3 comments
Q2. Source-selection criteria and aggregate results over all sources
Selection criteria: the submission states that the suite is selected and gives the criteria qualitatively (Section 4, Appendix F.3), and the revision adds the exact thresholds and that they were set after observing results (now Appendix F.5). A source is outside the suite if, on Gemma-4-31B, DA's accuracy falls by more than 10 pp or its attended tokens per response exceed those of the vanilla model.
Aggregate results: DA's efficiency gain holds over all 21 sources. With the six outside sources included, DA still attends to fewer tokens than the vanilla model on both models, per decode step and per response.
- Per decode step, attended tokens fall by 50% (Gemma) and 23% (Qwen).
- Per response, they still fall by 12% (Gemma) and 15% (Qwen), with all extra decode steps included.
- On Qwen-3.6-27B, adding the six outside sources actually improves the accuracy result, from −2.1 pp on the main suite to 0.0 pp over all 21 sources. The six sources are affected there by vanilla responses that reach the output limit. Where neither reaches the limit, the accuracy difference on them is −2.7 pp.
DA relative to the vanilla model. Main suite: three-run average. Other six: one run (Table 13 of the revision).
| Gemma | Qwen | |||||||
|---|---|---|---|---|---|---|---|---|
| Sources | Accuracy | Steps | Attended per step | Attended per response | Accuracy | Steps | Attended per step | Attended per response |
| Main suite (15) | −1.4 pp | 1.38× | 0.39× | 0.54× | −2.1 pp | 1.32× | 0.61× | 0.80× |
| Outside the suite (6) | −9.0 pp | 2.18× | 0.55× | 1.19× | +5.3 pp | 0.98× | 0.89× | 0.87× |
| All (21) | −3.6 pp | 1.75× | 0.50× | 0.88× | 0.0 pp | 1.10× | 0.77× | 0.85× |
.
W2-2 / Q3. Complete-response decode time under continuous batching, and the effect of prefill
Our additional experiment shows that DA does lower complete-response decode time, by 35% (Qwen) and 4% (Gemma) against the vanilla model, on tasks without extra decode steps (see details in Additional Experiment C of the general response). For each model, we submitted all requests of one task at once to vLLM on a single B200. The requests were batched continuously and ended naturally. Prompt processing is slower with DA, by 5% (Qwen) and 12% (Gemma), because of the extra input tokens. With prompt processing included, the total time falls by about 8% on Qwen and rises by 9% on Gemma.
The remaining overhead is extra tokens that we expect post-training to remove. The extra input tokens are prompt scaffolding, and most of the extra decode steps are redundant text that the protocol does not require (see details in Additional Analysis D and Additional Analysis E of the general response).
- Indeed, our additional experiment shows that adding one rule to the zero-shot prompt already removes more than half of the extra decode steps on Qwen
niah_multikey_3, with accuracy unchanged (see details inAdditional Experiment Fof the general response).
Roofline estimate in the target setting of large-batch serving at long contexts. Using our roofline-based cost model, we identify that KV cache reads account for 94% of decode time at a 1M-token context on Kimi-K3 [6] and 83% on DeepSeek-V4-Flash [5] (Appendix D.7). Under the same model, on the 15 main tasks, complete-response decode time falls by 24% (Gemma) and 13% (Qwen), with all extra decode steps included (Section 5.2, Appendix F.4).
.
References
[1] Jin et al. Self-Selected Attention Span for Accelerating Large Language Model Inference. arXiv:2404.09336, 2024.
[2] Yang et al. TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention. ICLR 2025.
[3] DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556, 2025.
[4] Qwen Team. Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency. Qwen blog, 2026.
[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.
[6] Kimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026.
We thank you again for your constructive feedback. As the discussion deadline is approaching, please let us know if you have any remaining concerns or questions. We would be very happy to promptly address them in the remaining time.
As summarized in our general response, our paper shows that DA's key mechanism already works zero-shot on off-the-shelf models. We believe DA has significant potential in production, where dedicated post-training can teach the protocol directly. In this light, our additional analyses and experiments support that the weaknesses raised are limitations of zero-shot elicitation, which we expect post-training to remove.
Our rebuttal addresses your main concerns, including the following:
- What DA adds over SSAS and simpler protocols (
W1-1 / Q1): SSAS fine-tunes a separate model for each task, while DA is a single task-agnostic protocol that off-the-shelf models follow zero-shot across 15 tasks. DA can also select again during a response and answer from its own notes in local mode, which we consider fundamental for broad applicability and robustness. - Accuracy loss on the excluded sources (
W2-1): our additional analysis shows that the loss mainly comes from simple errors of protocol use. Adding only three rules to the zero-shot prompt raises accuracy on Gemmacwefrom 69.8% to 84.0%. - Source selection (
Q2): the six outside sources were selected as failure cases on Gemma. Over all 21 sources, DA still attends to 12% (Gemma) and 15% (Qwen) fewer tokens per response than the vanilla model, and on Qwen its accuracy matches the vanilla model (0.0 pp). - Complete-response decode time (
W2-2 / Q3): our additional experiment shows that, on tasks without extra decode steps, DA lowers the measured wall time of decoding complete responses by 35% (Qwen) and 4% (Gemma) against the vanilla model, under continuous batching in vLLM. Most extra decode steps are redundant text, and adding one rule to the zero-shot prompt removes more than half of them on one task.
We thank the reviewer for the comments and are encouraged that the reviewer valued the evidence for zero-training transfer and the concrete vLLM implementation.
.
W1-1 / Q1. What DA adds over SSAS and over a simpler span-reference or one-shot selection protocol
DA responses use two capabilities beyond the simpler protocols: selecting again as the response goes on, which a one-shot selection lacks, and local mode, which neither has. SSAS [1] shares DA's premise that a model can select its own attention span (
Appendix A.4). However, SSAS demonstrates this premise only in a task-specific setting. It fine-tunes a 7B model separately for each of two tasks, on contexts within 2K tokens, with a partition of the context designed by hand for each task. DA's focus mode plays the role of the SSAS span reference in a task-agnostic protocol, elicited zero-shot with one prompt across 15 tasks.We have not run the two suggested protocols as separate prompts. Instead, we measured how often the stored DA responses use these two capabilities.
DA can select again as the response goes on, which a one-shot selection cannot do. 29% (Gemma) and 25% (Qwen) of responses make more than one focus declaration, and in 95% of these a later declaration names a segment outside the first selection. In the example below (Qwen,
singledoc_qa), the first focused segment only points to another section, so the model returns to global mode to locate it and then focuses on it:Local mode reduces attended tokens further, and SSAS has no counterpart to it. In local mode, the entire document is masked out while the model answers from its own notes. Replaying the stored responses with local mode replaced by focus mode on the segments already focused raises attended tokens per response by 13.7% (Gemma) and 12.9% (Qwen).
.
W1-2 / W2-3. Comparison with sparse attention methods such as TidalDecode, and the DA+DSA composition
DA is complementary to index-based selection by construction. TidalDecode [2] and DeepSeek Sparse Attention (DSA) [3] select tokens again at every decode step. Each selection has a cost that grows with the context. For example, the indexer reads 3% of the bytes of the full KV cache on Qwen3.8-Flash-Next [4] and 13% on DeepSeek-V4 [5] (
Appendix D.6). DA's selection instead stays fixed for 95 (Gemma) and 181 (Qwen) decode steps on average. The two act at different points. DA masks out segments before the index runs, so in focus mode and local mode the index scans only the segments that remain visible. In global mode, where DA attends to the whole context, the index reduces the attention reads.DA empirically reduces the per-step cost of DSA. On DeepSeek-V4-Flash [5], DA with DSA scans less than half as many index tokens per step as DSA alone, and is 3.1 pp more accurate (
Appendix I). Over complete responses, the total scan is 28% higher than with DSA alone, because DA generates more decode steps. On Gemma and Qwen, most extra decode steps are redundant text that the protocol does not require, which we expect post-training to remove (seeW2-2 / Q3below).We expect the same synergy with training-free methods. We have not run DA with a training-free method, or a comparison at matched accuracy, on Gemma and Qwen. TidalDecode also selects at every step. DA masks segments before this selection takes place, so we expect the same reduction of selection cost as with DSA.
.
W2-1. Accuracy loss on the excluded sources (
cwe,structured_data)Our additional analysis of every incorrect response shows that the accuracy loss mainly comes from simple errors of protocol use, which we expect post-training to correct (see details in
Additional Analysis Aof the general response). It covers the vanilla model and DA on the tasks with the largest accuracy drops, including these two. Comprehension or reasoning errors do not become more frequent with DA: they occur in 7% of DA responses against 13% of vanilla responses on Gemma, and in about 1% of both on Qwen.cweloss. Adding three short rules raises Gemma's accuracy with DA from 69.8% to 84.0% on the 96 held-out items (see details inAdditional Experiment Bof the general response). Note: the paper's 70.3% is the same DA run over all 128 items.