acceptodds
Withdrawn from ICLR 2027

On-Demand Attention: Language Models Know When to Recall

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, although the benefit of global access varies across prediction positions. We find that, before global attention is computed for the current step, the decoding states available after local computation in frozen pretrained models already contain information predictive of its benefit over local attention. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method: after local computation, a lightweight recall head decides whether to recompute the current step with global attention. ODA trains only the recall head with modest data and compute budgets, leaving pretrained weights unchanged and retaining the complete historical KV cache so that information skipped at one step remains available for later access. Experiments across model scales and families, including hybrid attention backbones, show that ODA recovers most of the performance lost under local attention while substantially reducing the frequency of global attention. Controlled long-context measurements in vLLM further show that GPU-side conditional execution translates fewer global reads into practical decoding speedups over full attention. These findings show that pretrained decoding states can support both token prediction and decisions about accessing distant information, allowing models to allocate global computation as needed during decoding. Code will be available upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, long-context inference, adaptive attention, conditional computation, efficient decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2eVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • paper proposes local attention first, with a learned head deciding whether to rerun the current step using full attention or not
  • authors exploit the fact that: decoding states available after local computation in pretrained models already contain signals predictive of the benefit of global attention over local attention
  • the method acheives upto 2.65x speedups while decoding due to reduced "full" attention calls
  • evaluations on long-context benchmarks show competitive performance
Critical strengths
  • the paper identifies that pretrained models can predict when to use full attention or not, and uses this finding for a method to speed up decoding. the method preserves quality across tasks and models: both attention only and hybrids

  • the authors ablate that this predictability of full attention access performs better than random access to full attention -- this means that this predictability is not noise

  • the method is very practical, and can be readily integrated with inference serving infra such as vLLM

Critical weaknesses
  • my major complaint is its hard to know where the accuracy-vs-decode-time trade-off when these are presented in different tables -- right now, decoding speed ups and near-quality to full attention are demonstrated in different configurations

  • its unclear whether the method provides speedups on the reasoning tasks such as AIME (math) or LiveCodeBenchv6 (code) where lots of tokens are required to decode -- right now RULER and LongBench requires decoding on ~100 tokens.

Questions for authors
  • can the authors provide a clear figure -- where x axis is decoding time, y axis is accuracy -- to show how different configurations trade-off accuracy with decoding time

  • could the authors provide evaluation on AIME and LCBv6 where lots of tokens are required to be decoded and show a similar accuracy-decoding time trade-off there? since the authors use pretrained models, hopefully it shouldn't be hard to eval on these tasks :)

Reviewer_67PQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes On-Demand Attention (ODA), a local-first decoding framework for efficient long-context language model inference. Experiments across five models from the Qwen3, Qwen3.5, and Gemma families demonstrate near-full-attention performance on RULER and LongBench with fewer global-attention calls. The authors also implement conditional execution in vLLM, reporting up to 2.65× decoding throughput under controlled long-context settings.

Critical strengths

ODA provides a meaningful approach to adaptive long-context computation without modifying pretrained backbone parameters. Its recall head exploits existing hidden representations to estimate the benefit of global attention, while retaining the complete KV cache for future access. And the evaluation covers five models across RULER and LongBench, providing evidence that selective global access can preserve most of the performance of full attention. Furthermore, the implementation of conditional CUDA Graph execution in vLLM demonstrates that reduced global-attention computation can translate into measurable decoding speedups under controlled conditions, rather than remaining a theoretical FLOPs reduction.

Critical weaknesses

The quality evaluations and runtime benchmarks use substantially different operating conditions. The reported 2.65× decoding speedup at 512K context relies on a prescribed 12.5% Full-call schedule rather than the recall head's actual decisions, and no corresponding generation-quality evaluation is provided at this context length. Although ODA substantially outperforms random global-access selection, random routing is a relatively weak baseline. The paper does not provide online generation comparisons against inexpensive adaptive policies based on local prediction confidence or entropy under matched Full-call budgets.

Questions for authors
  1. Can the authors report generation quality and actual decoding throughput using the same learned recall policy on identical workloads?
  2. How does ODA compare with simple confidence-based or entropy-based routing policies under matched Full-call budgets?
  3. The Qwen3-1.7B recall head contains approximately 28.3 million FP32 parameters, and a Local forward pass is executed even when Full recomputation is subsequently required. How do smaller or simpler recall heads affect the quality-efficiency trade-off?
Reviewer_kYA1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces On-Demand Attention, a decoding method that first runs the frozen language model with local sliding-window attention and uses a lightweight recall head to predict whether recomputing the step with full attention will improve next-token prediction. The authors evaluate generation quality across Qwen3 and Gemma models on RULER and LongBench and build a conditional CUDA Graph runtime in vLLM to measure decoding throughput.

Critical strengths
  • The paper shows that hidden states after a local forward pass in frozen pretrained models already carry enough signal to predict the benefit of full attention.
  • Evaluations across dense and hybrid architectures confirm that selective full-attention retries recover nearly all accuracy lost by purely local attention.
  • The custom vLLM runtime using device-side conditional CUDA Graphs and ring buffers is well engineered and avoids host-device synchronization overhead.
Critical weaknesses
  • Running a complete local forward pass through all backbone layers before discarding it and recomputing the entire model on every recall is structurally wasteful, especially since MLP costs are paid twice.
  • Quality tables require 42% to 88% full-attention calls across 4K to 64K contexts, yet Table 20 shows that even a 50% call rate causes net slowdowns at every context length up to 64K.
  • Experiments only compare against StreamingLLM and omit modern training-free or lightweight sparse KV retrieval baselines such as Quest, RefreshKV, or SeerAttention-R.
  • Despite motivating the work with extended reasoning workloads, all experiments disable thinking mode, cap outputs at 512 tokens, and measure runtime only on a 1.7B model at batch size 1.
  • Retaining the entire historical KV cache yields zero memory reduction, and Table 16 shows the recall rate oscillates between 76%, 50%, and 71% as training progresses.
Questions for authors
  • On LongBench, Qwen3-1.7B and Qwen3-8B require over 80% full-attention calls, while Table 20 shows that even a 50% call rate is slower than full attention up to 64K tokens. What is the actual end-to-end wall-clock speedup when running the learned policy directly on the 4K to 64K benchmarks in Tables 1 and 2?
  • How does recomputing the entire backbone compare in accuracy and latency against page-level sparse attention methods like Quest that select KV blocks per layer without running MLPs twice?
  • Why does the full-call rate in Table 16 fluctuate from 76.22% to 50.11% and back to 71.43% across training updates, and how should practitioners pick a checkpoint without evaluating on the target benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.