acceptodds
Under review as a conference paper at ICLR 2027

LASA: Learnable Adaptive Sparse Attention for Efficient and Accurate Reasoning

Abstract

In long chain-of-thought reasoning, every decoding step reads an ever-growing KV cache, and this memory traffic becomes the inference bottleneck. Sparse attention reduces this cost by accessing only a subset of the KV cache. Most existing methods focus on improving the quality of ranking KV tokens, while using a fixed token count (top-) for budgeting. Some adaptive-budget methods instead vary token counts according to a fixed target for retained attention mass (top-). Both rely on a priori targets shared across layers, KV heads, and queries, rather than learning from model output quality where more or less computation is needed. We introduce (Learnable Adaptive Sparse Attention), which makes budgeting a learned and fine-grained decision. Given a block ranking distilled from dense attention, a M-parameter predictor sets a threshold for each layer, KV head, and decoding step, and uses simple elementwise comparisons to select blocks whose normalized ranker scores exceed it. The predictor is trained end-to-end through output-level distillation under a relaxed token budget, with both the backbone and ranker frozen. Our extensive evaluation shows that substantially improves the accuracy–density Pareto frontier over existing sparse attention methods, retaining near-full-attention accuracy while under 1K average attended tokens. Our efficient kernels and vLLM integration translate these savings into up to attention-kernel and end-to-end decoding speedups over dense attention on H20 GPUs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, adaptive computation, efficient inference, large reasoning models, long-context decoding, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8SkGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LASA learns query-dependent sparse-attention budgets for individual layers and KV heads while freezing the backbone and a distilled block ranker. A lightweight predictor selects blocks through learned thresholds, trained using output-level distillation and a relaxed budget constraint. The paper combines this budgeting mechanism with hard-gated training and a vLLM implementation for efficient reasoning-model decoding.

Critical strengths
  1. The same-ranker comparisons and the trained layer–head-bias ablation provide useful evidence that query-dependent budgeting contributes beyond ranking quality alone.

  2. Evaluation covers four reasoning models and six benchmarks. Reporting generation lengths and including dense-layer accesses in the density metric makes the cost comparison more informative.

  3. The systems evaluation uses a strong dense-attention baseline and includes a complete AIME25 workload. The reported generation time decreases from 19.11 to 5.59 minutes at similar observed accuracy, demonstrating practical benefits beyond kernel-level sparsity.

Critical weaknesses
  1. The precision of the near-dense accuracy claim remains unclear. Repeated sampling improves estimation on the fixed problems, but AIME contains only 30 distinct problems per year. Paired, problem-level uncertainty estimates would help establish how reliably accuracy is retained at the lowest budgets.

  2. The deployment evidence is narrower than the quality evaluation. Most timing results concern Qwen3-8B on H20, prefill remains dense, and the 64K extension primarily establishes efficiency. These experiments support useful decoding gains but leave robustness across longer contexts and other deployment configurations less established. These limitations motivate a weak-accept recommendation rather than clear acceptance.

Questions for authors
  1. Can you provide paired, problem-clustered uncertainty intervals for accuracy differences from dense attention, particularly at the sparsest AIME24/25 operating points, and specify a tolerance for “near-full-attention accuracy”?

  2. Is corresponding quality and realized-budget evidence available for the 64K efficiency extension? Please distinguish this from the already demonstrated reasoning evaluations beyond the 16K training length.

1 comment

Thank you for the additional experiments and the paired analysis by problem. The low-budget AIME point estimates meet the proposed tolerance, but nine of the 16 reported 95% confidence intervals still include accuracy drops of more than two percentage points relative to dense attention. I am keeping my rating at 3.

Reviewer_Hr1vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LASA, a sparse decoding attention method that learns how much computation to allocate to each layer, KV head, and decoding step. LASA uses a frozen, attention-distilled block ranker and trains a lightweight query-conditioned predictor to select blocks by thresholding their estimated attention masses. The predictor is optimized through output-distribution distillation with a soft budget penalty. A multiplicative gating formulation allows excluded blocks to receive surrogate gradients during training. The authors also develop kernels and integrate LASA into vLLM. Experiments across four reasoning models and six benchmarks show a favorable accuracy–KV-density trade-off.

Critical strengths

The paper studies an important problem and is well-written. Easy to follow. The empirical performance is promising.

Critical weaknesses

Novelty of the paper is limited. The problem of dynamic/learnable budget is not new and well studies but the paper omits many existing related works and baselines. I can name a few here [1, 2].

Bui, Ngoc, et al. "Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction." arXiv preprint arXiv:2605.09649 (2026). Huang, Yen-Chieh, et al. "Kv admission: Learning what to write for efficient long-context inference." arXiv preprint arXiv:2512.17452 (2025).

Questions for authors
  • How does LASA compare to other learnable KV eviction methods such as DBTrimKV [1] in terms of performance and efficiency?

  • Budget predictability beyond the training range remains unclear. How do mean and upper-quantile selected-block counts evolve beyond the 16K training context?

  • Does the same-ranker advantage over top-p persist on coding tasks and R1-Llama-8B?

  • Why is 16-token LASA evaluated at only one budget per benchmark? Could you provide a budget sweep with accuracy, KV density, and latency, and clarify how the reported operating points were selected? Also plotting both 16 token block with 64-token block seems to be misleading. At matched accuracy, how do 16-token and 64-token blocks compare in complete-workload latency?

  • What additional KV-cache memory does the indexer require, and how does this affect maximum serving concurrency?

[1] Bui, Ngoc, et al. "Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction." arXiv preprint arXiv:2605.09649 (2026).

Reviewer_chZwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LASA, a learnable adaptive sparse attention method for efficient long reasoning. The authors argue that existing methods mainly focus on ranking important KV blocks, while the attention budget is usually controlled by fixed top-k or shared top-p rules. Since different layers, KV heads, and queries may have different attention requirements, such fixed budgeting rules can lead to inefficient computation allocation.

LASA builds on the SeerAttention-R block ranker and introduces a lightweight query-conditioned threshold predictor to dynamically select KV blocks. The predictor is trained using vocabulary-level KL distillation and a relaxed budget constraint, with the backbone and ranker frozen. The authors also propose a hard-gating training implementation and develop optimized indexing and sparse decoding kernels integrated into vLLM.

Experiments on four reasoning models and six benchmarks show improved accuracy–density trade-offs over several sparse attention baselines. LASA achieves up to 4.1× end-to-end decoding speedup on H20 GPUs and approximately 3.4× speedup on a complete AIME25 workload while maintaining accuracy close to full attention.

Critical strengths

1. Well-motivated fine-grained budgeting with convincing empirical improvements.

I find the separation between KV ranking and budgeting useful. Figure 1 shows that attention concentration and output sensitivity vary substantially across layers, heads, and decoding steps. This provides a reasonable motivation for learning different budgets instead of applying the same top-k or top-p target everywhere.

The method is also simple and practical. The same-ranker comparison in Figure 5 and the predictor ablation in Table 1 provide meaningful evidence that query-dependent budgeting can improve the accuracy–density trade-off. In particular, LASA performs better than a learned layer-head-only threshold while using fewer KV tokens.

2. Strong system implementation and practical evaluation.

The main strength of this paper is its practical efficiency. The authors implement fused indexing and dynamic sparse decoding kernels and integrate them into vLLM, rather than only reporting theoretical sparsity or FLOPs reduction.

The experiments cover multiple reasoning models, benchmarks, sequence lengths, and batch sizes. I also appreciate the comparisons with optimized dense and sparse attention implementations, as well as the analysis of generation-length inflation.

The complete AIME25 workload is particularly convincing: LASA reduces generation time from 19.11 to 5.59 minutes while maintaining comparable accuracy. This is a substantial practical improvement and an important reason why I lean toward acceptance.

Critical weaknesses

1. Limited methodological novelty, and insufficient evidence isolating the benefit of the proposed learning objective.

While the overall framework is effective, I am not fully convinced by the methodological novelty of the learned budgeting approach.

Several important components are already established in prior work. SeerAttention-R, which LASA directly builds upon, already supports threshold-based adaptive KV-block selection. Using output-level KL distillation to preserve model behavior under reduced computation is also a common approach. More generally, learned sparse selection combined with output distillation and computation-budget constraints has been explored in previous work such as DynamicViT. Elastic Attention also investigates learned adaptive sparsity in pretrained LLMs, although with a different routing granularity.

I agree that LASA combines these ideas in a different and useful way, particularly through its query-conditioned threshold predictor and efficient implementation. However, the paper could better clarify which aspects of this combination are responsible for the improvements.

The current ablations do not sufficiently isolate the benefit of vocabulary-level KL supervision. Figure 5 mainly compares LASA with fixed top-k, fixed top-p, and thresholds with less expressive conditioning. These results support learned fine-grained budgeting, but they do not establish that output-level supervision is particularly important compared with other objectives for learning the same policy.

A comparison using the same threshold predictor trained with attention-output reconstruction instead of vocabulary KL would be helpful. Similarly, a query-conditioned learned top-k or top-p predictor trained using the same output-level objective would clarify whether LASA's advantage comes mainly from the threshold parameterization, the training objective, or the efficiency of its implementation.

I consider this an important limitation because the specific algorithmic contribution is less clear than the practical system improvement.

2. Limited evaluation of budget stability and accuracy beyond the training context length.

The threshold predictor is trained on packed sequences of approximately 16K tokens. Appendix D.1 specifies length-bin boundaries of 16, 32, 64, 128, and 256 blocks for the 64-token-block setting. Attention rows beyond 256 blocks, corresponding to approximately 16K tokens, are therefore not explicitly included in the budget penalty.

However, the reasoning evaluation allows much longer generations, and the efficiency experiments extend to 32K and 64K contexts.

Although the LSE-normalized threshold has some useful properties for controlling the number of selected blocks, this does not directly guarantee that the learned budgeting policy preserves output quality or stays within its intended budget at longer context lengths.

The paper provides strong evidence of decoding efficiency at long contexts, but a more systematic evaluation of actual selected-block counts and model quality beyond the training length would strengthen the results. It is also unclear how the length-bin boundaries are configured for the 16-token-block variants.

This matters because stable budget allocation across long reasoning trajectories is one of the main motivations of the paper.

Questions for authors

Q1. Contribution of output-level supervision.

Can the authors compare vocabulary-level KL distillation against an attention-level objective, such as attention-output reconstruction, while keeping the ranker, threshold predictor, and budget constraint fixed?

Since output-level KL is already a standard distillation objective, I think this comparison is important for understanding its specific contribution to learned budgeting. Also, the proposed KL measures next-token distribution agreement rather than final reasoning correctness. It would be useful to clarify this distinction.

Q2. Comparison with other learned budgeting policies.

How does LASA compare with a query-conditioned learned top-k or top-p predictor trained using the same vocabulary-level KL and budget constraint?

Since threshold-based selection was already considered in SeerAttention-R, I would like to understand whether the main improvement comes from learning the threshold, its particular parameterization, or its lower inference overhead. A comparison at matched KV density and actual decoding latency would be especially useful.

Q3. Generalization beyond the training context length.

How do the selected-block counts and output quality change beyond the 16K training context, particularly at 32K and 64K? Can the authors report the mean and high-percentile budget usage across different context-length ranges?

Also, are the budget-bin boundaries adjusted for the 16-token-block variants? The current appendix explicitly describes these boundaries for 64-token blocks but does not clearly explain the corresponding 16-token configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.