- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LASA learns query-dependent sparse-attention budgets for individual layers and KV heads while freezing the backbone and a distilled block ranker. A lightweight predictor selects blocks through learned thresholds, trained using output-level distillation and a relaxed budget constraint. The paper combines this budgeting mechanism with hard-gated training and a vLLM implementation for efficient reasoning-model decoding.
Critical strengths
-
The same-ranker comparisons and the trained layer–head-bias ablation provide useful evidence that query-dependent budgeting contributes beyond ranking quality alone.
-
Evaluation covers four reasoning models and six benchmarks. Reporting generation lengths and including dense-layer accesses in the density metric makes the cost comparison more informative.
-
The systems evaluation uses a strong dense-attention baseline and includes a complete AIME25 workload. The reported generation time decreases from 19.11 to 5.59 minutes at similar observed accuracy, demonstrating practical benefits beyond kernel-level sparsity.
Critical weaknesses
-
The precision of the near-dense accuracy claim remains unclear. Repeated sampling improves estimation on the fixed problems, but AIME contains only 30 distinct problems per year. Paired, problem-level uncertainty estimates would help establish how reliably accuracy is retained at the lowest budgets.
-
The deployment evidence is narrower than the quality evaluation. Most timing results concern Qwen3-8B on H20, prefill remains dense, and the 64K extension primarily establishes efficiency. These experiments support useful decoding gains but leave robustness across longer contexts and other deployment configurations less established. These limitations motivate a weak-accept recommendation rather than clear acceptance.
Questions for authors
-
Can you provide paired, problem-clustered uncertainty intervals for accuracy differences from dense attention, particularly at the sparsest AIME24/25 operating points, and specify a tolerance for “near-full-attention accuracy”?
-
Is corresponding quality and realized-budget evidence available for the 64K efficiency extension? Please distinguish this from the already demonstrated reasoning evaluations beyond the 16K training length.
Thank you for the additional experiments and the paired analysis by problem. The low-budget AIME point estimates meet the proposed tolerance, but nine of the 16 reported 95% confidence intervals still include accuracy drops of more than two percentage points relative to dense attention. I am keeping my rating at 3.