- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets the serialized selection-to-attention dependency in dynamic sparse attention (DSA) for long-context LLM decoding. Its main idea is to overlap block selection with attention: an EMA-based predictor speculates the next top-K block set, speculative attention runs on the predicted set, and a FlashAttention-based online-softmax repair kernel folds in missed blocks after the true set is known. A profiling-guided speculation budget keeps speculative work off the critical path. Experiments across seven LLMs, three DSA methods (Quest, InfLLM-v2, SeerAttention-R), and five benchmarks report decoding speedups with essentially unchanged downstream accuracy.
Critical strengths
1: The motivation is grounded in measured temporal locality and GPU headroom, and the ablation shows that incremental repair is necessary for the gain. This is consequential because the paper's core claim depends on selection and attention being concurrently schedulable without extending the critical path.
2: The validation is broad: multiple LLMs, both training-free and trainable DSA methods, and several accuracy benchmarks. This matters because it makes the speedups less likely to be a single-configuration artifact and supports the claim that accuracy is preserved.
Critical weaknesses
1: The utilization evidence is too coarse. The paper reports low overall decode-stage SM/L2/DRAM utilization, but does not break this down by selection, attention, and repair. This matters because overlap is only justified if selection leaves specific resources idle that speculative attention can use; aggregate utilization cannot establish that.
2: Peak memory impact is not discussed. PRR adds predictor state, speculative indices, gather/repair workspaces, and extra KV reads, but the paper does not report peak memory or whether KV-cache capacity or maximum batch size changes. This matters for long-context serving, where KV cache capacity often limits batch size and a small memory regression can reduce throughput.
Questions for authors
- Can you break down SM, L2, and DRAM utilization separately for selection, speculative attention, and repair? This matters because the overlap argument requires showing that selection leaves the specific resources speculative attention needs, not just low aggregate utilization.
- Can you report peak GPU memory breakdown and the maximum batch size for baseline DSA versus PRR under the same context length? This matters because extra predictor state/workspaces or KV reads could reduce KV-cache capacity or serving batch size even if average latency improves.
- Beyond mean absolute error, can you provide a tighter error bound or statistics on error accumulation over long decoding steps? This matters because PRR retains over-predicted blocks P/A in the softmax and is not bitwise equivalent, so long-run numerical drift needs stronger evidence.
- How robust is PRR at large batch sizes, especially when per-request context length is shorter? This matters because the paper's gains shrink with batch size, and long-context serving may mix batch and length regimes where resource headroom and critical-path assumptions differ.