- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper tries to solve the problem of out-of-core execution of the attention kernel so that the attention won't cause OOM. It uses CQS decomposition to recursively divide the original attention computations into smaller subsequence attention computations, and uses the masks to avoid redundant computation of the diagonal blocks. On a single A100, its numerical accuracy is comparable to FlashAttention-2(FA2). The execution time of the kernel is substantially slower than the original FA2, and without CPU accumulators, the memory footprint can be even larger than FA2. While the paper proves that they can tackle cases where the original FA2 becomes OOM (16M seqlen), the runtime of the whole 16M forward+backward kernels seems to exceed 4.4 hours.
Critical strengths
- The paper provides an implementation of the out-of-core attention with both forward and backward support, and makes some kernel engineering efforts.
Critical weaknesses
- It seems that we don't need such complex decomposition for out-of-core attention. As shown by FlashAttention, the computation of attention can be naturally decomposed into blocks. Using such rectangular block decomposition, we can easily reuse the mature attention kernels without adding complicated masks, as shown by Ring Attention (Liu et al.,2024).
- This work is directly based on CQS-Attention (Bian&Somani2025), and only adds "new features such as recursive decomposition, a pluggable inner-kernel interface, workload scheduling, and OOM guardrails", which give little meaningful technical advances.
- The evaluation settings seem to be unrealistic; computing attention for more than hours is unacceptable for real-world workloads. The evaluation workloads only considered a single kernel, without showing the necessity of the out-of-core attention inside an end-to-end realistic LLM workload.
Questions for authors
- What is the advantage of CQS decomposition over conventional rectangular decomposition?
- Could the author clarify the contribution beyond the existing CQS-attention?
- Can the author provide a realistic workload where we need out-of-core attention?