acceptodds
Withdrawn from ICLR 2027

Stream-CQSA: Exact Out-of-Memory Recovery for Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context large language models are limited not only by attention cost but also by out-of-memory (OOM) failures. A selected attention call may not fit in available device memory even when the kernel is optimized. Exact and approximate attention methods reduce memory use, but every fixed implementation still has a device-specific capacity boundary. We introduce Stream-CQSA, an attention-level OOM recovery framework based on CQS decomposition, derived from the theory of cyclic quorum sets (CQS). Stream-CQSA recursively partitions an infeasible attention call into independent subsequence tasks, executes each with a compatible inner kernel, and recomposes the local statistics to recover the full attention output. This recovery is exact relative to the wrapped attention kernel, whether that kernel is exact or approximate. Compared with FlashAttention-2, the major baseline, our native Stream-CQSA kernel improves 16-bit forward-output error relative to a dense float64 reference and matches 16-bit backward-gradient error where FlashAttention-2 fits in the GPU memory. At the longest feasible baseline length, it costs – the forward runtime and – the forward–backward runtime. Beyond that sequence length boundary, our method continues to return an output while FlashAttention-2 OOMs. Stream-CQSA is therefore not a faster attention method. Instead, it converts memory-capacity failure into a recoverable execution path by trading extra compute, host-device transfer, and recomposition for completion.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
long-context attention, out-of-memory recovery, cyclic quorum sets, exact attention decomposition, stream attention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2dCGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to solve the problem of out-of-core execution of the attention kernel so that the attention won't cause OOM. It uses CQS decomposition to recursively divide the original attention computations into smaller subsequence attention computations, and uses the masks to avoid redundant computation of the diagonal blocks. On a single A100, its numerical accuracy is comparable to FlashAttention-2(FA2). The execution time of the kernel is substantially slower than the original FA2, and without CPU accumulators, the memory footprint can be even larger than FA2. While the paper proves that they can tackle cases where the original FA2 becomes OOM (16M seqlen), the runtime of the whole 16M forward+backward kernels seems to exceed 4.4 hours.

Critical strengths
  • The paper provides an implementation of the out-of-core attention with both forward and backward support, and makes some kernel engineering efforts.
Critical weaknesses
  • It seems that we don't need such complex decomposition for out-of-core attention. As shown by FlashAttention, the computation of attention can be naturally decomposed into blocks. Using such rectangular block decomposition, we can easily reuse the mature attention kernels without adding complicated masks, as shown by Ring Attention (Liu et al.,2024).
  • This work is directly based on CQS-Attention (Bian&Somani2025), and only adds "new features such as recursive decomposition, a pluggable inner-kernel interface, workload scheduling, and OOM guardrails", which give little meaningful technical advances.
  • The evaluation settings seem to be unrealistic; computing attention for more than hours is unacceptable for real-world workloads. The evaluation workloads only considered a single kernel, without showing the necessity of the out-of-core attention inside an end-to-end realistic LLM workload.
Questions for authors
  • What is the advantage of CQS decomposition over conventional rectangular decomposition?
  • Could the author clarify the contribution beyond the existing CQS-attention?
  • Can the author provide a realistic workload where we need out-of-core attention?
Reviewer_t8kXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed a Stream-CQSA to deal with the problem of extremely long attention computation to prevent OOM. Based on their designed experiments, the proposed method showed the better performance than the baseline.

Critical strengths

The paper is very poor. Like a auto research work or the work which lacks of completeness. No strength could be found.

Critical weaknesses
  1. Lack of end-to-end test. The experiments only evaluated the proposed method and the baseline on kernel level.
  2. Sequence is divided into sub-sequences, also breaking the causal relation, but the method is seems not recover the causal relation among the tokens.
  3. How the kernel deal with the different exact implementations of attention, such as MHA, GQA...
Questions for authors

Related to the weaknesses.

Reviewer_aZrrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Stream-CQSA partitions attention into cyclic-quorum-set subsequences and recomposes their local statistics. Recursive decomposition and controlled tensor residency allow otherwise infeasible calls to complete. On synthetic Q/K/V workloads, the method preserves the tested attention computation within numerical precision and extends the GPU-memory boundary, at additional computation and transfer cost.

Critical strengths
  1. The decomposition has a concrete correctness and implementation account. CQS masking assigns each retained token-pair contribution once, while global softmax statistics permit forward and backward recomposition. Stable merging and mask-related kernel studies address meaningful numerical and execution issues.
  2. Recovery and its cost are measured explicitly. At 16M tokens, the host-accumulator configuration completes forward attention in approximately 3277 seconds using 41.5 GiB, where FlashAttention-2 OOMs. Forward/backward error checks and dense/LongNet-style reference-kernel experiments provide evidence beyond this capacity result.
Critical weaknesses
  1. The benefit of CQS over a straightforward streamed blockwise baseline is not established. A natural alternative is to stream Q/K/V blocks through GPU memory and merge partial softmax statistics, keeping only the active input blocks resident. Section 2 discusses related out-of-core approaches, but the comparisons in Sections 4.1–4.4 do not include this matched alternative. Such a comparison is needed to attribute the memory/runtime trade-off to the CQS grouping introduced in Section 3. The synthetic attention workloads also leave its practical operating regime insufficiently demonstrated (Sections 4.1 and 5).
  2. Packed variable-length support is not demonstrated. Section 3.3 defines the CQS mask that removes duplicated interactions, while Section 4.1 evaluates causal attention with B=1. For packed variable-length inputs, the original attention mask and sequence boundaries must also be preserved through decomposition and gathering. The main text does not explain or evaluate this case. Clarifying the supported masks and checking forward/backward consistency and efficiency would establish the framework's practical scope; the CQS coverage argument alone does not provide that validation.
Questions for authors
  • How does CQS compare with simple streamed query/key/value blocks and partial-softmax merging under the same GPU/host memory budgets, precision, and output placement? Please report peak memory and complete runtime, including recovery overhead.
  • How is the CQS mask in Section 3.3 combined with the original task mask for packed variable-length inputs? Please explain boundary/index handling and provide a representative correctness and performance check.
  • Can a representative model prefill or training case illustrate when the recovery path is useful, with the relevant tensor sizes and end-to-end cost reported?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.