- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoWindow Attention (CoWA), which distributes long-range causal context across complementary KV-head windows while retaining shared local and sink windows. The attention pattern is determined by token positions without learned routing or additional indexing. The same design supports training, prefill, and autoregressive decoding, with exact backward computation and global KV-head indexing for tensor-parallel execution.
Critical strengths
S1: The paper provides a unified position-based rule for training, prefill, and autoregressive decoding, with exact backward gradients and consistent complementary windows under TP sharding. The implementation avoids additional routing or indexing overhead. The 128K/TP=8 experiments demonstrate substantial efficiency and memory benefits, while the authors also acknowledge relevant limitations. The main strength, in my view, is the consistent implementation across training and inference, rather than the attention pattern alone.
S2: The window-matched ablation provides useful evidence that complementary coverage improves recall independently of per-head window width, while the matched-budget experiments further support this interpretation. The scaling experiments from 0.6B to 14B and multi-seed evaluations at 14B/32B also strengthen the empirical results. The evaluation covers both efficiency and downstream quality. The results suggest that performance is largely preserved, although some tasks still show degradation.
Critical weaknesses
W1: I think the paper should more clearly distinguish its contributions from closely related approaches, particularly SPAttention (AAAI 2026) and Elastic Attention (ICML 2026). SPAttention already uses deterministic per-head distance bands whose union covers the causal history, which appears conceptually similar to the complementary coverage principle proposed here. The brief discussion in the footnote does not sufficiently clarify what is fundamentally new. Elastic Attention is also relevant, as it learns per-head attention mode assignments, providing a useful contrast to CoWA's position-defined design. Given the emphasis on tensor parallelism, a discussion of Star Attention and Ring Attention would further help establish the paper's scope, although these methods address somewhat different problems. I suggest expanding the related-work discussion to clarify the specific contributions in GQA support, training–decoding consistency, and parallel execution. A matched-budget comparison with SPAttention would also help establish whether CoWA offers an empirical advantage over this closely related design.
W2: While the reported operator-level speedups and memory savings are encouraging, the paper does not provide end-to-end training or serving throughput results. It is therefore unclear how much of the measured improvement translates into practical system-level gains. The efficiency evaluation is also limited to TP=8 and (H_k=8), without sensitivity analysis across other configurations.
Questions for authors
Q1. What is the main methodological difference between CoWA and SPAttention, beyond the GQA/KV-head formulation and implementation details? A matched-budget comparison would help clarify the practical significance of these differences. The footnote also states that the released MEHC implementation differs from its published description. Could the authors identify the specific differences and provide evidence for this claim?
Q2. Elastic Attention is particularly relevant because it also considers per-head attention pattern assignment, but uses a learned strategy rather than a deterministic one. How does CoWA compare under matched sparsity, for example on RULER or LongBench-E? Could the two approaches be complementary?
Q3. Could the authors report end-to-end training throughput or step time at 4K and 32K, as well as decoding throughput and peak memory at 128K? The current operator-level results do not establish the overall system benefit. How sensitive are these results to TP degree and KV-head count?
5 comments
Response (W1/Q2) to Reviewer sCCH
W1 and Q2. Elastic Attention and distributed execution
Elastic Attention selects head computation modes through input-dependent routing; CoWA assigns complementary windows through positions. These mechanisms offer a possible combination: a data-driven allocator could adapt window boundaries or assignments while enforcing collective coverage. An allocation learned offline could retain a regular execution schedule.
Our RULER comparisons use FullAttn, MoBA, and DSA at 14B, and FullAttn at 32B. Appendix B, Table 10 gives the task breakdown; the overall scores are:
| Model Size | Method | RULER 32K | RULER 128K with YaRN |
|---|---|---|---|
| 14B | FullAttn | 89.42 | 65.84 |
| 14B | MoBA | 87.17 | 61.52 |
| 14B | DSA | 87.67 | 64.05 |
| 14B | CoWA | 89.13 | 66.6 |
| 32B | FullAttn | 92.7 | 82.03 |
| 32B | CoWA | 92.58 | 81.78 |
These RULER results show that CoWA maintains performance close to its FullAttn controls at 32K and 128K. The table compares methods under our training protocol; Elastic Attention instead adapts a frozen pretrained backbone through a learned router. The approaches are complementary: a coverage-preserving window allocation could be learned for a target workload and then executed with CoWA's regular attention kernel.
Ring Attention distributes sequence blocks and overlaps KV communication with exact attention computation. Star Attention combines blockwise context processing with global attention for query and response tokens. CoWA distributes complementary supports across KV heads and preserves them under head tensor parallelism. We will clarify these distinct decompositions in the related-work discussion.
Response (W2/Q3) to Reviewer sCCH
W2 and Q3. End-to-end efficiency and parallel configurations
CoWA improves end-to-end throughput in both Megatron-LM and vLLM, where we integrate it by replacing the attention kernel.
Training
We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8 and the training settings in Appendix A.3. Batch size is matched at each model scale. FullAttn and CoWA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.
| Model | Method | 4K Step (MS) | 4K Tokens/s | 32K Step (MS) | 32K Tokens/s |
|---|---|---|---|---|---|
| 0.6B | FullAttn | 84 | 3,123,088 | 105 | 2,489,200 |
| 0.6B | MoBA | 185.2 | 1,416,699 | 228.6 | 1,143,530 |
| 0.6B | DSA | 82.9 | 3,162,770 | 76.4 | 3,420,169 |
| 0.6B | CoWA | 72.6 | 3,611,448 | 48.6 | 5,381,947 |
| 1.7B | FullAttn | 212 | 2,478,176 | 288 | 1,818,080 |
| 1.7B | MoBA | 323.4 | 1,624,575 | 529.5 | 988,849 |
| 1.7B | DSA | 210.8 | 2,491,814 | 232.1 | 2,255,608 |
| 1.7B | CoWA | 199.5 | 2,633,526 | 177.7 | 2,946,757 |
| 4B | FullAttn | 775 | 1,353,216 | 1,217.00 | 861,568 |
| 4B | MoBA | 1,175.60 | 892,082 | 2,229.10 | 470,390 |
| 4B | DSA | 770.8 | 1,360,542 | 982.9 | 1,066,771 |
| 4B | CoWA | 730 | 1,436,587 | 754.7 | 1,389,257 |
| 8B | FullAttn | 1,711.00 | 919,136 | 2,363.00 | 665,728 |
| 8B | MoBA | 2,229.40 | 705,425 | 3,804.80 | 413,452 |
| 8B | DSA | 1,705.60 | 922,046 | 2,029.50 | 775,129 |
| 8B | CoWA | 1,652.80 | 951,499 | 1,704.40 | 922,950 |
| 14B | FullAttn | 3,635.00 | 576,960 | 4,927.00 | 425,600 |
| 14B | MoBA | 4,499.00 | 466,159 | 7,482.00 | 280,263 |
| 14B | DSA | 3,626.00 | 578,392 | 4,336.00 | 483,610 |
| 14B | CoWA | 3,538.00 | 592,778 | 3,760.00 | 557,694 |
CoWA has the highest throughput among the four configurations at every scale and both lengths. At 4K, it achieves 1.03 to 1.16× FullAttn's throughput. At 32K, it achieves 1.31 to 2.16× FullAttn, 1.99 to 4.71× MoBA, and 1.15 to 1.57× DSA, showing larger system gains as attention work grows with context length.
Serving
We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.
| Model | Method | Prefill Input Tokens/S | Decode Output Tokens/S |
|---|---|---|---|
| 14B | FullAttn | 46,746 | 1,542 |
| 14B | MoBA | 21,743 | 471 |
| 14B | DSA | 66,310 | 1,987 |
| 14B | CoWA | 121,402 | 2,628 |
| 32B | FullAttn | 21,520 | 891 |
| 32B | CoWA | 60,352 | 1,574 |
CoWA's prefill/decode throughput gains over FullAttn are 2.60×/1.70× at 14B and 2.80×/1.77× at 32B. At 14B, the gains are 5.58×/5.58× over MoBA and 1.83×/1.32× over DSA.
TP/DP configurations and KV-head count
We keep eight GPUs, total concurrency 16, and eight KV heads while varying TP/DP. Concurrency per DP replica is 16 for TP8/DP1, eight for TP4/DP2, and four for TP2/DP4.
Prefill-heavy workload, input tokens/s
| Model | Configuration | FullAttn | CoWA | Speedup |
|---|---|---|---|---|
| 14B | TP8/DP1 | 46,747 | 121,402 | 2.60× |
| 14B | TP4/DP2 | 51,421 | 125,942 | 2.45× |
| 14B | TP2/DP4 | 55,215 | 144,845 | 2.62× |
| 32B | TP8/DP1 | 21,520 | 60,352 | 2.80× |
| 32B | TP4/DP2 | 23,672 | 62,789 | 2.65× |
| 32B | TP2/DP4 | 25,418 | 73,084 | 2.88× |
Decode-heavy workload, output tokens/s
| Model | Configuration | FullAttn | CoWA | Speedup |
|---|---|---|---|---|
| 14B | TP8/DP1 | 1,542 | 2,628 | 1.70× |
| 14B | TP4/DP2 | 1,404 | 2,605 | 1.86× |
| 14B | TP2/DP4 | 1,168 | 2,611 | 2.24× |
| 32B | TP8/DP1 | 891 | 1,574 | 1.77× |
| 32B | TP4/DP2 | 811 | 1,558 | 1.92× |
| 32B | TP2/DP4 | 675 | 1,561 | 2.31× |
Across TP=2, 4, and 8, CoWA achieves 2.45 to 2.88× prefill throughput and 1.70 to 2.31× decode throughput. With context length and shared widths fixed, more KV heads distribute the long-range history across more groups and reduce average per-head access (Section 2.2). The measured deployments use eight KV heads.
Memory
Our current vLLM integration retains the full KV history of each rank's local KV heads in HBM, following the same cache-residency policy as FullAttn. Consequently, CoWA has comparable serving-level memory requirements under the same configuration. At the operator level, Table 6 reports a 7.6× reduction in peak decoding memory at 128K, from 64.004 MiB for FullAttn to 8.441 MiB for CoWA.
Response (W1/Q1) to Reviewer sCCH
Thank you for your careful reading and the effort put into the detailed review. We appreciate your recognition of the consistent training and inference implementation, exact backward computation, global KV-head indexing, and controlled coverage experiments.
W1 and Q1. Collective coverage and the relationship to SPAttention
SPAttention and CoWA both use complementary head-wise access to cover the causal history, but they develop different attention architectures. SPAttention formally partitions distance bands among MHA heads, emphasizing exclusive head specialization. CoWA makes collective coverage a design constraint at the KV-head level in GQA: each KV head supplies a group of QO heads, all groups share near and sink access, and their long-range windows collectively cover the remaining history. This provides shared short-range context while reducing duplicated distant attention.
The distinction also extends to execution. CoWA uses one position-defined window rule for training forward/backward, prefill, and cached decoding, including unequal query/key lengths. Global KV-head indexing preserves complementary assignments across TP ranks. The paper evaluates this design in long-context training, retrieval, and end-to-end inference at 14B and 32B. Equal-width windows are a convenient, load-balanced realization, rather than the object of a model-specific window search.
The controlled evidence is direct. Table 1 varies collective long-range coverage at fixed per-head window width: at 8K, recall increases from 21.32% with one repeated window to 89.73% with eight complementary windows, close to FullAttn's 89.97%. Our new shared-window ablation follows Appendix A.1 (model dimension 512, sequence length 8K):
Sharing both regions raises recall by 10.60 percentage points over the exclusive-band control. Across four model dimensions and four sequence lengths, full CoWA achieves the highest or tied accuracy among these four sharing configurations. This tests shared locality within the same complementary-coverage framework rather than comparing an independently reimplemented SPAttention system.
Our inspection of
MEHC commitidentifies three concrete differences relevant to this control and its integration:Mask variant. SPAttention's Eqs. (4)/(8) specify exclusive supports, while the default code shares a local band alongside each head's distant band. Figure 2 likewise describes removing local sharing in ExclusiveBands. Our control follows the exclusive-band definition.
Cache interface. The default cache accepts 8 or 16 heads, supports lengths up to 4,096, and constructs equal query/key lengths. CoWA handles long-context GQA and unequal lengths in cached decoding.
Tensor layout.
model.pyforms[B,H,T,D]tensors, then its sparse call transposes them to[B,T,H,D]; the wrapper expects[B,H,T,D]and interpretsq.shape[2]as sequence length. This is a source-level interface mismatch.These findings motivate implementing the explicitly defined exclusive-band control in our common training setup. We will replace the broad implementation footnote with the variant definitions and pinned references above.