acceptodds
Under review as a conference paper at ICLR 2027

CoWindow Attention: Full Causal Coverage Is a Collective Property

Abstract

Long-context full attention (FullAttn) repeatedly exposes the complete causal history to every attention head, creating substantial redundant computation and memory traffic even with IO-efficient dense kernels. We introduce CoWindow Attention (CoWA), a structured attention architecture that distributes access to the causal history across KV heads. All heads share near-diagonal and prefix-sink windows, while complementary long-range windows partition the remaining history. Their union provides full causal coverage although each head attends sparsely to distant tokens. This position-defined attention pattern requires no learned router or indexer, is used consistently during training and inference, and aligns with KV-head tensor parallelism. A window-matched ablation at 8K isolates the effect of complementary long-range allocation: CoWA with 100% collective coverage reaches 89.73% accuracy, compared with 89.97% for FullAttn, while duplicated long-range windows perform substantially worse. Across a broader controlled associative-recall comparison with matched token budgets, CoWA closely tracks FullAttn as the context grows, whereas other sparse patterns lose a substantial fraction of the associations. In an attention-operator benchmark at 128K tokens on 8 GPUs with tensor parallelism, CoWA reduces forward and backward latency during training by and and decoding latency during inference by over FullAttn. Its per-rank peak operator memory matches FullAttn during training and is lower during decoding. Across scaling-law training from 0.6B to 14B parameters on 128 H100 GPUs, CoWA closely tracks FullAttn in perplexity while reducing total training FLOPs, with a 28.5% reduction at 14B during 32K-context training. The resulting 14B models and 32B models from separate continued training achieve comparable knowledge, reasoning, and long-context retrieval scores to FullAttn. These results show that full causal coverage can be a collective property of the head ensemble rather than a duplicated property of every head.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Long-Context Transformers, Structured Sparse Attention, Efficient Attention, Systems for Large Language Models, Tensor Parallelism, Training and Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sCCHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoWindow Attention (CoWA), which distributes long-range causal context across complementary KV-head windows while retaining shared local and sink windows. The attention pattern is determined by token positions without learned routing or additional indexing. The same design supports training, prefill, and autoregressive decoding, with exact backward computation and global KV-head indexing for tensor-parallel execution.

Critical strengths

S1: The paper provides a unified position-based rule for training, prefill, and autoregressive decoding, with exact backward gradients and consistent complementary windows under TP sharding. The implementation avoids additional routing or indexing overhead. The 128K/TP=8 experiments demonstrate substantial efficiency and memory benefits, while the authors also acknowledge relevant limitations. The main strength, in my view, is the consistent implementation across training and inference, rather than the attention pattern alone.

S2: The window-matched ablation provides useful evidence that complementary coverage improves recall independently of per-head window width, while the matched-budget experiments further support this interpretation. The scaling experiments from 0.6B to 14B and multi-seed evaluations at 14B/32B also strengthen the empirical results. The evaluation covers both efficiency and downstream quality. The results suggest that performance is largely preserved, although some tasks still show degradation.

Critical weaknesses

W1: I think the paper should more clearly distinguish its contributions from closely related approaches, particularly SPAttention (AAAI 2026) and Elastic Attention (ICML 2026). SPAttention already uses deterministic per-head distance bands whose union covers the causal history, which appears conceptually similar to the complementary coverage principle proposed here. The brief discussion in the footnote does not sufficiently clarify what is fundamentally new. Elastic Attention is also relevant, as it learns per-head attention mode assignments, providing a useful contrast to CoWA's position-defined design. Given the emphasis on tensor parallelism, a discussion of Star Attention and Ring Attention would further help establish the paper's scope, although these methods address somewhat different problems. I suggest expanding the related-work discussion to clarify the specific contributions in GQA support, training–decoding consistency, and parallel execution. A matched-budget comparison with SPAttention would also help establish whether CoWA offers an empirical advantage over this closely related design.

W2: While the reported operator-level speedups and memory savings are encouraging, the paper does not provide end-to-end training or serving throughput results. It is therefore unclear how much of the measured improvement translates into practical system-level gains. The efficiency evaluation is also limited to TP=8 and (H_k=8), without sensitivity analysis across other configurations.

Questions for authors

Q1. What is the main methodological difference between CoWA and SPAttention, beyond the GQA/KV-head formulation and implementation details? A matched-budget comparison would help clarify the practical significance of these differences. The footnote also states that the released MEHC implementation differs from its published description. Could the authors identify the specific differences and provide evidence for this claim?

Q2. Elastic Attention is particularly relevant because it also considers per-head attention pattern assignment, but uses a learned strategy rather than a deterministic one. How does CoWA compare under matched sparsity, for example on RULER or LongBench-E? Could the two approaches be complementary?

Q3. Could the authors report end-to-end training throughput or step time at 4K and 32K, as well as decoding throughput and peak memory at 128K? The current operator-level results do not establish the overall system benefit. How sensitive are these results to TP degree and KV-head count?

5 comments
AuthorsOpenReview ↗

Response (W1/Q1) to Reviewer sCCH

Thank you for your careful reading and the effort put into the detailed review. We appreciate your recognition of the consistent training and inference implementation, exact backward computation, global KV-head indexing, and controlled coverage experiments.

W1 and Q1. Collective coverage and the relationship to SPAttention

SPAttention and CoWA both use complementary head-wise access to cover the causal history, but they develop different attention architectures. SPAttention formally partitions distance bands among MHA heads, emphasizing exclusive head specialization. CoWA makes collective coverage a design constraint at the KV-head level in GQA: each KV head supplies a group of QO heads, all groups share near and sink access, and their long-range windows collectively cover the remaining history. This provides shared short-range context while reducing duplicated distant attention.

The distinction also extends to execution. CoWA uses one position-defined window rule for training forward/backward, prefill, and cached decoding, including unequal query/key lengths. Global KV-head indexing preserves complementary assignments across TP ranks. The paper evaluates this design in long-context training, retrieval, and end-to-end inference at 14B and 32B. Equal-width windows are a convenient, load-balanced realization, rather than the object of a model-specific window search.

The controlled evidence is direct. Table 1 varies collective long-range coverage at fixed per-head window width: at 8K, recall increases from 21.32% with one repeated window to 89.73% with eight complementary windows, close to FullAttn's 89.97%. Our new shared-window ablation follows Appendix A.1 (model dimension 512, sequence length 8K):

Attention LayoutAccuracy (%)
SPAttention-style exclusive bands, non-shared sink and near79.13
Shared sink only85.93
Shared near only80.75
CoWA, shared sink and near89.73
FullAttn89.97

Sharing both regions raises recall by 10.60 percentage points over the exclusive-band control. Across four model dimensions and four sequence lengths, full CoWA achieves the highest or tied accuracy among these four sharing configurations. This tests shared locality within the same complementary-coverage framework rather than comparing an independently reimplemented SPAttention system.

Our inspection of MEHC commit identifies three concrete differences relevant to this control and its integration:

  • Mask variant. SPAttention's Eqs. (4)/(8) specify exclusive supports, while the default code shares a local band alongside each head's distant band. Figure 2 likewise describes removing local sharing in ExclusiveBands. Our control follows the exclusive-band definition.

  • Cache interface. The default cache accepts 8 or 16 heads, supports lengths up to 4,096, and constructs equal query/key lengths. CoWA handles long-context GQA and unequal lengths in cached decoding.

  • Tensor layout. model.py forms [B,H,T,D] tensors, then its sparse call transposes them to [B,T,H,D]; the wrapper expects [B,H,T,D] and interprets q.shape[2] as sequence length. This is a source-level interface mismatch.

These findings motivate implementing the explicitly defined exclusive-band control in our common training setup. We will replace the broad implementation footnote with the variant definitions and pinned references above.

AuthorsOpenReview ↗

Response (W1/Q2) to Reviewer sCCH

W1 and Q2. Elastic Attention and distributed execution

Elastic Attention selects head computation modes through input-dependent routing; CoWA assigns complementary windows through positions. These mechanisms offer a possible combination: a data-driven allocator could adapt window boundaries or assignments while enforcing collective coverage. An allocation learned offline could retain a regular execution schedule.

Our RULER comparisons use FullAttn, MoBA, and DSA at 14B, and FullAttn at 32B. Appendix B, Table 10 gives the task breakdown; the overall scores are:

Model SizeMethodRULER 32KRULER 128K with YaRN
14BFullAttn89.4265.84
14BMoBA87.1761.52
14BDSA87.6764.05
14BCoWA89.1366.6
32BFullAttn92.782.03
32BCoWA92.5881.78

These RULER results show that CoWA maintains performance close to its FullAttn controls at 32K and 128K. The table compares methods under our training protocol; Elastic Attention instead adapts a frozen pretrained backbone through a learned router. The approaches are complementary: a coverage-preserving window allocation could be learned for a target workload and then executed with CoWA's regular attention kernel.

Ring Attention distributes sequence blocks and overlaps KV communication with exact attention computation. Star Attention combines blockwise context processing with global attention for query and response tokens. CoWA distributes complementary supports across KV heads and preserves them under head tensor parallelism. We will clarify these distinct decompositions in the related-work discussion.

AuthorsOpenReview ↗

Response (W2/Q3) to Reviewer sCCH

W2 and Q3. End-to-end efficiency and parallel configurations

CoWA improves end-to-end throughput in both Megatron-LM and vLLM, where we integrate it by replacing the attention kernel.

Training

We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8 and the training settings in Appendix A.3. Batch size is matched at each model scale. FullAttn and CoWA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.

ModelMethod4K Step (MS)4K Tokens/s32K Step (MS)32K Tokens/s
0.6BFullAttn843,123,0881052,489,200
0.6BMoBA185.21,416,699228.61,143,530
0.6BDSA82.93,162,77076.43,420,169
0.6BCoWA72.63,611,44848.65,381,947
1.7BFullAttn2122,478,1762881,818,080
1.7BMoBA323.41,624,575529.5988,849
1.7BDSA210.82,491,814232.12,255,608
1.7BCoWA199.52,633,526177.72,946,757
4BFullAttn7751,353,2161,217.00861,568
4BMoBA1,175.60892,0822,229.10470,390
4BDSA770.81,360,542982.91,066,771
4BCoWA7301,436,587754.71,389,257
8BFullAttn1,711.00919,1362,363.00665,728
8BMoBA2,229.40705,4253,804.80413,452
8BDSA1,705.60922,0462,029.50775,129
8BCoWA1,652.80951,4991,704.40922,950
14BFullAttn3,635.00576,9604,927.00425,600
14BMoBA4,499.00466,1597,482.00280,263
14BDSA3,626.00578,3924,336.00483,610
14BCoWA3,538.00592,7783,760.00557,694

CoWA has the highest throughput among the four configurations at every scale and both lengths. At 4K, it achieves 1.03 to 1.16× FullAttn's throughput. At 32K, it achieves 1.31 to 2.16× FullAttn, 1.99 to 4.71× MoBA, and 1.15 to 1.57× DSA, showing larger system gains as attention work grows with context length.

Serving

We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.

ModelMethodPrefill Input Tokens/SDecode Output Tokens/S
14BFullAttn46,7461,542
14BMoBA21,743471
14BDSA66,3101,987
14BCoWA121,4022,628
32BFullAttn21,520891
32BCoWA60,3521,574

CoWA's prefill/decode throughput gains over FullAttn are 2.60×/1.70× at 14B and 2.80×/1.77× at 32B. At 14B, the gains are 5.58×/5.58× over MoBA and 1.83×/1.32× over DSA.

TP/DP configurations and KV-head count

We keep eight GPUs, total concurrency 16, and eight KV heads while varying TP/DP. Concurrency per DP replica is 16 for TP8/DP1, eight for TP4/DP2, and four for TP2/DP4.

Prefill-heavy workload, input tokens/s

ModelConfigurationFullAttnCoWASpeedup
14BTP8/DP146,747121,4022.60×
14BTP4/DP251,421125,9422.45×
14BTP2/DP455,215144,8452.62×
32BTP8/DP121,52060,3522.80×
32BTP4/DP223,67262,7892.65×
32BTP2/DP425,41873,0842.88×

Decode-heavy workload, output tokens/s

ModelConfigurationFullAttnCoWASpeedup
14BTP8/DP11,5422,6281.70×
14BTP4/DP21,4042,6051.86×
14BTP2/DP41,1682,6112.24×
32BTP8/DP18911,5741.77×
32BTP4/DP28111,5581.92×
32BTP2/DP46751,5612.31×

Across TP=2, 4, and 8, CoWA achieves 2.45 to 2.88× prefill throughput and 1.70 to 2.31× decode throughput. With context length and shared widths fixed, more KV heads distribute the long-range history across more groups and reduce average per-head access (Section 2.2). The measured deployments use eight KV heads.

Memory

Our current vLLM integration retains the full KV history of each rank's local KV heads in HBM, following the same cache-residency policy as FullAttn. Consequently, CoWA has comparable serving-level memory requirements under the same configuration. At the operator level, Table 6 reports a 7.6× reduction in peak decoding memory at 128K, from 64.004 MiB for FullAttn to 8.441 MiB for CoWA.

Reviewer_d7wrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoWindow Attention (CoWA), a structured sparse attention mechanism that assigns complementary context coverage to different KV heads. The design combines shared attention-sink and local windows with head-specific long-range windows, allowing the heads to collectively cover the causal sequence. The paper evaluates the approach through controlled synthetic experiments, pretraining, and continued pretraining (CPT). The central idea is appealing, and the experimental coverage is reasonably broad, although the main experiments do not sufficiently isolate the contribution of each design component.

Critical strengths
  1. Allocating complementary context coverage across KV heads is a clever design choice.

Rather than applying the same sparse attention pattern to every head, CoWA distributes access to different parts of the context across KV heads. This offers an intuitive way to reconcile per-head sparsity with collective coverage of the sequence. The idea is conceptually clear and provides a useful direction for designing structured sparse attention.

  1. The controlled synthetic experiments provide meaningful support for the central idea.

The synthetic experiments are reasonably comprehensive and well aligned with the proposed mechanism. They help establish why complementary coverage across heads could be beneficial, rather than relying exclusively on aggregate downstream scores. These experiments provide a useful foundation for understanding the design.

  1. The main evaluation covers a reasonable range of settings.

The paper evaluates both pretraining and continued pretraining, alongside downstream and long-context tasks. This breadth is a positive aspect of the work and provides a more informative assessment than evaluation in a single training or application setting.

Critical weaknesses
  1. The main experiments lack sufficient component-level ablations.

Although the synthetic experiments support the central idea, it remains unclear how much each component contributes to performance in the main language-modeling experiments. In particular, the paper should separately examine the effects of attention-sink tokens, the local window, and the complementary long-range windows.

The key question is whether the benefit of complementary head coverage demonstrated in the synthetic setting also explains the gains in the main experiments. Useful controls would include removing individual components and replacing complementary long-range windows with duplicated or alternative sparse patterns while preserving the shared sink and local windows. Where possible, the comparisons should maintain a comparable attention budget, so that changes in performance are not explained simply by changes in the amount of accessible context.

  1. Differences in the baseline components make the architectural attribution less convincing.

Related to the previous concern, some sparse baselines do not include the same sink and local-window components as CoWA. Consequently, the comparisons change more than the allocation of long-range attention across heads. The results therefore do not cleanly establish whether the improvements come from the proposed complementary coverage, the shared sink and local windows, or their combination.

Comparisons against the original baseline configurations remain useful for evaluating the complete method. However, stronger attribution requires additional component-matched comparisons: the baselines should receive comparable sink and local windows, with the remaining attention budget allocated according to their respective sparse patterns. This would more directly test the paper’s central architectural claim.

  1. The downstream evaluation needs stronger evidence of capability preservation at the 14B and 32B scales.

The evaluation includes BBH, HellaSwag, OpenBookQA, WinoGrande, PIQA, GSM8K, and ARC-Challenge. These results are useful as general capability checks, but I am not convinced that the current evaluation sufficiently stresses the larger models or exposes potential weaknesses in demanding reasoning settings. The paper should demonstrate that the selected tasks provide enough headroom to distinguish the methods, rather than relying primarily on aggregate scores across the benchmark suite.

The reported MMLU-Pro and GPQA-Diamond scores also appear low and require further contextualization. In particular, the paper should clarify the starting checkpoints, whether the models are base or instruction-tuned, and the prompting, generation, and scoring protocols. Parameter count alone is not sufficient to establish an expected score. Comparisons with the original checkpoints under the same evaluation protocol would help determine whether the results reflect the starting models’ capabilities, the evaluation setup, or degradation introduced during CPT.

  1. The robustness of the comparison to the training regime is insufficiently established.

The paper should clarify whether the relative performance of CoWA and full attention depends on the chosen training-token budget. A comparison near a Chinchilla-style training budget addresses one operating regime, but it does not by itself establish the same ordering under substantially longer training. Prior work distinguishes training-compute-optimal allocation from deployment-oriented regimes in which smaller models are trained considerably longer; this motivates examining more than a single training regime, although it does not establish that Chinchilla-style training inherently disadvantages full attention.

I would therefore appreciate a clearer account of the token-to-parameter ratios, training compute, and the evolution of the performance gap during training. Results at additional token budgets, even at a smaller scale, would help assess whether the conclusions are robust or specific to the selected training duration. Matched-token and matched-compute comparisons should also be distinguished explicitly.

Questions for authors
  1. What explains the RULER-CWE result in the 14B YaRN 128K setting? CoWA appears to have a notable advantage in this configuration. Is this primarily attributable to its local-window component, its complementary long-range coverage, or an interaction between the two? Could the authors provide a targeted ablation, such as varying the local-window size or adding an equivalent local window to the relevant sparse baseline?

  2. What is the starting checkpoint for the 32B model, and how does its performance change during CPT? Please identify the exact checkpoint and report its performance before CPT under the same evaluation protocol. Does the original model outperform the CPT model on any of the reported tasks? A before-and-after comparison, together with a full-attention control continued from the same checkpoint on the same data, would help separate the effects of CPT from those of the attention modification.

4 comments
AuthorsOpenReview ↗

Response (W1/W2) to Reviewer d7wr

Thank you for the time invested in the review and the detailed suggestions. We appreciate your recognition of the complementary KV-head design, the controlled synthetic experiments, and the evaluation across training regimes and long-context tasks.

W1 and W2. Contributions of coverage and shared windows

The controlled experiments separate the benefit of complementary long-range coverage from that of shared sink and near windows. In Table 1, we hold the shared windows and per-head long-range width fixed while increasing the number of distinct remote windows across eight KV heads. At 8K and model dimension 512, recall rises from 21.32% with one repeated window to 32.92%, 52.32%, and 89.73% with two, four, and eight complementary windows, respectively. FullAttn reaches 89.97%.

We additionally evaluate four sharing configurations using the associative-recall protocol of Appendix A.1: 256 random key-value pairs, sequence lengths from 1K to 8K, and model dimensions from 64 to 512. The configurations are exclusive distance bands without shared sink or near windows (SPAttention-style), shared sink only, shared near only, and CoWA with both windows shared. The table reports recall accuracy (%).

Model DimensionShared Components1K2K4K8K
64Neither sink nor near40.2923.2214.26.35
64Sink only40.2931.3522.4111.72
64Near only40.2924.2816.649.63
64Sink and near, CoWA40.2933.1424.3415.41
128Neither sink nor near10068.7538.9538.8
128Sink only10079.5549.8448.53
128Near only10070.8340.2639.74
128Sink and near, CoWA10083.9152.3251.21
256Neither sink nor near10084.280.4271.06
256Sink only10094.1592.8482.34
256Near only10085.7282.9372.74
256Sink and near, CoWA10098.7797.9385.6
512Neither sink nor near10093.1190.8879.13
512Sink only10010010085.93
512Near only10094.4291.3280.75
512Sink and near, CoWA10010010089.73

CoWA gives the highest or tied recall in all 16 settings. At 8K and model dimension 512, the exclusive-band control reaches 79.13%, shared sink only reaches 85.93%, shared near only reaches 80.75%, and full CoWA reaches 89.73%. Even the exclusive-band control exceeds MoBA (50.12%) and DSA (53.71%) in the original 8K recall comparison. Sink sharing makes the larger individual contribution in this experiment. This is consistent with the account of attention sinks in Gu et al.: a common prefix position can absorb attention scores that carry little useful value information.

For the other sparse baselines, we retain the official architectures, sparse-selection rules, and kernels to compare each complete method's quality and execution cost. Adding CoWA-style shared windows to these methods would require modifying their selection mechanisms and corresponding kernels. The controlled comparisons above therefore use a common CoWA-family setup to attribute the effects of complementary coverage and shared windows, while the original baselines measure the performance of deployable methods in their native configurations.

AuthorsOpenReview ↗

Response (W3/W4/Q1/Q2) to Reviewer d7wr

W3 and Q2. Checkpoints, CPT, and evaluation

The 14B models are pretrained from scratch, with no post-training. Both 32B models start from Qwen/Qwen3-32B, which includes Qwen3's post-training, and undergo identical CPT on SmolLMCorpus. Appendix A.4 specifies 64B tokens, a 32,768-token context, a 4M-token batch, and a constant learning rate of 0.00001. Initial weights, data, and training settings are shared; attention is the experimental variable. FullAttn after the same CPT procedure is therefore the capability-preservation reference, controlling the common effects of continued training.

Evaluation uses lm-evaluation-harness v0.4.13 with standard task configurations held fixed across attention variants. Appendix A.5 and Appendix B document the protocol, shot counts, generation settings, and seeds. The 14B setting is non-thinking; the 32B setting is thinking. The five reported seeds are evaluation repeats, while scaling-law training uses seed 42. We will add the checkpoint identifier and harness version to the setup.

W3. Task-level capability and multi-hop reasoning

Tables 8 and 9 report the following task-level results:

Model SizeTaskFullAttnCoWA
14BMMLU-Pro64.96 ± 0.4664.89 ± 0.43
14BGPQA-Diamond39.87 ± 3.4742.33 ± 3.17
14BHendrycks-Math62.46 ± 0.6562.31 ± 0.62
32BMMLU-Pro68.57 ± 0.4168.51 ± 0.47
32BGPQA-Diamond56.06 ± 3.5455.05 ± 3.56
32BHendrycks-Math94.94 ± 0.8294.71 ± 0.84

CoWA closely tracks FullAttn on these tasks. At 14B, Hendrycks-Math also distinguishes the sparse methods: MoBA and DSA score 39.00 and 55.99, versus CoWA's 62.31 and FullAttn's 62.46. MMLU-Pro and GPQA retain ample headroom at both scales.

The separate RULER-QA components for 32B with YaRN at 128K are:

RULER Task, 32B at 128KFullAttnCoWA
QA-SQuAD51.4150.09
QA-HotpotQA45.846.13

HotpotQA evaluates multi-hop reasoning over multiple documents. These scores support comparable performance in this long-context setting. Cross-window information can combine through the output projection, which mixes QO-head outputs from different KV-head groups, and through subsequent attention and feed-forward layers.

W4. Training duration and compute accounting

Table 7 gives the training budgets below. Ratios use nominal model sizes and include both stages.

Target Model Size4K Pre-Training Tokens32K Long-Context TokensTotal Tokens Per Parameter, Approximately
0.6B12B1.5B22.5
1.7B34B4B22.4
4B80B10B22.5
8B164B20B23
14B296B36B23.7

FullAttn and CoWA receive equal training tokens at each scale. Their LM-loss and gradient-norm trajectories remain closely aligned over the observed training progress. At 14B, final long-context perplexities differ by less than 0.01; CoWA reduces total training FLOPs by 3.1% during 4K pretraining and 28.5% during the 32K stage.

Equal tokens control data exposure. Among sparse methods, approximately matched forward computation over selected connections aligns the core attention budget while retaining native parameterizations. Figure 4 reports perplexity against total training FLOPs, including whole-model computation, causally clipped attention, and routing or indexing costs. Operator measurements include those execution costs as well. A fixed-total-compute comparison measures quality under a common resource budget, potentially with different token counts.

Q1. The 14B, YaRN 128K CWE result

CWE scores are low for both models: 5.76 ± 0.46 for FullAttn and 13.22 ± 1.12 for CoWA. This gives the comparison limited diagnostic value for distinguishing local-window, long-range, and interaction effects. The controlled allocation and sharing experiments above provide direct evidence for those design choices. Table 10 retains the CWE result as part of the complete RULER evaluation.

Thank you for the detailed response and the additional experiments. I appreciate the authors’ efforts to address my comments. However, the new evidence does not sufficiently resolve my main concerns.

W1 and Q1: Component ablations in the main language-model training setting are still needed.

The additional associative-recall experiments are informative, but they do not establish how much the shared sink, local window, and complementary long-range coverage contribute to the main language-modeling results. My request is to train and evaluate the corresponding ablated variants under comparable conditions, ideally with comparable attention budgets. This would provide more direct evidence that the proposed complementary coverage explains the observed gains. For Q1, I agree that the low absolute CWE scores limit their diagnostic value, but this observation leaves the question about the source of CoWA’s advantage unresolved.

W3 and Q2: The CPT comparison does not sufficiently establish preservation of the original model’s capabilities.

The reported results leave me concerned about substantial capability degradation relative to the original Qwen3-32B checkpoint. I acknowledge that continuing FullAttn from the same checkpoint with identical data and training settings provides a useful control for comparing the attention variants within this CPT setup. However, matching this reference does not resolve whether both models have already lost substantial capability relative to the starting checkpoint. If that is the case, their similar performance provides only limited support for the broader capability-preservation claim. A direct evaluation of the original checkpoint under the same protocol remains necessary to establish the extent of any degradation and give the comparison a firmer basis.

Despite these limitations, I still consider the central design worthwhile and the overall contribution sufficient for weak acceptance. The response has not provided enough evidence to raise my assessment, but my positive view of the core idea remains. I therefore maintain my original score of 3 (weak acceptance).

Thank you for clarifying these two concerns.

Component attribution. Directly attributing the language-model results to shared sink, local windows, and complementary coverage would require additional ablated models trained under matched conditions. Completing these training runs is beyond the rebuttal timeframe. We therefore provided controlled associative-recall experiments to examine the components within a common setup. Together with the original coverage ablation, these experiments provide mechanistic evidence for the design; the large-scale language-model experiments evaluate the complete architecture.

The CPT reference. Our continued training uses SmolLMCorpus, a pretraining corpus, whereas the starting Qwen3-32B checkpoint has already undergone post-training. Further training on this corpus can change the balance of capabilities, particularly reasoning, code, and mathematics. This is why both CoWA and FullAttn undergo the same CPT procedure: the comparison measures the effect of the attention architecture under a common training trajectory.

Our capability-preservation claim refers to performance comparable to FullAttn under identical CPT conditions. Recovering the original checkpoint’s post-trained capability profile would involve a separate post-training stage; the original post-training data are not publicly available to us. We will make the scope of the claim explicit in the revised manuscript.

Reviewer_5DZhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CoWindow Attention (CoWA) a collective attention window that covers near-diagonal, prefix-sink, and long-range windows for all heads defined by CoWA's window calculation formula that depends on the position. Since it is based off of the position indexing, it is consistently used during training and inference, and aligns with KV-head tensor parallelism. The main goal of CoWA is to have a good coverage over the long context across heads while still being efficient and avoiding full attention. The union of the heads' attention recovers the full causal attention. The author shows that using CoWA the performance matches up to full Attention while keeping the computational cost comparable to full attention.

Critical strengths
  • The paper introduces an interesting Attention mechanism that distributes different attention windows through different heads so that collectively the full attention is recovered. The idea is novel and interesting.
  • The authors carefully analyzes the eligibility of CoWA for KV-head tensor parallelism as well as its generalization to other architectures.
  • The evaluation is thorough across different tasks, ablations on kernel execution, and distributed deployment for parallelism.
Critical weaknesses
  • The deployment of the different long-term attentions across heads seem like more of a heuristic than grounded by studies. Since it is almost deterministic on which heads get which parts of the attention, the method is limited to being not data-dependent.
  • Somewhat related to above point, since each KV head only attends to a specific slice of the long-range history, I'm not sure how the model can handle complex multi-hop reasoning tasks that might require a single head to attend to different long range windows?
Questions for authors

Please address the comments in the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5DZh

We thank the reviewer for the careful review and the time spent evaluating our work. We appreciate your recognition of the collective allocation of attention windows, the analysis of KV-head tensor parallelism, and the breadth of the task and execution evaluations.

W1. The basis for position-defined window allocation

CoWA's central principle is that full causal coverage is a collective property of the attention heads. Its construction makes every causal position accessible to at least one KV-head group. The access pattern follows position-defined rules; Attention weights within each window remain content-dependent through learned queries and keys.

Table 1 holds the shared sink and near windows and nominal per-head long-range width fixed. At model dimension 512 and sequence length 8K, complementary allocation improves recall as collective coverage increases:

Long-Range Allocation Across Eight KV HeadsCollective Long-Range CoverageRecall Accuracy
One window repeated across eight heads12.50%21.32%
Two windows, each repeated across four heads25%32.92%
Four windows, each repeated across two heads50%52.32%
Eight complementary windows, one per head100%89.73%
FullAttn100%89.97%

Equal-width windows provide a controlled realization and balance long-range access during decoding. The same coverage principle supports other allocations, including boundaries chosen to balance the causal attention area assigned to each head during training.

This regularity also serves our systems objective. Position-defined windows remove the content-selection stage and give predictable block traversal, memory access, and scheduling across training, prefill, and decoding. Global KV-head indexing preserves the allocation under tensor parallelism. For fixed execution shapes, the layout supports preallocated buffers, compilation, and CUDA Graph capture; it also provides predetermined access ranges for future context-parallel communication planning.

At 14B, CoWA finishes long-context training within 0.01 perplexity of FullAttn with 28.5% fewer total training FLOPs in the 32K stage. On 128 H100 GPUs, the additional Megatron-LM measurement gives 557,694 versus 425,600 tokens/s at 14B and 32K, a 1.31× gain.

W2. Combining evidence across windows

The existing long-context QA results support comparable multi-hop performance. Appendix B's RULER-QA column averages SQuAD and HotpotQA. Their separate scores for the 32B models with YaRN at 128K are:

RULER Task, 32B at 128KFullAttnCoWA
QA-SQuAD51.4150.09
QA-HotpotQA45.846.13

HotpotQA requires combining evidence from multiple documents, making it relevant to this concern. CoWA retains comparable performance under the same evaluation protocol.

The architecture combines information across windows through the head outputs and successive layers. Different KV-head groups expose different distant regions to their QO heads; the output projection combines the retrieved information in a common residual representation, and later attention and feed-forward layers process it further. Shared near and sink windows additionally provide common local and prefix context. This gives the model a direct computational path for combining evidence retrieved from different regions.

Dear Reviewer 5DZh,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate any follow-up on our response concerning position-defined allocation and cross-window information integration, including the task-level multi-hop QA results and additional end-to-end efficiency measurements.

Please let us know if any concerns remain or would benefit from further clarification. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: CoWA’s Contributions, Additional Evidence, and Review Discussion

Dear AC, SAC, and PC,

We thank the reviewers for their thoughtful evaluation. We summarize our contribution, the reviewers’ feedback, and the evidence provided during rebuttal.

TL;DR of Our Work

CoWA investigates whether full causal coverage can be maintained collectively across attention heads while reducing duplicated long-range computation. All KV-head groups share local and sink windows; complementary distant windows collectively cover the remaining history. This position-defined architecture supports training, prefill, and decoding without a learned router or indexer.

The controlled coverage experiment demonstrates the value of complementary allocation: at fixed shared windows and per-head long-range width, 8K recall increases from 21.32% with duplicated windows to 89.73% with complementary coverage, close to FullAttn’s 89.97%. Training from 0.6B to 14B and evaluation at 14B and 32B establish the complete architecture’s quality and efficiency relative to matched FullAttn controls.

Reviewers’ Recognition

We appreciate the reviewers’ recognition of the central design and its evidence. 5DZh describes the idea as novel and interesting and recognizes the tensor-parallel analysis and broad evaluation. d7wr highlights the clear complementary-coverage concept and the controlled experiments supporting it. sCCH recognizes the consistent training–decoding implementation, exact backward computation, and complementary assignments under tensor parallelism.

Reviewers’ Concerns and Our Responses

  • 5DZh and d7wr ask what supports the allocation and the shared components. We explained the content-dependent attention weights within position-defined windows and added shared-window ablations across four model dimensions and four context lengths. At 8K and dimension 512, exclusive bands achieve 79.13%, sink sharing 85.93%, local sharing 80.75%, and complete CoWA 89.73%.

  • sCCH asks for clearer positioning relative to SPAttention and related methods. We clarified the shared foundation of complementary coverage and CoWA’s shared-context GQA architecture, unified execution, and parallel deployment. We provided pinned source references for the implementation differences and explained the relationship to Elastic, Ring, and Star Attention.

  • 5DZh raises cross-window reasoning concerns; d7wr requests clearer capability evaluation. We explained information integration through output projections and successive layers, reported task-level results including comparable 128K HotpotQA performance, and clarified checkpoints, evaluation protocols, and training budgets. Both 32B variants start from Qwen3-32B and undergo identical CPT.

  • sCCH requests end-to-end efficiency and additional parallel configurations. We added Megatron-LM measurements on 128 H100 GPUs and vLLM measurements on eight H100 GPUs with concurrency 16. CoWA improves 32K training throughput by 1.31–2.16×. At 14B/32B, serving gains are 2.60×/2.80× for prefill-heavy workloads and 1.70×/1.77× for decode-heavy workloads. Gains persist across TP=2, 4, and 8 with eight KV heads.

Discussion Status

All three reviewers currently recommend weak acceptance. sCCH stated that most concerns were addressed and raised the score to 3. d7wr maintained 3, explicitly finding the central design worthwhile and the overall contribution sufficient for weak acceptance, while retaining requests for LM component ablations and original-checkpoint evaluation.

We clarified that AR ablations provide controlled mechanism evidence, while the LM experiments evaluate the complete architecture. Additional matched LM ablation training exceeds the rebuttal timeframe. The 32B capability comparison concerns CoWA and FullAttn under identical CPT conditions. These distinctions will be explicit in the revision.

We thank the reviewers for strengthening the paper and the chairs for considering its controlled evidence, model-scale validation, and practical system gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.