acceptodds
Under review as a conference paper at ICLR 2027

NCSA: Native Chunk Summary Attention

Abstract

Long-context language modeling faces two bottlenecks: attention computation grows rapidly with context length, and the key–value (KV) cache grows with every token during generation. Recent KV-compression methods address these bottlenecks by representing several tokens with a compact summary. Existing designs mainly follow two approaches. Dedicated-compressor designs generate compressed KV entries through an additional module and computation path. Summary-token designs instead insert a learned summary for each chunk and process it as an ordinary sequence position. Consequently, the extra positions pass through every layer’s feed-forward network, and a summary produced in one layer is not available to text attention until the next. In this work, we introduce NCSA (Native Chunk Summary Attention), a nearly parameter-free architecture that generates and uses chunk summaries within the same attention layer. NCSA keeps summaries outside the token sequence in a summary residual stream, thereby eliminating the one-layer delay and making summary-side feed-forward computation optional while preserving the compressed KV cache. At the 4K context length used for pretraining, NCSA reduces estimated Transformer-block FLOPs by 14.2% relative to the summary-token baseline. Across matched pretraining runs at 0.3B, 0.6B, and 4B parameters on 50B tokens, NCSA consistently improves long-context retrieval over the compressed-attention baselines while remaining competitive in language modeling and commonsense reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, kv cache, summary attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_J9q1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

NCSA introduces a native compressed-attention architecture that maintains one summary state per chunk outside the ordinary token sequence. Within each layer, it first updates chunk summaries and then exposes the updated summary keys and values to text attention in the same layer. This removes the one-layer read-before-write delay of summary-token architectures and makes summary-side feed-forward computation optional. Matched 50B-token pretraining experiments at 0.3B, 0.6B, and 4B compare NCSA with dense attention, a dedicated compressor, and summary-token compression.

Critical strengths
  1. The paper identifies a precise architectural limitation of summary-token compression: a summary written in one layer cannot be read by text positions until the following layer. The forward- and backward-graph analyses clearly motivate the proposed write-before-read ordering.

  2. The empirical comparison is well controlled. Architectures at each scale use the same backbone, data order, optimization recipe, context length, and number of training tokens. NCSA consistently improves RULER and real-document retrieval over the compressed baselines, with particularly strong scaling at 4B.

Critical weaknesses
  1. The efficiency claim is supported primarily by analytical FLOP estimates. NCSA introduces two sequential attention stages per layer—summary writing followed by text attention—which may have different kernel-launch, synchronization, and memory-access behavior from KSA’s more parallel computation. Actual pretraining throughput, prefill latency, decode latency, and peak KV memory are needed before concluding that the 14.2% block-FLOP reduction yields practical speedups.

  2. The principal models are trained at 4K, and most evaluations are at or below 4K. Although the 8K extrapolation is encouraging, practical long-context compression claims would be substantially stronger with evaluation or continued training at 16K–128K and with end-to-end generative tasks rather than predominantly retrieval-oriented diagnostics.

Questions for authors
  1. What wall-clock throughput and latency does NCSA achieve relative to KSA, CSA, and dense attention using equally optimized kernels? How much of the analytical FLOP saving survives the sequential write/read implementation?

  2. What are the exact prefill and decoding KV-cache sizes and memory bandwidth costs at long sequence lengths?

  3. Do the gains persist when models are trained directly at 16K or longer contexts and evaluated on natural long-document QA or generation?

  4. How variable are the reported gains across training seeds, particularly the smaller commonsense and JRT differences?

3 comments

We thank the reviewer for the valuable feedback. We address the concerns below.

W1/Q1: Wall-clock throughput and latency vs KSA, CSA and Dense under equally optimized kernels

All arms share the same block code and the same attention kernel, and differ only in mask/KV layout and in which tokens take which path. Prefill latency at batch=1 (4B, ms; parentheses give speedup over Dense, where >1 means faster than Dense):

arm4K8K16K32K64K128K
Dense53.6120.1300.2858.92786.99901.4
NCSA64.7 (0.83x)119.8 (1.00x)244.8 (1.23x)543.3 (1.58x)1343.6 (2.07x)3772.5 (2.62x)
KSA76.1 (0.70x)131.9 (0.91x)265.0 (1.13x)575.4 (1.49x)1410.2 (1.98x)3923.1 (2.52x)
CSA61.2 (0.88x)118.8 (1.01x)242.4 (1.24x)536.8 (1.60x)1330.4 (2.09x)3747.6 (2.64x)

Cross-arm comparison: KSA takes 1.18x NCSA's time at 4K, narrowing to 1.04x at 128K; CSA is about 5% faster than NCSA at 4K and within 1% from 8K onward.

Decode is compared at an equal memory budget rather than at an equal batch: for each arm we determine the largest batch that fits in GPU memory and report throughput at that point. Per-token latency is averaged over a full cycle of C steps, so the summary write that occurs every C steps is included in the reported cost rather than averaged away.

Sarmmax batchms/tokentok/svs Dense
8KDense14733.934332-
8KNCSA55442.23131183.03x
8KKSA55442.61130023.00x
8KCSA54842.88127792.95x
128KDense932.40278-
128KNCSA3840.389413.39x
128KKSA3842.638913.21x
128KCSA3845.838292.99x

At batch=1 the per-token latency is identical across arms (30-34 ms at every length, dominated by per-layer scheduling); the gain appears as a 3.8-4.2x larger feasible batch, which amounts to a 3.0-3.4x throughput gain once the amortized summary write is included. At 128K the measured prefill speedup (2.62x) matches the analytical bound implied by the FLOP reduction (2.63x); at short context a larger share of the analytical saving is consumed by the sequential stage-1 write, whose share of NCSA's attention time falls from 33% at 4K to 3.7% at 128K. Taken together, essentially the whole analytical saving survives at long context, and about 80% of it survives at 4K.

W1/Q2:Exact prefill and decode KV-cache sizes and memory bandwidth at long sequence lengths

Decode resident KV (4B, bf16):

SDense rows/layerNCSA rows/layerDense MiBNCSA MiBcompression
4K409611565761633.54x
8K8192218011523073.76x
32K327688324460811713.94x
128K131072329001843246273.98x

NCSA's cache is a 128-row text window, the in-flight chunk, and S/4 summary rows; these are closed-form and identical across machines. Prefill temporary KV is S columns per layer for Dense and S + S/C for NCSA. Each decode step reads 18432 MiB versus 4627 MiB at 128K (4.0x fewer bytes) and 576 versus 163 MiB at 4K. Measured effective bandwidth (amortized single-step SDPA over the resident KV at 128K) is ~6.6 TB/s for both arms, so the gain comes from reading 4x fewer bytes, not from reading faster.

W2/Q3: Whether the gains persist for models trained at 16K or longer and evaluated on natural long-document QA or generation

These runs are currently in progress: we are training at 16K+ and will evaluate on natural long-document QA/generation, and will report the results in a later version of this response.

Q4: Variability of the reported gains across training seeds, including the smaller commonsense and JRT margins

We first separate two sources of variance. The three seeds we currently report are evaluation seeds on fixed checkpoints: they bound evaluation noise, not training variance. Measured on RULER-13 (13 tasks, mean per evaluation seed):

modeleval seed 1eval seed 2eval seed 3mean +- sd
NCSA 0.3B34.5735.1035.4135.03 +- 0.42
KSA 0.3B32.5532.3431.9932.29 +- 0.28
NCSA 0.6B43.3042.7043.4043.13 +- 0.38
KSA 0.6B39.0337.6238.9038.52 +- 0.78

Paired differences (NCSA - KSA) are +2.73 +- 0.70 at 0.3B (95% CI [+0.99, +4.47]) and +4.62 +- 0.42 at 0.6B (95% CI [+3.58, +5.65]); both exclude zero. Within-arm evaluation spread is 0.3-0.8 points on the 13-task mean, so we make no significance claim for differences smaller than this spread, such as the smaller commonsense margins.

Training-seed variance requires retraining from different initializations and data orders. Those runs are in progress; we will report them in a later version of this response

W2/Q3: Whether the gains persist for models trained at 16K or longer and evaluated on natural long-document QA or generation

We continued pretraining the 0.3B Dense, NCSA and KSA checkpoints from their 4K schedule to a 16K context for 6B tokens, and evaluated the final checkpoint of each arm on 12 LongBench datasets (2,950 samples).

taskmetricnDenseNCSAKSANCSA - Dense
gov_reportROUGE20017.0416.0914.45-0.96
multi_newsROUGE20017.2113.8014.03-3.41
qmsumROUGE20019.2118.3516.95-0.85
vcsumROUGE-zh2005.215.766.62+0.56
narrativeqaF12002.782.712.92-0.08
qasperF12007.2410.047.71+2.80
2wikimqaF12007.618.418.87+0.80
hotpotqaF12005.975.654.94-0.32
multifieldqa_enF115015.4315.7912.31+0.36
lshtaccuracy20012.5011.0012.00-1.50
lccedit-sim50012.9718.7416.13+5.77
repobench-pedit-sim50023.2622.3823.26-0.88
mean-295012.2012.3911.68+0.19

On natural long-document summarization, QA and code completion, the comparison is unchanged after continued pretraining to 16K: NCSA (12.39) stays ahead of KSA (11.68) and level with Dense (12.20), with a mean per-task difference of +0.19 against Dense (median -0.20) and every task except lcc (+5.77) within 3.5 points. Continuing pretraining at 16K therefore neither erodes the lead over KSA nor costs quality relative to Dense.

We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

Reviewer_PJUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NCSA, a native KV-compression attention layer for long-context LMs. Each chunk of C=4 tokens gets one summary vector that lives in a separate "summary stream" outside the token sequence. In every layer, the summary first attends to its own chunk using the layer's existing Q/K/V/O projections, and text queries then read the updated summary K/V in the same layer. Summaries skip the FFN, and the only new parameter is one learned anchor vector. The motivation is that in summary-token designs such as KSA, a summary written in layer ℓ can only be read from layer ℓ+1 on ("read-before-write"), and every summary position pays for the FFN.

The authors train Dense, CSA (a DeepSeek-V4-style compressor without the indexer), KSA and NCSA from scratch with the same backbone, data order and recipe, at 0.3B/0.6B/4B parameters, 50B tokens and 4K context. Compared with KSA, NCSA does better on RULER at 4K (Avg13 +2.15/+4.11/+6.53) and on JRT (+2.87/+0.77/+0.39), is on par or slightly better on language modeling and commonsense tasks, extrapolates better to 8K at 4B, and needs an estimated 14.2% fewer block FLOPs. A single-layer MQAR probe illustrates the ordering argument.

Critical strengths
  1. The comparison is well controlled. Four attention variants share the backbone, tokenizer, data order and training recipe, at three scales up to 4B parameters and 50B tokens. That makes the head-to-head numbers more trustworthy than usual for architecture papers. The method itself is simple, and Algorithm 1 makes it easy to reimplement (summary positions aside, see Q1).

  2. NCSA beats KSA on RULER at every scale while also being cheaper than KSA, and its 4B model extrapolates to 8K much more gracefully (RULER Avg13 28.58 vs 0.49 for KSA and 20.94 for Dense). If these gains hold up, NCSA is an easy drop-in improvement over summary-token compression.

Critical weaknesses
  1. The evidence that NCSA beats KSA, and that the read-before-write ordering is the reason, is thinner than the text suggests.

    • Every pretraining run uses a single seed, with no confidence intervals.
    • At 4B, about 90% of the 6.53-point Avg13 gap comes from three tasks (Table 9): VT 22.12 vs 2.28, CWE 59.40 vs 36.62 and FWE 49.33 vs 15.73. KSA jumps around on two of them across scales (VT 6.60 → 1.08 → 2.28, FWE 3.53 → 31.07 → 15.73), and on CWE NCSA beats even Dense by 13.7 points (59.40 vs 45.70). Single runs on these tasks look volatile. On the eight needle-retrieval tasks NCSA and KSA are tied at 4B (70.49 vs 70.02), although NCSA leads there at 0.6B (54.65 vs 48.87).
    • The JRT margin shrinks as the models grow (2.87, 0.77, 0.39), which does not fit the claim that NCSA turns scale into larger gains.
    • NCSA changes several things at once relative to KSA: the write/read order, keeping summaries outside the sequence, summaries no longer attending to themselves, and possibly the position scheme (not described). No ablation changes only the ordering.
    • The MQAR probe uses a single layer, where KSA's summaries are constant by construction, as the paper notes. It confirms this layer-0 limitation but says little about whether ordering matters in a deep model. And CSA also writes and reads in the same layer (Table 7), yet it is the weakest compressed model on both RULER and JRT.

    I am not yet convinced that the ordering, rather than one of the other differences or run-to-run noise, drives the gains.

  2. Novelty and efficiency are overstated.

    • The closest prior designs are not discussed. CCA-Attention (ICML 2025) pools each token group into one "core" token in the same layer with the layer's own projections, adds no parameters, and mixes these tokens with a local window in one softmax. That is close to the "No stream" ablation, which already recovers most of NCSA's gain over KSA (42.38 vs 43.19, with KSA at 39.08, all at 0.6B). Luna's pack/unpack attention also keeps an extra stream outside the sequence that is written and read in the same layer.
    • KSA is run with C=4, a 128-token window and compression in every layer, while the KSA report's default is chunk size 8, a window of 128 chunks (about 1K tokens) and a 3:1 hybrid with full attention. There is no KSA or CSA hybrid to compare with the paper's own 3:1 NCSA hybrid.
    • Efficiency is argued only through analytic FLOPs against KSA at 4K, with no throughput, latency or memory measurements. With the same counting I get about 8.5 GFLOPs/token for Dense at 4K, so NCSA saves only about 4% against plain attention at this length, and KSA is actually more expensive than Dense. Contexts beyond 8K, where compression is supposed to matter, are not tested.
Questions for authors
  1. How are RoPE positions assigned to NCSA's summary queries/keys, to KSA's summary tokens and to CSA's compressed entries? (The KSA report gives each summary the position of its chunk's last token.) Raw-token distances never exceed the 128-token window, so this choice could explain both the RULER gaps and KSA's collapse beyond 4K at 4B, which does not appear at 0.6B.
  2. Please add an ablation that changes only the ordering, e.g. NCSA text queries reading the incoming, not-yet-updated summaries, and 2–3 seeds or paired confidence intervals for RULER and JRT at 0.3B/0.6B.
  3. Please compare against CCA-Attention (or a simple pooling compressor), and against KSA in its default setting.
  4. Please report measured prefill/decode throughput and memory against Dense, at contexts well beyond 4K (e.g., 32K–128K).
3 comments

We thank the reviewer for the valuable feedback. We address the concerns below.

W1/Q1: RoPE positions assigned to NCSA's summary queries/keys, KSA's summary tokens and CSA's compressed entries

  • KSA and NCSA: both assign each summary the RoPE position of its chunk's last token.
  • CSA: the group-head position, i.e. the first token of its group, as in the official DeepSeek implementation.

W1/Q2: An ordering-only ablation with seeds or paired confidence intervals for RULER and JRT at 0.3B/0.6B

We trained a control that changes only the ordering of the read, exactly as suggested: stage-2 text queries read the incoming summary state instead of the same-layer updated gist.the only difference is which gist rows the text queries attend to. On RULER-13 (4K, 0.6B, three evaluation seeds):

RULER-13 meaneval seed 1eval seed 2eval seed 3meanpaired diff (NCSA - control)
NCSA43.3042.7043.4043.13+2.59, 95% CI [+1.52, +3.66]
control (staleread)40.6240.5840.4340.54

Pairwise t-interval uses the paired seed-level differences (n=3, t_{0.975,2}=4.303). The gap exceeds the spread across evaluation seeds; a training-seed replication is in progress and will be reported in a later version of this response.

For completeness, Table 1 gives the RULER-13 mean scores of NCSA and KSA at 0.3B and 0.6B (three evaluation seeds each):

modeleval seed 1eval seed 2eval seed 3
NCSA 0.3B34.5735.1035.41
KSA 0.3B32.5532.3431.99
NCSA 0.6B43.3042.7043.40
KSA 0.6B39.0337.6238.90

NCSA is ahead of KSA in every one of the six seed-scale pairs, and the two arms do not overlap within either scale (0.3B: NCSA min 34.57 vs KSA max 32.55; 0.6B: 42.70 vs 39.03). Paired differences are +2.73 (95% CI [+0.99, +4.47]) at 0.3B and +4.62 (95% CI [+3.58, +5.65]) at 0.6B, so the margin grows with model size in this range.

JRT. JRT is a fixed, deterministically scored item set. Its uncertainty is therefore reported as a 95% paired bootstrap over the evaluation items for all four architectures:

scaleDENSENCSAKSACSA
0.3B38.17 [37.2, 39.2]34.73 [33.8, 35.7]31.86 [31.0, 32.8]28.36 [27.5, 29.3]
0.6B43.80 [42.9, 44.7]40.78 [39.9, 41.7]40.01 [39.1, 41.0]39.09 [38.2, 40.0]

At 0.3B, NCSA retains the most of Dense's accuracy among the compressed architectures (+2.9 over KSA and +6.4 over CSA on the 6-task mean), driven mainly by fda and swde. At 0.6B the gap to KSA narrows to +0.8 points, within the bootstrap intervals, while NCSA remains +1.7 over CSA. Dense remains about 3 points above NCSA on average.

W2/Q3: Comparison against CCA-Attention (or a pooling compressor) and against KSA in its default setting

We are training KSA, NCSA and CCA under the same KSA hybrid default architecture, we will report these results in a later version of this response

W2/Q4:Measured prefill/decode throughput and memory against Dense at 32K-128K

We report measured prefill/decode throughput and memory against Dense at 4K-128K (4B), from a unified implementation in which both architectures share the same block code and attention kernel, on an exclusive GPU (min over iterations; decode times are full-cycle means that include the periodic summary write).

Prefill latency (batch=1, ms; throughput = S/latency):

SDenseNCSADense/NCSA
4K53.664.70.83x
8K120.1119.81.00x
16K300.2244.81.23x
32K858.9543.31.58x
64K2786.91343.62.07x
128K9901.43772.52.62x

Decode (each arm at its own memory-limited batch):

SDense max_batchNCSA max_batchDense tok/sNCSA tok/sNCSA/Dense
8K1475544332131183.03x
32K38152109836673.34x
64K197355818283.28x
128K9382789413.39x

At batch=1 the per-token latency is identical across arms (30-34 ms at all lengths, dominated by per-layer scheduling); the throughput gain comes from the 3.8-4.2x larger feasible batch enabled by the smaller KV, net of the ~25% per-user latency overhead of the amortized summary write at each arm's max batch.

Memory:

SDense decode KVNCSA decode KVcompressionDense prefill peakNCSA prefill peak
32K4608 MiB1171 MiB3.94x12.72 GiB13.28 GiB
64K9216 MiB2323 MiB3.97x17.16 GiB18.28 GiB
128K18432 MiB4627 MiB3.98x26.05 GiB28.30 GiB

NCSA's resident decode KV is 3.9-4.0x smaller than Dense at 32K-128K; its prefill peak memory is ~8% higher at 128K because it holds both the raw window and the summaries during prefill. KV sizes are closed-form (Dense: S; NCSA: swc*C + C + S/C rows per layer) and identical across machines.

W2/Q3: Comparison against CCA-Attention (or a pooling compressor) and against KSA in its default setting

We trained CCA, KSA and NCSA at 0.3B using KSA's default architecture: hybrid 3:1 (one full-attention layer per three compressed layers), chunk size 8, and a sliding window of 128 chunks (1024 tokens), with the same data and training budget as the 0.3B model in the paper. All three arms, together with the previously trained Dense baseline, are evaluated on the same RULER-13 benchmark at 4K.

taskDenseNCSAKSACCA
niah_single_199.8097.7692.3297.76
niah_single_299.4097.4088.5293.44
niah_single_394.8086.9279.4837.12
niah_multiquery27.7525.7428.8127.47
niah_multivalue27.9523.6333.3125.87
niah_multikey_159.0049.4056.8447.52
niah_multikey_27.4026.727.6040.12
niah_multikey_33.802.045.282.96
vt1.409.653.942.17
cwe23.3417.7014.5930.62
fwe0.5323.8917.0824.57
qa_squad27.8722.6029.5122.52
qa_hotpot22.8019.4820.1618.72
Avg1338.1438.6936.7336.22

NCSA is on par with Dense (+0.55) and above both compressed alternatives (+1.96 over KSA in its default setting, +2.47 over CCA).

We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

Reviewer_cNYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NCSA proposes a native chunk-summary attention architecture that fixes two inefficiencies of summary-token compression (e.g., KSA): the read-before-write gap, where a chunk summary becomes visible to text attention only in the next layer, and mandatory feed-forward computation on summary positions. By keeping summaries in a residual stream outside the token sequence, NCSA writes and reads summaries within the same layer and makes the summary-side FFN optional.

Critical strengths
  1. The paper identifies a subtle execution-order defect in summary-token attention through explicit forward/backward computation-graph analysis: the read-before-write gap and its delayed credit assignment.
  2. All architectures share identical backbones, data order, optimizer, and 50B-token budgets across three scales, isolating the architectural variable.
Critical weaknesses
  1. NCSA still uses full attention over local tokens plus summary reads, so real training/inference speedups versus strong sparse or linear-attention baselines (e.g., NSA, Gated DeltaNet) are unmeasured.
  2. Benefits of the methods are attributed to write-before-read, but this mechanism is never disentangled from other confounds like summary self-attention removal.
Questions for authors

See weaknesses.

4 comments

We thank the reviewer for the valuable feedback. We address the concerns below.

W1/Q1: Measured speedup of NCSA against strong sparse and linear-attention baselines (NSA, Gated DeltaNet)

Our efficiency claims are relative to Dense and are supported by end-to-end measurements under a unified implementation: at 128K (4B), NCSA reduces prefill latency by 2.62x, raises decode throughput at its memory limit by 3.0-3.4x, and cuts resident KV by 3.5-4.0x.

Following the request, we measured the attention kernel itself (prefill, forward, batch=1) for Dense, NCSA, GDN and NSA at 4K-128K, at 4B and 0.6B. The baselines belong to different capability classes (GDN: fixed-capacity recurrent state; NSA: a fixed-budget block selection combined with a sliding window; NCSA: length-growing, exactly indexable summaries), so we primarily report each relative to Dense. NCSA's figure covers both its stage-1 (summary writing) and stage-2 (main attention).

Attention-kernel latency, 4B (ms, and speedup over Dense in parentheses; >1 means faster than Dense):

4B, batch=14K8K16K32K64K128K
Dense0.501.565.5521.0481.99323.52
GDN0.61 (0.82x)0.79 (1.97x)1.20 (4.62x)2.00 (10.52x)3.77 (21.75x)7.33 (44.14x)
NSA1.27 (0.39x)2.30 (0.68x)4.60 (1.21x)10.38 (2.03x)26.80 (3.06x)79.29 (4.08x)
NCSA0.48 (1.04x)0.80 (1.95x)1.91 (2.91x)5.51 (3.82x)18.14 (4.52x)65.76 (4.92x)

Attention-kernel latency, 0.6B (ms, and speedup over Dense in parentheses; >1 means faster than Dense):

0.6B, batch=14K8K16K32K64K128K
Dense0.340.913.0210.9841.78163.01
GDN0.61 (0.56x)0.80 (1.14x)0.92 (3.28x)1.47 (7.47x)2.70 (15.47x)5.14 (31.71x)
NSA0.89 (0.38x)1.42 (0.64x)2.44 (1.24x)5.28 (2.08x)13.32 (3.14x)39.20 (4.16x)
NCSA0.35 (0.97x)0.57 (1.60x)1.22 (2.48x)3.21 (3.42x)10.06 (4.15x)35.48 (4.59x)

Three observations follow, all at the attention-kernel level. (i) At 4K NCSA is at parity with Dense (1.04x/0.97x), because the per-chunk summary write is a fixed cost that dominates at short context; at 16K NCSA is 2.9x faster than Dense, rising to 4.9x at 128K (4B) and 4.6x at 128K (0.6B). These are kernel-level ratios; end-to-end prefill, which additionally includes projections and the MLP, is 2.62x at 128K and 0.83x at 4K. (ii) NCSA is faster than NSA at every length, by 2.5x at 4K and 1.2x at 128K (4.9x vs 4.1x over Dense at 128K). (iii) GDN is the fastest arm at long context, as expected for a linear-attention method with a fixed-capacity recurrent state that reads O(1) KV per step but cannot exactly retrieve arbitrary history, whereas NCSA's summaries provide an exact, per-chunk indexable representation; a wall-clock comparison across these two capability classes is not an architecture-level judgment.

GDN and NSA are measured with their production kernels from flash-linear-attention, while NCSA is measured with the unified implementation used throughout this work.

W2/Q2:Disentangling write-before-read from other confounds such as summary self-attention

We trained a control identical to NCSA except that stage-2 reads the gist produced by the previous layer (no same-layer write-then-read), with the same model size, training data, and training budget.

The table below reports the full RULER-13 benchmark evaluated at 4K context: each column is the mean score over all 13 tasks for one evaluation seed, and the last column averages the three seeds.

RULER-13 @ 4K (mean of 13 tasks)eval seed 1eval seed 2eval seed 3mean
Control (previous-layer gist)40.6240.5840.4340.54
NCSA43.3042.7043.4043.13

The +2.59-point gap, with non-overlapping seeds (control max 40.62 < NCSA min 42.70), isolates the benefit of same-layer write-then-read: all other confounds (summary self-attention, the gist-path MLP, head counts) are identical between the two.

We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

Thank you for your response.

After careful consideration, I decided to maintain my score. I remain open to the discussions and decisions of other reviewers.

We thank the reviewer for the continued consideration. We would be grateful to know whether the response addressed most of the reviewer's concerns and what reservations remain. Knowing which points are still in doubt would help us make the revision more useful, and we would be glad to provide further analysis if that would help. As the deadline approaches, a brief reply would be much appreciated if the reviewer is able to provide one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.