- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
NCSA introduces a native compressed-attention architecture that maintains one summary state per chunk outside the ordinary token sequence. Within each layer, it first updates chunk summaries and then exposes the updated summary keys and values to text attention in the same layer. This removes the one-layer read-before-write delay of summary-token architectures and makes summary-side feed-forward computation optional. Matched 50B-token pretraining experiments at 0.3B, 0.6B, and 4B compare NCSA with dense attention, a dedicated compressor, and summary-token compression.
Critical strengths
-
The paper identifies a precise architectural limitation of summary-token compression: a summary written in one layer cannot be read by text positions until the following layer. The forward- and backward-graph analyses clearly motivate the proposed write-before-read ordering.
-
The empirical comparison is well controlled. Architectures at each scale use the same backbone, data order, optimization recipe, context length, and number of training tokens. NCSA consistently improves RULER and real-document retrieval over the compressed baselines, with particularly strong scaling at 4B.
Critical weaknesses
-
The efficiency claim is supported primarily by analytical FLOP estimates. NCSA introduces two sequential attention stages per layer—summary writing followed by text attention—which may have different kernel-launch, synchronization, and memory-access behavior from KSA’s more parallel computation. Actual pretraining throughput, prefill latency, decode latency, and peak KV memory are needed before concluding that the 14.2% block-FLOP reduction yields practical speedups.
-
The principal models are trained at 4K, and most evaluations are at or below 4K. Although the 8K extrapolation is encouraging, practical long-context compression claims would be substantially stronger with evaluation or continued training at 16K–128K and with end-to-end generative tasks rather than predominantly retrieval-oriented diagnostics.
Questions for authors
-
What wall-clock throughput and latency does NCSA achieve relative to KSA, CSA, and dense attention using equally optimized kernels? How much of the analytical FLOP saving survives the sequential write/read implementation?
-
What are the exact prefill and decoding KV-cache sizes and memory bandwidth costs at long sequence lengths?
-
Do the gains persist when models are trained directly at 16K or longer contexts and evaluated on natural long-document QA or generation?
-
How variable are the reported gains across training seeds, particularly the smaller commonsense and JRT differences?
3 comments
W2/Q3: Whether the gains persist for models trained at 16K or longer and evaluated on natural long-document QA or generation
We continued pretraining the 0.3B Dense, NCSA and KSA checkpoints from their 4K schedule to a 16K context for 6B tokens, and evaluated the final checkpoint of each arm on 12 LongBench datasets (2,950 samples).
| task | metric | n | Dense | NCSA | KSA | NCSA - Dense |
|---|---|---|---|---|---|---|
| gov_report | ROUGE | 200 | 17.04 | 16.09 | 14.45 | -0.96 |
| multi_news | ROUGE | 200 | 17.21 | 13.80 | 14.03 | -3.41 |
| qmsum | ROUGE | 200 | 19.21 | 18.35 | 16.95 | -0.85 |
| vcsum | ROUGE-zh | 200 | 5.21 | 5.76 | 6.62 | +0.56 |
| narrativeqa | F1 | 200 | 2.78 | 2.71 | 2.92 | -0.08 |
| qasper | F1 | 200 | 7.24 | 10.04 | 7.71 | +2.80 |
| 2wikimqa | F1 | 200 | 7.61 | 8.41 | 8.87 | +0.80 |
| hotpotqa | F1 | 200 | 5.97 | 5.65 | 4.94 | -0.32 |
| multifieldqa_en | F1 | 150 | 15.43 | 15.79 | 12.31 | +0.36 |
| lsht | accuracy | 200 | 12.50 | 11.00 | 12.00 | -1.50 |
| lcc | edit-sim | 500 | 12.97 | 18.74 | 16.13 | +5.77 |
| repobench-p | edit-sim | 500 | 23.26 | 22.38 | 23.26 | -0.88 |
| mean | - | 2950 | 12.20 | 12.39 | 11.68 | +0.19 |
On natural long-document summarization, QA and code completion, the comparison is unchanged after continued pretraining to 16K: NCSA (12.39) stays ahead of KSA (11.68) and level with Dense (12.20), with a mean per-task difference of +0.19 against Dense (median -0.20) and every task except lcc (+5.77) within 3.5 points. Continuing pretraining at 16K therefore neither erodes the lead over KSA nor costs quality relative to Dense.
We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.
We thank the reviewer for the valuable feedback. We address the concerns below.
W1/Q1: Wall-clock throughput and latency vs KSA, CSA and Dense under equally optimized kernels
All arms share the same block code and the same attention kernel, and differ only in mask/KV layout and in which tokens take which path. Prefill latency at batch=1 (4B, ms; parentheses give speedup over Dense, where >1 means faster than Dense):
Cross-arm comparison: KSA takes 1.18x NCSA's time at 4K, narrowing to 1.04x at 128K; CSA is about 5% faster than NCSA at 4K and within 1% from 8K onward.
Decode is compared at an equal memory budget rather than at an equal batch: for each arm we determine the largest batch that fits in GPU memory and report throughput at that point. Per-token latency is averaged over a full cycle of C steps, so the summary write that occurs every C steps is included in the reported cost rather than averaged away.
At batch=1 the per-token latency is identical across arms (30-34 ms at every length, dominated by per-layer scheduling); the gain appears as a 3.8-4.2x larger feasible batch, which amounts to a 3.0-3.4x throughput gain once the amortized summary write is included. At 128K the measured prefill speedup (2.62x) matches the analytical bound implied by the FLOP reduction (2.63x); at short context a larger share of the analytical saving is consumed by the sequential stage-1 write, whose share of NCSA's attention time falls from 33% at 4K to 3.7% at 128K. Taken together, essentially the whole analytical saving survives at long context, and about 80% of it survives at 4K.
W1/Q2:Exact prefill and decode KV-cache sizes and memory bandwidth at long sequence lengths
Decode resident KV (4B, bf16):
NCSA's cache is a 128-row text window, the in-flight chunk, and S/4 summary rows; these are closed-form and identical across machines. Prefill temporary KV is S columns per layer for Dense and S + S/C for NCSA. Each decode step reads 18432 MiB versus 4627 MiB at 128K (4.0x fewer bytes) and 576 versus 163 MiB at 4K. Measured effective bandwidth (amortized single-step SDPA over the resident KV at 128K) is ~6.6 TB/s for both arms, so the gain comes from reading 4x fewer bytes, not from reading faster.
W2/Q3: Whether the gains persist for models trained at 16K or longer and evaluated on natural long-document QA or generation
These runs are currently in progress: we are training at 16K+ and will evaluate on natural long-document QA/generation, and will report the results in a later version of this response.
Q4: Variability of the reported gains across training seeds, including the smaller commonsense and JRT margins
We first separate two sources of variance. The three seeds we currently report are evaluation seeds on fixed checkpoints: they bound evaluation noise, not training variance. Measured on RULER-13 (13 tasks, mean per evaluation seed):
Paired differences (NCSA - KSA) are +2.73 +- 0.70 at 0.3B (95% CI [+0.99, +4.47]) and +4.62 +- 0.42 at 0.6B (95% CI [+3.58, +5.65]); both exclude zero. Within-arm evaluation spread is 0.3-0.8 points on the 13-task mean, so we make no significance claim for differences smaller than this spread, such as the smaller commonsense margins.
Training-seed variance requires retraining from different initializations and data orders. Those runs are in progress; we will report them in a later version of this response