- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Simplified Sparse Attention (SSA), which repurposes interleaved gist tokens as both compressed context representations and learned routing signals. During continued pretraining, an attention-mask bottleneck encourages each gist token to summarize a local chunk. During decoding, the current query scores the gist keys, and the raw KV pairs of the top-ranked chunks are selectively unfolded into the attention context. The paper also proposes H-SSA, a hierarchical gist-of-gist extension intended to provide coarse-to-fine routing and logarithmic per-token decoding complexity when hierarchy depth grows with context length and selection budgets remain constant.
The method is evaluated on LongBench, five multi-document RAG benchmarks, passkey retrieval, and H100 latency benchmarks. The main empirical results show consistent improvements over gist-compression baselines at matched nominal compression ratios, substantial RAG gains over inference-time and compression baselines, and up to a reported 3.37x end-to-end decoding speedup over Flash-Decoding at 44K context. Ablations compare different hybrid contexts, random/recent/gist routing, and adaptive top-k against top-p selection.
Critical strengths
-
SSA gives gist tokens a second role as learned routing keys and selectively restores full-resolution KV blocks only when needed. This is a simple, well-motivated connection between context compression and sparse attention. The paper does not merely propose the idea: it provides two training stages, a hierarchical construction, GQA-aware routing, and custom prefill/decode kernels. This makes the work substantially more concrete than a conceptual analogy.
-
The evaluation includes full-attention continued-pretraining/finetuning references, gist-compression baselines, inference-time sparse methods, BM25 at the same token budget, and random/recent routing ablations. In particular, replacing the gist score with random or recency-based rankings produces large drops, supporting the claim that the learned gist keys carry query-conditioned relevance. Reporting both operator-level and end-to-end latency is also valuable.
Critical weaknesses
-
Appendix B states that raw tokens in chunk attend to all preceding gist tokens. These raw-to-gist edges alone contribute
but Appendix D.1 counts only intra-chunk attention, , and gist-to-gist attention, . Consequently, single-level SSA prefill appears to be , not . For fixed , both expressions remain quadratic, but the claimed reduction factor changes. For H-SSA, Table 1 assumes growing depth, whereas the experiments use two levels; Appendix D.2 should explicitly count raw-query-to-summary edges and explain how its analysis corresponds to Appendix E's global-summary slab. The decoding claim also needs qualification: Appendix C makes grow with at fixed compression settings, while Appendix D assumes constant selection budgets, and Appendix E explicitly includes an compaction scan. These discrepancies affect the central scalability claims, although they do not by themselves invalidate finite-length latency measurements.
-
The existing LongBench, RAG, and routing ablations provide useful evidence, but the generation protocol, evaluation seeds, and variability are not fully reported. Single reported values do not establish whether the gains, especially those over full attention, are stable across evaluation randomness. The saturated passkey test does not assess whether sparse routing preserves information needed for distributed aggregation and question answering; targeted RULER evaluation would address this gap. On efficiency, Appendix E describes routing overhead and an optional CUDA Graph path, but the execution path used in Figure 4, raw-KV residency, preprocessing coverage, and measurement conditions remain unclear. Hardware utilization and performance across representative batch sizes are needed to distinguish reduced KV traffic from effects of parallelism and baseline tuning. These concerns limit the strength and scope of the quality and efficiency conclusions.
Questions for authors
-
Please provide the complete evaluation protocol: toolkit, greedy versus sampled decoding, temperature, generation sampling top-k/top-p, and maximum output length. I recommend using lm-eval-harness where the tasks are supported to improve reproducibility and comparability. If custom scripts are retained, document differences in prompting, truncation, stopping rules, answer extraction, and scoring. For non-log-likelihood evaluations with randomness, please report all five individual seed results, their mean and standard deviation, using the same predefined seeds and evaluation instances across methods.
-
Where are the gist/meta-gist and full raw KV caches stored during decoding: GPU HBM or CPU DRAM? Please report total GPU memory, including raw KV, summary KV, metadata, and temporary buffers. If raw KV is offloaded, explain the transfer path and whether its cost is included in the decoding measurements.
-
Does Figure 4 include gist scoring, mask/bitmap construction, top-k selection, compaction, and all host-side preprocessing? Was the optional CUDA Graph path enabled? Please identify the dense decoding implementation and compare with an optimized dense baseline, such as an appropriate FA3 implementation, under matched graph-capture, precision, warmup, and timing conditions.
-
Please report batch size, warmup/repetition counts, and split-K configuration, together with achieved SM occupancy for both SSA/H-SSA and the dense baseline. Compare batch 1 and a small set of higher-concurrency settings, including one near GPU saturation, reporting TPOT, throughput, and peak memory. Keep model head counts and other workload conditions identical between methods, allow each implementation appropriate split-K tuning.
-
Please evaluate the RULER tasks CWE, FWE, QA-SQuAD, and QA-Hotpot, covering representative context lengths within and beyond the continued-pretraining range. Please disclose context-extension configuration (e.g., YaRN), and use matched configurations across SSA/H-SSA and the full-attention baseline. I recommend using lm-eval-harness for a standardized, reproducible evaluation setup. NIAH variants can be omitted given the saturated passkey results. Use five predefined data-construction seeds, with identical instances across methods, and report per-task seed results, mean and standard deviation. These tasks test aggregation and evidence integration that successful passkey retrieval alone does not establish.
-
Please also include a limited long-output evaluation using the existing checkpoints. I recommend AIME, AMC, or GPQA with chain-of-thought generation as candidate benchmarks, selecting one or two tasks where the full-attention baseline performs meaningfully and generates sufficiently long outputs. Long-input evaluations with short answers, such as LongBench and RULER, provide limited evidence about routing errors accumulating over extended decoding. Under matched prompts, generation budgets, and the evaluation protocol in Q2, report task quality and actual generated lengths, together with how decoding latency and memory usage evolve as the generated suffix grows. This would also clarify whether the speedups persist as the always-attended uncompressed suffix described in Appendix E grows.
-
After the standardized evaluation requested above, do the gains over a comparably trained full-attention baseline remain consistent across seeds? If so, could the authors investigate whether noise filtering explains those gains using a controlled MQAR experiment, for example with no-padding and random-padding conditions, together with controls for sequence length and positional effects? The key comparison would be how distractors affect each method, rather than accuracy in the padded condition alone. Full attention can itself learn to suppress irrelevant content through query-dependent softmax weights; therefore, outperforming it does not by itself establish denoising as the cause. If the gains no longer hold, please narrow the corresponding performance and mechanism claims. If the gains persist but the mechanism is not isolated, please present noise filtering as a hypothesis rather than an established explanation.
11 comments
Author Response to Reviewer YbNs (2/5)
W2 & Q2: Where the caches are stored, and GPU memory
The raw KV cache, the gist and meta-gist KV entries and all metadata reside in GPU memory (HBM). Nothing is offloaded to CPU memory, so decoding involves no host–device transfer.
GPU memory at batch 1 on an H100 (the model weights take 14.2 GB):
| Context | SSA (L=16): during decoding / peak | H-SSA (L=4, J=4): during decoding / peak |
|---|---|---|
| 8K | 14.8 / 16.2 GB | 15.0 / 16.2 GB |
| 14K | 15.3 / 18.4 GB | 15.8 / 18.5 GB |
| 21K | 16.0 / 21.3 GB | 16.6 / 21.3 GB |
| 29K | 16.8 / 25.6 GB | 17.7 / 25.6 GB |
| 36K | 17.8 / 30.7 GB | 18.9 / 30.8 GB |
| 44K | 18.8 / 36.8 GB | 20.2 / 36.8 GB |
Breakdown for SSA (L=16), beyond the model weights:
| Context | Raw KV | Summary KV | Metadata and other | Temporary buffers (peak only) |
|---|---|---|---|---|
| 21K | 1.05 GB | 0.06 GB | 0.7 GB | 5.3 GB |
| 44K | 2.21 GB | 0.13 GB | 2.3 GB | 18.0 GB |
The raw and summary KV sizes follow from the shapes (56 KB of keys and values per token over 28 layers). "Metadata and other" covers the routing metadata, the boolean gist mask that the model takes as input, and allocator overhead. Temporary buffers exist only during prefill, where our current implementation materializes the attention masks, and are released before decoding.
W2 & Q3: What Figure 4 measures, and the dense baseline
What is included. The time per output token in Figure 4 is end-to-end. It is measured around each generation step, so for SSA it includes gist scoring, top-k selection, mask and bitmap construction, compaction and all host-side preprocessing. The optional CUDA Graph path of Appendix E was not enabled.
Dense baseline. The dense curve in Figure 4 was obtained by running full attention, which passes an explicit attention mask. Following your suggestion, we re-measured all methods under one protocol against an optimized dense baseline, FlashAttention-3.0.1: the same inputs, bf16, batch 1, no graph capture for any method, and one discarded warm-up generation followed by three timed generations of 32 tokens each.
SSA implementation. The SSA and H-SSA numbers below use an updated decode kernel. The method, the model weights, the selection rule, and prefill are unchanged; with the submitted implementation, the same protocol reproduces Figure 4. Two implementation changes account for the difference: (1) routing, i.e. gist scoring and top-k selection, now runs inside the CUDA kernel in the same call as the sparse attention, instead of as separate framework operations in every layer; (2) the first layer, which does not select, uses the sparse kernel instead of masked dense attention.
| Time per output token (ms) | Dense: full attention with explicit attention mask | Dense: FlashAttention-3 | SSA (L=16) | H-SSA (L=4, J=4) |
|---|---|---|---|---|
| 8K | 24.0 | 18.7 | 12.5 | 13.1 |
| 14K | 34.8 | 18.6 | 13.2 | 14.2 |
| 21K | 44.7 | 18.8 | 14.0 | 14.9 |
| 29K | 57.2 | 19.7 | 14.7 | 15.5 |
| 36K | 69.5 | 20.2 | 15.8 | 16.7 |
| 44K | 81.7 | 21.2 | 16.4 | 17.4 |
From the table, we find that against FlashAttention-3, SSA is 1.3–1.5× faster.
Author Response to Reviewer YbNs (3/5)
W2 & Q4: Batch size, split-K, occupancy and concurrency
Figure 4 uses batch 1, with the warm-up and repetitions given in Q3. Split-K is 128 per (sample, KV group) for SSA; its latency is insensitive to this setting (we swept 8, 32, 128 and 512 splits). The dense baseline is FlashAttention-3, as in Q3, with its split-KV decoding enabled. Both methods use the same model, so head counts and all other workload conditions are identical.
Achieved SM occupancy, measured with Nsight Compute at 36K as the mean number of active warps per SM over the limit of 64:
| Method | Kernel | Batch 1 | Batch 8 |
|---|---|---|---|
| FlashAttention-3 | split-KV attention | 18.6% | 18.7% |
| FlashAttention-3 | combine | 11.7% | 11.7% |
| SSA | gist scoring | 6.0% | 23.9% |
| SSA | top-k selection | 49.5% | 80.6% |
| SSA | compaction | 44.2% | 83.3% |
| SSA | sparse attention | 38.1% | 62.0% |
| SSA | combine | 6.2% | 10.5% |
| FlashAttention-3 | all kernels, weighted by time | 17.4% | 18.5% |
| SSA | all kernels, weighted by time | 21.8% | 46.9% |
For higher concurrency, we use a batched version of the decode kernel. Both methods prefill one sequence and replicate its cache to the batch. Each cell is ms per decoding step · output tokens per second.
| Context | Method | B=1 | B=2 | B=4 | B=8 | B=16 |
|---|---|---|---|---|---|---|
| 8K | Dense | 18.7 · 53 | 16.4 · 122 | 16.3 · 245 | 18.6 · 430 | 26.6 · 602 |
| 8K | SSA | 12.5 · 80 | 13.8 · 145 | 15.6 · 256 | 18.9 · 423 | 26.3 · 608 |
| 14K | Dense | 18.6 · 54 | 17.6 · 114 | 18.7 · 214 | 24.8 · 323 | 38.6 · 415 |
| 14K | SSA | 13.2 · 76 | 15.2 · 132 | 18.4 · 217 | 24.3 · 329 | 37.3 · 429 |
| 21K | Dense | 18.8 · 53 | 16.3 · 123 | 20.9 · 191 | 30.8 · 260 | 51.3 · 312 |
| 21K | SSA | 14.0 · 71 | 16.4 · 122 | 20.5 · 195 | 30.1 · 266 | 50.3 · 318 |
| 29K | Dense | 19.7 · 51 | 17.8 · 112 | 24.3 · 165 | 38.2 · 209 | 65.7 · 244 |
| 29K | SSA | 14.7 · 68 | 18.2 · 110 | 24.4 · 164 | 37.5 · 213 | 63.8 · 251 |
| 36K | Dense | 20.2 · 50 | 19.5 · 103 | 28.0 · 143 | 45.4 · 176 | 80.3 · 199 |
| 36K | SSA | 15.8 · 63 | 19.6 · 102 | 28.0 · 143 | 44.3 · 181 | 77.3 · 207 |
| 44K | Dense | 21.2 · 47 | 21.2 · 94 | 31.9 · 125 | 52.6 · 152 | 95.1 · 168 |
| 44K | SSA | 16.4 · 61 | 21.4 · 93 | 31.4 · 127 | 51.0 · 157 | 90.8 · 176 |
Output tokens per second is computed as 1000 × B / (ms per decoding step). SSA is the L=16 model.
From the table, we find: at batch 1, SSA is 1.3–1.5× faster than the dense baseline; at batch 2 and above, SSA ranges from 16% faster to 2% slower.
Peak GPU memory (GB) over the same grid:
| Context | Dense: B=1 | B=2 | B=4 | B=8 | B=16 | SSA: B=1 | B=2 | B=4 | B=8 | B=16 |
|---|---|---|---|---|---|---|---|---|---|---|
| 8K | 15.7 | 17.4 | 18.2 | 19.9 | 23.4 | 16.2 | 17.4 | 18.3 | 20.0 | 23.4 |
| 14K | 16.9 | 19.9 | 21.4 | 24.5 | 30.7 | 18.4 | 20.0 | 21.6 | 24.7 | 30.9 |
| 21K | 18.1 | 22.4 | 24.6 | 29.1 | 38.0 | 21.3 | 22.8 | 25.0 | 29.5 | 38.4 |
| 29K | 19.5 | 25.4 | 28.4 | 34.6 | 46.8 | 25.6 | 26.1 | 29.2 | 35.3 | 47.5 |
| 36K | 21.0 | 28.4 | 32.3 | 40.0 | 55.6 | 30.7 | 30.7 | 33.5 | 41.2 | 56.8 |
| 44K | 22.4 | 31.4 | 36.1 | 45.5 | 64.3 | 36.8 | 36.8 | 37.9 | 47.3 | 66.1 |
The largest setting (44K, B=16) uses 64–66 GB of the 80 GB GPU, so it is near GPU saturation. At batch 1 and 2, the SSA peak is above the dense one (36.8 GB against 22.4 GB at 44K) because our current prefill materializes the attention masks (the temporary buffers of Q2), which are released before decoding; during decoding SSA holds 18.8 GB (Q2), below the dense model's peak. These masks are an implementation choice rather than part of the method, since the decode kernel already evaluates the same mask on the fly.
Author Response to Reviewer YbNs (1/5)
Thank you for your detailed and rigorous review. We appreciate your recognition of SSA's simple and well-motivated connection between context compression and sparse attention, and of its concrete realization with two training stages, a hierarchical construction, and custom kernels.
Below, we address your concerns regarding (W1) the complexity analysis and scalability claims and (W2) the evaluation protocol, variability and efficiency measurements, along with each of your questions (Q1–Q7). In response, we corrected the complexity analysis, documented the full evaluation protocol, ran RULER (four tasks, three lengths, five seeds, six models) and long-output evaluations on AIME, GPQA and MATH under lm-eval-harness, and re-benchmarked latency, throughput, memory and SM occupancy against FlashAttention-3 at six context lengths and five batch sizes. SSA matches or exceeds full attention on RULER at every length, is 1.3–1.5× faster than FlashAttention-3 at batch 1, and stays on par with it up to batch 16, close to GPU saturation.
We hope this fully addresses your concerns and would be grateful if you could consider updating your score based on these improvements.
W1: Complexity analysis and scalability claims
Thank you for the careful review. We have corrected the paper (changes are marked in blue in the revised PDF).
Single-level prefill. Appendix D.1 did not count the raw-to-gist edges that Appendix B specifies. As you derive, they contribute Θ(n2/L) attention pairs, so single-level prefill is O(nL+n2/L), not O(nL+n2/L2). For fixed L it remains quadratic, with a reduction factor of L instead of L2.
H-SSA. Table 1 assumes a hierarchy whose depth grows with the context length, while our experiments use two levels, and Appendix D.2 did not count the raw-to-summary edges. With these edges counted, prefill for the ideal hierarchy is O(nL+nJlogJ(n/L)). For the two-level configuration we evaluate, each raw token attends to its own chunk, the gists of its current segment and all preceding meta-gists, which gives O(nL+nJ+n2/(LJ)). The last term is the global-summary slab of Appendix E: the slab holds the O(n/(LJ)) meta-gist columns that every later query can see, i.e. O(n2/(LJ)) entries, and the local windows form the diagonal band.
Decoding. The O(logn) bound assumes constant selection budgets and a depth that grows with the context length. As you point out, our experiments use the adaptive budget of Appendix C, where k grows with nkv, and Appendix E includes an O(n) compaction scan. Per-step decoding in the evaluated configuration is therefore O(n), and the logarithmic bound applies only to the ideal hierarchy. We have corrected Table 1 and Appendix D, added the costs of the evaluated configuration, and qualified the log-linear statement in the abstract and the conclusion.
W2 & Q1: Evaluation protocol and variability
Q1. All methods in a table are evaluated with the same script and settings.
Seeds and variability. Decoding is greedy in every evaluation, so there is no sampling and generation seeds have no effect. Randomness enters in two places:
lm-eval-harness. RULER (Q5) and the long-output evaluation (Q6) use lm-eval-harness 0.4.13 with one model wrapper for all methods, so tokenization, truncation, stopping and scoring are identical. For LongBench and RAG we keep our own scripts, since the baselines were run with them; the table above documents their prompting, truncation, stopping rules, answer extraction and scoring.