acceptodds
Under review as a conference paper at ICLR 2027

Simplified Sparse Attention via Gist Tokens

Abstract

Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler approach to sparse attention that requires no architectural changes. Concretely, we first perform continued pretraining on sequences interleaved with gist tokens. We optimize the standard next-token loss as usual, but the gist tokens use an attention mask to restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk's important information into the gist tokens. At inference time, SSA scores chunks via attention between the current query and the small set of gist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against the gist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the full KV cache, without requiring the auxiliary KV cache approach used by sparse attention methods. On LongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, in retrieval-augmented generation, SSA can even outperform full attention after continued pretraining by over 5.7 points. We attribute this to the ability of SSA's selective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. Beyond accuracy, SSA also improves efficiency: its decoding latency remains nearly flat as context length increases, yielding up to a 3.37x end-to-end decoding speedup over Flash-Decoding, while matching and eventually outperforming the FlashAttention baseline during prefill. SSA further extends to a hierarchical gist-of-gist variant (H-SSA) that achieves log-linear decoding complexity while maintaining or improving accuracy at high compression ratios up to 32x.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long context modeling, context compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YbNsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Simplified Sparse Attention (SSA), which repurposes interleaved gist tokens as both compressed context representations and learned routing signals. During continued pretraining, an attention-mask bottleneck encourages each gist token to summarize a local chunk. During decoding, the current query scores the gist keys, and the raw KV pairs of the top-ranked chunks are selectively unfolded into the attention context. The paper also proposes H-SSA, a hierarchical gist-of-gist extension intended to provide coarse-to-fine routing and logarithmic per-token decoding complexity when hierarchy depth grows with context length and selection budgets remain constant.

The method is evaluated on LongBench, five multi-document RAG benchmarks, passkey retrieval, and H100 latency benchmarks. The main empirical results show consistent improvements over gist-compression baselines at matched nominal compression ratios, substantial RAG gains over inference-time and compression baselines, and up to a reported 3.37x end-to-end decoding speedup over Flash-Decoding at 44K context. Ablations compare different hybrid contexts, random/recent/gist routing, and adaptive top-k against top-p selection.

Critical strengths
  1. SSA gives gist tokens a second role as learned routing keys and selectively restores full-resolution KV blocks only when needed. This is a simple, well-motivated connection between context compression and sparse attention. The paper does not merely propose the idea: it provides two training stages, a hierarchical construction, GQA-aware routing, and custom prefill/decode kernels. This makes the work substantially more concrete than a conceptual analogy.

  2. The evaluation includes full-attention continued-pretraining/finetuning references, gist-compression baselines, inference-time sparse methods, BM25 at the same token budget, and random/recent routing ablations. In particular, replacing the gist score with random or recency-based rankings produces large drops, supporting the claim that the learned gist keys carry query-conditioned relevance. Reporting both operator-level and end-to-end latency is also valuable.

Critical weaknesses
  1. Appendix B states that raw tokens in chunk CmC_m attend to all preceding gist tokens. These raw-to-gist edges alone contribute

    L∑m=1M(m−1)=Θ(LM2)=Θ(n2/L),L \sum_{m=1}^M (m - 1) = \Theta(L M^2) = \Theta(n^2 / L),

    but Appendix D.1 counts only intra-chunk attention, O(nL)O(nL), and gist-to-gist attention, O(n2/L2)O(n^2/L^2). Consequently, single-level SSA prefill appears to be O(nL+n2/L)O(nL + n^2/L), not O(nL+n2/L2)O(nL + n^2/L^2). For fixed LL, both expressions remain quadratic, but the claimed reduction factor changes. For H-SSA, Table 1 assumes growing depth, whereas the experiments use two levels; Appendix D.2 should explicitly count raw-query-to-summary edges and explain how its analysis corresponds to Appendix E's global-summary slab. The decoding claim also needs qualification: Appendix C makes kk grow with nkvn_{\mathrm{kv}} at fixed compression settings, while Appendix D assumes constant selection budgets, and Appendix E explicitly includes an O(n)O(n) compaction scan. These discrepancies affect the central scalability claims, although they do not by themselves invalidate finite-length latency measurements.

  2. The existing LongBench, RAG, and routing ablations provide useful evidence, but the generation protocol, evaluation seeds, and variability are not fully reported. Single reported values do not establish whether the gains, especially those over full attention, are stable across evaluation randomness. The saturated passkey test does not assess whether sparse routing preserves information needed for distributed aggregation and question answering; targeted RULER evaluation would address this gap. On efficiency, Appendix E describes routing overhead and an optional CUDA Graph path, but the execution path used in Figure 4, raw-KV residency, preprocessing coverage, and measurement conditions remain unclear. Hardware utilization and performance across representative batch sizes are needed to distinguish reduced KV traffic from effects of parallelism and baseline tuning. These concerns limit the strength and scope of the quality and efficiency conclusions.

Questions for authors
  1. Please provide the complete evaluation protocol: toolkit, greedy versus sampled decoding, temperature, generation sampling top-k/top-p, and maximum output length. I recommend using lm-eval-harness where the tasks are supported to improve reproducibility and comparability. If custom scripts are retained, document differences in prompting, truncation, stopping rules, answer extraction, and scoring. For non-log-likelihood evaluations with randomness, please report all five individual seed results, their mean and standard deviation, using the same predefined seeds and evaluation instances across methods.

  2. Where are the gist/meta-gist and full raw KV caches stored during decoding: GPU HBM or CPU DRAM? Please report total GPU memory, including raw KV, summary KV, metadata, and temporary buffers. If raw KV is offloaded, explain the transfer path and whether its cost is included in the decoding measurements.

  3. Does Figure 4 include gist scoring, mask/bitmap construction, top-k selection, compaction, and all host-side preprocessing? Was the optional CUDA Graph path enabled? Please identify the dense decoding implementation and compare with an optimized dense baseline, such as an appropriate FA3 implementation, under matched graph-capture, precision, warmup, and timing conditions.

  4. Please report batch size, warmup/repetition counts, and split-K configuration, together with achieved SM occupancy for both SSA/H-SSA and the dense baseline. Compare batch 1 and a small set of higher-concurrency settings, including one near GPU saturation, reporting TPOT, throughput, and peak memory. Keep model head counts and other workload conditions identical between methods, allow each implementation appropriate split-K tuning.

  5. Please evaluate the RULER tasks CWE, FWE, QA-SQuAD, and QA-Hotpot, covering representative context lengths within and beyond the continued-pretraining range. Please disclose context-extension configuration (e.g., YaRN), and use matched configurations across SSA/H-SSA and the full-attention baseline. I recommend using lm-eval-harness for a standardized, reproducible evaluation setup. NIAH variants can be omitted given the saturated passkey results. Use five predefined data-construction seeds, with identical instances across methods, and report per-task seed results, mean and standard deviation. These tasks test aggregation and evidence integration that successful passkey retrieval alone does not establish.

  6. Please also include a limited long-output evaluation using the existing checkpoints. I recommend AIME, AMC, or GPQA with chain-of-thought generation as candidate benchmarks, selecting one or two tasks where the full-attention baseline performs meaningfully and generates sufficiently long outputs. Long-input evaluations with short answers, such as LongBench and RULER, provide limited evidence about routing errors accumulating over extended decoding. Under matched prompts, generation budgets, and the evaluation protocol in Q2, report task quality and actual generated lengths, together with how decoding latency and memory usage evolve as the generated suffix grows. This would also clarify whether the speedups persist as the always-attended uncompressed suffix described in Appendix E grows.

  7. After the standardized evaluation requested above, do the gains over a comparably trained full-attention baseline remain consistent across seeds? If so, could the authors investigate whether noise filtering explains those gains using a controlled MQAR experiment, for example with no-padding and random-padding conditions, together with controls for sequence length and positional effects? The key comparison would be how distractors affect each method, rather than accuracy in the padded condition alone. Full attention can itself learn to suppress irrelevant content through query-dependent softmax weights; therefore, outperforming it does not by itself establish denoising as the cause. If the gains no longer hold, please narrow the corresponding performance and mechanism claims. If the gains persist but the mechanism is not isolated, please present noise filtering as a hypothesis rather than an established explanation.

11 comments
AuthorsOpenReview ↗

Author Response to Reviewer YbNs (1/5)

Thank you for your detailed and rigorous review. We appreciate your recognition of SSA's simple and well-motivated connection between context compression and sparse attention, and of its concrete realization with two training stages, a hierarchical construction, and custom kernels.

Below, we address your concerns regarding (W1) the complexity analysis and scalability claims and (W2) the evaluation protocol, variability and efficiency measurements, along with each of your questions (Q1–Q7). In response, we corrected the complexity analysis, documented the full evaluation protocol, ran RULER (four tasks, three lengths, five seeds, six models) and long-output evaluations on AIME, GPQA and MATH under lm-eval-harness, and re-benchmarked latency, throughput, memory and SM occupancy against FlashAttention-3 at six context lengths and five batch sizes. SSA matches or exceeds full attention on RULER at every length, is 1.3–1.5× faster than FlashAttention-3 at batch 1, and stays on par with it up to batch 16, close to GPU saturation.

We hope this fully addresses your concerns and would be grateful if you could consider updating your score based on these improvements.

W1: Complexity analysis and scalability claims

Thank you for the careful review. We have corrected the paper (changes are marked in blue in the revised PDF).

Single-level prefill. Appendix D.1 did not count the raw-to-gist edges that Appendix B specifies. As you derive, they contribute Θ(n2/L)\Theta(n^2/L) attention pairs, so single-level prefill is O(nL+n2/L)O(nL+n^2/L), not O(nL+n2/L2)O(nL+n^2/L^2). For fixed LL it remains quadratic, with a reduction factor of LL instead of L2L^2.

H-SSA. Table 1 assumes a hierarchy whose depth grows with the context length, while our experiments use two levels, and Appendix D.2 did not count the raw-to-summary edges. With these edges counted, prefill for the ideal hierarchy is O(nL+nJlog⁡J(n/L))O(nL+nJ\log_J(n/L)). For the two-level configuration we evaluate, each raw token attends to its own chunk, the gists of its current segment and all preceding meta-gists, which gives O(nL+nJ+n2/(LJ))O(nL+nJ+n^2/(LJ)). The last term is the global-summary slab of Appendix E: the slab holds the O(n/(LJ))O(n/(LJ)) meta-gist columns that every later query can see, i.e. O(n2/(LJ))O(n^2/(LJ)) entries, and the local windows form the diagonal band.

Decoding. The O(log⁡n)O(\log n) bound assumes constant selection budgets and a depth that grows with the context length. As you point out, our experiments use the adaptive budget of Appendix C, where kk grows with nkvn_{kv}, and Appendix E includes an O(n)O(n) compaction scan. Per-step decoding in the evaluated configuration is therefore O(n)O(n), and the logarithmic bound applies only to the ideal hierarchy. We have corrected Table 1 and Appendix D, added the costs of the evaluated configuration, and qualified the log-linear statement in the abstract and the conclusion.

W2 & Q1: Evaluation protocol and variability

Q1. All methods in a table are evaluated with the same script and settings.

LongBench (Table 2)RAG (Tables 3, 5)
ToolkitLongBench official prompts and metrics, in our own prediction scriptOur own scripts, following the KVLink setup
DecodingGreedy (no sampling, one beam)Greedy (no sampling, one beam)
Temperature, sampling top-k / top-pNot usedNot used
Maximum output lengthLongBench per-task limit: 32 (HotpotQA, 2WikiMQA, MuSiQue, TriviaQA), 64 (MF-en), 128 (NarrativeQA, Qasper, SAMSum), 512 (GovReport)200 tokens
Input truncation31,500 tokens, removing the middle, as in LongBenchNone (ten documents, about 2K tokens)
StoppingEnd-of-sequence tokenEnd-of-sequence token
Answer extractionThe generated continuationThe text after the final "Answer:"
ScoringLongBench official metrics: F1 (QA tasks, TriviaQA), ROUGE-L (SAMSum, GovReport)Substring exact match after normalization

Seeds and variability. Decoding is greedy in every evaluation, so there is no sampling and generation seeds have no effect. Randomness enters in two places:

  • The random-selection control of Table 7. Over five seeds it scores 44.28 ± 1.19 (HotpotQA), 38.96 ± 0.68 (2WikiMQA) and 18.41 ± 0.82 (MuSiQue); the per-seed values are 44.78 / 44.56 / 44.92 / 44.95 / 42.18, 39.95 / 38.72 / 38.78 / 39.24 / 38.13 and 19.13 / 17.43 / 17.66 / 19.16 / 18.68.
  • Data construction. The RULER results in Q5 use five predefined construction seeds, with identical instances across methods.

lm-eval-harness. RULER (Q5) and the long-output evaluation (Q6) use lm-eval-harness 0.4.13 with one model wrapper for all methods, so tokenization, truncation, stopping and scoring are identical. For LongBench and RAG we keep our own scripts, since the baselines were run with them; the table above documents their prompting, truncation, stopping rules, answer extraction and scoring.

AuthorsOpenReview ↗

Author Response to Reviewer YbNs (2/5)

W2 & Q2: Where the caches are stored, and GPU memory

The raw KV cache, the gist and meta-gist KV entries and all metadata reside in GPU memory (HBM). Nothing is offloaded to CPU memory, so decoding involves no host–device transfer.

GPU memory at batch 1 on an H100 (the model weights take 14.2 GB):

ContextSSA (L=16): during decoding / peakH-SSA (L=4, J=4): during decoding / peak
8K14.8 / 16.2 GB15.0 / 16.2 GB
14K15.3 / 18.4 GB15.8 / 18.5 GB
21K16.0 / 21.3 GB16.6 / 21.3 GB
29K16.8 / 25.6 GB17.7 / 25.6 GB
36K17.8 / 30.7 GB18.9 / 30.8 GB
44K18.8 / 36.8 GB20.2 / 36.8 GB

Breakdown for SSA (L=16), beyond the model weights:

ContextRaw KVSummary KVMetadata and otherTemporary buffers (peak only)
21K1.05 GB0.06 GB0.7 GB5.3 GB
44K2.21 GB0.13 GB2.3 GB18.0 GB

The raw and summary KV sizes follow from the shapes (56 KB of keys and values per token over 28 layers). "Metadata and other" covers the routing metadata, the boolean N×NN\times N gist mask that the model takes as input, and allocator overhead. Temporary buffers exist only during prefill, where our current implementation materializes the N×NN\times N attention masks, and are released before decoding.

W2 & Q3: What Figure 4 measures, and the dense baseline

What is included. The time per output token in Figure 4 is end-to-end. It is measured around each generation step, so for SSA it includes gist scoring, top-k selection, mask and bitmap construction, compaction and all host-side preprocessing. The optional CUDA Graph path of Appendix E was not enabled.

Dense baseline. The dense curve in Figure 4 was obtained by running full attention, which passes an explicit attention mask. Following your suggestion, we re-measured all methods under one protocol against an optimized dense baseline, FlashAttention-3.0.1: the same inputs, bf16, batch 1, no graph capture for any method, and one discarded warm-up generation followed by three timed generations of 32 tokens each.

SSA implementation. The SSA and H-SSA numbers below use an updated decode kernel. The method, the model weights, the selection rule, and prefill are unchanged; with the submitted implementation, the same protocol reproduces Figure 4. Two implementation changes account for the difference: (1) routing, i.e. gist scoring and top-k selection, now runs inside the CUDA kernel in the same call as the sparse attention, instead of as separate framework operations in every layer; (2) the first layer, which does not select, uses the sparse kernel instead of masked dense attention.

Time per output token (ms)Dense: full attention with explicit attention maskDense: FlashAttention-3SSA (L=16)H-SSA (L=4, J=4)
8K24.018.712.513.1
14K34.818.613.214.2
21K44.718.814.014.9
29K57.219.714.715.5
36K69.520.215.816.7
44K81.721.216.417.4

From the table, we find that against FlashAttention-3, SSA is 1.3–1.5× faster.

AuthorsOpenReview ↗

Author Response to Reviewer YbNs (3/5)

W2 & Q4: Batch size, split-K, occupancy and concurrency

Figure 4 uses batch 1, with the warm-up and repetitions given in Q3. Split-K is 128 per (sample, KV group) for SSA; its latency is insensitive to this setting (we swept 8, 32, 128 and 512 splits). The dense baseline is FlashAttention-3, as in Q3, with its split-KV decoding enabled. Both methods use the same model, so head counts and all other workload conditions are identical.

Achieved SM occupancy, measured with Nsight Compute at 36K as the mean number of active warps per SM over the limit of 64:

MethodKernelBatch 1Batch 8
FlashAttention-3split-KV attention18.6%18.7%
FlashAttention-3combine11.7%11.7%
SSAgist scoring6.0%23.9%
SSAtop-k selection49.5%80.6%
SSAcompaction44.2%83.3%
SSAsparse attention38.1%62.0%
SSAcombine6.2%10.5%
FlashAttention-3all kernels, weighted by time17.4%18.5%
SSAall kernels, weighted by time21.8%46.9%

For higher concurrency, we use a batched version of the decode kernel. Both methods prefill one sequence and replicate its cache to the batch. Each cell is ms per decoding step · output tokens per second.

ContextMethodB=1B=2B=4B=8B=16
8KDense18.7 · 5316.4 · 12216.3 · 24518.6 · 43026.6 · 602
8KSSA12.5 · 8013.8 · 14515.6 · 25618.9 · 42326.3 · 608
14KDense18.6 · 5417.6 · 11418.7 · 21424.8 · 32338.6 · 415
14KSSA13.2 · 7615.2 · 13218.4 · 21724.3 · 32937.3 · 429
21KDense18.8 · 5316.3 · 12320.9 · 19130.8 · 26051.3 · 312
21KSSA14.0 · 7116.4 · 12220.5 · 19530.1 · 26650.3 · 318
29KDense19.7 · 5117.8 · 11224.3 · 16538.2 · 20965.7 · 244
29KSSA14.7 · 6818.2 · 11024.4 · 16437.5 · 21363.8 · 251
36KDense20.2 · 5019.5 · 10328.0 · 14345.4 · 17680.3 · 199
36KSSA15.8 · 6319.6 · 10228.0 · 14344.3 · 18177.3 · 207
44KDense21.2 · 4721.2 · 9431.9 · 12552.6 · 15295.1 · 168
44KSSA16.4 · 6121.4 · 9331.4 · 12751.0 · 15790.8 · 176

Output tokens per second is computed as 1000 × B / (ms per decoding step). SSA is the L=16 model.

From the table, we find: at batch 1, SSA is 1.3–1.5× faster than the dense baseline; at batch 2 and above, SSA ranges from 16% faster to 2% slower.

Peak GPU memory (GB) over the same grid:

ContextDense: B=1B=2B=4B=8B=16SSA: B=1B=2B=4B=8B=16
8K15.717.418.219.923.416.217.418.320.023.4
14K16.919.921.424.530.718.420.021.624.730.9
21K18.122.424.629.138.021.322.825.029.538.4
29K19.525.428.434.646.825.626.129.235.347.5
36K21.028.432.340.055.630.730.733.541.256.8
44K22.431.436.145.564.336.836.837.947.366.1

The largest setting (44K, B=16) uses 64–66 GB of the 80 GB GPU, so it is near GPU saturation. At batch 1 and 2, the SSA peak is above the dense one (36.8 GB against 22.4 GB at 44K) because our current prefill materializes the N×NN\times N attention masks (the temporary buffers of Q2), which are released before decoding; during decoding SSA holds 18.8 GB (Q2), below the dense model's peak. These masks are an implementation choice rather than part of the method, since the decode kernel already evaluates the same mask on the fly.

Reviewer_w6cxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Simplified Sparse Attention (SSA), which interleaves gist tokens during continued pretraining to compress context chunks and uses these learned gists as query-conditioned routing signals. At decoding, SSA selects the top-k relevant chunks and selectively unfolds their raw tokens, requiring no architectural changes.

Critical strengths
  1. SSA repurposes standard continued pretraining with interleaved gist tokens to jointly learn context compression and query-adaptive routing, without new parameters, auxiliary losses, external indexers, or architectural modifications. The same gist tokens serve as both compressed summaries and Top-k selection signals, enabling effective sparse attention even after pretraining alone, with optional fine-tuning.
  2. SSA consistently outperforms gist-compression and inference-time sparse-attention baselines on LongBench.
Critical weaknesses
  1. Missing the most relevant trainable sparse-attention baselines. SSA requires continued pretraining, so its natural competitors are trainable sparse-attention methods such as NSA, MoBA, and DSA, yet all three are explicitly excluded.
  2. The attention patterns of SSA after the CPT can be revealed and discussed in detail.
Questions for authors
  1. This work [1] has been missed in your related work discussion.

[1] InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer w6cx (1/2)

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of SSA's joint learning of compression and query-adaptive routing without new parameters or architectural changes, and of its consistent gains on LongBench.

Below, we address your concerns regarding (W1) the missing trainable sparse-attention baselines and (W2) the attention patterns of SSA after continued pretraining, along with your question (Q1) on related work. In response, we trained MoBA on Qwen2-7B-Instruct through both stages (continued pretraining and finetuning) at three compression ratios and evaluated it on all nine LongBench tasks, traced at every decoding step and layer which answer tokens SSA's selection attends to, and added InComeS to the related work. SSA outperforms MoBA in all six settings, by up to 5.4 points, and its selection covers 5.7–6.4 times the share of answer tokens that a random selection of the same size would. We hope this fully addresses your concerns.

W1: Missing trainable sparse-attention baselines

NSA and DSA both add components outside the standard Transformer: NSA replaces the attention layer with three branches, a compression MLP and a gating module, and DSA adds a separate indexer with its own parameters. A comparison with them would therefore change the architecture as well as the attention pattern, and training either from the same checkpoint was also beyond our compute budget for the rebuttal period. MoBA adds no parameters and is the closest trainable sparse-attention method to our setting, so we trained it on Qwen2-7B-Instruct under the same protocol as SSA.

Setup. We train MoBA on the same corpora and with the same hyperparameters as SSA (one epoch of continued pretraining followed by one epoch of finetuning). While the original MoBA setup applies sparse routing only during prefill and reverts to full attention for generation, we retain its sparse routing during decoding as well for an apples-to-apples comparison.

Average over the nine LongBench tasks.

MethodCPT 8×CPT 16×CPT 32×FT 8×FT 16×FT 32×
MoBA44.6343.2938.6345.3844.2240.23
SSA46.2045.3944.0746.7445.2643.35
H-SSA–43.6442.56–46.4844.94

Per-task results.

MethodNrtvQAQasperMF-enHotpotQA2WikiMQAMuSiQueSAMSumTriviaQAGovRepAvg.
Continued pretraining, 8×
MoBA23.2834.7142.6153.6153.7431.3443.0588.9530.4244.63
SSA23.5536.8442.2161.0254.2332.4744.5990.5630.3646.20
Continued pretraining, 16×
MoBA21.9432.6143.7551.3748.6428.9943.4488.2430.6643.29
SSA24.3033.8643.7655.7051.4634.1744.3291.0229.9445.39
H-SSA21.8931.1137.4656.0150.9830.2544.6389.4131.0543.64
Continued pretraining, 32×
MoBA17.2031.5032.0446.7443.4422.2841.6882.2930.5238.63
SSA22.6134.0337.0854.1952.0330.5344.4290.4031.3144.07
H-SSA16.1630.8239.3053.6649.0830.6244.3589.1729.8942.56
Finetuning, 8×
MoBA26.0438.7950.3046.6451.2230.8743.6489.7031.2145.38
SSA26.2443.5454.2454.2648.1828.9544.3190.1130.7946.74
Finetuning, 16×
MoBA23.5736.7750.0249.5747.3027.9142.4788.5231.8944.22
SSA24.3842.4751.9854.6942.9128.5942.6688.4331.2745.26
H-SSA22.7243.9853.4056.1148.3129.0443.2890.3631.1246.48
Finetuning, 32×
MoBA20.8235.2942.8542.4043.6421.1242.6682.9930.2640.23
SSA24.5840.2445.6850.6439.3827.5542.6989.1030.3043.35
H-SSA22.0343.3250.7952.4745.1827.9342.5689.4730.6844.94

From the tables, we find:

  • SSA outperforms MoBA at every compression ratio, after both continued pretraining and finetuning, by 1.0–5.4 points on the nine-task average. H-SSA is also ahead of MoBA in all four of its settings, by up to 4.7 points.
  • The margin is largest at the highest compression ratio. At 32×, SSA is 5.4 points ahead of MoBA after continued pretraining and 3.1 after finetuning, and H-SSA is 3.9 and 4.7 points ahead.
AuthorsOpenReview ↗

Author Response to Reviewer w6cx (2/2)

W2: Attention patterns of SSA after continued pretraining

We have added an analysis of what SSA's selection attends to after continued pretraining alone. Continued pretraining uses a fixed attention mask and never performs selection, so the analysis shows whether the gist scores it produces already point to the relevant chunks.

Setup. On HotpotQA, 2WikiMQA and MuSiQue, we take the instances whose gold answer occurs verbatim in the context, mark the tokens of those occurrences, and record at every decoding step and every layer whether they are among the keys the step attends to. Recall is the share of answer tokens attended. As a reference, a random selection that reads the same share of the prefix would cover about the same share of the answer tokens (5.4% for H-SSA and 4.1% for SSA). The hit rate is the share of decoding steps that attend to at least one answer token; we give it over all layers and by layer range.

Model (after continued pretraining)Share of prefix readRecall of answer tokensHit rate, all layersLayers 1–3Layers 4–9Layers 10–18Layers 19–27
H-SSA, L=4, J=45.42%30.8%56.5%23.0%43.7%63.8%69.1%
SSA, L=164.12%26.4%50.5%19.4%36.5%58.1%62.6%

From the table, we find:

  • The selection concentrates on the evidence. H-SSA reads 5.4% of the prefix, and that part contains 30.8% of the answer tokens, 5.7 times what a content-agnostic selection of the same size would attend to. For SSA the ratio is 6.4 (26.4% of the answer tokens at 4.1% of the prefix).
  • The pattern is depth-dependent, and the two models agree on it. The hit rate rises from about 20% in layers 1–3 to 63–69% in layers 19–27. The same layers stand out in both models: layers 14, 16, 21 and 22 reach 76–79% for H-SSA and 71–73% for SSA.
  • The pattern is already in place after continued pretraining: the finetuned H-SSA checkpoint gives nearly the same numbers (29.2% recall at 5.2% of the prefix read, hit rate 55.0%).

Q1: Missing related work (InComeS)

Thank you for the pointer; we have added InComeS (Li et al., 2025) to the related work.

Thank you for the response.

I will keep my score unchanged.

AuthorsOpenReview ↗

Thank you for your reply

Thank you for taking the time to review our work and for maintaining the positive score. We appreciate your careful consideration and are glad our clarifications were helpful.

Reviewer_xg3LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SSA uses interleaved gist tokens both to summarize a context and to select which parts of it should be read at full resolution. Continued pretraining teaches the model to pass information through gist tokens; during decoding, query–gist scores select chunks whose original key/value entries are restored to the attention set. An optional finetuning stage trains with this selection procedure. H-SSA extends the idea to a hierarchy of summaries. Experiments cover LongBench with Qwen2-7B-Instruct, document QA with Llama-3.2-1B, retrieval tests, and inference timing. Selectively recovering raw tokens substantially improves on gist-only compression, and the reported decoding speedup over Flash-Decoding reaches 3.37× at 44K context.

Critical strengths
  1. Reusing gist tokens for routing is a useful and economical idea. The model needs no separate learned indexer, and the first training stage uses an ordinary language-modeling loss. Table 7 also shows that the scores carry useful relevance information: gist-based selection outperforms random selection by 8.5–11.1 points across three multi-document QA tasks, with the rest of the sparse-attention machinery held fixed. It is encouraging that selective unfolding already works after the compression stage, before training explicitly with unfolding.

  2. The accuracy gains over gist compression are substantial and reasonably broad. For example, at 16× compression after continued pretraining, SSA reaches 45.39 on LongBench versus 40.64 for ActivationBeacon and 40.89 for UniGist. The document-QA results on a second model family show a similar benefit, including experiments with independently encoded document caches. These results make a convincing case that recovering selected raw tokens is more useful than relying on compressed representations alone.

Critical weaknesses
  1. The scaling claims exceed what the analysis and implementation establish. Appendix B allows raw tokens to attend to all earlier gist tokens, but Appendix D.1 omits those connections. With M = n/L chunks, they contribute L·M(M−1)/2 attention pairs, giving an O(n²/L) term in prefill rather than only O(n²/L²). For hierarchical decoding, the logarithmic bound assumes constant selection budgets and a depth that grows with context length. The evaluated method uses two levels, a selection budget proportional to sequence length (Appendix C), and an O(n) metadata scan (Appendix E). The near-flat timing over 8K–44K is useful empirical evidence, but it does not validate logarithmic scaling for that implementation. The paper needs a corrected prefill analysis and a clear distinction between the ideal hierarchy and the configuration actually evaluated.

  2. The evaluation leaves the effective attention budget and the benefit of the routing score insufficiently separated. Retaining the original KV entries is a reasonable design for sparse attention, but its nominal “compression ratio” is then different from the storage reduction offered by gist-only compression. Actual KV traffic also depends on overlap between the query heads' selected chunks. Appendix J describes a 16× configuration that reads approximately 1% of the cache, without reporting the selected-token counts needed to connect this to the main accuracy tables. In addition, Table 7 compares learned routing only with random and recent selection. Those controls establish query relevance, but do not test the claimed advantage over query-dependent pooled-key routing. A comparison within the same trained model, with measured access budgets matched, would make the contribution of the learned gist score much clearer.

Questions for authors
  1. Can you reconcile the prefill expression in Table 1 and Appendix D.1 with the raw-to-previous-gist connections specified in Appendix B? What are the corrected costs for the actual two-level implementation, including adaptive selection and metadata compaction?
  2. For the main accuracy results and Figure 4, what are the selected-token counts after the GQA union, resident KV memory, and KV bytes accessed per decoding step? Does head overlap explain the approximately 1% access figure, and do the accuracy and timing runs use the same selection settings, including any serving-mode budget rounding?
  3. When only the ranking score is replaced with query–pooled-key similarity in the same trained model, how much of the Table 7 gain remains at a matched access budget? This would address the claimed benefit over fixed pooling without requiring a separate model architecture or training run.
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer xg3L (1/3)

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of SSA's economical gist-based routing, its learned relevance signals, and its broad gains over gist-only compression.

Below, we address your concerns regarding (W1) the scaling claims and complexity analysis and (W2) the effective attention budget and the contribution of the learned routing score, along with each of your questions (Q1–Q3). In response, we re-derived the complexity for all attention edges and for the two-level configuration we evaluate, instrumented the decode kernel to count the tokens and KV bytes each decoding step reads across all nine LongBench tasks and ten configurations, and ran the suggested query–pooled-key routing control in the same trained model. Single-level SSA attends to 71–92% of the tokens a gist-only method reads at the same nominal ratio, and the learned gist score beats pooled-key routing on all three tasks, by 2.6–4.9 points.

We hope this fully addresses your concerns and would be grateful if you could consider updating your score based on these improvements.

W1 & Q1: Scaling claims and complexity analysis

Thank you for pointing out the limitations of the old analysis, and we have corrected the paper (changes are marked in blue in the revised PDF).

Prefill. Appendix B lets every raw token attend to all preceding gist tokens, and Appendix D.1 did not count these edges. They contribute L∑m=1M(m−1)=Θ(n2/L)L\sum_{m=1}^{M}(m-1) = \Theta(n^2/L) attention pairs, so single-level prefill is O(nL+n2/L)O(nL + n^2/L), not O(nL+n2/L2)O(nL + n^2/L^2). For fixed LL it is still quadratic, with a reduction factor of LL instead of L2L^2 over full attention. The same omission affected the hierarchical row of Table 1: with raw-to-summary edges counted, prefill for the ideal hierarchy is O(nL+nJlog⁡J(n/L))O(nL + nJ\log_J(n/L)), not O(nL+nJ/L)O(nL + nJ/L).

Ideal hierarchy versus the evaluated configuration. The O(log⁡n)O(\log n) decoding bound assumes constant selection budgets and a hierarchy depth that grows with the context length. As you point out, our experiments use two levels, the adaptive budget of Appendix C (where kk grows linearly with the context length), and the O(n)O(n) metadata scan of Appendix E. The corrected costs for this two-level configuration are:

  • Prefill: O(nL+nJ+n2/(LJ))O(nL + nJ + n^2/(LJ)). Each raw token attends to its own chunk (LL tokens), the gists of its current segment (at most JJ), and all preceding meta-gists (up to n/(LJ)n/(LJ)).
  • Per-step decoding: O(n)O(n). Routing scores at most O(n/L)O(n/L) summary keys, the query attends to O(n/(LJ))O(n/(LJ)) unfolded raw tokens and selected summary tokens, and compaction scans O(n)O(n) bytes of metadata.

We agree that the near-flat latency over 8K–44K does not validate logarithmic scaling. It is an empirical measurement of the evaluated two-level configuration, whose per-step cost is O(n)O(n) as given above; the logarithmic bound applies only to the ideal hierarchy.

AuthorsOpenReview ↗

Author Response to Reviewer xg3L (2/3)

W2 & Q2: Effective attention budget

Nominal ratio versus storage. We agree that the two should be distinguished. SSA is a sparse-attention method: it keeps the raw KV cache and reduces what each decoding step attends to, so its 8×/16×/32× is an attention-budget ratio and not a storage reduction. For gist-only compression (ActivationBeacon, UniGist) the same ratio is both: at N, the raw tokens are discarded and each step attends to n/N gist tokens. What the comparison in Table 2 needs is that SSA does not attend to more tokens per step than these baselines at the same nominal ratio, and we now report this.

Selected-token counts after the GQA union. We instrumented the decode kernel to count the keys each decoding step keeps, after the union over the query heads of each KV group, and re-ran the LongBench evaluation of Table 2 with the counter on (all nine tasks, every instance, every step).

The table gives the mean number of tokens a decoding step attends to: the unfolded raw tokens, the selected summary tokens, and the suffix (the question and the generated tokens, which are never compressed). For the gist-only baselines (ActivationBeacon, UniGist), the count at ratio N is one summary token per N tokens of the compressed prefix, which is about 11,570 raw tokens on these instances, plus the same suffix.

ConfigurationNominalSSA: unfolded rawSSA: selected summariesSuffixSSA: attended per stepGist-only (prefix / N + suffix)SSA / gist-only
Continued pretraining
SSA, L=88×8941071791,1801,62473%
SSA, L=1616×4833018269590677%
SSA, L=3232×3051019150655292%
H-SSA, L=4, J=416×544245181970904107%
H-SSA, L=8, J=432×29765183545545100%
Finetuning
SSA, L=88×8701051721,1461,61971%
SSA, L=1616×4722917968090275%
SSA, L=3232×3031018749954891%
H-SSA, L=4, J=416×537222171930894104%
H-SSA, L=8, J=432×2866317452453698%

From the table, we find:

  • Single-level SSA attends to fewer tokens per step than a gist-only method at the same nominal ratio: 71–77% at 8× and 16×, and 91–92% at 32×. Its margin over ActivationBeacon and UniGist in Table 2 is therefore not obtained with a larger attention budget.
  • H-SSA matches the gist-only budget at 32× (0.98–1.00) and attends to 4–7% more at 16× (L=4, J=4), because it keeps the selected summary tokens of both levels in addition to the unfolded raw tokens.

Head overlap and the ≈1% figure. The ≈1% is the selection budget of a single query head. Appendix C gives each head k chunks of L tokens, i.e. nkv/(LeffG)nk_v/(L_{\text{eff}}G) tokens, which is about 0.9% of the context at 16× with G=7 heads per KV group. This is the figure quoted in Appendix J. The heads of a KV group share its keys and values, so what a decoding step reads is the union of their selections. Measured on LongBench, this union is 4.4% of the prefix for SSA (L=16), and the ≈1% is the per-head budget before the union. Appendix J did not state explicitly that the figure is per head; we have clarified the sentence and added the post-union numbers.

Selection settings for accuracy and timing. The main accuracy results and Figure 4 use the same selection settings: top-k per query head followed by the union within each KV group. No serving-mode budget rounding was used for any reported number.

Resident KV memory. SSA does not reduce storage. The raw KV cache is identical to the dense model's (57 KB per token over 28 layers). The summary tokens add 1/L to it for SSA (6.25% at L=16) and 1/L+1/(LJ) for H-SSA (31% at L=4, J=4). At a 44K context this is 2.54 GB of raw KV plus 0.16 GB of summary KV for SSA (L=16), against 2.54 GB for the dense model.

KV bytes accessed per decoding step. The attention operator reads the keys and values of the attended tokens in all 28 layers. Measured on the first 40 instances of each LongBench task with the finetuned models, this is 39.0 MB per step for SSA (L=16, 680 tokens) and 55.2 MB for H-SSA (L=4, J=4, 930 tokens), against 686 MB for dense attention over the same contexts, i.e. 5.7% and 8.0%. Routing additionally reads summary keys (keys only) to score chunks: each key is read once per KV group, and the two-level model scores only the gists of the selected segments, which is 20.0 MB (SSA) and 31.1 MB (H-SSA). In total, a decoding step reads 8.5% and 12.5% of what dense attention reads.

AuthorsOpenReview ↗

Author Response to Reviewer xg3L (3/3)

W2 & Q3: Learned gist score versus query-dependent pooled-key routing

We ran the control you suggest, with the access budget of the mean-pooled-key pivot matched to that of the other three pivots. In the same finetuned model as Table 7 (H-SSA, L=4, J=4), we replace only the ranking score: each chunk is ranked by the score between the query and the mean of the chunk's raw-token keys, per query head (for a meta-gist, the mean over the raw keys of its segment), so no gist key enters the score at either level. The top-k budget, the parent constraint, the GQA union, the local window and the hybrid context are unchanged, so every pivot selects the same number of chunks per head.

Selection pivotHotpotQA2WikiMQAMuSiQueAvg.
Recent43.1138.4417.1932.91
Random44.0339.4519.1334.20
Mean-pooled key (new)51.6943.0126.3840.36
Gist (ours)55.1247.9529.0244.03

From the table, we find:

  • Mean-pooled keys carry query relevance: they are 3.6–7.7 points above random selection on the three tasks.
  • The learned gist score is better on every task, by 2.6–4.9 points. When the gist score is replaced by pooled keys at a matched access budget, 42–73% of the Table 7 gain over random selection remains, depending on the task.

This supports the argument in Section 2.1 that the gist key, which is trained through the bottleneck, is a better routing signal than a fixed average of the raw keys. We have added the mean-pooled row to Table 7.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion and Rebuttal Summary

Dear Area Chair and Reviewers,

We thank the reviewers for their reviews and would like to briefly summarize the progress made during the rebuttal and discussion period.

All three reviewers (xg3L, w6cx, YbNs) replied to our rebuttal. Reviewer YbNs stated that the additional results address their main concerns and raised their score to 3 (weak acceptance). Reviewer w6cx kept their weak acceptance. Reviewer xg3L stated that their concerns are addressed and that they hold a positive attitude toward the paper, while maintaining their score. We briefly summarize the primary points addressed:

  1. Complexity analysis and scaling claims (xg3L, YbNs). We corrected the analysis to count the raw-to-gist edges: single-level prefill is O(nL + n²/L). We also added the costs of the two-level configuration we evaluate: O(nL + nJ + n²/(LJ)) for prefill and O(n) per decoding step.

  2. Effective attention budget (xg3L). We instrumented the decode kernel to count the tokens each decoding step attends to on LongBench tasks. Single-level SSA attends to 71–92% of the tokens a gist-only method reads at the same nominal ratio, and H-SSA to 98–107%. The gains over gist compression therefore do not come from a larger attention budget.

  3. Learned gist score versus pooled-key routing (xg3L). We ran the suggested control in the same trained model at a matched access budget. The learned gist score beats mean-pooled-key routing on all three tasks, by 2.6–4.9 points.

  4. Trainable sparse-attention baseline (w6cx). We trained MoBA, the closest trainable method that adds no parameters, under the same protocol as SSA. SSA outperforms it in all six settings, by 1.0–5.4 points on the LongBench average, and H-SSA is ahead in all four of its settings, by up to 4.7 points.

  5. Attention patterns after continued pretraining (w6cx). We traced which answer tokens SSA's selection attends to at every decoding step and layer. The selection covers 5.7–6.4 times the share of answer tokens that a random selection of the same size would.

  6. Evaluation protocol, variability, and RULER (YbNs). We documented the full evaluation protocol and ran RULER under lm-eval-harness: four tasks, three lengths (8K, 16K, 32K), five data-construction seeds. SSA and H-SSA are within the seed-to-seed spread of the comparably trained full-attention model on every task and length.

  7. Efficiency measurements (YbNs). We re-benchmarked all methods under one protocol against FlashAttention-3 and reported memory and five batch sizes. End-to-end, SSA is 1.3–1.5× faster at batch 1 and at least on par at batch 2–16. The peak memory now equals the dense model's at every length.

  8. Quality over long outputs (YbNs). On document-level translation with outputs of 2,750–2,960 tokens, no SSA variant scores below full attention: SSA is 1.2–2.5 chrF higher, and H-SSA is on par (+0.3).

We are sincerely grateful for the reviewers' detailed feedback, which has already helped us significantly strengthen the paper.

Thank you for your time and consideration.

Sincerely,

Authors of SSA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.