- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SANTA++, which is a training-free stochastic sparse-attention method for reducing KV Cache reads during long context autoregressive decoding. Prompt keys are divided into parent groups and then teams. It reads all original keys and values within selected teams by their inclusion probabilities. Results show promising accuracy-efficiency trade-offs and a 1.69x attention-kernel speedup, though benefits are task-dependent and exclude prefill, preparation overhead, and end-to-end latency.
Critical strengths
-
It can be applied to existing transformers without retraining or modifying model weights.
-
It scores only one representative key per team before deciding which parts of the prompt cache to read
-
The sampling budget directly controls how many teams are evaluated
-
It preserves near-dense performance on LongBench v2 and HELMET RAG while reporting lower logical KV access.
-
Well-motivated representative design
Critical weaknesses
-
The reported 1.69x gain only covers one attention-operator, not full-model generation latency or throughput
-
Most results report logical KV-vector accesses rather than measured bandwidth or hardware memory behavior
-
The GPU implementation maintains packed KV copies and metadata, adding about 71.98 MB per layer in the tested configuration.
-
Dense prefill, clustering, team construction, cache packing, temporary allocation, and related latency are excluded from the timing results
-
Increasing the sampling budget does not consistently improve accuracy, making budget selection less predictable
Questions for authors
-
Can you report end-to-end decoding latency and throughput, including team construction, k-means, KV packing, and workspace allocation?
-
Can you measure actual peak memory usage, rather than only logical KV accesses?
-
How does performance generalize beyond Qwen2.5-7B, 32K contexts, batch size one?
-
What explains the substantial degradation on specific tasks such as JSON KV retrieval and harder RULER variants
-
How sensitive are results to parent size P, representative count R, and grouping method.
3 comments
Thanks for the answers. I will maintain my positive score.
Thank you for your continued support and for taking the time to consider our revised results. We appreciate your constructive questions, which helped us improve both the experimental evaluation and the clarity of the paper.
Thank you for the encouraging assessment and concrete requests. We now measure up to 1.23× whole-model decoding speedup (INT4 weights, FP16 KV, batch eight), and report full-request latency, physical traffic, and memory use. References below are to the revised paper.
W1, W4, Q1 — Full-model latency and preparation.
Since submission, we have extended and optimized the kernels and integrated them into batched full-model generation. We now run contiguous S=1024 in all 28 Qwen2.5-7B layers, with approximately 32K inputs, 512 outputs, and exact attention to newly generated tokens. On the RTX 5090 Laptop GPU, matched dense Flash SDPA → SANTA++ measurements (replicated prompts; sequential prefill) are:
We measure 1.044×/1.232× whole-model decoding speedups and full-request ratios of 1.012×/1.032× (Tables 8, 36–38). FP16 full requests are roughly break-even; see Appendix G.9 for timing uncertainty. Full requests include dense prefill and all cache preparation. Preparation takes 118/284 ms; contiguous grouping avoids k-means. Prefill accounts for 65%/84% of request time, explaining the smaller total gains. INT4 weights enable batch eight within the GPU's 24 GB memory; both methods use the same quantized checkpoint and FP16 KV cache. With INT4 weights, SANTA++ decoding is slower than dense at batch one (0.81×), but faster at batch four (1.13×) and eight (1.23×; Table 8). Section 4 and Appendix G give the implementation and timing details.
Separately, FP16 contiguous S=1024 retains 95.1% of dense RULER aggregate score at 28.2% access and gives 1.38× attention-operator speedup (Table 7).
W2 — Physical memory traffic.
Nsight Compute measurements cover the complete attention operator, including routing and intermediate buffers. On the profiled 32K CWE input, S=1024 reduces total DRAM read/write traffic from 62.55 to 18.42 MiB, a 70.6% reduction. S=128 uses 9.44 MiB (Appendix G.7, Table 44).
W3, Q2 — Cache storage and measured memory.
Our updated implementation stores the prompt KV cache only once, reordered by team. At FP16 batch three, peak allocated memory is 22.134 GiB for both methods, with 0.755 GiB of additional routing data and workspace for SANTA++. At INT4 batch eight, peak allocated memory is 21.810 GiB for both, with 2.012 GiB of additional storage for SANTA++. Prefill determines the peak in these runs. The extra storage at batch one is 0.252 GiB for the full model (Table 9; Appendix G.4).
Q3 — Generalization.
The main timing sweep covers FP16 batches 1–3 and INT4-weight batches 1, 4, and 8. Accuracy evaluations cover 8K and 32K. Beyond Qwen, DeepSeek-V2-Lite-Chat retains 98.1% of the dense aggregate on three RULER-8K tasks at 86.2% logical access to its shared latent cache (Appendix H).
Q4 — Hard retrieval failures.
The revised paper makes the remaining task-level losses explicit: contiguous S=1024 scores 60.2% on MK-NIAH-3 versus 89.6% dense, despite retaining 95.1% of the aggregate (§3). To investigate the failure, matched 50-prompt diagnostics at 32K with k-means parents and S=128 give:
All sparse controls select 32 teams. Both interventions recover substantial accuracy, pointing to team selection as a promising improvement target. Appendix C.1 gives the intervention details.
W5, Q5 — Budget and grouping sensitivity.
Increasing the budget changes which teams are sampled and can change subsequent generation, so individual task scores need not improve monotonically. However, aggregate RULER-32K scores increase across both grouping sweeps (Tables 4–5).
We now evaluate sensitivity to parent size P and representative count R on two RULER-8K tasks at S=128 (Appendix C.3):
Our default P=16,R=4 provides a favorable accuracy–access trade-off: finer teams improve accuracy by only 1.4–1.9 points while increasing access by over 10 percentage points. Coarser teams reduce access but incur larger accuracy losses.
For grouping policy, contiguous S=1024 achieves 79.40% on RULER-32K at 28.16% access, versus 80.40% at 34.44% for k-means. Contiguous grouping also avoids k-means preparation, making it our preferred GPU implementation.