- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper discusses the bandwidth impact of long context decoding on KV caches and proposes Joint Subspace Sparse Attention (JSSA). Unlike FASA and Loki, where the key is PCA'd, JSSA can learn a single common subspace for queries and keys by minimizing the error of the bilinear scores. The entire KV-cache is used, but a small FP8 scoring cache that contains the projected keys is used. The projection is shared between queries and the keys and has a closed-form formula: it is obtained from the top-r eigenvectors of the sum of the trace-normalized second moments of queries and key. A Triton kernel is developed that can read the tokens directly from the original context, rather than copying them to a buffer.
Critical strengths
The motivation is solid. Figure 2 shows gathering can cost as much as the sparse attention itself; Figure 6 shows the gather-free kernel roughly doubles the attention-stage speedup and gives 1.07-2.22X end-to-end advantage; Table 15 shows per KV-head selection beats head-shared voting by 5.29 points at 128K.
Critical weaknesses
1-There is little that is new here: score-fidelity projections (KQ-SVD; InfiniGen, OSDI'24), channel scoring (SparQ, FASA, Double Sparsity), key PCA (Loki), and per-head selection (HeadKV-R2, RocketKV). Double Spar (arXiv 2408.07092) is nearly the same method but adds offline calibration, a small label cache, keeps all of the KV, and fuses the kernel. The main novelty is the symmetric closed form and the GQA kernel applied per-KV head. Regarding the claim that "it is the first to support gather-free token-level attention," it should be directly compared with FlashInfer's vector sparse decoder (MLSys 2025).
2-Several claims do not appear to be correct. Section 4.2.2 and E.4.2 state that JSSA "matches or exceeds" full-KV on Ministral at all lengths, but Table 17 shows 90.72 vs 92.58 at 32K and 89.02 vs 89.45 at 64K. Section 4.2.2 also says JSSA is closest to full-KV "across all three models," but at 32K, Loki on Qwen model and SqueezedAttention on Ministral are actually closer. Additionally, Section 4.5 claims JSSA consistently improves over full-KV, yet Fig. 8 shows a maximum of 1.02x at 8K with JSSA below FKV at small batch, and App.C reports 0.89x on DeepSeek.
3-Accuracy and speed are not evaluated together on the same model, omitting 128K accuracy-versus-speed trade-off. The speed test loads a long prompt (32K/128K) and generates 128 tokens, where a large initial cache makes picking 2,048 tokens instantly efficient. In contrast, reasoning tasks start with a short prompt and generate up to 40K tokens. While under 2,048 tokens, JSSA selects all tokens and adds scoring overhead without any savings. Speedups only emerge as the cache grows beyond 2,048 tokens, making real reasoning speedups much lower than Fig. 4's 128K figure.
4-The Baselines are not compared thoroughly. QUEST has one kernel run per KV head but is then compared in terms of latency (Figs. 4, 9). FASA on MLA uses 64 scoring dimensions, while JSSA uses 128, so the GLM-4.7-Flash difference in capacity is not matched. Loki and FASA are JSSA pipelines that use a different scorer, meaning their throughput matches JSSA by design; no baseline runs independently. The other four baselines are in HF Transformers and do not report throughput.
5-It appears that memory usage increases, but this is not reported (Section3.4, A.7). BF16 KV and a scoring cache of 6.25% FP8 are used. There is no table for peak memory or maximum batch size. Eviction baselines (SnapKV, TriAttention) use less memory for better accuracy, so one important axis is missing. It seems likely that the blank cells in Figure 6(b) are OOMs, but this is not explained.
6-The speedup is not fully detailed in the paper (Section3.1, Section4.5, Figure 4). The 6.43x figure refers to GLM-4.7-Flash (MLA) using DeepSeek's FlashMLA sparse kernel. The authors use their own kernel for the GQA algorithm, resulting in 1.49-1.69x (Qwen3-8B) and 1.88-2.82x (Gemma4, where only five global layers are sparsified, mentioned only in the caption of Table 11; this should be included in Section4.1). The traffic limit of the 6.25% scoring cache is about 13X. The GQA process achieves 1.5X, and the paper does not specify where the rest of the time goes at the full step level (Figure 9 shows only the attention pipeline).
7-The vLLM integration is not described. No vLLM version, no dense attention backend, no stock vLLM number at the same setting is given, so the reader cannot check how fast the method is compared the full-KV baselin given the accuracy drop at that setting. Nothing on whether the sparse path and the scoring cache go through the scheduler, paged allocation, chunked prefill, or prefix caching.
8-The effect of FP8 is not isolated. Only the scoring cache uses FP8, while all KV data is BF16, so the full-KV comparison is fair in that respect. However, E.2 removes the FP8 vs BF16 comparison only for accuracy, not for throughput. The precision of QUEST's page metadata is not specified. FASA does not require any additional cache, but this is not acknowledged. The standard competitor, full-KV with an FP8 KV cache, is not included, and JSSA's compatibility with an FP8 main cache has not been tested.
9-The native-sparse comparison is only for MLA (Section4.6, App. C). DeepSeek and GLM-5.2 run on FlashMLA, so the authors' GQA kernel is not tested against a natively sparse model. Table 4 shows parity, not an advantage, so the paper should clarify its scope: JSSA benefits full-attention models but does not improve models that already have native sparse attention.
10-Minors: In the abstract, "reach 96-99%" should be "reaches"; Eq. 5 and Eq. 20 are duplicates.
Questions for authors
0- Please check weakness point and answer them, first.
1-In Table 16, for the LLaMA LongBench, JSSA wins over full-KV comes mainly from LCC (+11.56) and RepoBench-P (+4.71); what is the reason for beating dense attention by 11 points.
2-Missing experiments. prefill cost and TTFT for building K_proj; which top-B algorithm and its cost beyond 128K (Fig. 9 shows top-B dominating at small batch); mean query vs other aggregation across the G query heads (Section3.1).
3- What is the reason on Table 4's identical 90.91 on GLM-5.2 GPQA. Any justification?
4- KQ-SVD at 0.83% on AIME@512 (Table 14) is surprisingly very low. So, I am not sure if it is correct.
5- What is normalized commutator ratio" in Section B.3?
6- I am also wondering how the method works for multi-turn scanario like sc-bench?
6-Minors: Abstract "reach 96-99%" to "reaches"; Eq. 5 and Eq. 20 are duplicates.
16 comments
Response to Reviewer atgN (2/14)
Response to W1 (2/3): Where JSSA differs from KQ-SVD and Double Sparsity
Calibration shorter than the deployment context (vs. KQ-SVD). KQ-SVD's key map is built through a pseudo-inverse of the key second moment, which amplifies directions that the calibration data under-represent. JSSA's shared orthonormal projection has unit gain in every direction. We built both bases from the same calibration statistics for each setting and evaluated them on the same data (LLaMA-3.1-8B, r = 32).
Table atgN-2. JSSA / KQ-SVD under different calibration data. Attention mass and score error: 128K oracle stream, B = 256. PG-19: perplexity at 32K/512 and 128K/2048 (lower is better).
| Calibration | Attention mass kept (%) | Relative score error | PG-19 32K | PG-19 128K |
|---|---|---|---|---|
| LongAlign, packed to 128K (deployed) | 84.3 / 84.4 | 0.26% / 0.24% | 11.27 / 11.29 | 11.49 / 11.30 |
| Qasper, packed to 128K | 84.3 / 84.3 | 0.27% / 0.25% | – | – |
| LongAlign, unpacked (median 13.5K, max 52K) | 82.7 / 59.4 | 2.6% / 20.1% | 11.29 / 11.28 | 12.87 / 57.63 |
| Qasper, unpacked (~4K) | 69.1 / 50.7 | 21.4% / 40.2% | 11.49 / 29.48 | 38.40 / 305.57 |
When the calibration covers the deployment length, the two are on par on either corpus (KQ-SVD is slightly better at 128K, 11.30 vs. 11.49). When it does not, KQ-SVD degrades far more: with the unpacked LongAlign calibration (median 13.5K tokens), its 128K perplexity is 57.63 vs. 12.87 for JSSA, and with 4K-token calibration it already breaks at 32K. The same holds on Qwen3-14B (YaRN): with unpacked LongAlign, JSSA keeps 73.1% of the attention mass and KQ-SVD 51.3% (76.9% and 76.8% when length-matched), with score errors of 7.0% vs. 34.5%. The failure requires an evaluation context far longer than the calibration context: on LongBench, whose prompts have a median of ~10K tokens, KQ-SVD stays within 49.06–49.12 across all four calibrations (JSSA 49.06–49.16).
This is neither too little data nor a matter of tuning KQ-SVD's pseudo-inverse. Eight times more documents of the same length (64 instead of 8) leave KQ-SVD at 57.9% of the mass with a 22.1% score error, so the cause is the calibration length, not the sample count. Raising the pseudo-inverse cutoff does not help either (at 1e-1 the mass kept falls to 54.1%). A ridge on both second moments, G + λ·tr(G)/d·I, does help, but only once it dominates them: with the same unpacked calibration, λ = 1 keeps 76.0% of the mass and λ = 100 keeps 82.2% (JSSA 82.7%). In the limit λ → ∞, KQ-SVD reduces exactly to JSSA (subspace overlap 1.000), and the same large λ also removes KQ-SVD's in-distribution edge in Table atgN-1 (LLaMA, B = 256: 72.04 → 71.39, JSSA 71.34). Its slight advantage in distribution and its fragility under short calibration thus come from the same whitening, and no single λ keeps both. JSSA sits at the fully regularized end of this family and has no λ to choose.
The unpacked LongAlign set has a median of 13.5K tokens but includes documents of 46K and 52K, which is why JSSA's 12.87 lies between the 32K- and 128K-calibrated values of the length ablation (16.05–18.31 and 11.31–11.48). JSSA is not immune either. With 4K calibration its 128K perplexity rises to 38.40, so we still recommend calibrating at the deployment length (Appendix A.5). JSSA's advantage is that it degrades much more slowly when this is not done.
Low scoring rank (vs. Double Sparsity). A channel subset can keep only the energy already present in the chosen channels. JSSA rotates the joint query–key energy into the leading dimensions, which matters most when few dimensions are available.
Table atgN-3. JSSA / Double Sparsity as the rank decreases. Recall, mass and score error at B = 256; PG-19 perplexity averaged over 32K/64K/128K.
| Model | r | Recall | Attention mass kept | Relative score error | PG-19 avg. |
|---|---|---|---|---|---|
| LLaMA-3.1-8B | 32 | 71.3 / 60.0 | 84.3 / 82.9 | 0.26% / 1.54% | 10.98 / 10.86 |
| 16 | 58.3 / 45.2 | 83.4 / 80.5 | 0.62% / 4.56% | 11.19 / 15.02 | |
| 8 | 48.8 / 35.3 | 82.5 / 77.9 | 1.08% / 8.80% | – | |
| Qwen3-14B | 32 | 71.2 / 67.3 | 76.9 / 76.4 | 1.99% / 3.26% | 11.01 / 11.04 |
| 16 | 57.0 / 49.6 | 75.1 / 73.8 | 4.75% / 9.80% | 11.55 / 11.85 | |
| 8 | 42.7 / 33.2 | 72.3 / 70.0 | 8.75% / 22.33% | – |
From r = 32 to r = 16, JSSA's PG-19 perplexity rises by 0.21 (LLaMA) and 0.54 (Qwen3-14B), and Double Sparsity's by 4.16 and 0.81. On LLaMA, Double Sparsity at r = 16 reaches 22.15 at 128K, while JSSA stays at 11.47, the same as its r = 32 value. The gap is smaller on Qwen3-14B, consistent with its smaller oracle gap. We will add this low-rank comparison to Section 4.7.
Response to Reviewer atgN (3/14)
Response to W1 (3/3): The kernel claim and FlashInfer's vector-sparse decode
We agree that "first" is not established against FlashInfer, and we will remove it. We will describe the kernel by what it does. It attends over per-KV-head token selections on vLLM's paged GQA cache in a single launch, without gathering the selected entries into a buffer. Each selected KV row is read once and shared by the G query heads of its group.
FlashInfer's vector-sparse decode (page size 1) can express the same computation. However, because each KV head selects a different set of tokens, it needs one decode wrapper per KV head, each with its own page table built from the selection. We compared the two on the same paged cache and the same selected indices. The setup was one layer of Qwen3-8B / LLaMA-3.1-8B shape, B = 2048, CUDA graphs, with the L2 cache flushed before every call.
Table atgN-4. Attention over the same selected tokens (µs, 128K; 32K is within 3%), RTX PRO 6000.
| Batch | Ours | FlashInfer, page size 1, per KV head | FlashInfer / ours |
|---|---|---|---|
| 1 | 26.6 | 115.7 | 4.3× |
| 4 | 50.2 | 154.6 | 3.1× |
| 16 | 154.6 | 230.4 | 1.5× |
| 64 | 433.2 | 543.7 | 1.26× |
The gap is largest at small batch, where H_kv separate launches leave the GPU underused, and it shrinks as the batch fills the GPU. Converting the selection into FlashInfer's page-table format adds a further 9–23%. So FlashInfer is a correct but slower way to run per-KV-head selections, and the gap would narrow if it accepted per-head indices in one launch. We will add this comparison, with B200 numbers, and cite FlashInfer's sparse decode as the closest kernel.
Double Sparsity's fused Triton kernel targets a different setting:
- It processes all selected tokens of a (sequence, query head) pair as one tile with a plain softmax. This suits its 256-token budget, but without a split over tokens or an online softmax it does not scale to B = 2048.
- It reads a contiguous, per-query-head cache, so GQA requires expanding K and V G-fold.
Ours reads the paged GQA cache in place, splits the selected tokens across thread blocks with an online softmax, and runs inside vLLM's CUDA graphs.
Response to Reviewer atgN (1/14)
Response to W1 (1/3): Novelty relative to the closest works
We agree that several ingredients are shared with prior work. Token-level selection, per-head selection, key PCA (Loki), channel scoring (SparQ, FASA, Double Sparsity) and score-fidelity projections (KQ-SVD, InfiniGen) all predate JSSA, and we do not claim any of them. Our contribution is narrower, and we will rewrite the introduction and related work to state it as such:
Double Sparsity, InfiniGen and KQ-SVD at the same rank and budget. We implemented all three selection rules in our pipeline (same r = 32, calibration, per-KV-head budget, and exact full-dimensional top-B oracle). For InfiniGen, the channels are picked per request from the request's own prefill keys, with the true current-layer query.
Table atgN-1. Top-B recall (%), r = 32.
We want to be explicit about two points:
We therefore do not claim higher accuracy than either at the deployed setting. The next comment shows where they differ.