acceptodds
Under review as a conference paper at ICLR 2027

Joint Subspace Sparse Attention for Efficient Long-Context Decoding

Abstract

Sparse attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity. Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query–key structure needed for accurate ranking, while gathering scattered selected tokens can offset the resulting latency benefit. We introduce Joint Subspace Sparse Attention (JSSA) to address both challenges. For selection quality, JSSA learns a shared low-dimensional query–key subspace by minimizing token-score preservation error. We upper-bound the resulting non-convex objective with a tractable marginal objective that admits a closed-form spectral solution; direct refinement of the exact objective provides no measurable improvement in the tested settings. For execution efficiency, we develop a gather-free token-level indexed-attention kernel that supports distinct per-KV-head selections under the standard grouped-query attention (GQA) KV-cache layout. Across standard GQA, hybrid sliding-window/global attention, and Multi-head Latent Attention (MLA) models, JSSA consistently remains close to full-KV accuracy while avoiding the substantial generation-length inflation and architecture-dependent failures observed with competing methods. On an NVIDIA B200 GPU, JSSA achieves up to the decoding throughput of full-KV attention. JSSA achieves reasoning accuracy comparable to model-native DSA without model training and reach 96–99% of its decoding throughput even at 700B scale large models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, Long-Context, Decoding, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_atgNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses the bandwidth impact of long context decoding on KV caches and proposes Joint Subspace Sparse Attention (JSSA). Unlike FASA and Loki, where the key is PCA'd, JSSA can learn a single common subspace for queries and keys by minimizing the error of the bilinear scores. The entire KV-cache is used, but a small FP8 scoring cache that contains the projected keys is used. The projection is shared between queries and the keys and has a closed-form formula: it is obtained from the top-r eigenvectors of the sum of the trace-normalized second moments of queries and key. A Triton kernel is developed that can read the tokens directly from the original context, rather than copying them to a buffer.

Critical strengths

The motivation is solid. Figure 2 shows gathering can cost as much as the sparse attention itself; Figure 6 shows the gather-free kernel roughly doubles the attention-stage speedup and gives 1.07-2.22X end-to-end advantage; Table 15 shows per KV-head selection beats head-shared voting by 5.29 points at 128K.

Critical weaknesses

1-There is little that is new here: score-fidelity projections (KQ-SVD; InfiniGen, OSDI'24), channel scoring (SparQ, FASA, Double Sparsity), key PCA (Loki), and per-head selection (HeadKV-R2, RocketKV). Double Spar (arXiv 2408.07092) is nearly the same method but adds offline calibration, a small label cache, keeps all of the KV, and fuses the kernel. The main novelty is the symmetric closed form and the GQA kernel applied per-KV head. Regarding the claim that "it is the first to support gather-free token-level attention," it should be directly compared with FlashInfer's vector sparse decoder (MLSys 2025).

2-Several claims do not appear to be correct. Section 4.2.2 and E.4.2 state that JSSA "matches or exceeds" full-KV on Ministral at all lengths, but Table 17 shows 90.72 vs 92.58 at 32K and 89.02 vs 89.45 at 64K. Section 4.2.2 also says JSSA is closest to full-KV "across all three models," but at 32K, Loki on Qwen model and SqueezedAttention on Ministral are actually closer. Additionally, Section 4.5 claims JSSA consistently improves over full-KV, yet Fig. 8 shows a maximum of 1.02x at 8K with JSSA below FKV at small batch, and App.C reports 0.89x on DeepSeek.

3-Accuracy and speed are not evaluated together on the same model, omitting 128K accuracy-versus-speed trade-off. The speed test loads a long prompt (32K/128K) and generates 128 tokens, where a large initial cache makes picking 2,048 tokens instantly efficient. In contrast, reasoning tasks start with a short prompt and generate up to 40K tokens. While under 2,048 tokens, JSSA selects all tokens and adds scoring overhead without any savings. Speedups only emerge as the cache grows beyond 2,048 tokens, making real reasoning speedups much lower than Fig. 4's 128K figure.

4-The Baselines are not compared thoroughly. QUEST has one kernel run per KV head but is then compared in terms of latency (Figs. 4, 9). FASA on MLA uses 64 scoring dimensions, while JSSA uses 128, so the GLM-4.7-Flash difference in capacity is not matched. Loki and FASA are JSSA pipelines that use a different scorer, meaning their throughput matches JSSA by design; no baseline runs independently. The other four baselines are in HF Transformers and do not report throughput.

5-It appears that memory usage increases, but this is not reported (Section3.4, A.7). BF16 KV and a scoring cache of 6.25% FP8 are used. There is no table for peak memory or maximum batch size. Eviction baselines (SnapKV, TriAttention) use less memory for better accuracy, so one important axis is missing. It seems likely that the blank cells in Figure 6(b) are OOMs, but this is not explained.

6-The speedup is not fully detailed in the paper (Section3.1, Section4.5, Figure 4). The 6.43x figure refers to GLM-4.7-Flash (MLA) using DeepSeek's FlashMLA sparse kernel. The authors use their own kernel for the GQA algorithm, resulting in 1.49-1.69x (Qwen3-8B) and 1.88-2.82x (Gemma4, where only five global layers are sparsified, mentioned only in the caption of Table 11; this should be included in Section4.1). The traffic limit of the 6.25% scoring cache is about 13X. The GQA process achieves 1.5X, and the paper does not specify where the rest of the time goes at the full step level (Figure 9 shows only the attention pipeline).

7-The vLLM integration is not described. No vLLM version, no dense attention backend, no stock vLLM number at the same setting is given, so the reader cannot check how fast the method is compared the full-KV baselin given the accuracy drop at that setting. Nothing on whether the sparse path and the scoring cache go through the scheduler, paged allocation, chunked prefill, or prefix caching.

8-The effect of FP8 is not isolated. Only the scoring cache uses FP8, while all KV data is BF16, so the full-KV comparison is fair in that respect. However, E.2 removes the FP8 vs BF16 comparison only for accuracy, not for throughput. The precision of QUEST's page metadata is not specified. FASA does not require any additional cache, but this is not acknowledged. The standard competitor, full-KV with an FP8 KV cache, is not included, and JSSA's compatibility with an FP8 main cache has not been tested.

9-The native-sparse comparison is only for MLA (Section4.6, App. C). DeepSeek and GLM-5.2 run on FlashMLA, so the authors' GQA kernel is not tested against a natively sparse model. Table 4 shows parity, not an advantage, so the paper should clarify its scope: JSSA benefits full-attention models but does not improve models that already have native sparse attention.

10-Minors: In the abstract, "reach 96-99%" should be "reaches"; Eq. 5 and Eq. 20 are duplicates.

Questions for authors

0- Please check weakness point and answer them, first.

1-In Table 16, for the LLaMA LongBench, JSSA wins over full-KV comes mainly from LCC (+11.56) and RepoBench-P (+4.71); what is the reason for beating dense attention by 11 points.

2-Missing experiments. prefill cost and TTFT for building K_proj; which top-B algorithm and its cost beyond 128K (Fig. 9 shows top-B dominating at small batch); mean query vs other aggregation across the G query heads (Section3.1).

3- What is the reason on Table 4's identical 90.91 on GLM-5.2 GPQA. Any justification?

4- KQ-SVD at 0.83% on AIME@512 (Table 14) is surprisingly very low. So, I am not sure if it is correct.

5- What is normalized commutator ratio" in Section B.3?

6- I am also wondering how the method works for multi-turn scanario like sc-bench?

6-Minors: Abstract "reach 96-99%" to "reaches"; Eq. 5 and Eq. 20 are duplicates.

16 comments
AuthorsOpenReview ↗

Response to Reviewer atgN (1/14)

Response to W1 (1/3): Novelty relative to the closest works

We agree that several ingredients are shared with prior work. Token-level selection, per-head selection, key PCA (Loki), channel scoring (SparQ, FASA, Double Sparsity) and score-fidelity projections (KQ-SVD, InfiniGen) all predate JSSA, and we do not claim any of them. Our contribution is narrower, and we will rewrite the introduction and related work to state it as such:

  1. One objective that covers the static projection family. We cast selection-score preservation as a single objective over projections, E[(q⊤k−q⊤Pk)2]\mathbb{E}[(q^\top k - q^\top P k)^2], with a closed-form spectral solution. Several prior scorers lie inside this family but choose PP by other criteria: Double Sparsity and FASA restrict PP to a channel subset (rows of the identity) picked by per-channel heuristics, and Loki uses key statistics only.
  2. A training-free closed form that is robust where the closest alternatives are not. It remains robust at low scoring rank and when the calibration data are shorter than the deployment context (next comment).
  3. An execution path for per-KV-head token selection on paged GQA caches (third comment).

Double Sparsity, InfiniGen and KQ-SVD at the same rank and budget. We implemented all three selection rules in our pipeline (same r = 32, calibration, per-KV-head budget, and exact full-dimensional top-B oracle). For InfiniGen, the channels are picked per request from the request's own prefill keys, with the true current-layer query.

Table atgN-1. Top-B recall (%), r = 32.

ModelBJSSADouble Sparsity (paper criterion)InfiniGenKQ-SVD
LLaMA-3.1-8B25671.3459.9769.5272.04
204875.9466.1673.9076.72
Qwen3-14B25671.2267.3162.6070.67
204877.0173.5670.0777.06

We want to be explicit about two points:

  • KQ-SVD was proposed for low-rank KV-cache compression, not for token selection; we adapted it as a scorer (Appendix B.6) as an alternative solution of our score-preservation objective. After the bug fix discussed under Q4, the two solutions are on par at the deployed calibration on RULER, AIME and GPQA, with KQ-SVD slightly higher in recall on LLaMA. JSSA's closed form is preferred for its robustness when calibration is shorter than deployment (next comment).
  • Double Sparsity matches JSSA downstream at r = 32, despite its lower recall. On RULER (LLaMA-3.1-8B, 32K/64K/128K), it averages 81.42 vs. JSSA's 81.36, and on PG-19 its 32K–128K average perplexity is within 0.12 of JSSA's on all three models.

We therefore do not claim higher accuracy than either at the deployed setting. The next comment shows where they differ.

AuthorsOpenReview ↗

Response to Reviewer atgN (2/14)

Response to W1 (2/3): Where JSSA differs from KQ-SVD and Double Sparsity

Calibration shorter than the deployment context (vs. KQ-SVD). KQ-SVD's key map is built through a pseudo-inverse of the key second moment, which amplifies directions that the calibration data under-represent. JSSA's shared orthonormal projection has unit gain in every direction. We built both bases from the same calibration statistics for each setting and evaluated them on the same data (LLaMA-3.1-8B, r = 32).

Table atgN-2. JSSA / KQ-SVD under different calibration data. Attention mass and score error: 128K oracle stream, B = 256. PG-19: perplexity at 32K/512 and 128K/2048 (lower is better).

CalibrationAttention mass kept (%)Relative score errorPG-19 32KPG-19 128K
LongAlign, packed to 128K (deployed)84.3 / 84.40.26% / 0.24%11.27 / 11.2911.49 / 11.30
Qasper, packed to 128K84.3 / 84.30.27% / 0.25%––
LongAlign, unpacked (median 13.5K, max 52K)82.7 / 59.42.6% / 20.1%11.29 / 11.2812.87 / 57.63
Qasper, unpacked (~4K)69.1 / 50.721.4% / 40.2%11.49 / 29.4838.40 / 305.57

When the calibration covers the deployment length, the two are on par on either corpus (KQ-SVD is slightly better at 128K, 11.30 vs. 11.49). When it does not, KQ-SVD degrades far more: with the unpacked LongAlign calibration (median 13.5K tokens), its 128K perplexity is 57.63 vs. 12.87 for JSSA, and with 4K-token calibration it already breaks at 32K. The same holds on Qwen3-14B (YaRN): with unpacked LongAlign, JSSA keeps 73.1% of the attention mass and KQ-SVD 51.3% (76.9% and 76.8% when length-matched), with score errors of 7.0% vs. 34.5%. The failure requires an evaluation context far longer than the calibration context: on LongBench, whose prompts have a median of ~10K tokens, KQ-SVD stays within 49.06–49.12 across all four calibrations (JSSA 49.06–49.16).

This is neither too little data nor a matter of tuning KQ-SVD's pseudo-inverse. Eight times more documents of the same length (64 instead of 8) leave KQ-SVD at 57.9% of the mass with a 22.1% score error, so the cause is the calibration length, not the sample count. Raising the pseudo-inverse cutoff does not help either (at 1e-1 the mass kept falls to 54.1%). A ridge on both second moments, G + λ·tr(G)/d·I, does help, but only once it dominates them: with the same unpacked calibration, λ = 1 keeps 76.0% of the mass and λ = 100 keeps 82.2% (JSSA 82.7%). In the limit λ → ∞, KQ-SVD reduces exactly to JSSA (subspace overlap 1.000), and the same large λ also removes KQ-SVD's in-distribution edge in Table atgN-1 (LLaMA, B = 256: 72.04 → 71.39, JSSA 71.34). Its slight advantage in distribution and its fragility under short calibration thus come from the same whitening, and no single λ keeps both. JSSA sits at the fully regularized end of this family and has no λ to choose.

The unpacked LongAlign set has a median of 13.5K tokens but includes documents of 46K and 52K, which is why JSSA's 12.87 lies between the 32K- and 128K-calibrated values of the length ablation (16.05–18.31 and 11.31–11.48). JSSA is not immune either. With 4K calibration its 128K perplexity rises to 38.40, so we still recommend calibrating at the deployment length (Appendix A.5). JSSA's advantage is that it degrades much more slowly when this is not done.

Low scoring rank (vs. Double Sparsity). A channel subset can keep only the energy already present in the chosen channels. JSSA rotates the joint query–key energy into the leading dimensions, which matters most when few dimensions are available.

Table atgN-3. JSSA / Double Sparsity as the rank decreases. Recall, mass and score error at B = 256; PG-19 perplexity averaged over 32K/64K/128K.

ModelrRecallAttention mass keptRelative score errorPG-19 avg.
LLaMA-3.1-8B3271.3 / 60.084.3 / 82.90.26% / 1.54%10.98 / 10.86
1658.3 / 45.283.4 / 80.50.62% / 4.56%11.19 / 15.02
848.8 / 35.382.5 / 77.91.08% / 8.80%–
Qwen3-14B3271.2 / 67.376.9 / 76.41.99% / 3.26%11.01 / 11.04
1657.0 / 49.675.1 / 73.84.75% / 9.80%11.55 / 11.85
842.7 / 33.272.3 / 70.08.75% / 22.33%–

From r = 32 to r = 16, JSSA's PG-19 perplexity rises by 0.21 (LLaMA) and 0.54 (Qwen3-14B), and Double Sparsity's by 4.16 and 0.81. On LLaMA, Double Sparsity at r = 16 reaches 22.15 at 128K, while JSSA stays at 11.47, the same as its r = 32 value. The gap is smaller on Qwen3-14B, consistent with its smaller oracle gap. We will add this low-rank comparison to Section 4.7.

AuthorsOpenReview ↗

Response to Reviewer atgN (3/14)

Response to W1 (3/3): The kernel claim and FlashInfer's vector-sparse decode

We agree that "first" is not established against FlashInfer, and we will remove it. We will describe the kernel by what it does. It attends over per-KV-head token selections on vLLM's paged GQA cache in a single launch, without gathering the selected entries into a buffer. Each selected KV row is read once and shared by the G query heads of its group.

FlashInfer's vector-sparse decode (page size 1) can express the same computation. However, because each KV head selects a different set of tokens, it needs one decode wrapper per KV head, each with its own page table built from the selection. We compared the two on the same paged cache and the same selected indices. The setup was one layer of Qwen3-8B / LLaMA-3.1-8B shape, B = 2048, CUDA graphs, with the L2 cache flushed before every call.

Table atgN-4. Attention over the same selected tokens (µs, 128K; 32K is within 3%), RTX PRO 6000.

BatchOursFlashInfer, page size 1, per KV headFlashInfer / ours
126.6115.74.3×
450.2154.63.1×
16154.6230.41.5×
64433.2543.71.26×

The gap is largest at small batch, where H_kv separate launches leave the GPU underused, and it shrinks as the batch fills the GPU. Converting the selection into FlashInfer's page-table format adds a further 9–23%. So FlashInfer is a correct but slower way to run per-KV-head selections, and the gap would narrow if it accepted per-head indices in one launch. We will add this comparison, with B200 numbers, and cite FlashInfer's sparse decode as the closest kernel.

Double Sparsity's fused Triton kernel targets a different setting:

  • It processes all selected tokens of a (sequence, query head) pair as one tile with a plain softmax. This suits its 256-token budget, but without a split over tokens or an online softmax it does not scale to B = 2048.
  • It reads a contiguous, per-query-head cache, so GQA requires expanding K and V G-fold.

Ours reads the paged GQA cache in place, splits the selected tokens across thread blocks with an online softmax, and runs inside vLLM's CUDA graphs.

Reviewer_rn4QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JSSA is a calibration-based, training-free sparse-attention method that retains the full KV cache but uses a low-dimensional auxiliary key cache to select tokens during decoding. It constructs a shared query–key projection from the leading eigenvectors of normalized query and key second moments, motivated by an upper bound on a bilinear score-error objective. For GQA, the paper introduces an indexed-attention kernel that directly consumes distinct per-KV-head token selections without materializing gathered KV buffers; MLA experiments instead use FlashMLA. The reported evaluations show a promising accuracy–throughput trade-off across architectures, including comparisons with native DSA at approximately 700B scale.

Critical strengths

The most compelling aspect of this work is that it treats token selection and sparse execution as equally important problems. A better selector is not especially useful if gathering the selected KV entries consumes much of the latency saved by sparsification. Likewise, an efficient kernel does little to help if the selected tokens are insufficient for accurate generation.

Critical weaknesses
  1. Missing the comparison with the closest works. Double Sparsity already uses offline-calibrated channel selection, a compact label cache, and Triton kernels for token-level sparse decoding. InfiniGen uses SVD-based weight skewing so that a few channels predict attention scores. ShadowKV, MagicPIG, RetrievalAttention, PQCache, and HashAttention are also missing. Without these, I can't evaluate the claim of being the "first gather-free per-KV-head kernel," and Loki on its own is not a demanding baseline for score preservation. If a well-tuned Double Sparsity or InfiniGen gets similar recall, most of what remains is engineering.

  2. The ablation experiments support using the spectral solution in practice, but they do not fully establish why it yields better downstream selection.

Questions for authors
  1. Can you isolate where the gain comes from? Please compare key-only PCA (Loki-style, in your pipeline), query-only PCA, raw-sum PCA, and normalized-sum PCA at r=32. Also, are Loki's and JSSA's projections computed before or after RoPE?
  2. How does JSSA compare with Double Sparsity and InfiniGen at the same scoring dimension and token budget? How is your kernel different from Double Sparsity's Triton kernel, or from FlashInfer sparse attention with page size 1 and per-head indices? The answer bears directly on the "first kernel" claim.
  3. How does a 128K-calibrated basis do at 256K, or at any length beyond calibration? What actually changes between the 32K and 128K bases? Is it mostly low-frequency RoPE dimensions?
  4. Why does JSSA beat full-KV by 11.6 points on LCC with LLaMA? Does this hold across seeds and prompts? What does the LongBench average look like without the code tasks?
  5. Could you add standard deviations over the 8 seeds to Tables 3 and 4 and say which differences are statistically meaningful?
  6. Was QUEST given an equally optimized single-launch kernel? At 128K on GQA, what share of decoding time goes to the score scan plus top-B, and how would that scale to 1M tokens?
  7. For GQA, why use the mean query instead of aggregating per-head projected scores (max or sum over the G heads)? Aggregating keeps per-head information at about the same cost.
11 comments
AuthorsOpenReview ↗

Response to Reviewer rn4Q (1/8)

Response to W1 and Q2 (1/4): Comparison with the closest works at the same scoring dimension and budget

Thank you for pointing to these works. We implemented the selection rules of Double Sparsity and InfiniGen inside our pipeline, so that every method uses the same rank r = 32, the same 128K calibration data, the same per-KV-head top-B budget and the same exact full-dimensional top-B oracle. Only the scoring projection differs.

  • Double Sparsity: an axis-aligned channel subset, i.e., M restricted to rows of the identity. We evaluate both its paper criterion (per-channel |q|·|k|) and the released code's default (per-channel |k|).
  • InfiniGen: queries are skewed by the right singular vectors of the post-RoPE query matrix, and the r skewed channels with the largest |q| + |k| magnitude are kept, chosen per request from that request's prefill keys. We give it the true current-layer query rather than its speculated one, and the same top-B budget instead of its threshold rule. Both choices favor InfiniGen.

Table rn4Q-1. Top-B recall (%) vs. the exact full-dimensional top-B, r = 32.

ModelBJSSADouble Sparsity (paper)Double Sparsity (code)InfiniGen
LLaMA-3.1-8B25671.3459.9755.2969.52
204875.9466.1662.1173.90
Qwen3-14B25671.2267.3159.5262.60
204877.0173.5667.9770.07

JSSA has the highest recall in every setting. Its margin is 3.5–11 pp over Double Sparsity and 2–9 pp over InfiniGen.

We also ran Double Sparsity end to end on RULER (LLaMA-3.1-8B, same budgets as our main evaluation):

Table rn4Q-2. RULER accuracy (%), LLaMA-3.1-8B.

Method32K / 51264K / 1024128K / 2048Average
JSSA86.1482.5575.3881.36
Double Sparsity (paper criterion)87.5982.4574.2181.42
Double Sparsity (code default)––69.98–

We want to be transparent here. At r = 32, with the paper's criterion and our calibration data, Double Sparsity matches JSSA on RULER for this model, and on PG-19 its 32K–128K average perplexity is within 0.12 of JSSA's on all three models. Its recall gap does not turn into an accuracy gap at this rank. The released default criterion is clearly weaker (−5.4 at 128K). The same holds for KQ-SVD, which the paper compares in Tables 13 and 14: after we fixed a bug in our integration of it (see our general comment), it matches JSSA in distribution. JSSA differs from both where the rank or the calibration length is constrained, which the next comment shows for Double Sparsity.

AuthorsOpenReview ↗

Response to Reviewer rn4Q (2/8)

Response to W1 and Q2 (2/4): The gap widens as the scoring cache shrinks

We repeated the comparison at smaller ranks on both models, keeping everything else fixed.

Table rn4Q-3. JSSA vs. Double Sparsity (paper criterion) as the rank decreases (B = 256). Each cell is JSSA / Double Sparsity.

ModelrRecallAttention mass keptRelative score errorPG-19 perplexity (avg. 32K–128K)
LLaMA-3.1-8B3271.3 / 60.0 (+11.4)84.3 / 82.9 (+1.4)0.26% / 1.54%10.98 / 10.86
1658.3 / 45.2 (+13.1)83.4 / 80.5 (+2.9)0.62% / 4.56%11.19 / 15.02
848.8 / 35.3 (+13.5)82.5 / 77.9 (+4.6)1.08% / 8.80%–
Qwen3-14B3271.2 / 67.3 (+3.9)76.9 / 76.4 (+0.4)1.99% / 3.26%11.01 / 11.04
1657.0 / 49.6 (+7.4)75.1 / 73.8 (+1.3)4.75% / 9.80%11.55 / 11.85
842.7 / 33.2 (+9.5)72.3 / 70.0 (+2.3)8.75% / 22.33%–

JSSA degrades much more gracefully at low rank, on both models. From r = 32 to r = 8, the gap in attention mass covered by the selected tokens grows from 1.4 to 4.6 pp on LLaMA and from 0.4 to 2.3 pp on Qwen3-14B. The recall gap also grows, from 3.9 to 9.5 pp on Qwen3-14B. Double Sparsity's score error is 2–8× higher than JSSA's at r ≤ 16. On LLaMA, JSSA at r = 16 even keeps more attention mass than Double Sparsity at r = 32 (83.4 vs. 82.9) with half the scoring cache.

The oracle gap now also appears downstream. From r = 32 to r = 16, JSSA's PG-19 perplexity rises by 0.21 on LLaMA and 0.54 on Qwen3-14B, while Double Sparsity's rises by 4.16 and 0.81. On LLaMA at 128K, Double Sparsity at r = 16 reaches 22.15, while JSSA stays at 11.47, the same as its r = 32 value (11.49). The downstream gap is smaller on Qwen3-14B, in line with its smaller oracle gap.

This is the expected behavior of a rotation versus a subset. A fixed channel subset can keep only the energy that already lies in the chosen channels, whereas JSSA rotates the joint query–key energy into the leading dimensions. This matters most when few dimensions are available.

In the revision, we will add this low-rank comparison against Double Sparsity, including the PG-19 results, to Section 4.7.

In summary:

  1. Double Sparsity's projection lies inside JSSA's search space. Restricting M to rows of the identity gives a channel subset like Double Sparsity's, which it chooses with a per-channel heuristic. JSSA instead minimizes its score-preservation surrogate over all orthonormal projections, which yields consistently higher recall and lower score error, and a widening advantage as the rank decreases.
  2. Both rules were adapted to GQA for this comparison. Double Sparsity selects tokens per query head, and its paper notes that GQA models are incompatible with its key-outlier channels. InfiniGen is likewise defined for multi-head attention. Both rules above run in our per-KV-head pipeline.
  3. The kernel contribution is separate. We address it in a following comment.
AuthorsOpenReview ↗

Response to Reviewer rn4Q (3/8)

Response to W1 and Q2 (3/4): Other related works

We will add ShadowKV, MagicPIG, RetrievalAttention, PQCache and HashAttention to the related work. They operate in different design spaces:

  • ShadowKV stores low-rank pre-RoPE keys and offloads values to CPU, selecting chunks through landmarks.
  • MagicPIG uses LSH-based sampling with CPU-side hash tables.
  • RetrievalAttention and PQCache retrieve tokens through approximate nearest-neighbor indexes or product-quantized keys, with the KV cache offloaded to CPU.
  • HashAttention learns hash functions and therefore requires training.

JSSA keeps the full KV cache on the GPU, needs no training, and scores with a single fixed linear projection. Its closest comparators are therefore the static-projection methods above (Double Sparsity, InfiniGen, Loki, FASA).

Response to W1 and Q2 (4/4): The kernel claim

We agree that "first" is not established, and we will drop it. We will describe the kernel by what it does: it attends over per-KV-head token selections on vLLM's paged GQA cache in a single launch, without gathering the selected entries into a buffer, and each selected KV row is read once and shared by the G query heads of its group. The numbers below were measured on an RTX PRO 6000 (one layer of Qwen3-8B/LLaMA-3.1-8B shape, B = 2048, CUDA graphs, L2 flushed before every call); we will add the same measurements on the B200 to the revision.

FlashInfer with page size 1. Because each KV head selects a different set of tokens, FlashInfer needs one paged-decode wrapper per KV head, with a per-head page table built from the selection.

Table rn4Q-4. Attention stage over the same selected tokens (µs, 128K; 32K is within 3%).

BatchOursFlashInfer, page size 1, per KV headFlashInfer / ours
126.6115.74.3×
450.2154.63.1×
16154.6230.41.5×
64433.2543.71.26×

The advantage is largest at small batch, where H_kv separate launches leave the GPU underused, and shrinks as the batch fills it. Converting the selection into FlashInfer's page-table format adds a further 9–23%.

Double Sparsity's Triton kernel. It was designed for a different setting:

  • One program per (sequence, query head) loads all selected tokens as a single tile and applies a plain softmax. This suits its 256-token budget, but there is no split over tokens or online softmax, so it does not scale to B = 2048.
  • It indexes a contiguous, non-paged cache laid out per query head. On GQA models, K and V must first be expanded G-fold, which is why its paper notes that GQA models are incompatible.
  • Selection runs as a separate dense matmul over the full label cache followed by torch.topk.

Our kernel reads the paged cache directly, splits the selected tokens across thread blocks with an online softmax, shares one selection per KV head across the G query heads, and is preceded by a fused score scan and radix top-B. All of it runs inside vLLM's CUDA graphs.

Reviewer_hS23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Joint Subspace Sparse Attention (JSSA), a training-free approach for accelerating long-context decoding through token-level sparse attention. The method addresses two challenges: identifying relevant tokens using low-dimensional query–key representations and efficiently accessing irregularly selected KV entries during decoding. JSSA constructs a shared query–key projection using a spectral solution derived from an upper bound on attention-score preservation error. It further introduces a gather-free indexed-attention kernel that supports distinct token selections across KV heads under the standard GQA cache layout.

The authors evaluate JSSA across GQA, hybrid attention, and MLA architectures, including models at approximately 700B scale. Experiments on long-context understanding, language modeling, reasoning, and decoding throughput demonstrate competitive accuracy and substantial efficiency improvements over existing sparse-attention approaches.

Overall, the paper presents a practically relevant approach with strong empirical results and a meaningful systems contribution. However, some aspects of the proposed scoring objective and token-selection strategy require further justification, and the dependence on long-context calibration and additional scoring-cache resources raises questions about deployment flexibility.

Critical strengths

S1. Relevant problem formulation and joint optimization of selection quality and execution efficiency.

The paper identifies two important challenges in token-level sparse attention: preserving accurate token rankings under low-dimensional scoring and avoiding the overhead of gathering irregularly selected KV entries. Addressing both the selection and execution aspects provides a coherent approach to efficient long-context decoding.

S2. Practical gather-free implementation with supporting ablations.

The proposed indexed-attention kernel supports distinct per-KV-head token selections under the standard GQA KV-cache layout without explicitly gathering selected entries. The evaluation provides evidence that eliminating this gathering overhead improves decoding efficiency. The controlled comparisons between gather-free execution and gather-then-attend further support the practical value of the implementation.

S3. Comprehensive empirical evaluation across architectures and scales.

The experiments cover multiple model architectures, long-context understanding, language modeling, reasoning, and decoding efficiency. The evaluation includes comparisons with relevant baselines such as Loki, FASA, QUEST, and SqueezedAttention, as well as additional studies on projection rank, calibration, scoring-cache precision, and selection granularity. The large-scale comparison with model-native DSA is also a valuable addition.

Critical weaknesses

W1. Limited Justification of the Surrogate Objective and Its Relationship to Token-Selection Quality

The paper motivates the joint query–key subspace through an objective that minimizes the expected squared error of attention scores. However, the proposed practical solution minimizes a marginal upper bound rather than the original objective. The resulting closed-form solution is obtained from the leading eigenvectors of the sum of normalized query and key second-moment matrices, which is closely related to standard spectral projection techniques.

Although the derivation is mathematically reasonable, the upper bound does not establish that the resulting projection is optimal for the original score-preservation objective or for the actual top-B token-selection task. In particular, minimizing average squared score error does not necessarily preserve the relative ordering of tokens near the selection boundary, which directly determines sparse-attention accuracy.

The authors provide useful comparisons with Loki, FASA, and an asymmetric KQ-SVD-based alternative, and show that directly refining the exact objective offers little additional improvement. Nevertheless, the necessity of the particular marginal surrogate and its equal weighting of the normalized query and key statistics could be further justified. Additional comparisons with simpler query-only or differently weighted joint projections would help clarify whether the proposed construction is particularly effective for token ranking.

W2. Insufficient Analysis of Group-Mean Query Selection in GQA

In the proposed GQA implementation, token selection is performed using the mean query of all query heads associated with a KV head. The selected token set is then shared among these query heads. While this design reduces selection overhead, averaging query representations may obscure differences in attention preferences across individual query heads.

In particular, a token that is highly relevant to one query head may receive a relatively low aggregated score when the associated query heads attend to different positions. The paper does not sufficiently investigate how this approximation affects the preservation of important tokens, especially under aggressive sparsity.

Although Appendix E.3 compares per-KV-head selection against layer-level head-shared selection, this experiment does not isolate the effect of averaging query representations within each GQA group. Comparisons with alternative within-group aggregation strategies, such as maximum scoring or aggregated attention mass, under matched token and computational budgets would help justify the current design.

W3. Dependence on Long-Context Calibration and Incomplete Characterization of Deployment Trade-offs

Although JSSA does not require model training, it relies on model-specific calibration to construct the projection bases. The calibration sensitivity results in Section 4.7 indicate that bases calibrated on 32K contexts can degrade substantially when evaluated at 128K. This suggests that the proposed method may depend on calibration sequences covering the intended deployment context length.

Consequently, it remains unclear how robust JSSA is when deployed under context lengths or attention distributions that differ substantially from calibration. Additional evaluations of calibration transfer and performance beyond the calibrated context range would help establish its practical flexibility.

Furthermore, JSSA maintains an auxiliary projected-key cache in addition to the original full-dimensional KV cache. Although its storage overhead is bounded and its decoding cost is included in the reported throughput measurements, the overall memory–accuracy–throughput trade-off is not fully characterized. This is particularly relevant when comparing against methods such as FASA that reuse channels from the original key cache without maintaining an additional projected representation.

Finally, the largest reported decoding speedups occur on MLA models using an existing FlashMLA sparse-attention kernel, whereas the newly proposed gather-free kernel specifically targets GQA. Although the paper provides component-level ablations for the GQA kernel, a clearer distinction between improvements attributable to the proposed scoring method and those enabled by the underlying sparse-attention backend would strengthen the systems contribution.

Questions for authors

How sensitive is the performance of JSSA to the relative weighting of query and key second-moment matrices in the spectral surrogate? Would simpler query-only or weighted joint projections achieve comparable token-selection accuracy under the same scoring dimension and computational budget?

7 comments
AuthorsOpenReview ↗

Response to Reviewer hS23 (1/4)

Response to W1 and Q1: Justification of the surrogate objective, Q/K weighting, and query-only projections

We agree that the marginal upper bound does not certify optimality. We do not have a proof that its solution is optimal for the original score-preservation objective or for top-B selection. Its closed form, the leading eigenvectors of the sum of the normalized query and key second moments, is also close to standard spectral projection. Instead of an additional theoretical claim, we ran the comparisons you asked for. Every variant uses the same r = 32, the same 128K calibration, and the same group-mean query used at deployment. Only the projection changes, so all variants have the same scoring dimension and the same computational cost. Recall is measured against the exact full-dimensional top-B.

Q/K weighting (Q1). We swept M=TopEig(α G^Q/tr+(1−α) G^K/tr)M = \mathrm{TopEig}(\alpha\,\hat G_{Q}/\mathrm{tr} + (1-\alpha)\,\hat G_{K}/\mathrm{tr}) for α = 0, 0.1, …, 1. α = 0 is key-only, α = 1 is query-only, and α = 0.5 is JSSA.

Table hS23-1. Top-B recall (%) vs. the exact full-dimensional top-B.

ModelBα=0 (key-only)α=0.5 (JSSA)α=1 (query-only)Best α in sweep
LLaMA-3.1-8B25663.2271.3470.9472.02 (α=0.8)
204869.1975.9475.2076.45 (α=0.7)
Qwen3-14B25654.5171.2266.0271.31 (α=0.4)
204866.5777.0172.8277.05 (α=0.4)

The best α lies on opposite sides of 0.5 for the two models (0.7–0.8 for LLaMA, 0.4 for Qwen3), and the curve is flat around its maximum. Equal weighting is within 0.7 pp of the per-model optimum in all four settings. Per-model tuning of α could gain up to 0.7 pp on LLaMA. We keep α = 0.5 because it is determined by the calibration statistics alone and needs no tuning.

Why not query-only (Q1). Query-only projection works well on LLaMA (−0.4 pp vs. JSSA) but drops by 5.2 pp on Qwen3-14B. Key-only drops by 8.1 and 16.7 pp. Using only one side of the score is therefore model-dependent. The joint construction is the only one that stays near the best value on both models, and we choose it for that stability.

Trace normalization. On LLaMA, summing the raw second moments without normalization lowers recall from 71.34% to 70.05% (B = 256). Without normalization the larger-magnitude moment dominates the basis. Normalization puts the two sides on an equal footing.

Ordering near the selection boundary (W1). Small average squared error does not by itself guarantee correct ordering near the cut-off. We therefore measured recall only over tokens whose exact rank lies in (B/2, B], i.e., the tokens just inside the boundary.

Table hS23-2. Boundary recall (%) over exact ranks (B/2, B], B = 256.

ModelJSSAquery-onlykey-only
LLaMA-3.1-8B53.5953.2145.31
Qwen3-14B54.0848.8141.54

The boundary ordering follows the overall recall: the projection with the smaller score error also orders the near-boundary tokens better. We did not observe a case where average error is small but boundary ordering breaks down. This is an empirical observation, not a guarantee.

Cost of using the upper bound. As noted in Appendix B.4, refining the closed-form basis on the exact objective changes little. Over 1,600 steps and learning rates spanning three orders of magnitude, the exact objective decreases by less than 1% and top-B recall moves by at most 0.02 pp. As a further control, we ran the same optimizer from a random orthonormal initialization. It converges to a point whose exact objective is 2.5× (r = 8) and 8.0× (r = 32) higher than the closed form, in a different subspace. The closed form is thus a better solution than direct optimization reaches at this budget.

Correction on the KQ-SVD comparison. While preparing this response, we found a bug in our vLLM integration of KQ-SVD: on all-decode steps, the query was projected with KQ-SVD's key map. After the fix, KQ-SVD is on par with JSSA in distribution (RULER, AIME 2025 and GPQA), and we retract the reported KQ-SVD collapse (details in our general comment). What remains is robustness: when the calibration sequences are shorter than the deployment context, KQ-SVD degrades far more (PG-19 at 128K with unpacked LongAlign calibration: 57.63 vs. 12.87 for JSSA). We will rewrite this comparison accordingly.

Revision. In Section 3.2 we will state explicitly that the bound does not certify optimality, and we will add the full α sweep and boundary-recall results (both budgets, all metrics) to the appendix.

AuthorsOpenReview ↗

Response to Reviewer hS23 (2/4)

Response to W2: Group-mean query selection in GQA

We agree that Appendix E.3 compares per-KV-head selection against layer-level sharing and does not isolate the effect of averaging within a GQA group. We added an experiment that isolates it. The JSSA basis is fixed and every KV head keeps the same B tokens. Only the way its G query heads are combined changes:

  • mean (ours): score with the group-mean query.
  • max: maximum over heads of the per-head projected scores.
  • softmax: sum over heads of the per-head softmax-normalized projected scores, i.e., aggregated attention mass.
  • union: tokens ranked by their best per-head rank, i.e., the union of per-head top lists, filled to B.

Mean is a linear aggregation of per-head scores. By linearity, ∑h(Mqh)⊤(Mk)=G (Mqˉ)⊤(Mk)\sum_h (Mq_h)^\top(Mk) = G\,(M\bar q)^\top(Mk). Scoring with the group-mean query is exactly equivalent to summing the per-head projected scores, so no per-head score is discarded. The question is therefore whether a nonlinear aggregation (max, softmax, union) preserves important tokens better than the sum.

Preservation of each head's important tokens. To test the concern directly, we score each method against every query head's own exact full-dimensional top-B and average the recall over heads.

Table hS23-3. Per-head recall (%) vs. each query head's own exact top-B.

ModelBmeansoftmaxunionmax
LLaMA-3.1-8B (G=4)25651.6051.2750.5648.63
204858.3357.7256.8155.23
Qwen3-14B (G=5)25640.2538.1238.5834.54
204850.1047.0847.7243.66

Mean is the best in all four settings. If heads were losing tokens that matter only to them, max or union should win this metric, but they are the lowest. A nonlinear aggregate is pulled toward the single largest per-head score and gives up slots that the other heads need.

Attention mass. Softmax aggregation keeps slightly more attention mass than mean: +0.26/+0.25 pp on LLaMA and +1.41/+1.50 pp on Qwen3-14B (B = 256/2048). This is expected, because softmax aggregation directly targets that quantity. It does not carry over to per-head recall (Table hS23-3) or to downstream accuracy (Table hS23-4).

Downstream. On Qwen3-14B, RULER 128K with B = 2048, we changed only the aggregation inside the decode kernel.

Table hS23-4. RULER accuracy (%), Qwen3-14B, 128K, B = 2048.

AggregationAveragemultikey_2multikey_3multiquerycwe
mean82.49100689742.0
softmax81.3892649445.6

The differences concentrate in multi-key retrieval, where the same token must be found by several heads; there, summing across heads helps. Softmax is better on word-frequency counting (cwe). With 25 samples per task on a single setting, we read this as mean being at least as good as softmax, not as a significant gain.

Compute is not matched, and this favors the alternatives. Mean needs one score scan per KV head. Max, softmax and union need G scans (4 on LLaMA, 5 on Qwen3-14B), and softmax adds a normalization over the full context. Matching compute would require shrinking their rank to r/G, which would only weaken them further. Even with G× the compute, the alternatives do not improve per-head recall or downstream accuracy over mean.

Revision. We will add the identity above and this comparison to Section 3.1 and the full tables to the appendix. We will also state in Appendix E.3 that it compares layer-level sharing, and that within-group aggregation is analyzed separately.

AuthorsOpenReview ↗

Response to Reviewer hS23 (3/4)

Response to W3 (1/2): Calibration dependence

Calibration length. We agree that the basis should be calibrated on sequences that cover the maximum deployment context, and Section 4.7 already says so. The dependence runs in one direction only, however. Degradation appears only when the evaluation context is longer than the calibration context. A basis calibrated at the maximum length transfers to every shorter length.

Table hS23-5. PG-19 perplexity (LLaMA-3.1-8B, same budgets as the main evaluation), range over three calibration corpora (LongAlign, NarrativeQA, LooGLE).

Calibration lengthEval 32KEval 64KEval 128K
8K11.35–11.4510.55–10.6444.54–46.00
32K11.26–11.3110.20–10.2316.05–18.31
128K11.27–11.3010.19–10.2211.31–11.48

At 32K and 64K evaluation, every calibration length gives nearly the same perplexity. The degradation appears only in the 128K column, for bases calibrated on shorter sequences. The practical recipe is therefore to calibrate once at the maximum context the deployment will serve.

Transfer to shorter contexts and different distributions. LongBench prompts are much shorter than the calibration sequences (median ≈10K tokens) and span QA, summarization, few-shot and code. We calibrated LLaMA-3.1-8B in four ways, crossing two corpora (LongAlign, Qasper) with two constructions (packed 128K sequences, and unpacked documents with median lengths of 13.5K for LongAlign and 4K for Qasper). Across these four calibrations, JSSA's LongBench average varies by only 0.10 points (49.06–49.16). FASA varies by 3.22 points under the same four calibrations. The subspaces themselves differ substantially, with overlaps of 0.75–0.93 against the deployed basis, yet accuracy does not change. In addition, the same LongAlign-calibrated bases are used unchanged for the reasoning evaluation (Table 3), where the model attends to its own long self-generated traces.

Beyond the calibrated range. In this paper, every model is calibrated at 128K and evaluated at up to 128K. Two of our models support longer contexts: Gemma4-26B-A4B-It (256K) and GLM-4.7-Flash (about 200K). We have not evaluated a 128K-calibrated basis beyond 128K on these models. Table hS23-5 suggests that such use would degrade, and that the remedy is to calibrate at the longer length. Because calibration is forward-only and inexpensive (below), this is practical. We will state this limitation explicitly.

Calibration cost. Recalibration is cheap. On one GPU, calibrating LLaMA-3.1-8B on 8 × 128K tokens takes 2 min 56 s end to end. This consists of 166 s of forward-only prefill to accumulate the second moments and 1.2 s of eigendecomposition for all 256 KV heads, with no gradient steps and no hyperparameters beyond r. Gemma4-26B-A4B-It takes about 8 min.

Comments on the paper

1 comment

General response: new results, corrections, and what still holds

We thank all reviewers for their careful reading. This comment summarizes what we added during the rebuttal, the errors we found in the paper, and what still holds. Details are in the individual responses.

New results added in the rebuttal

  • Closest prior work at the same rank and budget: Double Sparsity, InfiniGen and KQ-SVD in our pipeline (recall, RULER, PG-19), plus comparisons at lower rank and with calibration shorter than the deployment context.
  • Design choices isolated: the query/key weighting (α sweep), query-only and key-only bases, trace normalization, recall near the selection boundary, and within-group GQA aggregation (mean = sum vs. max, softmax mass and union, including RULER).
  • Systems: our attention kernel vs. FlashInfer's page-size-1 decode; QUEST with a single-launch kernel; decoding speed along a reasoning trace; accuracy and speed on the same model at 128K; a full decode-step breakdown; FP8 isolated (BF16 scoring cache, full-KV with FP8 KV); memory accounting; prefill cost; scaling to 1M tokens.
  • Statistics: per-seed standard-deviation bounds and significance tiers for Tables 3 and 4.
  • Calibration: cost (3 min for LLaMA-3.1-8B at 8 × 128K tokens) and which RoPE frequencies change between 32K- and 128K-calibrated bases.
  • MLA: a capacity-matched FASA vs. 64-dimensional JSSA comparison on GLM-4.7-Flash.

While preparing these, we re-checked the paper and found several errors, listed below.

Corrections

  1. KQ-SVD (Tables 13 and 14, Appendix B.6). A bug in our vLLM integration of KQ-SVD projected the query with KQ-SVD's key map on every all-decode step. After the fix, KQ-SVD is statistically indistinguishable from JSSA on AIME 2025 (Qwen3-8B, 8 runs per budget, Welch p ≥ 0.12 at every budget) and matches it on RULER and GPQA-Diamond. We retract the reported KQ-SVD collapse, its non-convergence, and the explanation we gave for it. Our HuggingFace-based KQ-SVD results (PG-19, oracle recall) are unaffected.
  2. Table 16, LLaMA-3.1-8B block. The full-KV, QUEST and SnapKV rows were run in HuggingFace Transformers and all other rows in vLLM, and the full-KV row scored LCC and RepoBench-P over all 500 samples instead of the 200 used everywhere else. JSSA's apparent gain over full-KV on code comes from this, not from JSSA. With a same-engine full-KV, the LLaMA averages are 48.92–49.09 (full-KV) vs. 49.16 (JSSA), and QUEST rises from 46.94 to 47.83.
  3. Transcription errors in Table 16. LLaMA full-KV Qasper/MuSiQue are 45.48/31.23 (printed 45.58/32.23; its average becomes 48.28). Qwen3-14B FASA HotpotQA is 59.96 (printed 56.96; average unaffected).
  4. JSSA's LLaMA-3.1-8B RULER score at 32K is 86.14, not 86.35 (an earlier basis variant). The RULER average is 81.36.
  5. Overstated claims that JSSA "matches or exceeds" full-KV on Ministral-3-8B, is "closest to full-KV across all three models", and "consistently improves over full-KV" in throughput (Sections 4.2.2, 4.5, E.4.2). We will restrict each to the settings where it holds.
  6. Figure 6(a) reports the speedup of attending to the selected tokens only, excluding scoring and top-B. We will label it so.
  7. QUEST in Figures 4 and 9 used one attention launch per KV head. With a single-launch kernel, QUEST is about as fast as JSSA, and we no longer claim a speed advantage over it.

What still holds

  • Accuracy at the same budget. Among QUEST, FASA, Loki, SnapKV and SqueezedAttention, JSSA has the highest RULER average on all three long-context models (81.36 / 86.51 / 88.57 on LLaMA-3.1-8B / Qwen3-14B / Ministral-3-8B, against 76.52 / 85.19 / 87.13 for the best of them) and the lowest PG-19 perplexity among the methods evaluated on it (QUEST, FASA, Loki). Double Sparsity at r = 32 is on par with JSSA in distribution, and we no longer claim higher accuracy than it. KQ-SVD, a KV-compression method that we adapted to token selection, is likewise on par once our integration bug is fixed.
  • Where JSSA differs from these two. When the calibration sequences are shorter than the deployment context, KQ-SVD degrades far more (PG-19 at 128K: 57.63 vs. 12.87). At lower rank, Double Sparsity degrades far more (PG-19 on LLaMA-3.1-8B at r = 16: 15.02 vs. 11.19). JSSA reaches this with a closed form and no tuned hyperparameter.
  • Reasoning. JSSA's differences from full-KV in Tables 3 and 4 are within seed noise, except one cell where JSSA is higher.
  • Kernel. On the same selected tokens, our single-launch per-KV-head attention is 4.3× faster than FlashInfer's page-size-1 decode with one wrapper per KV head at batch 1, and 1.26× faster at batch 64. We drop the claim that it is the first such kernel.

We will apply all of these corrections in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.