- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
JASP is a sparse attention method targeting autoregressive decoding using CPU offloading, by storing a small proxy for each token in KV cache and loading keys and values relevant to the current decode query (plus a recent token window, and attention sink). The authors make the empirical observation that score patterns are similar across KV heads for a given layer. This observation motivates storing a single proxy for each token on a given layer, rather than storing a separate proxy for every KV head in GQA architectures. At prefill, sampled queries and keys are used to perform a joint score factorization, obtaining matrices U_q and U_k that project queries and concatenated keys (across KV heads) into a shared low-rank space, while preserving the original attention scores QK^T as closely as possible. The authors deliver real systems gains over FlashAttention-2 on an RTX 6000D GPU + CPU system. For the 128K context operating point, JASP shows 1.89x higher aggregate decode throughput than FlashAttention-2. RULER results at the same context show a modest accuracy decrease (78.9 → 75.9), while LongBench shows matched accuracy (Llama 3.1 8B).
Critical strengths
- JASP delivers a concrete systems result, demonstrating 1.9–2.6x aggregate decode throughput of FlashAttention-2 under saturated serving across 16K–128K contexts, with a modest accuracy drop for RULER at 128K (dense 78.9 vs. JASP 75.9) and matched LongBench accuracy.
- While joint QK factorization is not new (LRQK), applying joint score-aligned QK factorization to produce a single proxy shared across KV heads, while preserving KV-group-specific retrieval, appears to be a meaningful algorithmic contribution.
- By virtue of having a single proxy shared across KV heads, JASP achieves higher effective compression of metadata stored on the GPU (Table 1: JASP uses substantially less on-GPU proxy memory than most other offloading methods, while capturing higher attention mass).
- In addition to FA2 comparisons, the authors compare serving throughput against other CPU offloading methods (App. G.2). In particular, the authors demonstrate 114.1 tok/s at 128K context vs. LRQK's 16.4 tok/s at the same 1% retrieval budget. LRQK is a baseline close in spirit, similarly using joint QK factorization. Of course, some of the speedup here is also attributable to JASP's different retrieval implementation. Though ParisKV at its native Top-100 setting is faster at 128K (130.7 tok/s), at a lower quality operating point (44.8 vs. JASP’s 54.2 on the Table 16 diagnostic subset).
Critical weaknesses
- end-to-end measurements. While the authors demonstrate aggregate decode throughput gains at saturated serving batches, it is not clear whether this translates into a net end-to-end latency or throughput advantage when prefill is accounted for. JASP prefill and setup: 48.8 seconds at 128K batch 1; FlashAttention-2 prefill: 46.0 seconds at 128K batch 1. Meanwhile, for decode, JASP reaches 114.1 tokens/s at 128K batch 8, compared with 60.3 tokens/s for FlashAttention-2 at 128K batch 4. These are aggregate throughput measurements at different batches, so they do not establish a per-request latency gain. The authors also note in App. G.1 that JASP is slightly slower at batch 1 for decoding. The evaluation settings of RULER and some LongBench tasks involve relatively short generations compared with their prompt lengths, so it is unclear whether there would be a net end-to-end performance advantage in these settings. Prefill-inclusive latency/throughput measurements across different generation lengths would be useful.
- Attribution of systems gains. JASP introduces several systems improvements beyond the joint proxy, including optimized UVA retrieval, HugePages, and capacity-balanced CPU/GPU KV placement. While the authors provide separate ablations of proxy quality, scoring latency, and retrieval performance, it is not clear how much of the overall decode throughput improvement is attributable specifically to the joint proxy. An end-to-end throughput comparison using an alternative per-head proxy under otherwise identical retrieval and memory placement implementations would help clarify this.
- In Table 2, while JASP's average accuracy across RULER tasks is relatively close to the dense attention baseline (dense 78.9 vs. JASP 75.9), JASP appears to struggle on FWE in particular (dense 69.7 vs. JASP 48.3), whereas Quest, for example, scores 68.2 on FWE. JASP very significantly outperforms Quest on MK3, though (JASP 49.0 vs. Quest 3.0), so this is perhaps a tradeoff rather than a strict downside. Even Oracle selection drops to 47.8 on FWE, while Quest scores 68.2, suggesting top-k-by-score selection at 1% performs poorly on aggregation tasks. does a larger budget recover FWE?
Questions for authors
Please see weaknesses
4 comments
Thank you again for taking the time to review our paper. Your feedback has helped us identify ways to make the paper clearer and stronger. If you have any further questions or if anything in our response could use clarification, please feel free to let us know. We would be happy to address any points you raise.
Thank you for the thoughtful response and additional experiments addressing my questions. My overall assessment remains positive, and I maintain my positive score.
Thank you for taking the time to read our response and consider the additional experiments. We appreciate your positive assessment and the constructive feedback you have provided throughout the discussion, which has helped us strengthen the paper. If any further questions arise, we would be happy to address them.
Response to Reviewer gyHY
Thank you for distinguishing serving throughput from request latency and for asking about the contribution of the shared proxy. We added a prefill-inclusive comparison at the same batch size and a shared-versus-per-head proxy comparison with fixed retrieval and KV placement, alongside the extended retrieval-budget experiment on difficult tasks.
Q1. Prefill-inclusive end-to-end performance
We additionally measured end-to-end performance including prefill and proxy construction. At the same batch size, B=4, four requests run concurrently, each with a 112K-token input and a 16K-token output.
JASP achieves a 1.75× end-to-end speedup even when prefill is included.
We agree with the reviewer's observation that RULER and some LongBench tasks require relatively short outputs. Since JASP and FlashAttention-2 have similar prefill overheads, while JASP's primary gains come from decoding, their end-to-end performance may be similar for long-prefill, short-decode workloads. As generation length increases, sustained decode acceleration increasingly amortizes the prefill cost, yielding a more pronounced end-to-end benefit.
However, long generation is an important target scenario for sparse KV-cache offloading. Such methods aim to reduce KV access and computation during sustained decoding over long contexts, including concurrent long-context serving, long-form code generation, and continued long-text generation. JASP's decoding speedup can therefore translate into end-to-end performance gains in these target workloads.
Q2. Contribution of the shared proxy to decode efficiency
We added an independent per-head control. At 128K and B=4, both variants use a 1% dynamic retrieval budget and the same gather path and KV placement. JASP shares one rank-128 code across the eight KV heads, whereas the independent per-head variant stores a separate rank-128 code for each head.
Throughputs are means of three runs and measure decode only. The shared representation uses one-eighth of the historical proxy storage, gives a 2.83× proxy-scoring speedup, and raises full-decode throughput to 1.27× that of the independent per-head variant.
This complements the equal-footprint ablations in Section 4.4 (Table 4): those tests show improved retrieval quality from joint representation at fixed total storage, while the present experiment compares the runtime cost of one shared code with a separate rank-128 code per head.
Q3. Does a larger budget recover FWE?
Yes. We further evaluated larger dynamic retrieval budgets on 50 FWE samples at 128K context and compared the results with Dense attention:
Increasing the retrieval budget from 1% to 8% improves FWE from 43.3 to 60.7, a gain of 17.4 points, leaving only a 4.0-point gap to Dense attention. Thus, at 8% retrieval, FWE approaches dense-attention performance.