acceptodds
Under review as a conference paper at ICLR 2027

JASP: A Joint Attention-Score Proxy for Long-Context KV-Cache Offloading

11 views

Abstract

Long-context LLM inference is increasingly important, but its growing KV cache consumes substantial GPU memory and slows autoregressive decoding. CPU offloading relieves this memory pressure, but sparse decoding must still identify important tokens from the complete history and retrieve their K/V states at every step. This process relies on a GPU-resident proxy whose accuracy determines how few tokens can be retrieved, while its footprint controls the history-dependent bytes read during scoring and is therefore a major contributor to scan latency. Existing projection-based proxies typically construct a separate index for each KV head, repeatedly storing and scanning related retrieval information. We show that KV heads share substantial query-conditioned attention-score structure across layers. Based on this observation, we introduce JASP, a joint attention-score proxy that learns one score-aligned code per historical token across KV heads. Unlike key reconstruction, its objective directly preserves the query–key scores that determine token rankings. Query-specific scoring preserves distinct token rankings, and each KV group retains its own selected token set. JASP therefore removes redundant historical proxy traffic while preserving head-specific retrieval behavior. We further combine GPU-driven selected-row retrieval with capacity-balanced CPU/GPU KV placement to translate the compact proxy into serving gains. Under saturated serving, JASP achieves 1.9–2.6× the aggregate decode throughput of FlashAttention-2 across 16K–128K contexts. JASP scores 75.9 on the complete 128K RULER suite, compared with dense attention’s 78.9, while matching dense quality on LongBench. Code is released at [Anonymous/JASP](https://anonymous.4open.science/r/JASP).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
long-context LLM inference, KV-cache offloading, sparse attention, retrieval proxy, LLM serving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gyHYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

JASP is a sparse attention method targeting autoregressive decoding using CPU offloading, by storing a small proxy for each token in KV cache and loading keys and values relevant to the current decode query (plus a recent token window, and attention sink). The authors make the empirical observation that score patterns are similar across KV heads for a given layer. This observation motivates storing a single proxy for each token on a given layer, rather than storing a separate proxy for every KV head in GQA architectures. At prefill, sampled queries and keys are used to perform a joint score factorization, obtaining matrices U_q and U_k that project queries and concatenated keys (across KV heads) into a shared low-rank space, while preserving the original attention scores QK^T as closely as possible. The authors deliver real systems gains over FlashAttention-2 on an RTX 6000D GPU + CPU system. For the 128K context operating point, JASP shows 1.89x higher aggregate decode throughput than FlashAttention-2. RULER results at the same context show a modest accuracy decrease (78.9 → 75.9), while LongBench shows matched accuracy (Llama 3.1 8B).

Critical strengths
  1. JASP delivers a concrete systems result, demonstrating 1.9–2.6x aggregate decode throughput of FlashAttention-2 under saturated serving across 16K–128K contexts, with a modest accuracy drop for RULER at 128K (dense 78.9 vs. JASP 75.9) and matched LongBench accuracy.
  2. While joint QK factorization is not new (LRQK), applying joint score-aligned QK factorization to produce a single proxy shared across KV heads, while preserving KV-group-specific retrieval, appears to be a meaningful algorithmic contribution.
  3. By virtue of having a single proxy shared across KV heads, JASP achieves higher effective compression of metadata stored on the GPU (Table 1: JASP uses substantially less on-GPU proxy memory than most other offloading methods, while capturing higher attention mass).
  4. In addition to FA2 comparisons, the authors compare serving throughput against other CPU offloading methods (App. G.2). In particular, the authors demonstrate 114.1 tok/s at 128K context vs. LRQK's 16.4 tok/s at the same 1% retrieval budget. LRQK is a baseline close in spirit, similarly using joint QK factorization. Of course, some of the speedup here is also attributable to JASP's different retrieval implementation. Though ParisKV at its native Top-100 setting is faster at 128K (130.7 tok/s), at a lower quality operating point (44.8 vs. JASP’s 54.2 on the Table 16 diagnostic subset).
Critical weaknesses
  1. end-to-end measurements. While the authors demonstrate aggregate decode throughput gains at saturated serving batches, it is not clear whether this translates into a net end-to-end latency or throughput advantage when prefill is accounted for. JASP prefill and setup: 48.8 seconds at 128K batch 1; FlashAttention-2 prefill: 46.0 seconds at 128K batch 1. Meanwhile, for decode, JASP reaches 114.1 tokens/s at 128K batch 8, compared with 60.3 tokens/s for FlashAttention-2 at 128K batch 4. These are aggregate throughput measurements at different batches, so they do not establish a per-request latency gain. The authors also note in App. G.1 that JASP is slightly slower at batch 1 for decoding. The evaluation settings of RULER and some LongBench tasks involve relatively short generations compared with their prompt lengths, so it is unclear whether there would be a net end-to-end performance advantage in these settings. Prefill-inclusive latency/throughput measurements across different generation lengths would be useful.
  2. Attribution of systems gains. JASP introduces several systems improvements beyond the joint proxy, including optimized UVA retrieval, HugePages, and capacity-balanced CPU/GPU KV placement. While the authors provide separate ablations of proxy quality, scoring latency, and retrieval performance, it is not clear how much of the overall decode throughput improvement is attributable specifically to the joint proxy. An end-to-end throughput comparison using an alternative per-head proxy under otherwise identical retrieval and memory placement implementations would help clarify this.
  3. In Table 2, while JASP's average accuracy across RULER tasks is relatively close to the dense attention baseline (dense 78.9 vs. JASP 75.9), JASP appears to struggle on FWE in particular (dense 69.7 vs. JASP 48.3), whereas Quest, for example, scores 68.2 on FWE. JASP very significantly outperforms Quest on MK3, though (JASP 49.0 vs. Quest 3.0), so this is perhaps a tradeoff rather than a strict downside. Even Oracle selection drops to 47.8 on FWE, while Quest scores 68.2, suggesting top-k-by-score selection at 1% performs poorly on aggregation tasks. does a larger budget recover FWE?
Questions for authors

Please see weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer gyHY

Thank you for distinguishing serving throughput from request latency and for asking about the contribution of the shared proxy. We added a prefill-inclusive comparison at the same batch size and a shared-versus-per-head proxy comparison with fixed retrieval and KV placement, alongside the extended retrieval-budget experiment on difficult tasks.

Q1. Prefill-inclusive end-to-end performance

We additionally measured end-to-end performance including prefill and proxy construction. At the same batch size, B=4, four requests run concurrently, each with a 112K-token input and a 16K-token output.

MethodEnd-to-end completion time
JASP672.52 s
FlashAttention-21177.45 s

JASP achieves a 1.75× end-to-end speedup even when prefill is included.

We agree with the reviewer's observation that RULER and some LongBench tasks require relatively short outputs. Since JASP and FlashAttention-2 have similar prefill overheads, while JASP's primary gains come from decoding, their end-to-end performance may be similar for long-prefill, short-decode workloads. As generation length increases, sustained decode acceleration increasingly amortizes the prefill cost, yielding a more pronounced end-to-end benefit.

However, long generation is an important target scenario for sparse KV-cache offloading. Such methods aim to reduce KV access and computation during sustained decoding over long contexts, including concurrent long-context serving, long-form code generation, and continued long-text generation. JASP's decoding speedup can therefore translate into end-to-end performance gains in these target workloads.

Q2. Contribution of the shared proxy to decode efficiency

We added an independent per-head control. At 128K and B=4, both variants use a 1% dynamic retrieval budget and the same gather path and KV placement. JASP shares one rank-128 code across the eight KV heads, whereas the independent per-head variant stores a separate rank-128 code for each head.

ProxyHistorical proxy storage (bytes/token/layer)Proxy scoring (ms/step)Full-decode throughput (tokens/s)
JASP: shared rank-1281305.5094.39
Independent per-head: rank-128 per head1,04015.5674.15

Throughputs are means of three runs and measure decode only. The shared representation uses one-eighth of the historical proxy storage, gives a 2.83× proxy-scoring speedup, and raises full-decode throughput to 1.27× that of the independent per-head variant.

This complements the equal-footprint ablations in Section 4.4 (Table 4): those tests show improved retrieval quality from joint representation at fixed total storage, while the present experiment compares the runtime cost of one shared code with a separate rank-128 code per head.

Q3. Does a larger budget recover FWE?

Yes. We further evaluated larger dynamic retrieval budgets on 50 FWE samples at 128K context and compared the results with Dense attention:

Dynamic retrieval budgetFWE
1%43.3
8%60.7
Dense attention64.7

Increasing the retrieval budget from 1% to 8% improves FWE from 43.3 to 60.7, a gain of 17.4 points, leaving only a 4.0-point gap to Dense attention. Thus, at 8% retrieval, FWE approaches dense-attention performance.

Thank you again for taking the time to review our paper. Your feedback has helped us identify ways to make the paper clearer and stronger. If you have any further questions or if anything in our response could use clarification, please feel free to let us know. We would be happy to address any points you raise.

Thank you for the thoughtful response and additional experiments addressing my questions. My overall assessment remains positive, and I maintain my positive score.

Thank you for taking the time to read our response and consider the additional experiments. We appreciate your positive assessment and the constructive feedback you have provided throughout the discussion, which has helped us strengthen the paper. If any further questions arise, we would be happy to address them.

Reviewer_2SgBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes JASP (Joint Attention-Score Proxy), a compact retrieval proxy for long-context LLM inference with CPU-offloaded KV caches. The key idea is to exploit correlations in attention-score patterns across different KV heads. Instead of maintaining a separate low-rank proxy for each KV head, JASP concatenates the keys across heads and constructs a shared low-dimensional representation. Unlike conventional key-reconstruction approaches, the proposed factorization directly minimizes the reconstruction error of query-key attention scores. Query-specific scoring is preserved, and dynamic Top-k token selection is performed at the KV-group level. The authors further integrate the shared proxy with GPU-driven UVA retrieval, HugePages, and capacity-balanced CPU/GPU KV placement to improve serving throughput. Experiments on Llama-3.1-8B-Instruct and Qwen2.5-7B show promising proxy quality and inference efficiency. Under saturated serving, JASP reports 1.89–2.60× higher aggregate decoding throughput than FlashAttention-2 across 16K–128K contexts. At 128K, it achieves 75.9 on RULER compared with 78.9 for dense attention, while maintaining comparable performance on the evaluated LongBench tasks.

Critical strengths
  1. A compact and effective shared retrieval proxy. The paper proposes a score-aligned joint low-rank representation across KV heads. This reduces redundant historical proxy storage while retaining query-specific scoring and KV-group-specific retrieval decisions. The controlled ablation in Table 4 provides useful evidence for this design.
  2. Good retrieval quality under a small proxy footprint. The quality–footprint comparison is relatively comprehensive and demonstrates the effectiveness of the proposed representation.
  3. Extensive evaluation of the retrieval proxy and serving implementation. The paper evaluates retrieval quality, proxy footprint, downstream performance, and decoding throughput. It also includes comparisons under a common retrieval budget and several component-level ablations. The authors additionally report the overhead of proxy construction during prefill and provide cross-model evaluations, which strengthen the empirical analysis.
Critical weaknesses
  1. Limited algorithmic novelty relative to existing low-rank and cross-head compression methods My primary concern is the limited novelty of the core algorithm. Several important components of JASP have already been investigated in prior work: • Loki and ShadowKV have explored low-rank representations for efficient sparse attention and KV-cache management. • LRQK jointly factorizes queries and keys to approximate attention scores in a low-dimensional space. • SALS (NeurIPS 2025) already proposes a shared low-rank representation across multiple attention heads. In particular, SALS formally discusses the advantage of joint projection over restricted block-diagonal per-head projections under a fixed rank budget. JASP combines cross-head shared representations with a query-aware, score-aligned low-rank factorization. While this combination is reasonable and empirically effective, its individual conceptual components are closely related to existing approaches. The paper cites SALS but does not include a direct evaluation against the complete method. The Joint Key PCA control in Appendix E.1 is useful, but it is not equivalent to a full SALS comparison. Importantly, Appendix E.1 already shows that a shared joint-key representation can achieve reasonably strong retrieval performance. JASP further improves Top-k IoU from 0.514 to 0.610 and oracle-normalized attention mass from 0.978 to 0.988. While these gains support the score-aligned objective, it remains unclear how much additional downstream accuracy or serving efficiency they provide. I would encourage the authors to more explicitly distinguish the new contributions from existing cross-head projection and score approximation methods.
  2. The system-level novelty is relatively limited, and the speedup attribution is insufficient The proposed serving system combines several techniques, including GPU-driven UVA retrieval, HugePages, row-coalesced KV gathering, and heterogeneous CPU/GPU KV placement. However, GPU-native UVA retrieval for CPU-offloaded KV caches has already been explored in ParisKV. The use of large memory pages and optimized memory-access kernels also builds upon established systems techniques. Although the engineering optimizations may be practically useful, they should be distinguished from the core algorithmic contribution. More importantly, the reported end-to-end throughput improvement combines several changes: • A more compact shared retrieval proxy. • Optimized KV gathering and memory access. • CPU/GPU KV placement. • Increased maximum serving batch size. The paper provides component-level ablations, but it does not sufficiently isolate how much end-to-end speedup is directly attributable to the proposed joint score-aligned factorization. A controlled comparison using identical retrieval kernels, KV placement, and serving configurations while varying only the proxy design would strengthen the main efficiency claim.
  3. Lack of strictly quality-matched serving comparisons The paper reports strong decoding throughput relative to FlashAttention-2. However, the practical advantage over existing KV retrieval systems is less clear under quality-matched conditions. For example, Table 15 shows that at 128K: • ParisKV Native achieves 130.7 tokens/s with dynamic Top-100 retrieval. • JASP achieves 114.1 tokens/s with dynamic 1% retrieval. • ParisKV under the common 1% budget achieves 46.5 tokens/s. These results show that JASP is substantially faster than ParisKV under the common 1% retrieval budget, but not faster than its native Top-100 configuration at 128K. The configurations operate at different quality levels. Table 16 also shows that JASP achieves higher average accuracy than native ParisKV on the selected diagnostic tasks. Therefore, neither the native-configuration comparison nor the common-budget comparison provides a strict iso-quality throughput comparison. Furthermore, the aggregate serving throughput measurements depend on the batch sizes supported by the respective systems. Although the paper separately compares JASP and FlashAttention-2 at matched batch sizes, the comparison against other retrieval systems could be more comprehensive. I would encourage the authors to report throughput–accuracy Pareto curves and compare different methods at matched quality targets and batch sizes.
  4. Linear full-history proxy scanning limits the demonstrated scalability Although JASP substantially compresses the historical scoring representation, it still computes approximate attention scores against the compressed representations of all historical tokens during every decoding step. Consequently, the proxy scoring cost remains linear in context length. This differs from coarse-to-fine or candidate-based retrieval approaches that avoid performing fine-grained scoring over the entire history. The experiments primarily cover contexts up to 128K. It remains unclear whether the compact dense-scanning strategy retains its practical advantage at substantially longer context lengths, especially compared with retrieval methods using hierarchical filtering or approximate nearest-neighbor search. A more comprehensive evaluation of proxy scoring and selection latency as a function of context length would help clarify the scalability of the approach.
  5. Insufficient evaluation of long-generation and challenging reasoning workloads The evaluation mainly focuses on long-context retrieval and the conventional LongBench benchmark. I would encourage additional evaluation on challenging long-context reasoning tasks such as LongBench v2, as well as reasoning benchmarks including AIME 2025, MATH-500, and GPQA Diamond when evaluated with models that produce sufficiently long reasoning trajectories. This is particularly relevant because JASP fits the shared low-rank projection during prefill and keeps it fixed throughout decoding. The appendix includes an encouraging analysis of proxy stability over 2,048 generation steps. However, it remains unclear whether similar ranking accuracy, downstream task quality, and serving efficiency are maintained for substantially longer generations. The authors could evaluate different prompt and generation lengths, reporting both task accuracy and decoding efficiency. Appropriate reasoning-capable models should be used so that differences are not dominated by the base model's limited reasoning ability.
  6. Missing fine-grained breakdown of the complete attention pipeline The paper reports proxy scoring latency, retrieval microbenchmarks, and overall decoding throughput. However, a complete per-layer latency breakdown would provide a clearer picture of the actual efficiency bottlenecks. In particular, I would like to see the following costs measured separately: • One-time low-rank factorization, proxy construction, and quantization during prefill. • Query projection and quantization during decoding. • Full-history proxy scoring. • Softmax, KV-group aggregation, and Top-k token selection. • GPU/CPU KV retrieval and transfer. • Final sparse attention computation. • Incremental proxy updates for newly generated tokens. This would help determine how much of the overall layer latency is actually reduced by the shared proxy, as opposed to the existing serving optimizations. The analysis should distinguish one-time setup costs from per-token decoding costs, and should report measurements across different batch sizes and context lengths.
  7. Noticeable quality degradation on some difficult RULER tasks Although JASP achieves a relatively small average degradation on the full RULER suite, certain tasks exhibit substantially larger drops. For example, at 128K: • MK3 decreases from 63.5 with dense attention to 49.0. • FWE decreases from 69.7 to 48.3. The authors reasonably observe that oracle Top-k retrieval under the same budget also suffers substantial degradation, suggesting that the limited retrieval budget accounts for much of the remaining quality loss. Nevertheless, this raises an important practical question: how much additional retrieval and serving cost is required to recover dense-attention quality on these challenging tasks? A throughput–quality analysis across retrieval budgets would be more informative than relying primarily on the 1% operating point.
Questions for authors

The following questions would materially affect my assessment of the paper.

Q1. Novelty relative to SALS and LRQK SALS has already investigated cross-head joint low-rank projection, while LRQK has explored low-rank query-key score approximation. Could the authors clarify which theoretical or algorithmic properties of JASP are fundamentally new beyond these existing formulations? The Joint Key PCA experiment in Appendix E.1 is helpful. Could the authors further quantify how much improvement the score-aligned objective provides in downstream accuracy and end-to-end serving efficiency under an otherwise identical system configuration?

Q2. Independent contribution of the joint proxy to end-to-end efficiency Could the authors report a controlled end-to-end throughput comparison among per-head score proxies, block-diagonal score proxies, joint-key PCA proxies, and JASP, while keeping the retrieval kernel, CPU/GPU placement, Top-k budget, and batch size identical? This would directly establish the system-level benefit of the proposed factorization independently of the engineering optimizations.

Q3. Quality-matched comparison with existing retrieval systems Could the authors provide an iso-quality throughput comparison with ParisKV and other relevant KV retrieval methods? In particular, how does JASP compare when the methods are configured to achieve approximately the same RULER or LongBench accuracy, rather than simply using the same dynamic retrieval ratio? Please also report matched-batch throughput alongside maximum-throughput configurations.

Q4. Complete per-layer latency breakdown What fraction of the total sparse attention layer latency is spent on proxy scoring, Top-k selection, KV retrieval, and the final attention computation? Could the authors report these components separately, together with the one-time proxy construction cost during prefill, under different context lengths and batch sizes? This would clarify whether the proposed compressed proxy is the dominant source of practical acceleration.

Q5. Scalability and long-generation behavior How does the proxy scoring and token-selection latency scale beyond 128K contexts, given that JASP still scans all historical compressed codes? Furthermore, does the prefill-calibrated, fixed projection remain effective for substantially longer generation trajectories beyond the 2,048-step analysis currently reported in the appendix? A combined accuracy–throughput evaluation on long-generation workloads would help establish the practical operating range of the method.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 2SgB [1/3]

Thank you for recognizing the shared proxy, retrieval quality, and experimental analysis. We clarify the design differences from prior work and use controlled experiments to distinguish the roles of proxy representation, retrieval budget, and serving capacity.

Q1. Differences from LRQK and SALS, and the contribution of the score-aligned objective

Novelty and differences from prior work. JASP constructs a single attention-score retrieval index shared across KV heads, while retaining query-specific scoring and KV-group-specific token selection. We jointly represent post-RoPE keys and embed each query only in the block of its assigned KV group. This shares the history code that grows with context length without changing the original query–KV associations or independent retrieval decisions.

LRQK and JASP target the same dynamic KV retrieval problem but use different algorithmic designs. LRQK performs low-rank score approximation independently within each attention head and uses iterative updates. JASP instead constructs a history index shared across KV heads through joint score fitting. It preserves independent retrieval decisions for each group while eliminating redundant storage and scanning across separate head-wise indexes.

SALS and JASP address different problems with different optimization objectives. SALS targets compression of the KV cache itself: it compresses the cache using a joint low-rank representation and reconstructs selected keys. JASP targets dynamic retrieval from CPU-offloaded KV caches: only the historical retrieval index is compressed, while the original-precision K/V is preserved and fetched for final attention. The two works merely exploit the same property of LLMs—correlated structure across different KV heads—which does not mean that they solve the same problem. Their optimization objectives, compressed objects, method designs, and execution paths are fundamentally different.

Section 4.4 and Appendix E.1 of our paper already evaluate JASP's design contributions through controlled ablations.

Independent contribution of the score-aligned objective. To isolate the fitting objective, we compare only Joint Key PCA and JASP while keeping the shared-index structure, historical storage budget, retrieval ratio, and execution backend unchanged. Appendix E.1 already shows that score alignment reduces score error from 0.069 to 0.053, improves M/O from 0.978 to 0.988, and increases IoU from 0.514 to 0.610.

We further evaluate downstream quality on 50 new samples each from MK2, MK3, FWE, and QA2 at 128K, and measure decode efficiency under the same execution backend:

MethodDownstream average ↑128K, B=4 decode throughput ↑
Joint Key PCA54.194.5 tokens/s
JASP55.894.4 tokens/s

With identical storage and execution configurations, the score-aligned objective improves the downstream average by 1.7 points while leaving decode throughput essentially unchanged. Both methods use the same historical code size, scoring layout, and retrieval budget, so changing the fitting objective does not alter the principal computation or memory-access workload during decoding. Thus, score alignment primarily improves attention-score approximation, token-selection quality, and downstream task performance rather than providing additional decode acceleration.

AuthorsOpenReview ↗

Response to Reviewer 2SgB [2/3]

Q2. What does the joint proxy contribute under fixed execution conditions?

We compare the four proxy variants requested by the reviewer, holding the rank-128 packed kernel, historical index footprint (130 bytes/token/layer), 1% dynamic retrieval budget, CPU/GPU KV placement, gather path, and batch size fixed. Only the proxy representation changes.

ProxyThroughput at 128K, B=1 (tokens/s)M/O ↑
Per-head Score36.60.959
Block-diagonal Score36.70.962
Joint Key PCA36.60.978
JASP36.70.988

Throughput is the mean of three runs in the new controlled experiment, timing decoding only. M/O values come from Section 4.4, Table 4, and Appendix E.1 of the submission.

Under identical storage budgets and execution configurations, all four proxies achieve essentially the same decode throughput. The history scan size, number of dynamically retrieved rows, KV transfer volume, and final attention workload are held fixed. Changing the proxy fitting method does not directly reduce those computation or memory-access costs.

However, identical execution costs do not imply identical retrieval quality. Section 4.4 and Appendix E.1 show that at the same 1% dynamic retrieval budget, JASP attains an M/O of 0.988, exceeding the other proxies. It therefore preserves more of the relevant attention mass for the same retrieval cost.

The algorithmic benefit of JASP is not a faster execution kernel at an identical retrieval budget; rather, it is the ability to meet a given quality target with fewer KV rows retrieved. Other proxy designs require a larger retrieval budget to attain comparable quality, increasing KV retrieval, data movement, and final attention costs, and consequently reducing decode throughput. JASP translates more accurate shared retrieval into practical serving acceleration by reducing the retrieval work required to reach the target quality.

Additional throughput comparison at matched per-head rank. We also compare JASP’s shared rank-128 index with an independent rank-128 index for each of the eight KV heads at 128K and B=4. Both use the same 1% dynamic retrieval budget, CPU/GPU KV placement, and gather path. Including quantization scales, the shared index stores 130 rather than 1,040 bytes per token per layer. Decode throughput, averaged over three runs, is 94.39 tokens/s for JASP versus 74.15 tokens/s for the independent per-head proxy (1.27×).

Q3. Quality–throughput comparison with existing retrieval systems

A strictly iso-quality operating point is not straightforward to define. RULER and LongBench comprise multiple tasks with different objectives. Equal aggregate scores need not imply similar performance on each task, while requiring all subtasks to match closely does not yield a single unambiguous comparison criterion.

More importantly, JASP does not trade quality for speed: at the tested common retrieval budget, it achieves both higher task quality and higher decoding throughput. We therefore report quality across retrieval budgets together with matched-batch throughput, rather than relying solely on an arbitrarily selected iso-quality point.

128K four-task evaluation averages:

Dynamic retrieval budgetJASPParisKVLRQK
1%55.852.554.7
2%57.855.857.5

128K decode throughput (tokens/s):

MethodDynamic retrieval budgetB=1B=4
JASP1%36.794.4
ParisKV1%27.942.0
LRQK1%6.516.2

At the common 1% retrieval budget, JASP achieves a higher average task score than both ParisKV and LRQK, as well as higher decoding throughput at both B=1 and B=4. ParisKV must increase its budget to 2% to reach JASP's 1% average score of 55.8; its B=4 throughput then falls to 25.0 tokens/s, substantially below JASP's 94.4 tokens/s.

These results show that JASP is not merely another speed–quality trade-off: it attains equal or better task quality with a smaller retrieval budget while achieving higher practical decoding throughput. In addition to the matched-batch comparisons above, the original paper reports aggregate serving throughput at each system's maximum admissible batch, capturing the benefits of greater concurrency capacity.

AuthorsOpenReview ↗

Response to Reviewer 2SgB [3/3]

Q4. Stage-wise latency and one-time construction cost

We profile JASP at context lengths of 32K and 128K with batch sizes of 1 and 4. The table reports cumulative time across all 30 sparse attention layers per decoding step, in milliseconds.

Context lengthBatchProxy scoringtop-kKeep-set constructionKV gatherSparse attention kernel
32K14.8703.4850.6952.8270.240
32K44.7623.3840.6744.1360.250
128K14.7843.4330.6786.4160.282
128K45.5023.1130.58219.0360.504

Proxy scoring and KV gathering are the two key components of retrieval efficiency.

Proxy scoring. JASP reduces historical data scanned through its compact shared codes. Across these settings, the cumulative proxy-scoring time over 30 sparse layers remains within 4.76–5.50 ms, demonstrating low and stable overhead. Figure 5(b) of the submission further shows JASP's scoring-efficiency advantage over other low-rank retrieval proxies.

KV gather. The total gathering time increases with retrieved KV volume as context length and batch size grow. Figure 6(a–b) of the submission already provides controlled comparisons at the same retrieval volume: compared with the strongest existing UVA baseline (ParisKV), JASP's optimized gather achieves a 2.6× speedup, with effective PCIe bandwidth utilization reaching 90.3% of the platform's measured saturated UVA bandwidth.

One-time initialization cost. At 128K, JASP takes 49.20 s for prefill and initialization, including 2.43 s for the joint score factorization. The factorization is performed once during prefill; the projection is kept fixed throughout decoding without repeated fitting.

Q5. Context scalability and long-generation behavior

Context scalability. First, Llama-3.1-8B-Instruct, the model used in our main experiments, has a native 128K context window. Thus, 128K is the longest context configuration that we can directly evaluate with this model.

Although JASP scans the entire historical proxy, its compact shared low-rank code substantially reduces the bytes scanned per historical token. The measurements in Q4 show that proxy scoring and top-k selection times change only modestly as context grows from 32K to 128K. Across the tested batch configurations, proxy scoring takes 4.76–5.50 ms and top-k takes 3.11–3.49 ms. Thanks to the compact shared proxy, JASP maintains low and stable retrieval-decision overhead throughout the evaluated long-context range.

Long-generation behavior. We additionally evaluate ten LongBench v2 inputs with 64K-token prefixes followed by 16,384 generated tokens. JASP's projection remains fixed throughout decoding, and we compare against ParisKV and Quest using the same 1% dynamic retrieval budget.

MethodM/O after 16K generated tokens ↑
JASP0.9667
ParisKV0.9602
Quest0.8497

Notably, ParisKV explicitly incorporates a drift-robust design for retrieval representations during long-context decoding, yet JASP achieves higher M/O without refitting its projection throughout generation. After 16K generated tokens, JASP retains 96.67% of the attention mass captured by the same-budget oracle, demonstrating that the fixed shared score projection preserves high retrieval accuracy over these long-generation trajectories.

Reviewer_fbAfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

JASP is a joint attn-score proxy for sparse KV offload. JASP can make each KV group select its own important tokens while make the acc good. JASP can reach almost oracle quality and get around 2x decode tps over FA2 under certain serving settings.

Critical strengths
  1. Fitting the proxy to scores instead of reconstructing keys is a good idea.
  2. The authors show that the proxy quality is very good with very small footprint.
  3. For the system impl, I think it is solid. The uva gather and capacity-balanced placement are practical and ablated well.
Critical weaknesses
  1. First, the novelty is limited for this paper. Shared low--rank structure is not a new idea: ShadowKV, Palu, xKV do the similar things. Therefore, the new part is limited.
  2. The paper uses ShadowKV as a baseline but it did not test the TPS for it. The ShadowKV is released and designed for offloading tps. I think the author should add this.
  3. The paper only calibrated the proxy for the prefill. For longer generation settings, the distribution may drift and it is not tested. (ShadowKV did some related tests though)
  4. It is better to test more GPUs or more models. Now it looks like slower than the FA2 under small bsz.
Questions for authors
  • How does the proxy performs for long gen settings like 16K+ generated tokens?
  • Can ShadowKV be added to the tps table?
  • How does it work on models with very few KV heads or MLA?
5 comments
AuthorsOpenReview ↗

Response to Reviewer fbAf [1/2]

Thank you for the feedback on JASP's proxy design and serving implementation. We address the contribution, long-generation stability, the ShadowKV comparison, and cross-model serving below.

W1. Novelty and the contribution of score alignment

JASP's specific contribution is a shared, score-fitted retrieval index with query-specific scoring and KV-group-specific selection. It jointly represents post-RoPE keys while embedding each query only in its original KV-group block. One historical code can therefore support different query-head scores and different group-level retrieval sets. The selected original, model-precision K/V rows are then fetched for attention; the proxy is used for selection rather than KV reconstruction.

The equal-footprint ablations in Section 4.4 (Table 4) separate the effects of score fitting, cross-head structure, and query-specific scoring. With the shared representation, storage budget, and retrieval conditions fixed, Appendix E.1 (Table 7) shows that score alignment lowers attention-score error and improves token selection.

Q1. Fixed projections during long generation

Figure 7 in Appendix E.2 shows that with a fixed projection over 2,048 generation steps, mean M/O remains above 98% and drops by 0.29 percentage points from the first window to the last.

To further evaluate longer generations, we select 10 LongBench v2 inputs, each with a 64K-token prefix, and generate 16,384 tokens while keeping JASP's projection fixed throughout decoding. We compare against ParisKV and Quest under the same 1% dynamic retrieval budget.

MethodM/O after 16K generated tokens ↑
JASP0.9667
ParisKV0.9602
Quest0.8497

Notably, ParisKV explicitly incorporates a drift-robust retrieval mechanism for long-context decoding, yet JASP achieves a higher M/O without refitting its projection during generation. After 16K generated tokens, JASP still retains 96.67% of the attention mass captured by the same-budget Oracle, demonstrating that the fixed shared score projection maintains high retrieval accuracy over long generation trajectories.

Q2. ShadowKV: throughput and downstream quality

Although ShadowKV and JASP both accelerate long-context KV offloading, they use different KV handling strategies. ShadowKV reconstructs selected keys from low-rank representations and retrieves only the corresponding values, reducing KV transfer volume and improving throughput. JASP instead uses the low-rank proxy solely for token selection and retrieves the original, model-precision K/V for attention, avoiding approximation error from key reconstruction. We omitted ShadowKV from the original throughput comparison because the methods use different KV representations and transfer volumes, making the workloads different even at the same nominal retrieval budget.

To address the requested comparison, we measured both methods at the same nominal dynamic retrieval budget. At approximately 128K context, batch size 4, and 1% dynamic retrieval, JASP achieves 114.49 tokens/s versus 124.63 tokens/s for ShadowKV, reaching 91.9% of ShadowKV's decode throughput.

JASP scores higher on the downstream evaluations below. Tables 2 and 3 report these results at the same 1% dynamic retrieval budget:

Quality evaluationJASPShadowKV
RULER (13-task average)75.972.7
RULER MK286.571.0
RULER MK349.023.5
LongBench (14-task average)36.434.8

These results highlight the different trade-offs of the two offloading designs: ShadowKV reduces data transfer by reconstructing keys and achieves higher throughput; JASP preserves the original keys and achieves higher downstream quality, especially on multi-key retrieval, while retaining throughput close to ShadowKV.

AuthorsOpenReview ↗

Response to Reviewer fbAf [2/2]

Q3. Cross-model validation, concurrent serving, and MLA applicability

Cross-model validation. In addition to Llama-3.1-8B, Appendix E.2 reports proxy retrieval quality across context lengths, quality on 14 LongBench tasks, and serving throughput at 8K–32K on Qwen2.5-7B. These results demonstrate strong retrieval quality, downstream performance close to dense attention, and practical serving speedups on another model family.

Serving batch and throughput. We agree that JASP has lower decode throughput than FlashAttention-2 at small batch sizes. The objective in this serving setting is to maximize aggregate throughput for concurrent long-context requests, rather than minimize single-request latency. At each system's maximum admissible batch on the same hardware, JASP achieves 1.89–2.60× the aggregate decode throughput of FlashAttention-2.

MLA and models with fewer KV heads. JASP fits a compact historical retrieval proxy to attention scores and does not inherently require a particular number of KV heads. With MLA, attention scores can likewise be evaluated from the shared latent KV representation, enabling a corresponding score-aligned retrieval proxy while preserving query-head-specific scoring. MLA's cache representation therefore does not preclude applying this mechanism at the algorithmic level. We additionally evaluated Qwen2.5-1.5B-Instruct with only 2 KV heads. JASP achieves M/O of 0.9956 and 0.9962 at 1% and 2% dynamic retrieval budgets, respectively, demonstrating high retrieval quality even with very few KV heads.

Thank you again for taking the time to review our paper. Your feedback has helped us identify ways to make the paper clearer and stronger. If you have any further questions or if anything in our response could use clarification, please feel free to let us know. We would be happy to address any points you raise.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Thanks

As the discussion period draws to a close, we would like to briefly summarize what we added in response to your feedback. We evaluated retrieval stability over 16K generated tokens, added a throughput comparison with ShadowKV at the same retrieval budget, and measured end-to-end performance including prefill and proxy construction. We also ran controlled experiments to isolate the contribution of the shared proxy, compared throughput at similar task quality, and clarified how JASP differs from prior work.

We sincerely thank all three reviewers for your careful assessments, constructive suggestions, and engagement throughout the discussion. Your feedback helped us strengthen the evaluation and present our contributions more clearly. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.