acceptodds
Under review as a conference paper at ICLR 2027

Finding the Needles in Long-Context Attention through Sparse Features

Abstract

In long-context decoding, high attention scores often concentrate on a small fraction of historical tokens scattered throughout the context. Attending only to these tokens can preserve generation quality, but finding them efficiently is a needle-in-a-haystack problem. We show that these tokens are not arbitrarily distributed in feature space: although they may differ in many respects, they share query-relevant features in a high-dimensional sparse representation. To expose this structure, we develop a shared query-key sparse autoencoder (SAE) that decomposes dense queries and keys into a common dictionary of sparse features. Most high-scoring keys share active features with the query, whereas such overlap is uncommon among the remaining keys. Building on this finding, we introduce PrismKV, which indexes historical keys by their active SAE features to form overlapping buckets. Compact bucket summaries identify relevant tokens without scanning the entire history, allowing attention to operate only on the selected original KV entries. Across RULER and LongBench on the 27B model, PrismKV retains an average of 99.90% of full-attention performance. Relative to optimized full attention, it achieves attention-path speedups of 3.37x on Qwen3-8B at a 128K context length and 2.78x on the 27B model with a 256K total window.

Primary area:
foundation or frontier models, including LLMs
Keywords:
kv cache, attention, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MFCEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PrismKV, a selective KV access method for long-context autoregressive decoding. The authors start from the observation that historically important tokens for attention, while scattered across the original sequence, tend to share a small set of features with the current query in high-dimensional sparse representations. To exploit this structure, PrismKV trains a shared query-key sparse autoencoder per attention layer and per KV group, assigning each historical key to multiple overlapping feature buckets. Each bucket stores token indices together with the count, mean, and per-dimension variance of the original keys. During decoding, the system estimates the attention quality of each bucket using the current dense query and bucket statistics, selects the highest-scoring buckets, merges and deduplicates tokens from the selected buckets, and then computes attention over the original KV entries.

Critical strengths
  1. An insightful overlapping-feature indexing design. Unlike methods that assign each key to a single cluster or contiguous position block, PrismKV allows one token to belong to multiple feature buckets simultaneously. This enables the representation that “the same token can be relevant to different queries via distinct features”, which aligns well with the compositional nature of attention relevance and constitutes the paper’s most valuable technical contribution.
  2. Strong quality results with indexing overhead taken into account. PrismKV achieves competitive performance across multiple benchmarks for long-context understanding, continual generation, and context reuse. The final attention computation uses raw Q/K/V, preventing SAE reconstruction error from directly corrupting attention scores. The paper also reports indexing construction, periodic update, attention-path, and end-to-end latency, rather than only idealized candidate-retrieval cost.
Critical weaknesses
  1. There is a notable disconnect between the paper’s core mechanistic claim and the actual inference algorithm. The paper attributes performance gains to “shared sparse features” between queries and keys, yet the bucket scoring formula during inference uses the original dense query along with the mean and variance of keys within each bucket, without explicitly using the SAE encoding of the query �0��0�. The method can therefore alternatively be interpreted as “using an SAE to produce overlapping key clusters, then routing over cluster summaries with the dense query”. Existing experiments do not establish that shared query–key features themselves are a necessary ingredient for successful retrieval.

  2. The shared-feature analysis carries a risk of circular reasoning, and the associated ablations are insufficient to establish causality. The SAE matching loss trains sparse query/key encodings directly using full-attention outcomes. Thus the observation that high-attention keys tend to share features with the query under the supervised SAE is, to some extent, an expected consequence of the training objective. The paper includes an analysis without matching supervision, but this analysis covers only 14 contexts and a subset of layers/groups. More importantly, removing matching supervision degrades RULER and LongBench scores by merely 0.405 and 0.051 points, suggesting that this supervision and its corresponding “shared-feature” explanation may not be the primary source of performance gains.

  3. Critical ablations and quality–efficiency Pareto analysis supporting the design choices are missing. The paper does not systematically study key hyperparameters such as dictionary size, activation count per key/query, number of selected buckets, and unindexed tail length. It also does not ablate the three scoring terms — bucket count, mean, and variance — separately. The fixed default configuration cannot reveal whether performance is sensitive to these choices, nor whether PrismKV consistently outperforms simpler clustering, LSH, or learned routing methods under equal actual token-read budgets.

  4. Baseline coverage and budget fairness are inadequate. The paper states that all methods use the same token-reading budget, yet it primarily reports the realized read fraction for PrismKV. Since bucket sizes and deduplicated candidate counts selected by PrismKV vary dynamically, it remains unclear how budgets are strictly aligned per sample, per layer, and per KV group. Experiments do not include direct comparisons against closely related methods such as RetrievalAttention, Squeezed Attention, or DeepSeek Sparse Attention, and latency comparisons are largely limited to the authors’ adapted Quest implementation. This is insufficient to support a claim of comprehensive advantages over state-of-the-art retrieval-based attention methods.

  5. System evaluation and reproducibility do not yet meet top-conference standards. Latency is measured only on the PPU-ZW810 with batch size near 1 and no dynamic/continuous batching. Results on mainstream GPUs, serving throughput, concurrent requests, tail latency, and real memory overhead are absent. The reported SAE training cost also excludes backbone Q/K trajectory collection. Details on training data sources, baseline configurations, and train–test separation are insufficient. Notably, for the 27B model, the 2.78× speedup on the attention path translates into only 1.16× end-to-end speedup, meaning the practical system-level benefit remains modest.

Questions for authors
  1. Is the sparse query encoding �0��0� computed or used during inference? If not, how do the authors demonstrate that performance arises from query–key shared features, rather than overlapping key clusters produced by the SAE together with dense bucket-summary routing?

  2. Could the authors add the following critical controls: bucket selection directly using �1��1�, vanilla k-means, LSH, a learned router without SAE, and ablated variants that use only bucket mean or bucket size? How do these alternatives perform under the same number of candidate tokens?

  3. Removing query–key matching supervision yields only a tiny downstream performance drop. How do the authors explain the observation that shared-feature coverage decreases substantially while task performance remains nearly unchanged? Is the matching loss truly necessary?

  4. How exactly is the “identical token-reading budget” enforced? Do all baselines read exactly the same number of unique historical tokens per sample, per layer, and per KV group? Could the authors report the realized read fraction for every method together with full quality–latency Pareto curves?

  5. Could the authors supplement end-to-end results on mainstream NVIDIA GPUs, including continuous batching, multi-request throughput, peak memory usage, index storage overhead, candidate merging, and the cost of irregular KV gather? In addition, what is the full offline training and data-preparation cost when Q/K trajectory collection is included?

9 comments
AuthorsOpenReview ↗

Comment 1 of 7

W1 / Q1 — Dense Query Routing and SAE Bucket Memberships

The SAE training code, trained SAE encoder weight matrices, inference implementation, optimized kernels for NVIDIA H200 and PPU-ZW810, and experimental results are all publicly available in the anonymous PrismKV repository.

Changes in the revised manuscript. Appendix D covers direct sparse-Q routing (W1/Q1, Table 20) and the design ablations (W3/Q2, Tables 21–25). Appendix B.2 (Tables 16–17) covers feature sharing and the matching-loss comparison (W2/Q3). Appendices A.5–A.6 and D.4 give the read-budget, baseline, and quality–latency comparisons (W4/Q4). Appendices E (Tables 26–29) and C.2 (Table 19) report H200 serving results and offline costs including Q/K capture (W5/Q5).

  • W1 / Q1

There is a notable disconnect between the paper’s core mechanistic claim and the actual inference algorithm. The paper attributes performance gains to “shared sparse features” between queries and keys, yet the bucket scoring formula during inference uses the original dense query along with the mean and variance of keys within each bucket, without explicitly using the SAE encoding of the query . The method can therefore alternatively be interpreted as “using an SAE to produce overlapping key clusters, then routing over cluster summaries with the dense query”. Existing experiments do not establish that shared query–key features themselves are a necessary ingredient for successful retrieval.

Is the sparse query encoding computed or used during inference? If not, how do the authors demonstrate that performance arises from query–key shared features, rather than overlapping key clusters produced by the SAE together with dense bucket-summary routing?

  • A: We use dense queries with bucket summaries at decode time. The SAE features determine which keys belong to each overlapping bucket. The router then scores those buckets using statistics of the original keys at the same asymptotic cost as projecting the query into SAE space. With the same bucket memberships and an 8% cap on unique old-history reads, bucket-summary routing scores 54.057 vs. 53.550 on LongBench and 96.731 vs. 95.901 on RULER, gains of 0.507 and 0.830 points over direct sparse-Q routing.

On all 3,250 Qwen3.8-27B RULER examples, SAE bucket assignments also outperform overlapping buckets built with multi-assignment K-means or sparse random projection under the same per-step read budget. See W3/Q2 below for the protocol and results by length.

MethodMacro average
PrismKV (SAE)96.838
Multi-assignment K-means90.585
Sparse random projection89.071
AuthorsOpenReview ↗

Comment 2 of 7

W2 / Q3 — Feature Sharing With and Without Q–K Matching

  • W2 / Q3

The shared-feature analysis carries a risk of circular reasoning, and the associated ablations are insufficient to establish causality. The SAE matching loss trains sparse query/key encodings directly using full-attention outcomes. Thus the observation that high-attention keys tend to share features with the query under the supervised SAE is, to some extent, an expected consequence of the training objective. The paper includes an analysis without matching supervision, but this analysis covers only 14 contexts and a subset of layers/groups. More importantly, removing matching supervision degrades RULER and LongBench scores by merely 0.405 and 0.051 points, suggesting that this supervision and its corresponding “shared-feature” explanation may not be the primary source of performance gains.

Removing query–key matching supervision yields only a tiny downstream performance drop. How do the authors explain the observation that shared-feature coverage decreases substantially while task performance remains nearly unchanged? Is the matching loss truly necessary?

  • A: The circular-reasoning objection conflates the effect of matching supervision with a relationship we observed independently. We clarify the motivation for adding a q-k matching term as below:
  1. We observe that high-attention keys share SAE features with the query. Even without Q–K matching supervision, 31.86% of the top 2% highest-scoring keys in the Qwen 27B SAE share at least one feature with the query, compared with 4.96% of the remaining keys.
  2. The observation directly motivated our choice of a Q–K matching loss. We first observed that high-attention keys were more likely to share SAE features with the query. We chose this particular loss because it directly reinforces the alignment we had already observed, helping the SAE organize tokens for retrieval.
  3. Practical benefit. We retain Q–K matching as a training-only objective that reinforces the observed alignment and adds no operations to the decoding path. On Qwen3-8B, the checkpoint trained with matching scores 0.406 points higher on RULER and 0.051 points higher on LongBench. On Qwen3.8-27B, the two checkpoints score within 0.22 points on both benchmarks, with small differences in opposite directions.

The SAE without Q–K matching still shows enrichment in shared features. Using the seven task families and decode-step sampling from the original experiment, we extend the 27B analysis to 200 new contexts, with 100 each at 32K and 64K. This covers all 16 standard attention layers and 64 KV groups. Both SAE checkpoints use the same Full Attention trajectories.

SAE checkpointHistorical-key splitTop-scoring keys sharing ≥1 feature with current QRemaining keys sharing ≥1 feature with current Q
Without Q–K matchingTop 2% / rest31.86%4.96%
Without Q–K matchingTop 1% / rest37.66%5.17%
With Q–K matchingTop 2% / rest48.26%4.43%

Without matching, the top-scoring keys share more features at both cutoffs in all 200 contexts. This supports shared features as a motivation for overlapping bucketing. The checkpoints have different training histories, so their difference cannot be attributed to the matching loss alone. 200-context results and 32K/64K heatmaps

AuthorsOpenReview ↗

Comment 3 of 7

W3 / Q2 (Part 1 of 2) — Hyperparameters and Direct Sparse-Q Routing

  • W3 / Q2

Critical ablations and quality–efficiency Pareto analysis supporting the design choices are missing. The paper does not systematically study key hyperparameters such as dictionary size, activation count per key/query, number of selected buckets, and unindexed tail length. It also does not ablate the three scoring terms — bucket count, mean, and variance — separately. The fixed default configuration cannot reveal whether performance is sensitive to these choices, nor whether PrismKV consistently outperforms simpler clustering, LSH, or learned routing methods under equal actual token-read budgets.

Could the authors add the following critical controls: bucket selection directly using z(q)z(q), vanilla k-means, LSH, a learned router without SAE, and ablated variants that use only bucket mean or bucket size? How do these alternatives perform under the same number of candidate tokens?

  • A: We added equal-read ablations, bucket construction controls, and a quality–latency curve for Qwen3.8-27B.

    1. Hyperparameter sweep.

Qwen3.8-27B. Quality and old-history reads come from all 3,750 LongBench examples. Graph decode is measured separately at 128K, batch size 1.

SweepSettingLongBenchOld readsGraph decode (ms/step)
Dictionary width1,00053.8517.991%54.360
Dictionary width2,00053.8147.991%54.343
Dictionary width3,00053.9257.991%54.763
Dictionary width4,00053.9537.991%55.092
Fixed memberships1 per key54.0043.353%47.064
Fixed memberships2 per key53.9774.704%47.251
Fixed memberships4 per key54.0736.568%47.545
Fixed memberships8 per key54.1419.150%48.188
Fixed memberships16 per key54.12313.022%48.981
Bucket countFull Flash54.034100%50.756
Bucket countTop149.6330.80%46.541
Bucket countTop251.7811.44%46.591
Bucket countTop452.9592.58%47.024
Bucket countTop853.5274.71%47.354
Bucket countTop1653.9968.67%48.062

Qwen3-8B. Index updates are measured at 64K on a fixed Full continuation. Reads include the unindexed tail, and latency includes amortized updates.

Update interval (tokens)Reads (tokens/KV group)Conditional PPLEager decode incl. updates (ms/step)
No update7,513.51.275022.131
5125,601.91.283622.539
2,0486,243.21.273922.291
4,0967,161.11.306522.311
  • Dictionary width. At 7.991% reads, LongBench scores span 0.139 points. Mean index build time rises from 0.083 to 0.144 s.
  • Key memberships. With Top16 fixed, more memberships increase reads and decode latency. Scores vary by only 0.164 points, with eight memberships scoring highest.
  • Selected buckets. This sweep shows the largest quality change. Top8 reads 3.96 percentage points less than Top16 and scores 0.469 points lower.
  • Index update interval. Updating every 512 tokens cuts reads by about 25% versus no updates, while amortized decode rises from 22.131 to 22.539 ms/step.

Hyperparameter sweeps and timings

  1. Direct bucket selection using z(q)z(q). We use up to 16 positive SAE features per query head to rank their corresponding buckets, aggregating normalized activation weights across the six query heads in each KV group. Both arms share the SAE, key memberships, attention kernel, and 8% cap on unique old-history reads. The default router scores bucket summaries with dense queries.
BenchmarkBucket summariesDirect z(q)z(q)
LongBench54.05753.550
RULER96.73195.901

Direct Q routing comparison

Reviewer_YcTqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PrismKV, a selective KV-access method for long-context decoding. A shared query–key sparse autoencoder (SAE) assigns keys to overlapping feature buckets. At inference time, queries score bucket summaries consisting of counts, means, and diagonal variances, and attention is evaluated on the union of selected original KV entries. The paper connects this design to measured feature overlap among high-attention keys and evaluates accuracy, decoding latency, and index amortization on several benchmarks and model backbones.

While the observation is interesting, I find the broader significance of the resulting KV-selection method limited given recent progress in training models with sparse attention, and its implementation details remain unclear.

Critical strengths
  1. The observation that high-attention keys are more likely to share active sparse features with the query, even when their other features differ, is interesting. Figure 2 provides strong empirical support across the evaluated data and layers, while Appendix B.1 shows that the association persists without explicit query–key matching supervision. This observation is also intuitively consistent with dot-product attention: shared feature directions can contribute positively to query–key alignment, as illustrated by the decomposition in Section 2.1. Together, these results provide a compelling motivation for feature-based KV indexing.

  2. The experimental evaluation covers multiple models and long-context benchmarks, with impressive results in both quality retention and decoding efficiency.

Critical weaknesses
  1. The paper does not provide a concrete GPU kernel design, making it difficult to assess the hardware efficiency of the proposed algorithm. PrismKV relies on overlapping feature buckets, dynamically sized candidate sets, and indexed access to non-contiguous KV entries. Section 4.1 states that PrismKV and Quest share an indexed sparse-attention kernel, but does not explain how these implementation challenges are addressed. In particular, it is unclear how the kernel maintains efficient memory access and handles variable candidate counts across KV groups without excessive padding or synchronization overhead.

  2. PrismKV introduces several interacting hyperparameters, including the dictionary size, key sparsity, bucket selection budget, and index-update interval, but their sensitivity is not evaluated. The existing ablations mainly examine random bucket membership and removal of Q–K matching, leaving it unclear how robust the reported quality and speedups are to these choices.

Questions for authors
  1. Could the authors elaborate on the GPU kernel design that enables efficient execution of PrismKV? Specifically, how are overlapping bucket indices merged and deduplicated, how are non-contiguous KV accesses organized, and how are variable candidate sizes across KV groups handled? Details on memory layout and workload scheduling, together with a latency breakdown of bucket selection, candidate construction, and attention, would help clarify the hardware efficiency of the approach.

  2. Could the authors clarify Quest's GQA adaptation? Its public accuracy implementation uses KV repetition, while selective KV access under GQA requires balancing head-specific retrieval with KV reuse. How are per-head selections aggregated, and how is a fair quality–latency comparison with PrismKV ensured?

  3. How sensitive are the results to the dictionary size, key sparsity, and bucket selection budget? Please report task quality, actual KV reads, and decoding latency for a few representative settings.

  4. Could the authors clarify how the token-reading budget is matched across methods, given that selecting B=16 buckets yields a variable number of unique tokens? Is matching performed per step or on average, and how are generated tokens and cache-eviction baselines handled in repeated-context tasks? Please report the baselines' actual historical-token reads alongside PrismKV's in Tables 2–5.

9 comments
AuthorsOpenReview ↗

Comment 1 of 5

Summary — Significance for Existing Checkpoints

The SAE training code, trained SAE encoder weight matrices, inference implementation, optimized kernels for NVIDIA H200 and PPU-ZW810, and experimental results are all publicly available in the anonymous PrismKV repository.

Changes in the revised manuscript. For W1/Q1/Q2, Appendix E explains the GPU kernel and reports H200 timings, including candidate merging and irregular KV access (Table 27) and continuous-batching throughput (Table 28). For W2/Q3, Appendices D.3–D.4 (Tables 23–25) cover hyperparameter sensitivity. For Q4, Appendix A.5 (Table 7) gives the budget-matching procedure and measured reads.

  • Summary: I find the broader significance of the resulting KV-selection method limited given recent progress in training models with sparse attention.
  • A: Recent advances in trained sparse attention address how to build models with sparsity. PrismKV addresses how to add selective KV access to existing checkpoints while keeping the backbone frozen. This gives models deployed with dense attention a path to long-context decoding speedups. The indexing principle applies to MHA and GQA and can be adapted to MLA through its query and key representations.
AuthorsOpenReview ↗

Comment 2 of 5

W1 / Q1 / Q2 (Part 1 of 2) — GPU Kernel, Quest Adaptation, and Single-Request Timing

  • W1 / Q1 / Q2 The paper does not provide a concrete GPU kernel design, making it difficult to assess the hardware efficiency of the proposed algorithm. PrismKV relies on overlapping feature buckets, dynamically sized candidate sets, and indexed access to non-contiguous KV entries. Section 4.1 states that PrismKV and Quest share an indexed sparse-attention kernel, but does not explain how these implementation challenges are addressed. In particular, it is unclear how the kernel maintains efficient memory access and handles variable candidate counts across KV groups without excessive padding or synchronization overhead.
  • Could the authors elaborate on the GPU kernel design that enables efficient execution of PrismKV? Specifically, how are overlapping bucket indices merged and deduplicated, how are non-contiguous KV accesses organized, and how are variable candidate sizes across KV groups handled? Details on memory layout and workload scheduling, together with a latency breakdown of bucket selection, candidate construction, and attention, would help clarify the hardware efficiency of the approach.
  • Could the authors clarify Quest's GQA adaptation? Its public accuracy implementation uses KV repetition, while selective KV access under GQA requires balancing head-specific retrieval with KV reuse. How are per-head selections aggregated, and how is a fair quality–latency comparison with PrismKV ensured?
  • A:

    The kernel code documentation covers NVIDIA H200 and PPU. PrismKV and Quest share the sparse attention kernel that runs after candidate token IDs have been selected. Each method uses its own selector to generate those candidates. The kernel uses each KV group's valid candidate count and reuses K/V loads within the group.

    1. Candidate construction. PrismKV merges the selected bucket bitmaps with bitwise OR, which deduplicates overlapping candidates, then compacts the result into sorted token IDs. Quest scores 16-token pages from key extrema and expands the selected pages into token IDs. The timing adapter applies softmax to each query head's page scores and then averages them within its KV group. The original 27B quality evaluation averages the raw scores within each group.

    2. Attention execution. The kernel reads K/V directly from their original positions using the valid IDs, with vectorized loads along the contiguous head dimension. Query heads in the same group reuse these loads. Each group computes over its own valid candidate count. Partial results are merged with softmax rescaling.

    3. H200 measurements. Both methods run Qwen3.8-27B BF16 on the same H200 and use CUDA Graph for decoding. Paired entries report Full FlashAttention → PrismKV.

      Single request (batch 1). Each request produces 4,096 tokens. Prefill produces the first token, and the remaining 4,095 steps are timed as decode. A 252K input plus 4K output fills the 256K window. Complete attention includes routing and sums the 16 standard attention layers.

Input lengthComplete attention (ms/step)Attention speedupWhole-model decoding (ms/step)Decode speedupOne-time index build (s)Decode speedup including index build
32K0.893 → 0.9480.94×21.027 → 21.0700.998×0.2930.995×
64K1.429 → 1.0231.40×21.619 → 21.1521.022×0.4451.017×
128K2.514 → 1.1782.13×22.516 → 21.3311.056×0.7241.047×
252K4.661 → 1.4643.18×24.441 → 21.5281.135×1.2771.119×

H200 single-request results

AuthorsOpenReview ↗

Comment 3 of 5

W1 / Q1 / Q2 (Part 2 of 2) — Continuous Batching and Kernel Stage Timings

Continuous batching. Both methods use the same requests, arrival times, and output caps. The long-output setting continues to a fixed cap and includes tokens after the first EOS. The 252K row is the long-output test at a 256K window.

Input lengthMaximum batch sizeWhole-model decode throughput (tokens/s)Decode speedupDecode speedup including index buildPeak GPU memory (GiB)
32K278.54 → 79.111.007×1.000×58.07 → 60.64
32K4122.74 → 124.571.015×1.006×62.43 → 67.70
32K8238.15 → 256.381.077×1.055×72.10 → 81.90
32K16286.99 → 314.721.097×1.066×92.16 → 110.44
64K275.09 → 78.991.052×1.040×61.68 → 66.60
64K4115.60 → 123.821.071×1.055×70.96 → 79.04
64K8214.55 → 252.371.176×1.138×88.25 → 104.26
128K269.53 → 78.001.122×1.099×69.82 → 78.56
128K4103.11 → 121.941.183×1.151×87.09 → 101.90
252K260.63 → 76.761.266×1.220×85.96 → 101.59

Attention latency comes from separately timed snapshots. For the near-256K short-output control, we average decode positions 1, 8, and 16 with both requests active.

H200 continuous-batching results

PrismKV kernel stage timings. The table uses separate attention snapshots. Each stage was timed independently in ms per batch step.

Input lengthSnapshot batch sizeBucket scoring and Top16 selectionCandidate merge and deduplicationIndexed attention
32K10.5430.0910.254
128K10.5360.1060.482
252K10.5370.1360.758
Near 256K20.7180.1861.105

In the near-256K batch-2 snapshot, the complete attention path takes 8.989 → 2.022 ms per batch step (4.45×). H200 routing and attention timings

Reviewer_8afpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an inference-time sparse attention method, called PrismKV. It works by training a sparse autoencoder (SAE) on attention input (queries and keys) and use its key activations to group tokens into overlapping buckets. Therefore, a first stage is needed for training the SAE itself (the LLM itself remains frozen). During decoding, the original query scores are used to compact bucket summaries, and attention is computed over the retrieved original KV entries. Experiments include RULER, LongBench, LongProc, and SCBench, alongside latency measurements. The results show near-full-attention quality on the 27B model and modest efficiency speedups.

Critical strengths

I really enjoy reading works that are able to transform interpretability insights into practical methods. The connection between shared sparse features and attention relevance is interesting. The overlapping feature buckets idea, although convoluted, seems to be a concrete way to turn this observation into an efficient retrieval mechanism. I also liked the analysis without QK matching supervision (see more below).

The downstream task results are strong, especially on the 27B model. PrismKV stays very close to full attention on RULER and LongBench while reading a small fraction of historical tokens. On LongProc and SCBench results are particularly stronger. The runtime evaluation also includes full-model decoding and a continued-generation experiment that accounts for index construction and periodic updates.

Critical weaknesses

With that said, I have found a few shortcomings. Some are easy to fix, while others require more work.

First, the experimental comparisons miss several closely related baselines. Squeezed Attention and RetrievalAttention are discussed in related work, but neither is evaluated. These comparisons seem particularly important because they also organize keys into an index for query-dependent retrieval. Methods such as Twilight, SeerAttention-R, and BLASST would also strengthen the evaluation as they are strong dynamic attention methos.

Second, I believe the very large LongProc gap deserves some explanation. PrismKV scores 67.3, while the strongest evaluated sparse baseline scores 29.6. How much of this comes from better retrieval, and how much comes from differences in cache retention and access to newly generated tokens?

Third, the method has several components, but their individual value is not fully established. In particular, removing QK matching barely changes results on RULER and LongBench. I am not convinced that these results establish the need for the additional matching objectives and training schedule. In the same spirit, random bucketing is a weak control here. Sparse random projections and clustering with multiple assignments would provide more informative comparisons, alongside further analysis of what the SAE features represent.

Finally, I think the evaluation does not fully show the quality-latency trade-off. I would rather like to see RULER results by task category and context length, rather than only an aggregate, as well as quality-latency curves for different bucket-selection budgets. The attention speedups are interesting, but full model gains are smaller, particularly on the hybrid 27B model for which only 16 of 64 layers are modified.

Questions for authors

Since queries are not encoded by the SAE during decoding, what advantage does its learned feature structure provide over sparse random projections or clustering with multiple assignments? Can you also show interpretable examples of what these features capture?

What explains the strong gains, particularly the large LongProc gap?

7 comments
AuthorsOpenReview ↗

Comment 1 of 5

W1 — Additional Baselines and Decode Latency

The SAE training code, trained SAE encoder weight matrices, inference implementation, optimized kernels for NVIDIA H200 and PPU-ZW810, and experimental results are all publicly available in the anonymous PrismKV repository.

Changes in the revised manuscript. For W1, Appendix A.6 (Table 8) adds LongBench baselines and 128K latency. For W3 and Q1, Appendices B.2 (Tables 16–17) and D.1–D.2 (Tables 20–22) add the matching-loss and SAE routing controls. For W4, Appendix A.7 (Tables 9–13) breaks down RULER by task and length, and Appendix D.4 (Figure 9 and Table 25) shows the quality–latency curve.

  • W1 First, the experimental comparisons miss several closely related baselines. Squeezed Attention and RetrievalAttention are discussed in related work, but neither is evaluated. These comparisons seem particularly important because they also organize keys into an index for query-dependent retrieval. Methods such as Twilight, SeerAttention-R, and BLASST would also strengthen the evaluation as they are strong dynamic attention methos.

All evaluations use Qwen3.8-27B. We report the macro average over all 16 LongBench tasks (3,750 examples). Each sparse baseline targets PrismKV's average number of unique old-history tokens read on the same example and retains access to the full generated tail. For latency, we use the same 128K prompt on PPU-ZW810 in BF16 at batch size 1 and report the median of three 128-step runs. Full Attention and PrismKV were timed together, and the five additional baselines were timed in separate runs.

MethodLongBench average ↑128K whole-model decode (ms/token) ↓
Full Attention54.03150.38
PrismKV53.96347.85
Twilight54.05474.25
RetrievalAttention54.0041,404.02
Squeezed Attention53.90191.81
BLASST53.84576.73
SeerAttention-R53.23961.25

PrismKV achieves nearly the same LongBench quality as Twilight and RetrievalAttention, trailing them by just 0.091 and 0.041 points, respectively, and scores above the other three added baselines. It has the lowest measured decode latency in the table, 21.9% lower than SeerAttention-R, the fastest of the five added baselines (47.85 vs. 61.25 ms/token). In the paired run, PrismKV is 1.053× as fast as Full Attention. Appendix A.6 (Table 8) summarizes decode latency. The timing link gives the detailed measurements. LongBench reference evaluation code · Timing protocol and results

AuthorsOpenReview ↗

Comment 2 of 5

W2 / Q2 — LongProc and Cache Retention

  • W2 / Q2 Second, I believe the very large LongProc gap deserves some explanation. PrismKV scores 67.3, while the strongest evaluated sparse baseline scores 29.6. How much of this comes from better retrieval, and how much comes from differences in cache retention and access to newly generated tokens?

What explains the strong gains, particularly the large LongProc gap?

  • A: We think the benefit mainly comes from the SAE bucketing and bucket overlapping. LongProc requires the model to shift its attention across tokens repeatedly during long outputs while continuing to attend to some fixed tokens. PrismKV's overlapping buckets support this access pattern. Fixed partitioning, as in Quest, can miss important information.

In HTML-to-TSV, for example, the model must read the current movie entry for each row it generates while following the conversion rules given at the beginning. SAE features can group keys from distant positions, and the same key can be retrieved through several buckets. Quest's page budget makes it harder to follow the shift from one entry to the next. Important keys spread across different pages each consume part of that budget, so key rows can be missed. The table below gives the fraction of old history read and row-level F1, which support this explanation.

MethodOld-history read fractionRow-level F1
PrismKV9.90%84.29
Quest9.85%14.85
AuthorsOpenReview ↗

Comment 3 of 5

W3 — Q–K Matching and Bucket Construction

  • W3 Third, the method has several components, but their individual value is not fully established. In particular, removing QK matching barely changes results on RULER and LongBench. I am not convinced that these results establish the need for the additional matching objectives and training schedule. In the same spirit, random bucketing is a weak control here. Sparse random projections and clustering with multiple assignments would provide more informative comparisons, alongside further analysis of what the SAE features represent.
  • A: We clarify the motivation for Q–K matching loss below:
  1. We observe that high-attention keys share SAE features with the query. Even without Q–K matching supervision, 31.86% of the top 2% highest-scoring keys in the Qwen 27B SAE share at least one feature with the query, compared with 4.96% of the remaining keys.
  2. This motivated us to strengthen Q–K matching in the SAE to help select tokens and accelerate attention. The observed relationship between feature sharing and high attention scores motivated the choice of a Q–K matching loss to strengthen that alignment and improve the method.
  3. Practical benefit. We retain Q–K matching as a training-only objective that reinforces the observed alignment and adds no operations to the decoding path. On Qwen3-8B, the checkpoint trained with matching scores 0.406 points higher on RULER and 0.051 points higher on LongBench. On Qwen3.8-27B, the two checkpoints score within 0.22 points on both benchmarks, with small differences in opposite directions.

We compare overlapping buckets built with SAE, multi-assignment K-means, and sparse random projection on all 3,250 Qwen3.8-27B RULER examples. The three methods share a per-step, per-layer, per-KV-group budget for unique old-history reads.

MethodMacro average
PrismKV (SAE)96.838
Multi-assignment K-means90.585
Sparse random projection89.071

See our response to Q1 below for an illustration of what the SAE represents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.