- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PrismKV, a selective KV access method for long-context autoregressive decoding. The authors start from the observation that historically important tokens for attention, while scattered across the original sequence, tend to share a small set of features with the current query in high-dimensional sparse representations. To exploit this structure, PrismKV trains a shared query-key sparse autoencoder per attention layer and per KV group, assigning each historical key to multiple overlapping feature buckets. Each bucket stores token indices together with the count, mean, and per-dimension variance of the original keys. During decoding, the system estimates the attention quality of each bucket using the current dense query and bucket statistics, selects the highest-scoring buckets, merges and deduplicates tokens from the selected buckets, and then computes attention over the original KV entries.
Critical strengths
- An insightful overlapping-feature indexing design. Unlike methods that assign each key to a single cluster or contiguous position block, PrismKV allows one token to belong to multiple feature buckets simultaneously. This enables the representation that “the same token can be relevant to different queries via distinct features”, which aligns well with the compositional nature of attention relevance and constitutes the paper’s most valuable technical contribution.
- Strong quality results with indexing overhead taken into account. PrismKV achieves competitive performance across multiple benchmarks for long-context understanding, continual generation, and context reuse. The final attention computation uses raw Q/K/V, preventing SAE reconstruction error from directly corrupting attention scores. The paper also reports indexing construction, periodic update, attention-path, and end-to-end latency, rather than only idealized candidate-retrieval cost.
Critical weaknesses
-
There is a notable disconnect between the paper’s core mechanistic claim and the actual inference algorithm. The paper attributes performance gains to “shared sparse features” between queries and keys, yet the bucket scoring formula during inference uses the original dense query along with the mean and variance of keys within each bucket, without explicitly using the SAE encoding of the query . The method can therefore alternatively be interpreted as “using an SAE to produce overlapping key clusters, then routing over cluster summaries with the dense query”. Existing experiments do not establish that shared query–key features themselves are a necessary ingredient for successful retrieval.
-
The shared-feature analysis carries a risk of circular reasoning, and the associated ablations are insufficient to establish causality. The SAE matching loss trains sparse query/key encodings directly using full-attention outcomes. Thus the observation that high-attention keys tend to share features with the query under the supervised SAE is, to some extent, an expected consequence of the training objective. The paper includes an analysis without matching supervision, but this analysis covers only 14 contexts and a subset of layers/groups. More importantly, removing matching supervision degrades RULER and LongBench scores by merely 0.405 and 0.051 points, suggesting that this supervision and its corresponding “shared-feature” explanation may not be the primary source of performance gains.
-
Critical ablations and quality–efficiency Pareto analysis supporting the design choices are missing. The paper does not systematically study key hyperparameters such as dictionary size, activation count per key/query, number of selected buckets, and unindexed tail length. It also does not ablate the three scoring terms — bucket count, mean, and variance — separately. The fixed default configuration cannot reveal whether performance is sensitive to these choices, nor whether PrismKV consistently outperforms simpler clustering, LSH, or learned routing methods under equal actual token-read budgets.
-
Baseline coverage and budget fairness are inadequate. The paper states that all methods use the same token-reading budget, yet it primarily reports the realized read fraction for PrismKV. Since bucket sizes and deduplicated candidate counts selected by PrismKV vary dynamically, it remains unclear how budgets are strictly aligned per sample, per layer, and per KV group. Experiments do not include direct comparisons against closely related methods such as RetrievalAttention, Squeezed Attention, or DeepSeek Sparse Attention, and latency comparisons are largely limited to the authors’ adapted Quest implementation. This is insufficient to support a claim of comprehensive advantages over state-of-the-art retrieval-based attention methods.
-
System evaluation and reproducibility do not yet meet top-conference standards. Latency is measured only on the PPU-ZW810 with batch size near 1 and no dynamic/continuous batching. Results on mainstream GPUs, serving throughput, concurrent requests, tail latency, and real memory overhead are absent. The reported SAE training cost also excludes backbone Q/K trajectory collection. Details on training data sources, baseline configurations, and train–test separation are insufficient. Notably, for the 27B model, the 2.78× speedup on the attention path translates into only 1.16× end-to-end speedup, meaning the practical system-level benefit remains modest.
Questions for authors
-
Is the sparse query encoding computed or used during inference? If not, how do the authors demonstrate that performance arises from query–key shared features, rather than overlapping key clusters produced by the SAE together with dense bucket-summary routing?
-
Could the authors add the following critical controls: bucket selection directly using , vanilla k-means, LSH, a learned router without SAE, and ablated variants that use only bucket mean or bucket size? How do these alternatives perform under the same number of candidate tokens?
-
Removing query–key matching supervision yields only a tiny downstream performance drop. How do the authors explain the observation that shared-feature coverage decreases substantially while task performance remains nearly unchanged? Is the matching loss truly necessary?
-
How exactly is the “identical token-reading budget” enforced? Do all baselines read exactly the same number of unique historical tokens per sample, per layer, and per KV group? Could the authors report the realized read fraction for every method together with full quality–latency Pareto curves?
-
Could the authors supplement end-to-end results on mainstream NVIDIA GPUs, including continuous batching, multi-request throughput, peak memory usage, index storage overhead, candidate merging, and the cost of irregular KV gather? In addition, what is the full offline training and data-preparation cost when Q/K trajectory collection is included?
9 comments
Comment 2 of 7
W2 / Q3 — Feature Sharing With and Without Q–K Matching
- W2 / Q3
The shared-feature analysis carries a risk of circular reasoning, and the associated ablations are insufficient to establish causality. The SAE matching loss trains sparse query/key encodings directly using full-attention outcomes. Thus the observation that high-attention keys tend to share features with the query under the supervised SAE is, to some extent, an expected consequence of the training objective. The paper includes an analysis without matching supervision, but this analysis covers only 14 contexts and a subset of layers/groups. More importantly, removing matching supervision degrades RULER and LongBench scores by merely 0.405 and 0.051 points, suggesting that this supervision and its corresponding “shared-feature” explanation may not be the primary source of performance gains.
Removing query–key matching supervision yields only a tiny downstream performance drop. How do the authors explain the observation that shared-feature coverage decreases substantially while task performance remains nearly unchanged? Is the matching loss truly necessary?
- A: The circular-reasoning objection conflates the effect of matching supervision with a relationship we observed independently. We clarify the motivation for adding a q-k matching term as below:
- We observe that high-attention keys share SAE features with the query. Even without Q–K matching supervision, 31.86% of the top 2% highest-scoring keys in the Qwen 27B SAE share at least one feature with the query, compared with 4.96% of the remaining keys.
- The observation directly motivated our choice of a Q–K matching loss. We first observed that high-attention keys were more likely to share SAE features with the query. We chose this particular loss because it directly reinforces the alignment we had already observed, helping the SAE organize tokens for retrieval.
- Practical benefit. We retain Q–K matching as a training-only objective that reinforces the observed alignment and adds no operations to the decoding path. On Qwen3-8B, the checkpoint trained with matching scores 0.406 points higher on RULER and 0.051 points higher on LongBench. On Qwen3.8-27B, the two checkpoints score within 0.22 points on both benchmarks, with small differences in opposite directions.
The SAE without Q–K matching still shows enrichment in shared features. Using the seven task families and decode-step sampling from the original experiment, we extend the 27B analysis to 200 new contexts, with 100 each at 32K and 64K. This covers all 16 standard attention layers and 64 KV groups. Both SAE checkpoints use the same Full Attention trajectories.
| SAE checkpoint | Historical-key split | Top-scoring keys sharing ≥1 feature with current Q | Remaining keys sharing ≥1 feature with current Q |
|---|---|---|---|
| Without Q–K matching | Top 2% / rest | 31.86% | 4.96% |
| Without Q–K matching | Top 1% / rest | 37.66% | 5.17% |
| With Q–K matching | Top 2% / rest | 48.26% | 4.43% |
Without matching, the top-scoring keys share more features at both cutoffs in all 200 contexts. This supports shared features as a motivation for overlapping bucketing. The checkpoints have different training histories, so their difference cannot be attributed to the matching loss alone. 200-context results and 32K/64K heatmaps
Comment 3 of 7
W3 / Q2 (Part 1 of 2) — Hyperparameters and Direct Sparse-Q Routing
- W3 / Q2
Critical ablations and quality–efficiency Pareto analysis supporting the design choices are missing. The paper does not systematically study key hyperparameters such as dictionary size, activation count per key/query, number of selected buckets, and unindexed tail length. It also does not ablate the three scoring terms — bucket count, mean, and variance — separately. The fixed default configuration cannot reveal whether performance is sensitive to these choices, nor whether PrismKV consistently outperforms simpler clustering, LSH, or learned routing methods under equal actual token-read budgets.
Could the authors add the following critical controls: bucket selection directly using , vanilla k-means, LSH, a learned router without SAE, and ablated variants that use only bucket mean or bucket size? How do these alternatives perform under the same number of candidate tokens?
-
A: We added equal-read ablations, bucket construction controls, and a quality–latency curve for Qwen3.8-27B.
- Hyperparameter sweep.
Qwen3.8-27B. Quality and old-history reads come from all 3,750 LongBench examples. Graph decode is measured separately at 128K, batch size 1.
| Sweep | Setting | LongBench | Old reads | Graph decode (ms/step) |
|---|---|---|---|---|
| Dictionary width | 1,000 | 53.851 | 7.991% | 54.360 |
| Dictionary width | 2,000 | 53.814 | 7.991% | 54.343 |
| Dictionary width | 3,000 | 53.925 | 7.991% | 54.763 |
| Dictionary width | 4,000 | 53.953 | 7.991% | 55.092 |
| Fixed memberships | 1 per key | 54.004 | 3.353% | 47.064 |
| Fixed memberships | 2 per key | 53.977 | 4.704% | 47.251 |
| Fixed memberships | 4 per key | 54.073 | 6.568% | 47.545 |
| Fixed memberships | 8 per key | 54.141 | 9.150% | 48.188 |
| Fixed memberships | 16 per key | 54.123 | 13.022% | 48.981 |
| Bucket count | Full Flash | 54.034 | 100% | 50.756 |
| Bucket count | Top1 | 49.633 | 0.80% | 46.541 |
| Bucket count | Top2 | 51.781 | 1.44% | 46.591 |
| Bucket count | Top4 | 52.959 | 2.58% | 47.024 |
| Bucket count | Top8 | 53.527 | 4.71% | 47.354 |
| Bucket count | Top16 | 53.996 | 8.67% | 48.062 |
Qwen3-8B. Index updates are measured at 64K on a fixed Full continuation. Reads include the unindexed tail, and latency includes amortized updates.
| Update interval (tokens) | Reads (tokens/KV group) | Conditional PPL | Eager decode incl. updates (ms/step) |
|---|---|---|---|
| No update | 7,513.5 | 1.2750 | 22.131 |
| 512 | 5,601.9 | 1.2836 | 22.539 |
| 2,048 | 6,243.2 | 1.2739 | 22.291 |
| 4,096 | 7,161.1 | 1.3065 | 22.311 |
- Dictionary width. At 7.991% reads, LongBench scores span 0.139 points. Mean index build time rises from 0.083 to 0.144 s.
- Key memberships. With Top16 fixed, more memberships increase reads and decode latency. Scores vary by only 0.164 points, with eight memberships scoring highest.
- Selected buckets. This sweep shows the largest quality change. Top8 reads 3.96 percentage points less than Top16 and scores 0.469 points lower.
- Index update interval. Updating every 512 tokens cuts reads by about 25% versus no updates, while amortized decode rises from 22.131 to 22.539 ms/step.
Hyperparameter sweeps and timings
- Direct bucket selection using . We use up to 16 positive SAE features per query head to rank their corresponding buckets, aggregating normalized activation weights across the six query heads in each KV group. Both arms share the SAE, key memberships, attention kernel, and 8% cap on unique old-history reads. The default router scores bucket summaries with dense queries.
| Benchmark | Bucket summaries | Direct |
|---|---|---|
| LongBench | 54.057 | 53.550 |
| RULER | 96.731 | 95.901 |
Comment 1 of 7
W1 / Q1 — Dense Query Routing and SAE Bucket Memberships
The SAE training code, trained SAE encoder weight matrices, inference implementation, optimized kernels for NVIDIA H200 and PPU-ZW810, and experimental results are all publicly available in the anonymous PrismKV repository.
Changes in the revised manuscript. Appendix D covers direct sparse-Q routing (W1/Q1, Table 20) and the design ablations (W3/Q2, Tables 21–25). Appendix B.2 (Tables 16–17) covers feature sharing and the matching-loss comparison (W2/Q3). Appendices A.5–A.6 and D.4 give the read-budget, baseline, and quality–latency comparisons (W4/Q4). Appendices E (Tables 26–29) and C.2 (Table 19) report H200 serving results and offline costs including Q/K capture (W5/Q5).
On all 3,250 Qwen3.8-27B RULER examples, SAE bucket assignments also outperform overlapping buckets built with multi-assignment K-means or sparse random projection under the same per-step read budget. See W3/Q2 below for the protocol and results by length.