- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose Self-Indexing Attention, a sparse attention method that uses a shared retrieval mechanism for both prefill and decoding. The method applies a randomized Hadamard transform to post-RoPE queries and keys, then extracts their signs to obtain bipolar (+1/−1) representations. These representations enable inexpensive approximate scoring of candidate tokens, which are then selected through top-k retrieval. Attention is subsequently computed using the corresponding keys and values. The authors report up to 6.1× prefill and 10.3× decode attention-operator speedups on Qwen3.5-9B, translating to a 1.35× improvement in TTFT and 1.23× higher decode throughput. The authors also demonstrate compatibility with low-bit KV-cache quantization using TurboQuant.
Critical strengths
-
The proposed method is a clever and technically sound retrieval mechanism for efficiently identifying which keys and values to access for attention computation during both prefill and decoding.
-
The method achieves competitive accuracy on the LongBench and RULER benchmarks.
-
The systems evaluation demonstrates meaningful performance benefits, with several-fold reductions in prefill and decode attention-operator latency, translating to 1.35× faster TTFT and 1.23× higher decode throughput on Qwen3.5-9B. Notably, Qwen3.5-9B is a hybrid-attention model in which only 25% of layers use full attention. An additional systems evaluation on DeepSeekV4-Flash demonstrates a nontrivial 16.2% throughput improvement.
Critical weaknesses
-
Limited novelty relative to prior work. SI-KV (Self-Indexing KVCache), which the authors acknowledge as a close predecessor and include as an experimental baseline, also employs sign-based retrieval and separately quantizes key magnitudes to low-bit (2-bit) precision. Thus, the use of 1-bit sign representations for indexing, alongside low-bit KV-cache quantization, has already been established in prior work. The present method differs in its retrieval mechanism and extends sparse attention to both prefill and decoding, whereas SI-KV demonstrates sparse retrieval only during decoding. These are meaningful distinctions, but the central idea of sign-based self-indexing within a low-bit KV representation is not entirely new.
-
Insufficient systems evaluation of the proposed integration with low-bit KV compression. The experiments do not convincingly demonstrate the practical benefits of combining Self-Indexing Attention with a low-bit KV representation. The reported latency benchmarks appear to use 16-bit KV representations rather than the proposed low-bit compressed configuration. More generally, sparse attention and KV-cache compression are not inherently incompatible and can, in principle, be combined independently. The more specific contribution of this work is the reuse of bits from the compressed key representation as a retrieval index. However, the experiments do not clearly isolate or quantify the practical advantage of this integration relative to combining sparse attention and KV-cache compression separately.
-
Lack of direct systems comparisons with closely related methods. Although accuracy comparisons are provided against SI-KV and Quest, the paper does not directly compare their systems efficiency. Given the similarities between SI-KV and the proposed method, a systems-level comparison would be particularly informative. Without such comparisons, it is difficult to assess the extent to which the proposed innovations improve upon existing approaches in practice.
Questions for authors
Please see the weaknesses above for the main concerns.
-
Given that SI-KV is a close predecessor, would it be feasible to provide a direct systems-level efficiency comparison between SI-KV and the proposed method?
-
Could the systems evaluation be extended to include the combined setting emphasized by the paper: self-indexing sparse attention with low-bit KV-cache compression? The current latency experiments appear not to include low-bit KV compression, making it difficult to assess the performance benefits of the proposed integration.
5 comments
W3 Q1
W3 Q1 Direct systems comparison with SI-KV
Question Q1
- Given that SI-KV is a close predecessor, would it be feasible to provide a direct systems-level efficiency comparison between SI-KV and the proposed method?
Response
To answer this question, we added a full-model comparison with reimplemented SI-KV retrieval in the same model and backend. Single-token verification is similar, while 64-token verification is substantially faster.
Under the same timing scope, while 64-token verification falls from 68.33 to 34.37 ms (1.99×). The benefit of multiple-query grouping is therefore explicit.
Complete-target-model verification for one token
| Method | Single-token verification latency ms |
|---|---|
| SI-KV | 30.18 |
| Ours | 29.89 |
Complete-target-model verification for 64 tokens
| Method | 64-token verification latency ms | Speedup |
|---|---|---|
| SI-KV | 68.33 | - |
| Ours | 34.37 | 1.99× |
We compared the methods using Qwen3.5-9B/H800, 64K cached contexts, identical inputs and the same low-bit cache. Timing includes all model layers, the LM head and KV writes. SI-KV retrieval is independently reimplemented within the same backend.
The results show that symmetric grouped retrieval is most advantageous in multi-token verification, providing direct evidence that it can reduce target-model verification costs in speculative decoding.
Supplementary evidence
SI-KV selector and original paper quality evidence
The first table reports standalone selector latency. The second reproduces the LongBench averages from paper Table 1 at 5% attention density.
Standalone selector timing from 16K to 128K
| Context | Ours shared selector<br>ms | SI-KV selector<br>ms |
|---|---|---|
| 16K | 0.02955 | 0.04413 |
| 32K | 0.03901 | 0.06562 |
| 64K | 0.04905 | 0.09632 |
| 128K | 0.05806 | 0.14773 |
LongBench averages reported in paper Table 1
| Model | SI-KV | Ours | Ours + TQ1 |
|---|---|---|---|
| Qwen3.6-35B-A3B | 61.01 | 61.84 | 60.88 |
| Qwen3.5-9B | 56.04 | 56.10 | 56.03 |
W1 && W2 Q2
W2 Q2 Low-bit KV integration and index reuse benefits
Question Q2
Response
We added full-model results with low-bit KV, sparse prefill, and sparse decode enabled together, plus a separate ablation quantifying the storage benefit of index reuse. We report two configurations using different linear-attention backends for Qwen3.5-9B’s linear-attention layers: the Transformers PyTorch fallback and FLA. Our sparse-attention and KV-compression implementations remain unchanged. Timing includes retrieval, dequantization, and KV writes.
Joint sparse execution and compressed KV at B8: Transformers PyTorch linear-attention backend
With the Transformers PyTorch linear-attention backend, Ours achieves 1.075× faster TTFT and 1.213× higher decode throughput, while reducing KV allocation by 4.49×.
Joint sparse execution and compressed KV at B8: FLA linear-attention backend
With the FLA linear-attention backend, Ours achieves 1.309× faster TTFT and 1.049× higher decode throughput, retaining the same 4.49× KV reduction.
Both experiments use Qwen3.5-9B/H800, 64K inputs, B8, and 64 output tokens per request. The tables represent separate measurement cohorts; all methods within each table use the same linear-attention backend, environment, and timing protocol. Speedups are computed within each table.
Index-storage ablation with identical outputs
This independent paired ablation uses FLA for the linear-attention layers, holds the other components fixed, and changes only whether an additional index is stored. Index reuse saves 0.501 GiB (12.31%), with identical output tokens across all 24 paired measurements.
Together, these results demonstrate the full-model benefits of joint sparse execution and compressed KV under both linear-attention backends, while separately quantifying the storage savings from index reuse.