acceptodds
Under review as a conference paper at ICLR 2027

Self-Indexing Attention for Compression-Compatible Sparse Long-Context LLM Inference

Abstract

Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies for the two stages, preventing one retrieval representation from being reused throughout inference. We propose Self-Indexing Attention, a training-free framework built on a shared transform-domain sign-magnitude representation. The key signs provide a reusable token-level index for grouped prefill selection and decode retrieval, while the same representation remains compatible with external KV-cache compression without separate indexer metadata. This 1-bit index enables efficient retrieval through bitwise operations widely supported by modern accelerators. At 5% attention density, Self-Indexing Attention remains close to dense attention on LongBench and RULER and achieves up to 6.1× prefill and 10.3× decode attention-operator speedups. Experiments with TurboQuant and DeepSeekV4-Flash further demonstrate compatibility with low-bit KV-cache compression and pretrained sparse-attention indexers.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, information retireval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jnV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose Self-Indexing Attention, a sparse attention method that uses a shared retrieval mechanism for both prefill and decoding. The method applies a randomized Hadamard transform to post-RoPE queries and keys, then extracts their signs to obtain bipolar (+1/−1) representations. These representations enable inexpensive approximate scoring of candidate tokens, which are then selected through top-k retrieval. Attention is subsequently computed using the corresponding keys and values. The authors report up to 6.1× prefill and 10.3× decode attention-operator speedups on Qwen3.5-9B, translating to a 1.35× improvement in TTFT and 1.23× higher decode throughput. The authors also demonstrate compatibility with low-bit KV-cache quantization using TurboQuant.

Critical strengths
  1. The proposed method is a clever and technically sound retrieval mechanism for efficiently identifying which keys and values to access for attention computation during both prefill and decoding.

  2. The method achieves competitive accuracy on the LongBench and RULER benchmarks.

  3. The systems evaluation demonstrates meaningful performance benefits, with several-fold reductions in prefill and decode attention-operator latency, translating to 1.35× faster TTFT and 1.23× higher decode throughput on Qwen3.5-9B. Notably, Qwen3.5-9B is a hybrid-attention model in which only 25% of layers use full attention. An additional systems evaluation on DeepSeekV4-Flash demonstrates a nontrivial 16.2% throughput improvement.

Critical weaknesses
  1. Limited novelty relative to prior work. SI-KV (Self-Indexing KVCache), which the authors acknowledge as a close predecessor and include as an experimental baseline, also employs sign-based retrieval and separately quantizes key magnitudes to low-bit (2-bit) precision. Thus, the use of 1-bit sign representations for indexing, alongside low-bit KV-cache quantization, has already been established in prior work. The present method differs in its retrieval mechanism and extends sparse attention to both prefill and decoding, whereas SI-KV demonstrates sparse retrieval only during decoding. These are meaningful distinctions, but the central idea of sign-based self-indexing within a low-bit KV representation is not entirely new.

  2. Insufficient systems evaluation of the proposed integration with low-bit KV compression. The experiments do not convincingly demonstrate the practical benefits of combining Self-Indexing Attention with a low-bit KV representation. The reported latency benchmarks appear to use 16-bit KV representations rather than the proposed low-bit compressed configuration. More generally, sparse attention and KV-cache compression are not inherently incompatible and can, in principle, be combined independently. The more specific contribution of this work is the reuse of bits from the compressed key representation as a retrieval index. However, the experiments do not clearly isolate or quantify the practical advantage of this integration relative to combining sparse attention and KV-cache compression separately.

  3. Lack of direct systems comparisons with closely related methods. Although accuracy comparisons are provided against SI-KV and Quest, the paper does not directly compare their systems efficiency. Given the similarities between SI-KV and the proposed method, a systems-level comparison would be particularly informative. Without such comparisons, it is difficult to assess the extent to which the proposed innovations improve upon existing approaches in practice.

Questions for authors

Please see the weaknesses above for the main concerns.

  1. Given that SI-KV is a close predecessor, would it be feasible to provide a direct systems-level efficiency comparison between SI-KV and the proposed method?

  2. Could the systems evaluation be extended to include the combined setting emphasized by the paper: self-indexing sparse attention with low-bit KV-cache compression? The current latency experiments appear not to include low-bit KV compression, making it difficult to assess the performance benefits of the proposed integration.

5 comments
AuthorsOpenReview ↗

W1 && W2 Q2

W2 Q2 Low-bit KV integration and index reuse benefits

Question Q2

  1. Could the systems evaluation be extended to include the combined setting emphasized by the paper: self-indexing sparse attention with low-bit KV-cache compression? The current latency experiments appear not to include low-bit KV compression, making it difficult to assess the performance benefits of the proposed integration.

Response

We added full-model results with low-bit KV, sparse prefill, and sparse decode enabled together, plus a separate ablation quantifying the storage benefit of index reuse. We report two configurations using different linear-attention backends for Qwen3.5-9B’s linear-attention layers: the Transformers PyTorch fallback and FLA. Our sparse-attention and KV-compression implementations remain unchanged. Timing includes retrieval, dequantization, and KV writes.

Joint sparse execution and compressed KV at B8: Transformers PyTorch linear-attention backend

B8 metricNative BF16Ours
TTFT ms70370.8465475.22
Decode token/s223.479271.165
KV GiB16.0153.570

With the Transformers PyTorch linear-attention backend, Ours achieves 1.075× faster TTFT and 1.213× higher decode throughput, while reducing KV allocation by 4.49×.

Joint sparse execution and compressed KV at B8: FLA linear-attention backend

B8 metricNative BF16Ours
TTFT ms23822.7518205.88
Decode token/s251.10263.51
KV GiB16.0153.570

With the FLA linear-attention backend, Ours achieves 1.309× faster TTFT and 1.049× higher decode throughput, retaining the same 4.49× KV reduction.

Both experiments use Qwen3.5-9B/H800, 64K inputs, B8, and 64 output tokens per request. The tables represent separate measurement cohorts; all methods within each table use the same linear-attention backend, environment, and timing protocol. Speedups are computed within each table.

Index-storage ablation with identical outputs

BatchIndex storageKV + index GiB
8Reuse K sign bits3.570
8Separate index4.071

This independent paired ablation uses FLA for the linear-attention layers, holds the other components fixed, and changes only whether an additional index is stored. Index reuse saves 0.501 GiB (12.31%), with identical output tokens across all 24 paired measurements.

Together, these results demonstrate the full-model benefits of joint sparse execution and compressed KV under both linear-attention backends, while separately quantifying the storage savings from index reuse.

W3 Q1 Direct systems comparison with SI-KV

Question Q1

  1. Given that SI-KV is a close predecessor, would it be feasible to provide a direct systems-level efficiency comparison between SI-KV and the proposed method?

Response

To answer this question, we added a full-model comparison with reimplemented SI-KV retrieval in the same model and backend. Single-token verification is similar, while 64-token verification is substantially faster.

Under the same timing scope, while 64-token verification falls from 68.33 to 34.37 ms (1.99×). The benefit of multiple-query grouping is therefore explicit.

Complete-target-model verification for one token

MethodSingle-token verification latency ms
SI-KV30.18
Ours29.89

Complete-target-model verification for 64 tokens

Method64-token verification latency msSpeedup
SI-KV68.33-
Ours34.371.99×

We compared the methods using Qwen3.5-9B/H800, 64K cached contexts, identical inputs and the same low-bit cache. Timing includes all model layers, the LM head and KV writes. SI-KV retrieval is independently reimplemented within the same backend.

The results show that symmetric grouped retrieval is most advantageous in multi-token verification, providing direct evidence that it can reduce target-model verification costs in speculative decoding.

AuthorsOpenReview ↗

Supplementary evidence

SI-KV selector and original paper quality evidence

The first table reports standalone selector latency. The second reproduces the LongBench averages from paper Table 1 at 5% attention density.

Standalone selector timing from 16K to 128K

ContextOurs shared selector<br>msSI-KV selector<br>ms
16K0.029550.04413
32K0.039010.06562
64K0.049050.09632
128K0.058060.14773

LongBench averages reported in paper Table 1

ModelSI-KVOursOurs + TQ1
Qwen3.6-35B-A3B61.0161.8460.88
Qwen3.5-9B56.0456.1056.03
Reviewer_BYxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Self-Indexing Attention uses key signs as a shared index for sparse prefill and decoding. The sign bits also form part of the compressed keys, avoiding a separate retrieval index. Query grouping reduces token selection cost. The method requires no extra training. Experiments report scores close to dense attention at about 5% attention density, along with faster attention computation and compatibility with low-bit KV compression.

Critical strengths
  1. The shared sign index supports both prefill and decoding and reuses bits stored in compressed keys. This reduces the need for separate retrieval metadata.
  2. Tables 1 and 2 show scores close to dense attention at about 5% density. Figure 6 also reports gains in full-model inference, supporting the method’s practical value.
Critical weaknesses
  1. The joint efficiency of sparse attention and KV compression remains unclear. Figure 6 uses BF16 Q/K/V, while the low-bit experiments mainly report quality and storage. Timings with compressed KV, including dequantization, are missing. This leaves the practical benefit of combining sparsity and compression insufficiently tested.
  2. The implementation builds on DeepSeekV4’s sparse attention kernel, adds GQA support, and integrates a new selector. However, the Qwen speed tests compare only against dense attention. No controlled timing comparison uses the same attention backend with alternative selectors. It is therefore unclear how much speed gain comes from the proposed index and selector rather than the underlying sparse attention kernel.
Questions for authors
  1. What are the end-to-end speed gains with low-bit KV compression enabled, including dequantization overhead?
  2. With the same DeepSeek-derived attention backend, how does the proposed selector compare with existing selectors in latency and accuracy? How much speedup comes from the new selector?
2 comments
AuthorsOpenReview ↗

W1 Q1 && W2 Q2

Question Q1

  1. What are the end-to-end speed gains with low-bit KV compression enabled, including dequantization overhead?

Response

We evaluated the full model with sparse prefill, sparse decode, and low-bit KV enabled together under both the Transformers Torch and FLA backends. Timing includes retrieval, dequantization, and KV writes. Each table reports a separate measurement cohort, with the same environment and timing protocol used for all methods within that table.

Complete-model results at 64K: Transformers Torch backend

BatchConfigurationTTFT msDecode token/s
1Native BF168801.2535.142
1Ours sparse prefill + compressed decode8507.0934.310
8Native BF1670370.84223.479
8Ours sparse prefill + compressed decode65475.22271.165

At B8, Ours achieves 1.075× faster TTFT and 1.213× higher decode throughput than Native BF16, demonstrating full-model benefits with dequantization overhead included.

Complete-model results at 64K: FLA backend

BatchConfigurationTTFT msDecode token/s
1Native BF162959.9533.89
1Ours sparse prefill + compressed decode2466.6933.78
8Native BF1623822.75251.10
8Ours sparse prefill + compressed decode18205.88263.51

With FLA, Ours achieves 1.309× faster TTFT and 1.049× higher decode throughput at B8. Thus, the joint configuration also improves full-model performance with an optimized linear-attention backend.

KV allocation across all full-attention layers, shared by both configurations

BatchNative BF16 KV GiBOurs compressed KV GiB
12.0020.446
816.0153.570

Both configurations reduce KV allocation by 4.49×. These experiments use Qwen3.5-9B/H800, 64K inputs, and 64 output tokens per request; KV allocation covers all eight full-attention layers. All speedups are calculated against the matched baseline within each backend.

AuthorsOpenReview ↗

Supplementary evidence

S1 Packed decode with dense prefill

To isolate packed-decode throughput, this ablation holds prefill dense and uses 64K inputs with the same Torch backend.

Packed-decode ablation with dense prefill

BatchNative BF16 token/sOurs packed decode token/sSpeedup
135.1435.401.007×
4134.47138.861.033×
8223.48272.761.221×

S2 Separate attention-path performance benchmark

Next, we report independent decode attention-path timings that include selection and dequantization, covering multiple context lengths and batch sizes.

Separate decode attention-path benchmark

ContextBatchBF16 msPacked K2V4 msPath speedup
16K10.04450.04600.968×
16K40.10060.05061.988×
16K80.18300.06872.664×
64K10.10870.08101.341×
64K40.35650.12382.880×
64K80.68910.19993.447×
128K10.19390.10681.816×
128K40.68970.22433.075×
128K81.34970.34583.903×

S3 Additional retrieval diagnostics at 128K

At 128K, we additionally compare selector latency, retained attention mass and output error.

Retrieval diagnostic for decode at 128K

MethodFull selector msAttention massOutput NRMSE
Ours0.0580686.61%0.0836
Quest B320.0363869.76%0.1730
Quest B160.0529174.23%0.1338
Reviewer_UerwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Indexing Attention, a training-free framework that reuses transformed key signs as a shared retrieval index for sparse prefill and decoding. It combines sign-based token selection, query grouping, and low-bit KV-cache compression, achieving near-dense accuracy with practical inference speedups.

Critical strengths

S1. Unified retrieval and compression design. The method reuses one-bit key signs for both prefill and decode without maintaining a separate retrieval index in the compressed representation. This provides a practical way to combine sparse attention with compatible KV-cache compression methods.

S2. High accuracy with high sparsity and high compression. With 5% attention density, it is still able to approach the performance of dense attention for LongBench and RULER. Under the same logical KV-cache memory budget, the Ours+TQ1 configuration is also able to obtain accuracy comparable to TQ2.

S3. Overall efficiency. The measured speedups reach up to 6.1× and 10.3× for prefill and decode respectively, when considering transformation and selection overhead. In addition to this, the paper achieves 1.35× TTFT acceleration and 1.23× steady-decode throughput improvement, and shows deployment results on DeepSeekV4-Flash.

Critical weaknesses

W1. Incomplete evaluation for end-to-end sparse-attention and compression system. Though it presents the accuracy of Self-Indexing Attention with TurboQuant, the shown full-model speedups are achieved without low-bit KV compression. In DeepSeekV4-Flash, it only replaces the native FP8 indexer cache and leaves the original KV cache untouched. Consequently, the end-to-end efficiency and memory benefits of the fully integrated system remain unclear.

W2. Limited comparison with unified sparse-attention baselines. The primary baselines, e.g., AnchorAttention, Quest and SI-KV, focus on different inference stages, while the proposed method supports both prefill and decode. Although the paper explains the difficulties of comparing with existing full-stage methods, the current experiments do not directly establish the advantage of sharing one index over alternative combined prefill-decode approaches.

W3. Limited evidence for consistent task-level accuracy at longer contexts. The 128K LongBench Pro experiment shows nearly unchanged overall accuracy, while some individual tasks, e.g., consistency checking and dialogue memory, show noticeable drops (Figure 8). Considering the single run and limited cases in each task in this evaluation, further evidence is desired to demonstrate whether such difference is general.

Questions for authors

What are the end-to-end speedup and memory savings when sparse prefill, sparse decode, and low-bit KV compression are enabled together?

Can the authors compare against a combined prefill-decode baseline under matched sparsity and memory budgets?

Are the task-level accuracy differences at 128K consistent across repeated runs or larger evaluation sets?

4 comments
AuthorsOpenReview ↗

W1 Q1 && W2 Q2 && W3 Q3

W1 Q1 Joint sparse attention and KV compression

Question Q1

What are the end-to-end speedup and memory savings when sparse prefill, sparse decode, and low-bit KV compression are enabled together?

Response

To answer this question directly, we evaluated the full model with sparse prefill, sparse decode and low-bit KV enabled together. The results show that retrieval and compressed KV can work together in one system.

At B8, we obtain a 1.075× TTFT speedup and decode throughput of 1.213× the BF16 baseline. KV allocation falls from 16.015 to 3.570 GiB, a 4.49× reduction.

Complete-model results for the joint configuration at 64K

BatchConfigurationTTFT msDecode token/sKV GiB
1Native BF168801.2535.1422.002
1Ours sparse prefill + packed decode8507.0934.3100.446
8Native BF1670370.84223.47916.015
8Ours sparse prefill + packed decode65475.22271.1653.570

These measurements use Qwen3.5-9B/H800 under the same Torch backend; timing includes retrieval, dequantization and KV writes.

Under this common timing scope, the B8 results improve request efficiency while reducing KV storage, showing the joint benefit of sparse execution and low-bit compression across both stages.

Question Q2

Can the authors compare against a combined prefill-decode baseline under matched sparsity and memory budgets?

Response

For this comparison, we added an AnchorAttention-prefill + Quest-decode combined baseline. It uses the same TQ K2/V4 low-bit KV format, matched sparsity and a common KV-plus-index capacity limit, with a quality comparison and an index-reuse ablation.

The results show that at B8, Ours uses 21.92% less KV-plus-index storage than the Anchor + Quest baseline, with a 1.157× TTFT speedup and similar decode throughput. The detailed comparison follows.

Complete prefill and decode composition at 64K

BatchMethodTTFT msDecode token/sKV + index GiB
1Native BF162959.9533.892.002
1Ours + K2/V42466.6933.780.446
1Anchor + Quest + K2/V42830.6034.080.571
8Native BF1623822.75251.1016.015
8Ours + K2/V418205.88263.513.570
8Anchor + Quest + K2/V421058.51262.844.572

Paired quality control on 65 HotpotQA questions at 16K

MethodOfficial QA F1 %
Native BF1663.55
Ours + K2/V461.87
Anchor + Quest + K2/V436.53

Independent index-storage ablation

BatchIndex storageKV + index GiB
8Reuse K sign bits3.570
8Separate index4.071

The combined-baseline comparison uses Qwen3.5-9B/H800, 64K inputs, a common backend and the same low-bit format. We match causal computation and the KV-plus-index capacity limit and report actual allocation. The quality comparison uses HotpotQA questions with official QA F1.

Together, these results show that the unified index reduces storage while maintaining decode speed close to the Anchor + Quest baseline. The separate ablation further confirms the direct storage benefit of index reuse.

Question Q3

Are the task-level accuracy differences at 128K consistent across repeated runs or larger evaluation sets?

Response

Our repeated LongBench Pro evaluations reproduced the qualitative trends reported in the paper. To extend this evidence, we evaluated 211 new LongMemEval-S questions with approximately 107K–117K-token histories.

LongMemEval-S evaluation

New taskNNative %Ours %Delta pp
Knowledge update7873.0878.21+5.13
Multi-session memory13363.1661.15-2.01

The evaluation uses an independent local Qwen3.5-9B judge and the official prompt; results are reported as local-judge accuracy.

Overall, aggregate accuracy is unchanged, while knowledge-update performance is better.

AuthorsOpenReview ↗

Supplementary evidence

S1 Additional native prefill-selector configuration

As an additional check, this 128K selector comparison uses an approximately 5% causal budget. Ours has lower selector latency than Anchor Attention.

Additional native prefill-selector configuration at 128K

TargetMethodFull selector ms
5%Ours87.47
5%Anchor121.83

S2 Independent TurboQuant and Quest composition

We also compared Ours with an independently reimplemented TurboQuant + Quest baseline using 64K inputs, dense prefill and a common Torch backend. The tables report full-model throughput, independent attention-module timing and actual cache allocation, including metadata, padding and indices.

Dense-prefill comparison at B8 under the Torch backend

MetricOursTQ + Quest
Model decode token/s272.76266.92
Additional index GiB0.0001.002
KV + index storage GiB3.5704.383

Complete-model decode comparison after dense prefill

BatchOurs decode token/sTQ + Quest decode token/sThroughput ratio
135.4034.781.018×
4138.86134.941.029×
8272.76266.921.022×

Separate attention-module measurement and cache allocation

B8 metricOursTQ + Quest
Complete attention module ms0.32901.1982
KV including metadata and fixed parameters GiB3.5703.381
Additional retrieval index GiB0.0001.002
Total KV storage GiB3.5704.383

Thanks for the extra experiments. I have one follow-up about the TTFT numbers. At 64K and batch size 8, the Native BF16 baseline takes about 70.4 seconds in the Q1 table and 23.8 seconds in the Q2 table. Were the runs configured or timed differently? Could you clarify what accounts for the gap?

Thank you for pointing this out. We apologize for not clearly specifying the backend configurations in the original tables.

Q1 and Q2 were originally evaluated in separate experimental environments. Q1 used the Transformers PyTorch fallback for the linear-attention layers, whereas Q2 used FLA. Our sparse-attention and KV-compression implementations remained unchanged, but the different linear-attention backends substantially affected full-model prefill time, accounting for the TTFT gap.

Within each original table, all methods used the same environment and backend, so the comparisons were internally matched. Both sets of results support the effectiveness of our method; the backend difference changes the absolute timings but does not alter the conclusions of the within-table comparisons.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We summarize the additional experimental evidence and further clarify the scope of our contribution.

  • Joint evaluation of sparse attention and low-bit KV compression. We added full-model experiments enabling sparse prefill, sparse decode, and KV compression together, with timing including retrieval, dequantization, and KV writes. On Qwen3.5-9B/H800 at 64K context and batch size 8, using the FLA backend, our method improves TTFT and decode throughput over the matched BF16 baseline while reducing KV allocation across the full-attention layers by 4.49×.

  • Isolating the benefit of index reuse. In a controlled ablation that changes only whether a separate index is stored, index reuse saves 0.501 GiB (12.31%) of storage. We also added a comparison with an AnchorAttention-prefill + Quest-decode baseline under matched sparsity and a common memory-capacity limit to evaluate our method against a separately composed alternative.

  • Contribution relative to SI-KV. We acknowledge that SI-KV established sign-based self-indexing. Building on this idea, our contribution uses a shared prefill–decode index and symmetric grouped retrieval to support multi-token verification, while accommodating compatible external KV compressors. The additional experiments demonstrate that our method is more efficient and effective than SI-KV.

Reviewer jnV1 acknowledged that the additional experiments strengthen the empirical evaluation, while retaining a concern about novelty. We hope that the clarified contribution boundary and controlled comparisons help assess the practical value and technical contributions of these extensions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.