acceptodds
Under review as a conference paper at ICLR 2027

CISA: Retrofitting Dense Attention Models with Chunk-Indexed Sparse Attention

Abstract

As context becomes grows, LLM inference becomes increasingly expensive because attention repeatedly reads a growing key-value (KV) history. We introduce Chunk-Indexed Sparse Attention (CISA), a method for retrofitting pretrained dense attention models without gradient updates. CISA selects fixed-size chunks of the KV history using compact summaries derived from the model's existing keys, avoiding token-level indexing and a separate full-length index cache. For indexing, CISA uses a single query per KV group and shares the selected chunks across the query in that group. During prefill, we reuse chunk selections across adjacent queries to reduce indexing cost and improve KV reuse and GPU utilization. We call this technique *QShare*. We evaluate CISA on the NVIDIA Nemotron 3 Nano and Ultra models, as well as on Inkling-Small. In one-layer BF16 benchmarks at batch size 32 using Ultra's per-device attention dimensions, CISA accelerates core attention by for prefill queries near the end of a 1M-token prompt and during decode. Including chunk scoring, selection, and all other measured GPU work, the corresponding speedups are and . On Ultra NVFP4, at , CISA keeps aggregate scores within 1.5 percentage points of dense attention on MRCR, LongBench v2, and GraphWalks. RULER sweeps on Nano and Ultra BF16 up to one million tokens show scores within 2.05 percentage points of dense attention with 8–16 fewer KV reads. When serving million-token prompts with 32 concurrent requests and 4K output tokens per request, CISA achieves the end-to-end throughput of dense attention using Nemotron 3 Ultra NVFP4 on four NVIDIA B300 GPUs. Selecting fewer chunks increases throughput to compared with dense attention; in separate GraphWalks evaluations, the smaller budget widens the aggregate score gap from 1.49 to 3.59 percentage points. These results expose a practical quality–speed tradeoff without requiring sparse training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention, sparse attention, efficiency

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_S2x9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework that can turn dense attention models into chunk-index sparse attention training. Unlike DSA, this method uses a single query per KV-group, which reduces the complexity of the indexer. Moreover, the paper presents QShare, which makes consecutive groups of tokens in prefill share the same chunks so that the fetched chunks can be reused by multiple tokens.

Critical strengths
  • The paper is well-written, easy to follow, its contributions are clear. The authors are upfront about the method's limitations, novelty claims etc., which I really appreciate.

  • This paper proposes a practical solution to an important problem. The current sparse attention mechanisms are not efficient: DSA still requires quadratic complexity for the indexer. The others require pretraining or fine-tuning. This paper's proposal can be simply applied to the existing methods and would reduce the attention's complexity.

  • Evaluation is thorough, covers a sufficient range of models and benchmarks. I especially liked the speedup adjusted for the change in generation length, which is often overlooked or hidden in other papers.

Critical weaknesses
  • The evaluation of QShare for prefill and chunk-index sparse attention for decode is convoluted. If I'm in a scenario where prefill time is not significant, I might want to disable QShare and focus on decode. However, there is no evaluation that shows which one of these two changes contributes most and how much to the degradation of model quality.
  • Figure 1 could be improved in terms of clarity.
  • A table that shows the complexity in terms of compute and I/O of the proposed method versus prior methods as a function of sparsity ratio, model dims etc would make it clear to understand theoretical gains.
Questions for authors

See the weaknesses above.

4 comments

We thank the reviewer for the positive assessment of the paper.

1. Separating chunk indexing from positional query sharing

If I'm in a scenario where prefill time is not significant, I might want to disable QShare and focus on decode. However, there is no evaluation that shows which one of these two changes contributes most and how much to the degradation of model quality.

We addressed this option in Appendix B.2 and have now reorganized it to distinguish chunk-indexed sparse attention from the additional effect of positional query sharing. We have also added additional ablations shown below, using the Ultra NVFP4 model and benchmark subsets.

BenchmarkKK (chunks)DenseCISA, G=1G=1CISA, G=4G=4
RULER-1M4,09694.5393.7693.76
AA-LCR1,02464.1964.0064.56
MRCR2004,0969.219.189.99
GraphWalks-2002,04824.2217.5716.99

To examine QShare's quality effect across context lengths, we additionally report Nano BF16 RULER ablations, varying only positional sharing within each pair.

ContextKK (chunks)Nano, G=1G=1Nano, G=4G=4
128K1,02493.4592.79
256K2,04891.2390.60
512K4,09689.6189.24
1M4,09687.0186.22

Across the four Nano comparisons, enabling QShare decreases accuracy by up to 0.79 percentage points, while the Ultra NVFP4 results show no uniform decrease. As shown in Appendix B.2, Table 9(b) we report 18,063 tokens/s without QShare (G=1), compared with 33,543 tokens/s with G=4, both at K=4,096.

2. Figure 1

Figure 1 could be improved in terms of clarity.

Thank you for the suggestion. We have improved the clarity as in this revised Figure 1.

3. Compute and I/O complexity

A table that shows the complexity in terms of compute and I/O of the proposed method versus prior methods as a function of sparsity ratio, model dims etc would make it clear to understand theoretical gains.

The table below gives theoretical costs per query position and complete attention layer, counting two FLOPs per multiply-add. We use:

  • NN: visible history length in tokens.
  • SS and α=S/N\alpha=S/N: retained-token budget and fraction; CISA counts remote tokens here.
  • HqH_q: total number of query heads in the layer.
  • LL: total number of KV groups in the layer.
  • dd: attention head width; for absorbed MLA, the latent width.
  • dId_I: method-specific index dimension.
  • hIh_I: number of DSA index query heads.
  • CC: CISA chunk size in tokens.
  • GG: number of adjacent queries sharing a selection; single-token decode uses G=1G=1.
  • A=4HqNdA=4H_qNd: dense-attention compute reference, in FLOPs.
  • V=2LNdV=2LNd: reference payload for separate K and V tensors, in elements.
  • J=NdIJ=Nd_I: payload of one full token-level index-key cache, in elements.

For clarity, we omit MLA's additional RoPE dimensions. DSA's absorbed MLA uses one shared latent group (L=1L=1); the latent serves both K and V, giving a main-cache payload of V/2V/2.

MethodAttention FLOPsScoring FLOPsModeled reads (elements)
Dense GQAAA00VV
DSA (MLA)αA\alpha A2hIJ2h_IJαV/2+J\alpha V/2+J
MSAαA\alpha A2LJ2LJαV+J\alpha V+J
CISA, G=1G=1αA\alpha A2LJ/C2LJ/CαV+LJ/C\alpha V+LJ/C
CISA + QShareαA\alpha A2LJ/(CG)2LJ/(CG)αV+LJ/(CG)\alpha V+LJ/(CG)

DSA aggregates multiple index query heads against one shared index-key cache. MSA uses one index query per KV group with a shared index-key cache; it scores all token-level keys before max-pooling into blocks. CISA instead scores compressed chunk representatives. At equal KV-group counts and index dimensions, CISA reduces MSA's scoring arithmetic by CGCG and modeled index-key reads by CG/LCG/L; both reductions are CGCG with one KV group.

All three sparse methods retain quadratic index-scoring work over a complete prefill for fixed architecture parameters. CISA reduces this work to O(LN2dI/(CG))O(LN^2d_I/(CG)) through chunk compression and query sharing, improving the constant factor without changing the asymptotic order.

AuthorsOpenReview ↗

reply part 2

To illustrate the theoretical costs at N=220N=2^{20} (1M context), we consider each model's complete attention layer executing on one GPU. We use the native (Hq,L,d)(H_q,L,d) configurations: (128,1,512)(128,1,512) for DeepSeek-V3.2's absorbed MLA, (64,4,128)(64,4,128) for MiniMax-M3, and (64,2,128)(64,2,128) for Nemotron Ultra. All use dI=128d_I=128; DSA uses hI=64h_I=64, and CISA uses C=16C=16.

The following costs are per query position and layer, using each method's retained-token budget SS.

Model / methodSSAttention GFLOPsScoring GFLOPsReads (M elements)
DeepSeek-V3.2 / DSA2,0480.536917.1799135.27
MiniMax-M3 / MSA2,0480.06711.0737136.31
Ultra / CISA, G=1G=165,5362.14750.033650.33
Ultra / CISA, G=4G=465,5362.14750.008437.75

For these configurations, CISA reduces index-scoring arithmetic relative to MSA by 32× without QShare and 128× with G=4G=4. Modeled index-key reads decrease by 8× and 32×, respectively, while CISA retains a larger attention budget. We note that as context length increases, scoring becomes the bottleneck for end-to-end latency which is why we designed CISA around optimizing the scoring.

I thank the authors for their response. I can see that Table 10 shows a comparison of response quality for chunk-indexing by disabling QShare (G=1). You claim that "while the Ultra NVFP4 results show no uniform decrease." but the quality reduces from 24.22 (Dense) to 17.57 (G=1) for GraphWalks-200 benchmark. You also claim that "...Appendix B.2, Table 9(b) we report 18,063 tokens/s..." but in the version of the manuscript that I have, Table 9 does not show any throughput numbers. Am I looking at a different version than yours? (openreview does not show any revision for the uploaded manuscript for me)

AuthorsOpenReview ↗

followup reply

Thank you for pointing this out. By "results show no uniform decrease" we meant that enabling QShare (G=4G=4) shows no uniform decrease in results compared to CISA with G=1G=1, rather than a comparison with dense attention. On GraphWalks-200, the drop from 24.22 to 17.57 is already present without QShare; enabling it changes the score to 16.99, an additional 0.58-point decrease. On the other Ultra benchmarks, G=4G=4 is unchanged on RULER and higher on AA-LCR and MRCR compared to G=1G=1 (Table 10). We have clarified this comparison in the manuscript.

We apologize for the confusion and have uploaded the corrected file with all revision additions, the correct reference is Table 12. For convenience, we show the table below:

SettingKK (chunks)Total tokens/s
Dense—14,677
CISA, G=1G=14,09618,063
CISA, G=4G=42,04838,375
CISA, G=4G=44,09633,543
CISA, G=8G=84,09638,075
Reviewer_VJbDOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CISA, a novel method of generating sparse attention indexer via a light-weight recalibration stage of an existing LLM with dense attention, and includes a set of efficient kernels as CISA's implementation. The recalibration stage selects the best query linear projection from a candidate set, which is composed of all q weights in a GQA's KV group, and an averaged query projection, by minimizing the loss between selected indexer and target attention probability distribution. The efficient kernels utilize the phenomenon that adjacent query tokens usually share a large proportion of same selected KV pages. Extensive experiments show that CISA is able to achieve near-lossless performance compared with the original dense attention with at most ~10x acceleration ratio.

Critical strengths
  1. The training free recalibration method of generating the indexer's weight is novel and effective. This mechanism shows that indexers can be obtained without a heavy CPT stage, and the recalibration method is elegant and simple.

  2. Experiments are conducted in very large scales, and across multiple models on various datasets. The results can fully demonstrate the efficiency and effectiveness of the proposed method.

  3. The presentation of this paper is clear and easy to understand.

Critical weaknesses
  1. Lack of baselines and some discussions. Although the experiments are conducted at very large scales, there are certain settings have been overlooked.
  • Deepseek DSA [1]: DSA uses two stage to train the indexers. During the warmup stage, a KL loss is used to direct the attention distillation process, and after the warmup stage, a CPT stage is conducted to enable the model to inference under attention sparsity. Note that heavy training is involved in DSA, therefore, DSA should be used as an upper bound of obtaining sparse attention indexers. By comparing the proposed method with the original DSA's training, we will be able to know how much performance degradation is caused by undertrained indexers, or simply caused by attention sparsity.

  • Minimax MSA [2]: This is a block wise version of DSA. Note that since the proposed method conducts sparse attention calculation in a block-wise manner, MSA should be a more relevant oracle method compared with DSA. However, since they train indexers in the same way, they should be compared with together.

  • Other training free sparse attention methods, for example, oracle topk sparse attention that selects KV using each query head's attention logits, traditional sparse attention methods such as MInference [3] , etc.

  1. Some highly related findings are not discussed with or cited properly. QShare relies on the locality of sparse attention's KV selection, which has been detailedly discussed by previous papers. ShadowKV [4] identifies this phenomenon and use it to accelerate KV cache offloading, NOSA [5] also validates this phenomenon in trainable sparse attentions, and Hils-attention [6] uses this finding to accelerate sparse attention kernels. These works should be discussed.

[1] DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

[2] MiniMax Sparse Attention

[3] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

[4] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

[5] NOSA: Native and Offloadable Sparse Attention

[6] Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

reply part 1

We thank the reviewer for the thorough review and encouraging feedback. The suggested additional experiments, comparisons and discussions are incorporated and detailed below.

1. Comparison with trained DSA and MSA

By comparing the proposed method with the original DSA's training, we will be able to know how much performance degradation is caused by undertrained indexers, or simply caused by attention sparsity.

DSA and MSA are important comparisons for understanding how much of the quality gap stems from imperfect selection and how much from attention sparsity. Comparing with their trained indexers also helps assess the remaining scope for quality recovery. We will expand the discussion of both methods in the paper.

We compared random and CISA initialization of Nano's indexer using DSA-style stage-1 training: a frozen dense backbone, token-level KL, and no chunking or query sharing during training. Each run used 1,000 steps with 16 sequences of 128K tokens (2.10B tokens).

We swept constant learning rates 10−310^{-3} and 10−410^{-4} using the same data and seed. Both achieved lower mean KL over the final 100 steps at 10−410^{-4} (random: 0.1325 vs. 0.1781; CISA: 0.0982 vs. 0.1787). We therefore report step-1,000 results at 10−410^{-4}, with training-loss curves for both initializations.

Nano configurationAA-LCRRULER 128KRULER 256KRULER 512KRULER 1M
Dense (paper)34.9494.6192.7591.3786.60
CISA, init. (paper)34.2592.9390.7089.5285.87
DSA Stage 1 + KL35.0092.8691.1489.3485.73

Training gives modest, benchmark-dependent gains, while training-free CISA remains competitive.

The takeaway is that CISA's initialization lets us skip stage-1 indexer training.

Stage 2: Sparse training stage. DSA's second stage updates both the indexer and backbone with roughly 1T tokens of continued pre-training (CPT). Our CPT trials reduced accuracy, as we expected for fully post-trained models. We therefore instead used supervised fine-tuning with knowledge distillation (KD-SFT). After 50B tokens, KD-SFT improves all long context benchmarks, as shown in the table below. This shows that with suitable training, CISA's attention sparsity does not contain an inherent gap to dense attention.

Nano configurationAA-LCRk=1K\underset{k=1K}{\text{AA-LCR}}RULER 128Kk=1K\underset{k=1K}{\text{RULER 128K}}RULER 256Kk=2K\underset{k=2K}{\text{RULER 256K}}RULER 512Kk=4K\underset{k=4K}{\text{RULER 512K}}RULER 1Mk=4K\underset{k=4K}{\text{RULER 1M}}LongBench v2k=2K\underset{k=2K}{\text{LongBench v2}}MRCRk=2K\underset{k=2K}{\text{MRCR}}GraphWalksk=2K\underset{k=2K}{\text{GraphWalks}}
Dense34.9494.6192.7591.3786.6043.097.8329.40
CISA, init.34.2592.9390.7089.5285.8742.457.8629.92
KD-SFT, 50B tokens37.8193.8394.1592.9288.2144.2312.1430.60
AuthorsOpenReview ↗

reply part 2

2. Comparing MSA, MInference and CISA.

MSA should be a more relevant oracle method compared with DSA.

MSA is a particularly relevant comparison given its block-wise selection. Following your feedback we added its max-token scoring criterion to CISA's calibrated indexer, replacing CISA's default mean-key chunk scoring.

Other training free sparse attention methods, … traditional sparse attention methods such as MInference [3], etc.

MInference's block-sparse variant pools queries and keys over time and uses each query head directly for selection. Motivated by this approach, we added a per-head control using each original query projection with CISA's mean-key chunks. This compares per-head selection with CISA's selection shared across heads within each KV group.

The tables below uses Inkling-Small with CISA.

MethodAA-LCRk=1K\underset{k=1K}{\text{AA-LCR}}RULER-1Mk=4K\underset{k=4K}{\text{RULER-1M}}MRCR200k=4K\underset{k=4K}{\text{MRCR200}}GraphWalks200k=4K\underset{k=4K}{\text{GraphWalks200}}
CISA (paper reference)65.6376.0317.7615.29
Per-head selection65.5074.3417.0215.76
MSA-style max scoring64.3168.5014.5511.98

In this setting, CISA scores higher than per-head selection on three of four benchmarks and higher than max scoring on all four. A chunk's highest token score need not reflect its overall contribution.

We follow the paper's full-model throughput methodology (Section 4.4), with three repeats per method (K=4,096K=4,096, G=32G=32).

MethodMean total tokens/sMethod throughput / CISA throughput
Dense55,198.870.65×
CISA84,283.521.00×
Per-head selection (Minference-style)56,682.850.67×
MSA-style max scoring63,983.870.76×

These results evaluate MInference-style per-head selection and MSA-style max scoring within CISA, measuring their effects on accuracy and throughput. They use our implementations of these selection strategies.

3. Exact-attention oracle

oracle topk sparse attention that selects KV using each query head's attention logits

Exact attention-mass selection. Our existing Nano RULER-256K diagnostic selects 2,048 chunks using exact dense-attention mass aggregated within each KV group and each 16-token chunk:

Attention / selection methodAccuracy (%)
Dense attention92.747
CISA approximate selector91.088
Exact attention-mass chunk selector91.686

Exact-mass selection recovers 0.597 points. This group/chunk diagnostic is not a per-head token oracle: retained mass need not minimize attention-output or LM-head error, so the remaining gap is not necessarily unavoidable sparsity loss.

4. Locality and related work

QShare relies on the locality of sparse attention's KV selection, which has been detailedly discussed by previous papers.

Thank you for pointing out these works, particularly HiLS-Attention. We will cite and discuss ShadowKV's locality-based cache transfers, NOSA's locality constraints, and HiLS-Attention's reuse of independently selected chunks across adjacent queries. QShare builds on this established locality by pooling index queries before retrieval to create one shared plan, saving indexing as well as attention work. We will clarify this distinction and narrow the novelty claim.

Reviewer_VJbDOpenReview ↗

Great work, looking to see this paper at acceptance list!

Thanks for your detailed response. All my weaknesses and questions have been fully resolved. Additional results can fully show the effectiveness and efficiency of the proposed method. The solid work would largely advance the community of sparse attention and efficient inference. I have adjusted my rating accordingly, and I wish you all the best to acceptance!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.