- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a framework that can turn dense attention models into chunk-index sparse attention training. Unlike DSA, this method uses a single query per KV-group, which reduces the complexity of the indexer. Moreover, the paper presents QShare, which makes consecutive groups of tokens in prefill share the same chunks so that the fetched chunks can be reused by multiple tokens.
Critical strengths
-
The paper is well-written, easy to follow, its contributions are clear. The authors are upfront about the method's limitations, novelty claims etc., which I really appreciate.
-
This paper proposes a practical solution to an important problem. The current sparse attention mechanisms are not efficient: DSA still requires quadratic complexity for the indexer. The others require pretraining or fine-tuning. This paper's proposal can be simply applied to the existing methods and would reduce the attention's complexity.
-
Evaluation is thorough, covers a sufficient range of models and benchmarks. I especially liked the speedup adjusted for the change in generation length, which is often overlooked or hidden in other papers.
Critical weaknesses
- The evaluation of QShare for prefill and chunk-index sparse attention for decode is convoluted. If I'm in a scenario where prefill time is not significant, I might want to disable QShare and focus on decode. However, there is no evaluation that shows which one of these two changes contributes most and how much to the degradation of model quality.
- Figure 1 could be improved in terms of clarity.
- A table that shows the complexity in terms of compute and I/O of the proposed method versus prior methods as a function of sparsity ratio, model dims etc would make it clear to understand theoretical gains.
Questions for authors
See the weaknesses above.
4 comments
reply part 2
To illustrate the theoretical costs at (1M context), we consider each model's complete attention layer executing on one GPU. We use the native configurations: for DeepSeek-V3.2's absorbed MLA, for MiniMax-M3, and for Nemotron Ultra. All use ; DSA uses , and CISA uses .
The following costs are per query position and layer, using each method's retained-token budget .
| Model / method | Attention GFLOPs | Scoring GFLOPs | Reads (M elements) | |
|---|---|---|---|---|
| DeepSeek-V3.2 / DSA | 2,048 | 0.5369 | 17.1799 | 135.27 |
| MiniMax-M3 / MSA | 2,048 | 0.0671 | 1.0737 | 136.31 |
| Ultra / CISA, | 65,536 | 2.1475 | 0.0336 | 50.33 |
| Ultra / CISA, | 65,536 | 2.1475 | 0.0084 | 37.75 |
For these configurations, CISA reduces index-scoring arithmetic relative to MSA by 32× without QShare and 128× with . Modeled index-key reads decrease by 8× and 32×, respectively, while CISA retains a larger attention budget. We note that as context length increases, scoring becomes the bottleneck for end-to-end latency which is why we designed CISA around optimizing the scoring.
I thank the authors for their response. I can see that Table 10 shows a comparison of response quality for chunk-indexing by disabling QShare (G=1). You claim that "while the Ultra NVFP4 results show no uniform decrease." but the quality reduces from 24.22 (Dense) to 17.57 (G=1) for GraphWalks-200 benchmark. You also claim that "...Appendix B.2, Table 9(b) we report 18,063 tokens/s..." but in the version of the manuscript that I have, Table 9 does not show any throughput numbers. Am I looking at a different version than yours? (openreview does not show any revision for the uploaded manuscript for me)
followup reply
Thank you for pointing this out. By "results show no uniform decrease" we meant that enabling QShare () shows no uniform decrease in results compared to CISA with , rather than a comparison with dense attention. On GraphWalks-200, the drop from 24.22 to 17.57 is already present without QShare; enabling it changes the score to 16.99, an additional 0.58-point decrease. On the other Ultra benchmarks, is unchanged on RULER and higher on AA-LCR and MRCR compared to (Table 10). We have clarified this comparison in the manuscript.
We apologize for the confusion and have uploaded the corrected file with all revision additions, the correct reference is Table 12. For convenience, we show the table below:
| Setting | (chunks) | Total tokens/s |
|---|---|---|
| Dense | — | 14,677 |
| CISA, | 4,096 | 18,063 |
| CISA, | 2,048 | 38,375 |
| CISA, | 4,096 | 33,543 |
| CISA, | 4,096 | 38,075 |
We thank the reviewer for the positive assessment of the paper.
1. Separating chunk indexing from positional query sharing
We addressed this option in Appendix B.2 and have now reorganized it to distinguish chunk-indexed sparse attention from the additional effect of positional query sharing. We have also added additional ablations shown below, using the Ultra NVFP4 model and benchmark subsets.
To examine QShare's quality effect across context lengths, we additionally report Nano BF16 RULER ablations, varying only positional sharing within each pair.
Across the four Nano comparisons, enabling QShare decreases accuracy by up to 0.79 percentage points, while the Ultra NVFP4 results show no uniform decrease. As shown in Appendix B.2, Table 9(b) we report 18,063 tokens/s without QShare (G=1), compared with 33,543 tokens/s with G=4, both at K=4,096.
2. Figure 1
Thank you for the suggestion. We have improved the clarity as in this revised Figure 1.
3. Compute and I/O complexity
The table below gives theoretical costs per query position and complete attention layer, counting two FLOPs per multiply-add. We use:
For clarity, we omit MLA's additional RoPE dimensions. DSA's absorbed MLA uses one shared latent group (L=1); the latent serves both K and V, giving a main-cache payload of V/2.
DSA aggregates multiple index query heads against one shared index-key cache. MSA uses one index query per KV group with a shared index-key cache; it scores all token-level keys before max-pooling into blocks. CISA instead scores compressed chunk representatives. At equal KV-group counts and index dimensions, CISA reduces MSA's scoring arithmetic by CG and modeled index-key reads by CG/L; both reductions are CG with one KV group.
All three sparse methods retain quadratic index-scoring work over a complete prefill for fixed architecture parameters. CISA reduces this work to O(LN2dI/(CG)) through chunk compression and query sharing, improving the constant factor without changing the asymptotic order.