acceptodds
Under review as a conference paper at ICLR 2027

LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attention

Abstract

Indexer-TopK, which computes scores and selects the top-k candidates, is a key primitive in long-context sparse attention for large language models and vector retrieval for recommendation systems and vector databases. However, existing GPU-based Indexer-TopK kernels like DeepSeek Sparse Attention (DSA) typically separate scoring from selection and materialize the full score matrix in high-bandwidth memory. This strategy is inefficient for large score matrices, such as those encountered in long-context sparse attention, due to excessive global-memory traffic, costly synchronization, and prohibitive memory overhead. In this study, motivated by the curse of dimensionality, we observe that sparse attention scores exhibit a concentration phenomenon, with scores falling within a narrow range. Based on this observation, we propose LiteTopK, an efficient fused Indexer-TopK kernel. LiteTopK samples a small subset of data to estimate query-data score ranges, then partitions candidates into bins accordingly. This design allows LiteTopK to maintain a slightly relaxed threshold online and write back only promising candidates, reducing unnecessary I/O and memory overhead while preserving exact top-k correctness. We integrate LiteTopK into three widely used LLM training and inference systems: vLLM, SGLang, and Megatron-LM. Experiments in a production environment with eight NVIDIA B200 GPUs demonstrate prefill speedups of up to 1.49× across GLM-5.2, DeepSeek-V4-Flash, and Hy-4-Preview at a context length of 1M tokens, without compromising model capabilities.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Kernel, LLM Systems, LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_REDZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents LiteTopK, a fused Indexer-TopK GPU kernel for long-context sparse attention. Motivated by the concentration of sparse-attention scores, it adopts a sample–filter–select workflow with score binning, online threshold updates, and local buffering to avoid materializing the full score matrix while preserving exact top-k selection. This design reduces intermediate memory traffic and the workload of the final selection stage. Experiments across multiple models and frameworks demonstrate kernel-level acceleration, reduced auxiliary memory usage, and end-to-end prefill speedups of up to 1.49×.

Critical strengths

I consider this work to have practical value. It fuses indexer scoring with top-k selection, reducing intermediate memory traffic while preserving exact top-k results. It demonstrates both kernel-level acceleration and actual end-to-end prefill speedups, along with reduced auxiliary memory usage, so the benefits are not limited to isolated kernel benchmarks. These improvements are meaningful for long-context inference and may also benefit training, although the paper does not demonstrate end-to-end training speedups.

Critical weaknesses

1.Modest novelty. The main contribution centers on kernel-level optimization of Indexer-TopK, while the algorithmic novelty appears relatively limited. I would like the authors to distinguish more clearly which design choices are new relative to existing selection methods and which are adaptations tailored to DSA. Given the paper’s emphasis, I personally feel that a systems or ML systems venue might be a more natural fit.

2.Insufficient comparison with other optimized top-k baselines. Comparisons against vLLM and SGLang are meaningful. However, while the paper states that the baselines are evaluated using their official implementations, it does not clearly specify the framework versions or the particular top-k backends used. This makes it difficult to determine which optimized implementations are already represented in the comparisons.

Questions for authors

1.Could the authors specify the versions of vLLM and SGLang and the top-k backends used in the experiments? Which optimized exact GPU top-k implementations are already covered by these baselines? For representative methods that are not covered, could the authors provide additional comparisons?

2.Could the authors provide targeted ablations to isolate the contributions of sampling, threshold refresh, and local buffering? In particular, how much does each component contribute to the overall performance gains, beyond the results reported for the complete implementation?

Reviewer_CnQzOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LiteTopK, a fused Indexer-TopK kernel for long-context sparse attention. Motivated by the observed concentration of indexer scores, it uses sampling, histogram-based filtering, and online threshold updates to retain promising candidates without materializing the full score matrix, while aiming to preserve exact Top-k results. Integrated into vLLM, SGLang, and Megatron-LM, LiteTopK achieves reported end-to-end prefill speedups of up to 1.49× on eight NVIDIA B200 GPUs and substantially reduces auxiliary memory usage.

Critical strengths

The paper addresses a real and practically important bottleneck in long-context sparse attention: materializing the full indexer score matrix incurs substantial memory traffic and workspace overhead, even though only a small fraction of candidates are ultimately selected. LiteTopK directly targets this inefficiency by fusing scoring and selection. Its integration into multiple LLM frameworks, together with the reported end-to-end prefill speedups and memory savings, demonstrates practical value beyond isolated kernel benchmarks.

Critical weaknesses
  1. Empirical evidence for score concentration. The concentration of aggregated multi-head indexer scores is an interesting observation and a key motivation for LiteTopK. However, Figure 2 only shows the distribution for the first layer of GLM-5.2. Could the authors provide more empirical evidence across different models and layers to establish how consistently this phenomenon holds?

  2. Ablation and sensitivity studies. Could the authors provide ablation studies comparing the proposed sampling strategy with random sampling, together with sensitivity analyses for the number of bins and the threshold refresh frequency? Reporting both latency and the number of retained candidates would help clarify the benefits of these design choices and the trade-off between filtering effectiveness and maintenance overhead.

  3. Stronger baselines. Could the authors include comparisons with specialized GPU Top-k methods, such as RTop-K? Combining such methods with the same indexer scoring implementation and reporting total scoring-and-selection latency and peak auxiliary memory would help establish LiteTopK’s advantage over stronger baselines.

Questions for authors

See weaknesses.

Reviewer_xUfUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LiteTopK which fuses scoring and exact top-k selection for sparse attention. It avoids storing the full score matrix and shows good speedup on vLLM SGLang and Megatron-LM with up to 1M context.

Critical strengths

The systems idea is simple but quite useful. The end to end results are strong and it also reduces memory a lot while keeping exact top-k results.

Critical weaknesses

The evaluation is mostly around DSA style sparse attention and three models. Also I did not see enough ablation on sample size bin count or refresh frequency so it is hard to know how sensitive the speedup is to these choices.

Questions for authors

How sensitive is LiteTopK to the number of bins and sample size? Does the speedup stay similar if the score distribution is less concentrated than the models tested here?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.