- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper accelerates the DSA indexer through certified candidate screening. A seed set establishes a lower bound on the Top-K threshold, while score upper bounds allow candidates to be rejected before full-head scoring. The method combines screening, full-score refinement, and sparse writeback, and evaluates Indexer latency and time to first token.
Critical strengths
S1. The problem is clear. The indexer still requires substantial full-score computation and can become a bottleneck in sparse attention.
S2. The upper-bound screening method is reasonable and can safely reject candidates that cannot enter Top-K. The experiments show computational savings and actual speedups.
Critical weaknesses
W1. Experimental coverage and empirical correctness validation are insufficient. The main performance evaluation covers limited datasets and application scenarios. Although theoretical guarantees are provided, there are no empirical Top-K agreement checks or downstream task-quality evaluations to verify that the numerical implementation preserves the expected results. This leaves the practical reliability and generality of the method insufficiently established.
W2. Missing hardware details limit interpretation and reproducibility. The paper does not specify the GPU model and memory capacity. These are essential for assessing the reported performance and comparing it with other implementations.
W3. The ablation figures also need better layout and readability. Labels such as "Base," "Full," and "Dense" are not intuitive; "Dense" can easily be confused with dense attention.
Questions for authors
Q1. What GPU model and memory capacity were used for the microbenchmarks and model-level experiments?
Q2. Have you checked Top-K agreement against the original scorer under the actual FP8 implementation, including boundary ties and candidate-buffer recovery? If so, what were the results?
3 comments
Thank you for your response and additional experiments. I will raise my rating to 3.
Thank you for considering our responses and additional experiments. We appreciate your time and constructive feedback, and will incorporate these additions into the revised manuscript.
Response to Reviewer dEhw
Thank you for recognizing the indexing bottleneck and the computational savings of certified screening. We appreciate your suggestions on experimental coverage, reproducibility, and presentation.
W1. Experimental coverage, GPU correctness, and downstream behavior
Our evaluation includes GLM-5.2 and DeepSeek-V3.2, English and Chinese books, and source code. The prefill workloads use InfiniteBench, Project Gutenberg, and SQLite, with contexts from 128K to approximately one million tokens. The GLM-5.2 microbenchmarks span all 21 indexer-producing layers and four prefill chunk sizes. Decoding experiments use document-analysis requests from HELMET, InfiniteBench, and LongBench v2, and code-review requests from SQLite. The SciFact example from LOFT supplies an additional retrieval workload for the scoring/writeback ablation.
We also evaluated the current DoSA kernel implementation through a replay of captured inference workloads. The captured query, key, and scoring-weight tensors cover 18,711 layer-level workload cases across context lengths and query-batch sizes. Two complete passes produced 85,050 complete calls and 74,844 output checks, all of which passed.
The actual GPU microbenchmark validation covers 77,414,400 query checks with zero selection errors and 91,342,848 individual score comparisons with 100% bitwise agreement. DoSA preserves the original scoring function and exact attention-token selection. The mathematical guarantee and these GPU checks directly address the effect of the indexing optimization on the model's attention inputs. We will present this validation prominently, describe the datasets and workload construction together, and clarify the scope of each experiment.
W2 and Q1. Hardware and reproducibility
Our experiments use the same settings as LiteTopK. All methods are evaluated on our platform with matched inputs, numerical precision, and timing scope. Model-level experiments use prefill chunks of 32K tokens for GLM-5.2 and 16K for DeepSeek-V3.2.
Complete-Indexer latency is measured with CUDA events after warmup and includes query/key preparation and any triggered recovery. Each microbenchmark case has six repetitions per method, with the median used for the reported case latency. The median relative standard deviations across cases are 1.11% for SGLang, 1.40% for LiteTopK, and 1.46% for DoSA. End-to-end time to first token is measured on individual requests with the prefix cache cleared. We will consolidate the platform comparison, software versions, parallelism, timing protocol, and memory accounting in the revised experimental setup.
W3. Readability of the ablation figures
Thank you for pointing out the ambiguity in the labels. We will replace them with descriptions of the operations performed, enlarge the labels, and make each caption specify the workload, timing scope, and aggregation.
Q2. FP8 execution, score boundaries, and candidate-buffer recovery
Yes. The output checks above run on the actual GPU implementation and compare against the original scorer on the same inputs. The current DoSA kernels also pass the following focused cases.
The upper bounds account for low-precision representation and arithmetic errors. Filtering retains equality at the threshold, candidates at the selection boundary are resolved using their original full scores, and a full candidate buffer invokes exact recovery. We will add these measurements and a plain-language explanation of the lossless selection rule to the revised manuscript.