- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates sparse prefill in frozen GQA models through comparisons among dense attention, oracle selection, and learned indexers. It constructs a top-k oracle from head-averaged dense attention distributions and trains a single-head indexer via KL distillation while keeping the backbone frozen. By comparing task performance and runtime efficiency across different support-selection granularities, the paper examines the quality degradation introduced by sparse selection and the speedups achieved in practice.
Critical strengths
On Qwen3.5-27B with a 128K context, the paper reports a 1.46× single-request TTFT speedup, providing some empirical support for the practical value of the evaluated configuration.
Critical weaknesses
1.Limited methodological novelty and an unclear core technical contribution.
The method largely follows the established approach of freezing the backbone, distilling an indexer from dense attention, and using it for sparse selection. The authors also explicitly acknowledge that their training objective is closely related to DSA’s warm-up objective. The paper does not sufficiently clarify what substantive new mechanism it introduces beyond existing indexer-distillation approaches. The single-head indexer appears more like a particular implementation choice than a well-substantiated methodological innovation.
2.The diagnostic analysis remains descriptive and does not yield sufficiently deep new insights.
The paper organizes its analysis around the oracle, indexer, and realization gaps. However, a central finding remains insufficiently explained: some key-block configurations preserve task performance under oracle selection but degrade substantially when using the learned indexer. The authors do not provide targeted controlled experiments to identify the source of this discrepancy, nor a stage-by-stage runtime breakdown to explain why the key-block implementation does not deliver additional speedups.
Questions for authors
1.Beyond applying frozen-backbone KL distillation with a single-head indexer, what substantive advance does this work introduce over existing indexer-based sparse attention approaches?
2.Why does learned key-block selection suffer a substantial accuracy drop when the corresponding oracle maintains strong task performance?
2 comments
Thank you for your response,and I will keep my rating.
Authors' response: controlled key-block experiments, fidelity diagnostics, and runtime profiling
Thank you for distinguishing a useful diagnostic organization from the need for deeper explanations. We agree that the key-block discrepancy requires more than a descriptive observation.
Frozen-backbone KL distillation and top-k selection are established components. We do not present the single-head implementation alone as the core novelty. The contribution is the empirical investigation of oracle feasibility, learned-selection error, and runtime under explicit support granularities and budgets.
We added a matched-budget teacher/student by token/key-block control with query sharing fixed to 32 and key-block width 128. On three selected tasks (niah_multikey_2, niah_multikey_3, cwe; ten exactly hash-matched prompts each), the respective three-task mean scores are:
These controls show that the teacher–student discrepancy persists for key blocks when query sharing and token budget are held fixed. They strengthen the evidence that the evaluated block support is feasible but the deployed learned selector fails to realize it. The scores cover selected tasks, not the full RULER macro. They do not yet distinguish score-estimation error, block aggregation, and training-objective mismatch; we retain these as hypotheses rather than claim a causal mechanism.
A further diagnostic holds dense trajectories fixed on nine shared prompts, three layers, and 32 tail queries at context 61,440. Student/teacher AMR is 0.4624/0.4769 for token support and 0.3599/0.3531 for key blocks; relative AMR is 0.9694 and 1.0192. Thus, these sampled aggregate metrics do not explain the key-block failure by lower relative mass retention. MAX-over-query selection need not maximize mean per-query AMR, so a ratio above one is permitted. This motivates examining task-critical support and recursive sparse trajectories, but does not establish either as the cause.
We now profile the current normalized two-pass implementation on one synthetic 64K request after warmup, with matched budget 2048, query sharing 32, and 4K chunks. Summed GPU kernel durations for normalization plus candidate scoring are 0.941 seconds for token support and 0.943 for key blocks. Sparse attention takes 1.294 and 1.386 seconds; named top-k/gather kernels take 0.097 and 0.053 seconds. All-model GEMMs contribute about 68% of summed kernel duration in both sparse cells. The revised appendix provides the breakdown and classification method.
In this trace, reduced top-k/gather time is offset by other costs, with no meaningful overall key-block advantage. We do not infer a general kernel limitation or a statistically established speed ranking from one instrumented synthetic request. These profiles are distinct from the real-prompt baseline timing protocol and do not isolate auxiliary projection time from backbone GEMMs.