- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FlashIndexer proposes dynamically selecting a subset of indexer heads to reduce indexing overhead. A lightweight router selects a small number of heads for each query, and only the selected heads are used to score tokens in the causal prefix. The method freezes both the backbone model and the original indexer, trains only the newly introduced router, and uses a specialized kernel implementation to accelerate execution. The reported experiments show that, after router training, FlashIndexer largely preserves downstream task performance with fewer active heads while achieving end-to-end prefill speedups.
Critical strengths
1.Good downstream performance after router training.
With only eight active indexer heads, FlashIndexer achieves overall performance close to the original full-head DSA baseline on RULER and LongBench v2 for both DSV4-Flash and GLM-5.1. These results suggest that training a lightweight router can reduce indexing computation while largely preserving the capabilities of the original models.
2.Demonstrated end-to-end inference benefits.
The paper evaluates end-to-end time to first token (TTFT), rather than reporting only isolated kernel speedups. At a context length of 128K tokens, FlashIndexer achieves TTFT speedups of 1.20× on GLM-5.1 and 1.12× on DSV4-Flash. These results demonstrate that the computational savings from using fewer indexer heads can translate into practical inference benefits.
Critical weaknesses
1.Limited novelty over MISA (primary concern)
MISA already introduced query-dependent selection of a small subset of indexer heads for token scoring. Thus, exploiting head redundancy through dynamic head selection is not a new contribution of FlashIndexer.
The main change is replacing routing based on pooled prefix-key statistics with a trainable query-side gate. While avoiding prefix-key access during routing is useful, the paper does not convincingly establish a substantial methodological advance beyond this modification.
The introduction’s characterization of the head dimension as “largely unexplored” also understates this direct precedent. The paper needs to establish the router’s added value over existing head-routing methods, rather than mainly reaffirm that fewer heads can preserve performance. Despite the practical speedups, this insufficiently demonstrated novelty is my main reason for recommending rejection.
Questions for authors
1.Benefit of dynamic head selection.
Could the authors compare FlashIndexer against a baseline that fixes eight heads per layer using calibration data, while learning query-dependent score reweighting under the same training setup? This would clarify whether changing the selected head subset across queries provides a meaningful benefit beyond adapting a fixed subset.
2.Comparison with MISA under matched adaptation.
Could the authors evaluate whether the reported routing-quality advantage over MISA persists under matched adaptation budgets? For example, an adapted MISA-based baseline could retain MISA’s head-selection rule while learning score reweighting, using the same training data, objective, token budget, and active-head count as FlashIndexer. This would help distinguish the benefit of the proposed routing design from that of additional adaptation.
2 comments
(continued)
Q2 (matched adaptation). The learned gate has one purpose: ranking the heads. Because TopK is not differentiable, we multiply each selected head's indexer weight by its gate value during training, so that the KL loss can back-propagate to the router, as in standard MoE gating. At inference we keep the multiplication only for train-inference consistency. We revised the description in Sec 3.2, which previously stated that the gate also modulates each head's contribution. Table A shows that removing it changes little: FlashIndexer w/o g differs from FlashIndexer by at most 0.4 points in any setting, with no consistent sign. The gate therefore helps through which heads it selects, not through rescaled scores.
For a MISA baseline with matched adaptation, the head-selection rule stays MISA's, so adaptation can only change the rescaling, not which heads are selected. We trained this variant as requested: "MISA w/ g" in Table A keeps MISA's head selection and multiplies each selected head's by a gate trained with the same data, objective, and token budget as FlashIndexer's router. The effect is again unsystematic: MISA w/ g differs from MISA by at most 0.34 points, with no consistent sign, and its average changes by 0.04 points on DSV4-Flash and 0.03 on GLM-5.1. Matched adaptation therefore does not narrow the gap to FlashIndexer.
Q1 (fixed head subset). Figure 1(b) suggests that no fixed subset can cover the heads different inputs rely on, because the union of per-sample top- heads keeps growing with the number of samples. To test this directly, we built Static-8: for each layer, we fix 8 heads calibrated on one tenth of the router training data (2K samples). For each calibration sample, we rank the heads by their individual attention recall, as the Oracle in Figure 5(a) does, and score the top 10 heads 1.0, 0.9, ..., 0.1; we accumulate the scores over samples and keep the 8 heads with the highest total. The clean comparison is Static-8 vs FlashIndexer w/o g, since both score with the original weights and differ only in head selection.
Static-8 is the lowest of the five routed variants in all 8 settings. Per-token selection improves on it by 0.2–0.9 points on DSV4-Flash and 0.7–1.1 points on GLM-5.1. Averaged over the three RULER lengths, Static-8 loses 0.55 points to DSA on DSV4-Flash and 1.27 on GLM-5.1, while FlashIndexer w/o g loses 0.06 and 0.35. Individual differences of this size are small, but the direction is the same in every setting, which is what the input-dependent redundancy of Figure 1(b) predicts. Changing the head subset across queries thus provides a consistent benefit over a fixed subset.
We thank the reviewer for the focused critique. Summary of changes: (i) we clarify the role of the learned gate and show that FlashIndexer's routing advantage over MISA comes from which heads it selects, not from additional adaptation (Q2); (ii) we add a fixed-subset baseline calibrated on one tenth of the router training data (Q1) and a MISA baseline with a gate trained as in FlashIndexer (Q2), reported with the gate ablation in new Appendix A.11 (Table A); (iii) we report results at 1M context, where the head-axis methods separate; (iv) we revised the abstract, introduction, method, and conclusion to credit MISA directly and to describe the gate accurately.
W1 (novelty over MISA). We agree that MISA is a direct precedent for dynamic head selection in the DSA indexer, and that describing the head axis as "largely unexplored" understated it. We revised the text accordingly. The abstract now states that MISA has begun to exploit the head axis by routing indexer heads with prefix-side statistics, at a routing cost that grows with the context. The second paragraph of the introduction replaces "largely unexplored" with: MISA already selects a subset of indexer heads for each query, but the structure of head-axis redundancy has not been characterized, and identifying the important heads without a context-dependent cost remains open. The conclusion no longer calls the head axis "largely untapped".
What we add relative to MISA is the following.
The router's added value is thus a more accurate head selection obtained at a cost that does not depend on context length. Q1 and Q2 below separate this value from the effect of adaptation.
Static-8 and FlashIndexer w/o g require no training; MISA w/ g trains a gate for MISA. All variants are evaluated under the setting of Table 1 (h=8, RULER at 32K/64K/128K and LongBench v2 at 128K):
Table A. Head selection vs. score rescaling (%, h=8). "w/o g" removes the gate rescaling at inference, so the indexer scores use the original weights wI; "w/ g" scales the heads selected by MISA with a gate trained as in FlashIndexer. Avg is the mean over the four columns.
(continued in the next comment)