acceptodds
Under review as a conference paper at ICLR 2027

FlashIndexer: Dynamic Head-Routed Indexer for Token-Level Sparse Attention

Abstract

Token-level sparse attention, exemplified by DeepSeek Sparse Attention (DSA), significantly reduces the downstream attention computation, but shifts the computational burden to the lightweight indexer. For every query, the indexer must compute scores for all context tokens using its entire set of heads before selecting the most relevant top- tokens, resulting in a per-layer cost that grows with both the context length and the number of indexer heads. Recent work has reduced indexer computation along the token and layer axes, but the head axis has received comparatively little attention. We revisit the head axis and make two key observations. First, the contributions of indexer heads are highly imbalanced and vary substantially across samples. Second, the important heads for each query can be reliably identified using only the query itself, without accessing any context tokens. Motivated by these observations, we propose FlashIndexer, a plug-and-play head-routing module for the DSA indexer. For each query, it dynamically selects a small subset of important heads from all indexer heads, activating only the selected heads for scoring context tokens, and thereby substantially reducing indexer computation. This query-adaptive routing preserves the original token-level top- selection while avoiding unnecessary computation from less informative heads. With only a small subset of indexer heads activated, FlashIndexer matches the accuracy of the original full-head DSA indexer on DeepSeek-V4-Flash-0731 and GLM-5.1 across RULER and LongBench v2, while achieving up to end-to-end time-to-first-token speedup at K context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long-context inference, efficient LLM inference, attention head routing, LLM serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8a1TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlashIndexer proposes dynamically selecting a subset of indexer heads to reduce indexing overhead. A lightweight router selects a small number of heads for each query, and only the selected heads are used to score tokens in the causal prefix. The method freezes both the backbone model and the original indexer, trains only the newly introduced router, and uses a specialized kernel implementation to accelerate execution. The reported experiments show that, after router training, FlashIndexer largely preserves downstream task performance with fewer active heads while achieving end-to-end prefill speedups.

Critical strengths

1.Good downstream performance after router training.

With only eight active indexer heads, FlashIndexer achieves overall performance close to the original full-head DSA baseline on RULER and LongBench v2 for both DSV4-Flash and GLM-5.1. These results suggest that training a lightweight router can reduce indexing computation while largely preserving the capabilities of the original models.

2.Demonstrated end-to-end inference benefits.

The paper evaluates end-to-end time to first token (TTFT), rather than reporting only isolated kernel speedups. At a context length of 128K tokens, FlashIndexer achieves TTFT speedups of 1.20× on GLM-5.1 and 1.12× on DSV4-Flash. These results demonstrate that the computational savings from using fewer indexer heads can translate into practical inference benefits.

Critical weaknesses

1.Limited novelty over MISA (primary concern)

MISA already introduced query-dependent selection of a small subset of indexer heads for token scoring. Thus, exploiting head redundancy through dynamic head selection is not a new contribution of FlashIndexer.

The main change is replacing routing based on pooled prefix-key statistics with a trainable query-side gate. While avoiding prefix-key access during routing is useful, the paper does not convincingly establish a substantial methodological advance beyond this modification.

The introduction’s characterization of the head dimension as “largely unexplored” also understates this direct precedent. The paper needs to establish the router’s added value over existing head-routing methods, rather than mainly reaffirm that fewer heads can preserve performance. Despite the practical speedups, this insufficiently demonstrated novelty is my main reason for recommending rejection.

Questions for authors

1.Benefit of dynamic head selection.

Could the authors compare FlashIndexer against a baseline that fixes eight heads per layer using calibration data, while learning query-dependent score reweighting under the same training setup? This would clarify whether changing the selected head subset across queries provides a meaningful benefit beyond adapting a fixed subset.

2.Comparison with MISA under matched adaptation.

Could the authors evaluate whether the reported routing-quality advantage over MISA persists under matched adaptation budgets? For example, an adapted MISA-based baseline could retain MISA’s head-selection rule while learning score reweighting, using the same training data, objective, token budget, and active-head count as FlashIndexer. This would help distinguish the benefit of the proposed routing design from that of additional adaptation.

2 comments

We thank the reviewer for the focused critique. Summary of changes: (i) we clarify the role of the learned gate and show that FlashIndexer's routing advantage over MISA comes from which heads it selects, not from additional adaptation (Q2); (ii) we add a fixed-subset baseline calibrated on one tenth of the router training data (Q1) and a MISA baseline with a gate trained as in FlashIndexer (Q2), reported with the gate ablation in new Appendix A.11 (Table A); (iii) we report results at 1M context, where the head-axis methods separate; (iv) we revised the abstract, introduction, method, and conclusion to credit MISA directly and to describe the gate accurately.

W1 (novelty over MISA). We agree that MISA is a direct precedent for dynamic head selection in the DSA indexer, and that describing the head axis as "largely unexplored" understated it. We revised the text accordingly. The abstract now states that MISA has begun to exploit the head axis by routing indexer heads with prefix-side statistics, at a routing cost that grows with the context. The second paragraph of the introduction replaces "largely unexplored" with: MISA already selects a subset of indexer heads for each query, but the structure of head-axis redundancy has not been characterized, and identifying the important heads without a context-dependent cost remains open. The conclusion no longer calls the head axis "largely untapped".

What we add relative to MISA is the following.

  • A systematic analysis of head redundancy in the DSA indexer. Figure 1 shows that per-head attention recall is highly imbalanced within every layer, and that the union of each sample's top-hh heads keeps growing as more samples are added, in shallow, middle, and deep layers alike. The redundancy is therefore input-dependent. To our knowledge, this structure had not been characterized for the DSA indexer before.
  • Head selection is the step that matters, and FlashIndexer selects more accurately. The selected heads determine the token top-kk set, and through it every downstream attention computation. At h=8h=8, the attention-score recall of the routed heads is 0.883 for FlashIndexer vs 0.785 for MISA (Sec 4.4.1). The exact, unpooled version of MISA's importance score also reaches 0.785, so the gap is a property of the prefix-side signal, not of MISA's pooling approximation. The difference becomes visible in accuracy at long context: on RULER-1M (DSV4-Flash, 250 samples), FlashIndexer scores 0.877 vs 0.873 for full-head DSA, whereas MISA drops to 0.836.
  • No context-dependent routing cost. MISA's router reads pooled prefix keys, so its per-query cost grows with LL; FlashIndexer's router reads only the current token. On DSV4-Flash, the per-layer router cost grows from 2.3 to 8.6 ms between 256K and 1M for FlashIndexer, against 11.6 to 68.2 ms for MISA, and MISA falls below 1×1\times TTFT on short inputs (0.84×0.84\times at 8K).

The router's added value is thus a more accurate head selection obtained at a cost that does not depend on context length. Q1 and Q2 below separate this value from the effect of adaptation.

Static-8 and FlashIndexer w/o g require no training; MISA w/ g trains a gate for MISA. All variants are evaluated under the setting of Table 1 (h=8h=8, RULER at 32K/64K/128K and LongBench v2 at 128K):

Table A. Head selection vs. score rescaling (%, h=8h=8). "w/o g" removes the gate rescaling at inference, so the indexer scores use the original weights wIw^I; "w/ g" scales the heads selected by MISA with a gate trained as in FlashIndexer. Avg is the mean over the four columns.

ModelMethodHead selectionRescalingRULER 32K64K128KLB v2Avg
DSV4-FlashDSAall 64 heads–96.0996.1693.5965.8587.92
Static-8fixed per layernone95.8195.3593.0364.7787.24
MISAper tokennone96.0895.6993.8265.8587.86
MISA w/ gper tokeng95.9195.9693.7965.6987.82
FlashIndexer w/o gper tokennone96.0396.1593.4965.7187.85
FlashIndexerper tokeng96.0895.7593.8565.9287.90
GLM-5.1DSAall 32 heads–95.6294.0294.0466.6287.58
Static-8fixed per layernone94.4692.4592.9665.7786.41
MISAper tokennone95.6192.9993.0566.7987.11
MISA w/ gper tokeng95.3992.9993.3966.5587.08
FlashIndexer w/o gper tokennone95.5393.1093.9966.6287.31
FlashIndexerper tokeng95.4492.9994.1666.4687.26

(continued in the next comment)

(continued)

Q2 (matched adaptation). The learned gate gg has one purpose: ranking the heads. Because TopK is not differentiable, we multiply each selected head's indexer weight by its gate value during training, so that the KL loss can back-propagate to the router, as in standard MoE gating. At inference we keep the multiplication only for train-inference consistency. We revised the description in Sec 3.2, which previously stated that the gate also modulates each head's contribution. Table A shows that removing it changes little: FlashIndexer w/o g differs from FlashIndexer by at most 0.4 points in any setting, with no consistent sign. The gate therefore helps through which heads it selects, not through rescaled scores.

For a MISA baseline with matched adaptation, the head-selection rule stays MISA's, so adaptation can only change the rescaling, not which heads are selected. We trained this variant as requested: "MISA w/ g" in Table A keeps MISA's head selection and multiplies each selected head's wIw^I by a gate trained with the same data, objective, and token budget as FlashIndexer's router. The effect is again unsystematic: MISA w/ g differs from MISA by at most 0.34 points, with no consistent sign, and its average changes by 0.04 points on DSV4-Flash and 0.03 on GLM-5.1. Matched adaptation therefore does not narrow the gap to FlashIndexer.

Q1 (fixed head subset). Figure 1(b) suggests that no fixed subset can cover the heads different inputs rely on, because the union of per-sample top-hh heads keeps growing with the number of samples. To test this directly, we built Static-8: for each layer, we fix 8 heads calibrated on one tenth of the router training data (2K samples). For each calibration sample, we rank the heads by their individual attention recall, as the Oracle in Figure 5(a) does, and score the top 10 heads 1.0, 0.9, ..., 0.1; we accumulate the scores over samples and keep the 8 heads with the highest total. The clean comparison is Static-8 vs FlashIndexer w/o g, since both score with the original weights and differ only in head selection.

Static-8 is the lowest of the five routed variants in all 8 settings. Per-token selection improves on it by 0.2–0.9 points on DSV4-Flash and 0.7–1.1 points on GLM-5.1. Averaged over the three RULER lengths, Static-8 loses 0.55 points to DSA on DSV4-Flash and 1.27 on GLM-5.1, while FlashIndexer w/o g loses 0.06 and 0.35. Individual differences of this size are small, but the direction is the same in every setting, which is what the input-dependent redundancy of Figure 1(b) predicts. Changing the head subset across queries thus provides a consistent benefit over a fixed subset.

Reviewer_22ikOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlashIndexer reduces the cost of the DeepSeek Sparse Attention (DSA) indexer along the head axis. A linear router reads the current token's hidden state and selects h≪HIh \ll H_I indexer heads, and only those heads score the full prefix. Its routing cost does not depend on context length. Only the router is trained (~1B tokens, KL distillation against the frozen full-head indexer). A token-packing kernel batches the active heads of several queries so that fewer heads translate into wall-clock savings. On DeepSeek-V4-Flash and GLM-5.1, using h=8h=8 heads matches full-head DSA on RULER (32K–128K) and LongBench v2. Prefill-only TTFT speedup at 128K is 1.12× and 1.20× respectively, rising to 1.75× when combined with IndexCache.

Critical strengths
  • Simple, practical method with clean, well-attributed evidence. The router adds one d×HId \times H_I projection per token, the model stays frozen, and the method drops into existing DSA models. Accuracy parity holds on two production-scale models. The efficiency analysis follows the speedup from kernel (2.78–3.04×) to indexer layer (~1.5×) to end-to-end (1.12–1.20×), and the numbers are consistent with Amdahl's law, which makes the efficiency claims credible.
  • The routing-quality analysis supports the core claim. A gate computed from the query alone recovers about as much attention mass as a per-query oracle (recall 0.883 vs. 0.873) and beats prefix-dependent ranking (0.785), including MISA's scoring rule without pooling. The ablations explaining why the native indexer weights perform worse than random (A.5) and why load balancing hurts (A.6) are informative. Together, these justify the design choices rather than just reporting final numbers.
Critical weaknesses
  • Incremental over MISA, with modest end-to-end gains. MISA already exploits head-axis sparsity in the DSA indexer. The main difference is routing from the query side instead of the prefix side. The end-to-end benefit is limited: prefill only, 1.12–1.20× TTFT at 128K, little gain at shorter contexts, and decode unchanged. Most of the 1.75× headline comes from IndexCache. The contribution is a solid engineering refinement more than a new idea.
  • The accuracy evaluation can't separate the methods, and the key regime is missing.
    • LongBench v2 Overall scores for DSA, MISA and FlashIndexer differ by ≤0.4 points, with no variance reported.
    • Domain sizes are tiny (Code n=11, Struct n=4), so per-domain differences such as Struct 40 vs. 45 come down to a single example.
    • RULER is largely saturated.
    • "Matches DSA" is supported, but the evidence doesn't favor FlashIndexer over MISA on accuracy. The advantage over MISA therefore rests on latency and a 20-sample probe at 16K.
    • The paper argues that savings grow with context, yet it stops at 128K even though DSV4-Flash targets contexts up to 1M. There are also no long-generation or reasoning workloads.
Questions for authors
  1. What are accuracy and TTFT at 256K–1M on DSV4-Flash? This regime should be the method's main use case.
  2. Is the latency comparison with MISA fair? Does MISA use an equally optimized kernel (e.g., token packing), or the reference implementation?
  3. How does the speedup hold up under realistic serving: batch size > 1 and chunked prefill, which change how many query tokens can be packed together?
  4. How well does the router generalize across domains? On DSV4-Flash it is trained on in-house code; how does it do on reasoning/stem or multi-turn agentic inputs?
4 comments

We thank the reviewer for the precise and constructive critique. Summary of new results: (i) TTFT and per-layer kernel latency at 256K/512K/1M on DSV4-Flash (Table A); (ii) RULER-1M accuracy on DSV4-Flash (Table B); (iii) SWE-Bench-Verified (multi-turn agentic) and AIME26 (reasoning) tasks (Table C); (iv) batch size 1–16 under chunked prefill (Q3); (v) a first decode-side result with speculative verification on DSV4-Flash (W1).

Table A. Prefill TTFT speedup over full-head DSA on DSV4-Flash (bsz=1\mathrm{bsz}=1, chunked prefill 32K).

LengthFlashIndexerMISA
256K1.22×1.22\times1.16×1.16\times
512K1.37×1.37\times1.29×1.29\times
1M1.52×1.52\times1.43×1.43\times

W1 (incremental over MISA; modest gains).

  • Gains are not modest at the lengths DSV4-Flash targets. Standalone, without IndexCache, FlashIndexer reaches 1.52×1.52\times TTFT at 1M on DSV4-Flash. We revised the contribution list to report FlashIndexer and IndexCache separately (FlashIndexer alone 1.20×1.20\times + IndexCache alone 1.61×1.61\times →\rightarrow combined 1.75×1.75\times at 128K on GLM-5.1).
  • The routing signal is a different design point with measurable consequences. MISA routes from block-pooled prefix keys and all HIH_I query heads; FlashIndexer routes from the current hidden state only. (a) Router cost: from 256K to 1M on DSV4-Flash, FlashIndexer's per-layer router grows 2.3$\rightarrow$8.6 ms (linear in LL), MISA's grows 11.6$\rightarrow$68.2 ms (super-linear, as its O(HIL/B)O(H_I L/B) per-query cost implies). (b) Routing quality: at h=8h=8, recall is 0.883 vs 0.785 for MISA, and 0.785 is also the no-pooling limit, so the gap is a property of the prefix-side signal rather than of MISA's approximation (Sec 4.4.1). (c) At 1M, this gap becomes visible in accuracy (Table B).
  • Decode. Query-side routing needs no pooled-key state, so it carries over to speculative verification, where the next_n draft tokens can be packed. On DSV4-Flash's logits kernel (hh: 64→864\rightarrow 8), net per-step savings after the 48 μ\mus router cost are +30 to +284 μ\mus for next_n=8 at $L=$8K–32K and +842 μ\mus for next_n=16 at 32K. We report this as a kernel-level result in the new Appendix A.9 and keep prefill as the scope of the main claims.

W2 (accuracy cannot separate the methods). We agree that LongBench v2 at 128K cannot separate the head-axis methods. On DSV4-Flash, Overall over 5 runs is DSA 65.85±1.6265.85\pm1.62, MISA 65.85±1.1365.85\pm1.13, and FlashIndexer 65.92±1.0765.92\pm1.07, all within run-to-run noise, and per-domain differences on Code (n=11n=11) and Struct (n=4n=4) are not meaningful. At $\leq$128K we therefore claim only that FlashIndexer matches full-head DSA. The 1M regime is where the methods separate:

Table B. RULER-1M on DSV4-Flash. Built from RULER-llama3-1M, keeping sub-tasks whose prompts fit the context limit without truncation, first 25 samples each (10 sub-tasks, 250 samples, single run).

TasknDSAFlashIndexerMISA
niah_single_2250.9600.9600.880
niah_single_3251.0001.0001.000
niah_multikey_1250.9201.0000.920
niah_multiquery250.9600.9600.950
niah_multivalue250.9900.9400.930
vt250.9120.9520.880
cwe250.9400.9080.992
fwe250.6800.6670.520
qa_squad250.7230.7000.683
qa_hotpotqa250.6400.6800.600
Mean2500.8730.8770.836

FlashIndexer matches full-head DSA on average (0.877 vs 0.873); its per-task differences go both ways (+8.0 on multikey_1, $-5.0onmultivalue)andnoneexceedstwoof25examples.MISAfalls3.7pointsbelowDSA,withthelargestdropsonfwe(5.0 on multivalue) and none exceeds two of 25 examples. MISA falls 3.7 points below DSA, with the largest drops on fwe (-16.0),single2(16.0), single_2 (-8.0),andmultivalue(8.0), and multivalue (-$6.0). With 25 samples per sub-task, single-task differences are not meaningful on their own, so we compare the 250-sample averages.

(continued in the next comment)

(continued)

Q1 (256K–1M). See Tables A and B. On DSV4-Flash, FlashIndexer's TTFT speedup over full-head DSA grows from 1.22×1.22\times at 256K to 1.37×1.37\times at 512K and 1.52×1.52\times at 1M, against 1.16×1.16\times, 1.29×1.29\times, and 1.43×1.43\times for MISA (Table A). The difference comes from the router: from 256K to 1M, FlashIndexer's per-layer router cost grows from 2.3 to 8.6 ms, MISA's from 11.6 to 68.2 ms (Appendix A.7). We also report accuracy at 1M (Table B): FlashIndexer scores 0.877, on par with full-head DSA (0.873), while MISA scores 0.836, 4.1 points below FlashIndexer.

Q2 (fairness). FlashIndexer and MISA share every component except the router, including the token-packing logits kernel and top-kk. Fig. 4(b) breaks down the indexer latency by component, and the 8-head logits bars of the two methods are identical. For the router, MISA uses its official implementation, while FlashIndexer's router is a single GEMM, an operation that most inference frameworks already optimize well. The comparison is therefore fair. We state this in the efficiency setup of Sec 4.3.

Q3 (realistic serving). All latency results in the paper and in Table A were measured in SGLang with chunked prefill enabled at a chunk size of 32K, which matches common serving configurations (32K or larger). Token packing operates within a request, so batching does not reduce packing opportunities: at bsz=2,4,8\mathrm{bsz}=2, 4, 8, and 1616 the speedup is the same as at bsz=1\mathrm{bsz}=1. We added the framework and these serving settings to Sec 4.3.

Q4 (router generalization). Although the DSV4-Flash router is trained only on code, FlashIndexer matches full-head DSA on RULER and LongBench v2 in Table 1, which lie outside its training domain. Table C adds multi-turn agentic and reasoning tasks:

Table C. Agentic and reasoning tasks.

ModelModeSWE-Bench-Verified (499 instances)AIME26
DSV4-FlashDSA80.695.42
DSV4-FlashFlashIndexer80.696.46
GLM-5.1DSA75.395.21
GLM-5.1FlashIndexer74.994.98

SWE-Bench-Verified trajectories average 87 agent steps and 56.9K tokens (max 272 steps, 267K tokens), so head routing is applied to long, multi-turn contexts.

We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

I appreciate the authors' reply, but my concerns in Critical Weaknesses still exist. I'm going to keep my score.

Reviewer_AUBQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlashIndexer, a lightweight head-routing mechanism to reduce the indexing overhead in DeepSeek Sparse Attention (DSA) for long-context inference. The key observation is that indexer-head importance is highly input-dependent and unevenly distributed, suggesting that only a small subset of heads is needed for each query token. Based on this, the method uses a router conditioned on the current token’s hidden state to select a small set of active heads, while keeping the original DSA token-selection pipeline unchanged. The router is trained with a distillation objective that matches the behavior of the original indexer, and the system is complemented by a token-packing kernel to convert head sparsity into practical prefill speedup.

Critical strengths

1)The paper introduces a new optimization axis for DSA-style sparse indexing: head-axis sparsity. Prior work mainly reduces indexing cost along the token axis or layer axis, whereas this work shows that indexer-head importance is highly input-dependent and exploits this with per-token head routing. 2)Important heads are selected directly from the current token representation, without accessing the prefix. This keeps the routing overhead independent of context length.

Critical weaknesses

1)The novelty appears limited. Head-axis redundancy is not a new issue in itself, and dynamic head selection in the DSA/indexer setting already has a direct precedent in MISA[1]. Relative to such prior work, the main contribution here seems to be replacing prefix-dependent head routing with a lighter router, which is a meaningful simplification but less of a new methodological paradigm. 2)The realized efficiency gains are bounded. The method only accelerates prefill stage and leaves decode unchanged, so the overall benefit for realistic online generation is inherently limited. Moreover, the paper reports much larger gains at the logits-kernel level than at end-to-end TTFT, suggesting that only part of the algorithmic savings translates into system-level improvements, and that these gains depend substantially on a specialized token-packing kernel. 3)The empirical scope is relatively narrow. The method is validated on only two DSA-based models and primarily on long-context benchmarks, which makes it difficult to assess whether the approach would remain effective across other sparse-attention variants, model scales, or training distributions. 4)The claim of orthogonality is only partially validated. The paper argues that FlashIndexer is orthogonal to token-axis and layer-axis optimizations, but empirically only demonstrates composition with IndexCache. This is enough to show compatibility with one layer-axis method, but not yet sufficient to support a broader claim of robust composability across token-axis and layer-axis optimizations in general.

Note: [1] Zhou, Ruijie, et al. "MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference." arXiv preprint arXiv:2605.07363 (2026).

Questions for authors

1)Could the authors clarify more explicitly what they consider to be the paper’s primary conceptual novelty relative to MISA, beyond replacing prefix-dependent head routing with a lignter design? 2)The method is applied only during prefill, while decode remains unchanged. Could the authors discuss more explicitly how much end-to-end benefit they expect in realistic serving scenarios with longer generations, and whether there is a plausible path to extending the method to decode as well? 3)The paper attributes the speedup partly to the token-packing kernel. Could the authors disentangle how much of the observed gain comes from the core routing idea itself versus the specialized kernel implementation? For example, what is the speedup without token packing, or under a more standard implementation? 4)The paper claims orthogonality to token-axis and layer-axis optimizations, but only demonstrates composition with IndexCache. Could the authors include results combined with a representative token-axis method and discuss whether there are any failure modes or non-trivial interactions when combining these approaches?

3 comments

We thank the reviewer for the careful assessment. Summary of new results: (i) composition of FlashIndexer with the token-axis method HISA on DSV4-Flash (Table A); (ii) a first decode-side result, head routing inside speculative-decoding verification on DSV4-Flash (Table B); (iii) results beyond 128K and on SWE-Bench-Verified and AIME26, which we cite below where relevant.

W1/Q1 (novelty relative to MISA). We agree that head-axis redundancy has been discussed before, and that MISA already selects indexer heads dynamically. To our knowledge, however, no prior work systematically analyzes and visualizes this redundancy in the DSA indexer. Our analysis (Figure 1) does so: per-head attention recall is highly imbalanced within every layer, and the union of each sample's top-hh heads keeps growing as more samples are added, in shallow, middle, and deep layers alike. This shows that the redundancy is input-dependent, which is what motivates per-token routing over a fixed head subset.

We also agree that, methodologically, FlashIndexer differs from MISA mainly in how heads are selected. Our point is that head selection is the step that matters: the selected heads determine the token top-kk set, and through it every downstream attention computation. FlashIndexer improves this step in two ways.

  • More accurate selection. At h=8h=8, the attention-score recall of the routed heads is 0.883 for FlashIndexer vs 0.785 for MISA (Sec 4.4.1). Because 0.785 is also what MISA reaches without pooling, the gap comes from the prefix-side signal itself, not from MISA's approximation. The difference shows up in accuracy at long context: on RULER-1M (DSV4-Flash, 250 samples), FlashIndexer scores 0.877 vs 0.873 for full-head DSA, whereas MISA drops to 0.836.

  • No context-dependent overhead. MISA's router reads pooled prefix keys, so its per-query cost grows with LL; FlashIndexer's router reads only the current token. On DSV4-Flash, the per-layer router cost grows from 2.3 to 8.6 ms between 256K and 1M for FlashIndexer, against 11.6 to 68.2 ms for MISA. MISA also falls below 1×1\times on short inputs (0.84×0.84\times TTFT at 8K, Sec 4.3).

The conceptual finding is that the important indexer heads can be identified from the current token alone, and more accurately than from prefix statistics, so accurate head routing does not have to pay a context-dependent cost.

W2/Q2 (prefill only; decode). First, the prefill gains grow with context and are substantial at the lengths these models target. Without IndexCache, FlashIndexer's TTFT speedup over full-head DSA is 1.22×1.22\times, 1.37×1.37\times, and 1.52×1.52\times at 256K, 512K, and 1M on DSV4-Flash. For long-context requests, prefill is a large share of the request latency, and TTFT is itself a serving objective.

Second, the submitted version applies routing in prefill only, because a single decode query cannot fill the MMA tile (Sec 3). We have since continued to explore decode acceleration, and FlashIndexer combines naturally with speculative decoding: a verification step scores the next_n draft tokens of one request against the same KV cache, so their routed heads can be packed as in prefill. Table B reports logits-kernel latency per verification step on DSV4-Flash (hh: 64→864\rightarrow 8, router cost 48 μ\mus per step).

Table B. Head routing in speculative verification (DSV4-Flash, μ\mus per step).

next_nL64 heads8 headsSavingNet gain
48K80.357.722.6−25-25
416K150.297.652.6+5
432K297.3187.7109.6+62
84K76.351.624.7−23-23
88K137.659.877.8+30
816K276.0108.9167.1+119
832K533.9201.8332.1+284
164K137.652.685.0+37
1632K1151.5261.3890.2+842

Routing pays off once a verification step packs 8 or more draft tokens at L≥L\geq 8K (e.g., next_n=8 in DSpark), and the gain grows with both next_n and LL. These are kernel-level results; we add them as Appendix A.9 and keep prefill as the scope of the main claims.

(continued in the next comment)

(continued)

W2/Q3 (kernel vs. end-to-end gains; role of token packing). Head routing and the kernel play different roles.

  • Routing reduces the computation; the kernel realizes it. With h=8h=8 of HIH_I heads, the indexer logits FLOPs drop by a factor of HI/8H_I/8 (8×8\times on DSV4-Flash, 4×4\times on GLM-5.1). Token packing adds no algorithmic saving of its own; it is the implementation that turns this FLOP reduction into wall-clock time.
  • Without packing there is no speedup. In standard per-token execution (Figure 3(a)), each query token issues its own GEMM over the prefix, so the number of GEMM calls is fixed by the query length and does not change with the head count. With only hh active heads, each call leaves most of the MMA tile empty and the latency stays the same. Packing (Figure 3(b)) merges the active heads of n=Bm/hn=B_m/h queries into one tile, which cuts the number of GEMM calls by nn. MISA uses the same packing kernel in our experiments, so the FlashIndexer–MISA gap reflects the router alone.

How much of the kernel-level gain reaches the end-to-end TTFT depends on the logits kernel's share of TTFT. Over a prefill of length LL, the logits kernel scores every query against its prefix, so its cost grows quadratically with LL, while the rest of the prefill pass, including sparse attention with a fixed top-kk, grows roughly linearly. The logits kernel therefore takes a larger share of TTFT as the context grows, and the kernel optimization yields a larger TTFT speedup: on DSV4-Flash, the TTFT speedup rises from 1.12×1.12\times at 128K to 1.52×1.52\times at 1M (see W2/Q2).

W3 (empirical scope). Our work focuses on DSA, the token-level sparse-attention design that is deployed in production models, including both models we evaluate. Other sparse-attention variants are a worthwhile direction, but they are beyond the scope of this paper; we will state this scope explicitly and plan to explore other sparse structures in future work. Within the DSA family, the two models differ along the dimensions that matter for routing: model size (530 GB vs 1.4 TB of weights), the number of indexer heads (64 vs 32), top-kk (512 vs 2048), and the indexer keys (C4-compressed on DSV4-Flash). On training distribution, the DSV4-Flash router is trained only on code, so every evaluation is out of its training domain. Beyond RULER and LongBench v2, we added a multi-turn agentic benchmark and a reasoning benchmark, and FlashIndexer matches full-head DSA on both:

ModelMethodSWE-Bench-Verified (499 issues)AIME26
DSV4-FlashDSA / FlashIndexer80.6 / 80.695.42 / 96.46
GLM-5.1DSA / FlashIndexer75.3 / 74.995.21 / 94.98

W4/Q4 (orthogonality). We added the combination with the token-axis method HISA (block size 128, top-64 blocks), using FlashIndexer's h=8h=8 routed heads in all indexing stages of HISA.

Table A. Composition with HISA on DSV4-Flash. RULER at 128K (%); TTFT at 128K (speedup over full-head DSA).

MethodRULER-128KTTFT (ms)
DSA93.592302.4 (1.00×1.00\times)
HISA83.972052.1 (1.12×1.12\times)
HISA + FlashIndexer84.412008.4 (1.15×1.15\times)

Two interactions appear:

  • Accuracy is bounded by the weaker component. HISA + FlashIndexer stays within 0.5 points of HISA (84.41 vs 83.97), so head routing adds no further loss, but it cannot recover tokens that HISA's block filter has already discarded.
  • Speedups are sub-additive. HISA compresses along the key axis, which already lowers the logits computation, so the part that head routing removes is smaller relative to the whole. At 128K on DSV4-Flash, adding FlashIndexer to HISA gives 1.02×1.02\times (2052.1 →\rightarrow 2008.4 ms), compared with 1.12×1.12\times when added to full-head DSA (Appendix A.4), so the combination (1.15×1.15\times) is only slightly faster than HISA (1.12×1.12\times) or FlashIndexer (1.12×1.12\times) alone. With IndexCache, FlashIndexer raises the speedup from 1.61×1.61\times to 1.75×1.75\times at 128K on GLM-5.1, because IndexCache skips whole indexer layers and FlashIndexer accelerates the ones that remain.

We will add these results to the appendix and revise the orthogonality statement accordingly: FlashIndexer combines with token- and layer-axis methods without additional accuracy loss, while its speed gains are sub-additive when the other method already shrinks the set of scored tokens.

We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.