acceptodds
Under review as a conference paper at ICLR 2027

MISA:Mixture of Indexer Sparse Attention for Long-Context LLM Inference

Abstract

DeepSeek Sparse Attention (DSA) uses a learned multi-head indexer to score prefix tokens and select the top- tokens for sparse attention, but indexing becomes costly at long context lengths. We propose **MISA** (**M**ixture of **I**ndexer **S**parse **A**ttention), a training-free, drop-in replacement that treats indexer heads as a pool of experts. A lightweight router uses block-level statistics to select a query-dependent subset of active heads for token-level scoring, reducing the per-query indexing cost from to , where . A hierarchical variant, MISA, further re-ranks an enlarged candidate set with the original DSA indexer. On the 13-subtask RULER benchmark across six context lengths from 4K to 128K, MISA achieves an average score of 94.94, second only to DSA at 95.46. End-to-end prefill evaluation shows that MISA reduces mean time to first token by a factor of on the full model at 256K and on a pruned 8-layer model at 512K. These results demonstrate that head-axis routing provides a practical quality–efficiency trade-off for fine-grained sparse attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5WSnOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MISA, an MoE-inspired approach to reducing indexer overhead in DSA. MISA uses dot products between queries and block-pooled keys to select 4 or 8 indexer heads, substantially reducing the indexer’s GEMM cost. It achieves a 1.42× TTFT speedup at a context length of 256K. To mitigate the resulting quality degradation, the paper further introduces a hierarchical variant, MISA†, which first performs coarse candidate filtering with fewer heads and then refines the scores through more precise computation. In the training-free setting, MISA† outperforms IndexCache and HISA.

Critical strengths
  • S1: Addresses a relevant bottleneck. Indexer overhead has become a key bottleneck in long-context sparse attention. MISA’s MoE-inspired head selection offers a well-motivated approach to reducing this overhead.
  • S2: Uses query–key interactions for routing. Compared with selecting heads solely based on whw_h, routing through query–block-pooled-key dot products explicitly accounts for query–key interactions. The reported experiments support the effectiveness of this design.
  • S3: Demonstrates competitive quality. The reported improvements over IndexCache provide evidence for the value of the proposed approach.
Critical weaknesses
  • W1: Absence of fair compassion. MISA can be interpreted as a form of dynamic indexer-head pruning, but the paper lacks a comparison with a fixed subset of indexer heads under the same active-head budget. This baseline would help clarify how much of the benefit comes from dynamic routing rather than simply reducing the number of heads.
Questions for authors
  • Q1: Why do MISA† and MISA incur different Top-K runtimes? Could the authors clarify the number and sizes of the Top-K operations in each method?
  • Q2: Since MISA reduces indexer computation along the head dimension, can it be combined with methods such as IndexCache?
4 comments

We thank the reviewer for the questions on a fixed-head baseline, the number and sizes of Top-K operations, and compatibility with IndexCache.

Answer for W1

The only thing MISA loses is the number of activated heads, which in DSA would have required pooling anyway. The reason DSA maintains multiple heads is primarily to preserve the representational capacity of the indexer. The MoE mechanism naturally meets the need for fewer activations with stronger expressiveness. To verify these claims, we conducted pre-training experiments:

We directly compared head pruning with MISA in terms of training effectiveness. We used a 30B-A3B DeepSeek-V3-like model with 64-head MLA for core attention. From scratch, we trained DSA indexers with 1–32 heads and MISA with 32 heads selecting 1–32 heads. The training data comprised 0.1B tokens with a sequence length of 32K.

HDSAMISAΔ (MISA−DSA)
Warmup144.7176.52+31.81
242.8976.63+33.74
444.1080.76+36.66
844.1778.70+34.52
1674.9177.53+2.62
3275.3776.92+1.54
Sparse149.6886.73+37.05
249.0687.10+38.04
452.5386.14+33.61
852.6987.24+34.55
1686.8988.21+1.32
3286.6585.38-1.27

Results show that DSA with 8 heads suffers a significant performance drop, while MISA maintains good results even when activating only a single head, confirming MISA's representational power. Moreover, MISA 4/32 and 8/32 outperform the 32-head DSA, demonstrating that MISA not only reduces computation but also improves quality.

Answer for Q1

Two factors should be distinguished when interpreting the different Top-K times of MISA and MISA† in the original results: first, the original table combines the Top-K times of both stages of MISA†, whereas MISA performs Top-K only once; second, the first stage of MISA† selects 8192 candidates, whereas MISA selects only 2048 tokens. In its second stage, MISA† rescores these 8192 candidates using all heads and then selects the final 2048 tokens.

To determine whether combining the two stages' times is the main reason for the difference, we conducted additional component-level tests with separate timings for each stage; the tests run the indexer directly on synthetic data with key lengths of 256K/512K and a fixed query length of 8K, and report runtimes in milliseconds. DSA, MISA, and the second stage of MISA† use the same vLLM Top-K kernel, while the first stage of MISA† still uses an external kernel to select 8192 candidates. At 256K, the first- and second-stage Top-K operations of MISA† take 8.26 ms and 0.31 ms, respectively; at 512K, they take 14.71 ms and 0.33 ms. The second stage takes only approximately 3.8% and 2.2% of the first-stage time, respectively, making its overhead negligible by comparison. This indicates that the additional second-stage Top-K operation is not the main source of overhead. These supplementary results therefore support the explanation that the original difference mainly stems from the different first-stage selection sizes: MISA† selects 8192 candidates from the long history, whereas MISA selects only 2048 tokens. We have updated the “Component latency breakdown” section of the revised MISA manuscript with these stage-wise timings, the measurement settings, and the Top-K explanation, and marked the section in blue.

Indexer component runtimes: key length 256K, query length 8K (ms)

componentDSAMISAMISA†
routing-1.121.12
logits35.4112.9813.07
Top-K (stage 1 for MISA†)4.744.438.26
logits (stage 2)--1.97
Top-K (stage 2)--0.31
other0.030.600.54
indexer total40.1919.1325.28

Indexer component runtimes: key length 512K, query length 8K (ms)

componentDSAMISAMISA†
routing-1.281.28
logits75.9926.3726.31
Top-K (stage 1 for MISA†)9.538.6814.71
logits (stage 2)--2.16
Top-K (stage 2)--0.33
other0.030.630.57
indexer total85.5536.9745.36

Answer for Q2

Yes. IndexCache reduces the number of indexer computations by reusing index results across layers, while MISA reduces the cost of each indexer computation along the head dimension, so the two can be combined. Our new GLM-5.2 experiments retain the base model's IndexCache cross-layer reuse mechanism and apply MISA or MISA† on top of it, directly demonstrating the feasibility of this combination. Single-stage MISA achieves an average RULER score of 87.87, below HISA's 90.45, whereas MISA† recovers to 97.08, only 0.49 points below the base model's 97.57. These results support that combining MISA† with IndexCache maintains task quality close to the base model.

The table below reports RULER results on GLM-5.2, where IndexCache denotes the base model and HISA, MISA, and MISA† follow the experimental settings in the paper. Avg. is the equally weighted mean over the six context lengths.

Method4K8K16K32K64K128KAvg.
IndexCache (base model)97.9197.8097.6797.8397.3196.8797.57
HISA98.1498.1896.9791.5582.6275.2190.45
MISA98.0297.3997.1596.1375.5762.9487.87
MISA†98.1797.9597.5497.4696.9694.3997.08
Reviewer_5WSnOpenReview ↗

Response to authors' rebuttal

Thank you for the additional experiments. Q1 is resolved: the stage-wise timings explain the Top-k difference between MISA and MISA†.

On Q2, the GLM-5.2 results show that MISA† stays close to the base model (97.08 versus 97.57) on top of IndexCache. This second-model evidence is what moves me.

Given that MISA† retains near-base quality on a second model and composes with IndexCache, I am raising my rating to 4.

We sincerely thank you for the thoughtful feedback, recognition of our clarifications and additional experiments, and positive reassessment of our work.

Reviewer_Q1LXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MISA reduces the cost of DeepSeek Sparse Attention's token indexer by selecting a small subset of indexer heads for each query. A router estimates head importance from block-pooled indexing keys, after which the selected heads score every prefix token. This preserves token-level selection and leaves the downstream sparse-attention operator unchanged. MISA† adds a second stage that re-scores a larger candidate set with all original indexer heads. Both variants are evaluated on DeepSeek-V3.2 using RULER and needle retrieval, with prefill latency measured on the full model at 256K and an eight-layer slice at 256K and 512K. The two-stage variant retains more of DSA's accuracy, while the single-stage variant provides the larger speedup.

Critical strengths
  1. Head selection offers a sensible way to reduce indexing work while retaining fine-grained token selection. The method reuses the pretrained indexer without additional training and changes a well-defined interface. Appendix E is a useful ablation: selecting heads from block-pooled query–key evidence works much better on needle retrieval than using gating weights or query norms alone. This supports the particular routing signal rather than merely showing that some heads can be dropped.

  2. The speedup is demonstrated beyond an isolated kernel. Table 2 reports full-model TTFT speedups of 1.42× for MISA and 1.27× for MISA† at 256K. The component measurements expose the costs of routing, extra top-k work, and candidate refinement, while Appendix A checks the eight-layer configuration across several concurrency limits. MISA† also reaches 91.19 on RULER at 128K compared with 93.65 for DSA, a useful result for a change requiring no further training.

Critical weaknesses
  1. The main quality and speed results do not establish a common operating point. RULER quality is measured through 128K, whereas full-model TTFT is measured only at 256K. This matters because MISA's RULER gap relative to DSA grows from 0.47 points at 32K to 8.21 points at 128K; the speedup at 256K cannot simply be paired with the average quality across shorter lengths. MISA† narrows the 128K gap to 2.46 points, at additional cost. Moreover, IndexCache with half the indexers retained scores 94.84 on average, close to MISA†'s 94.94, but is absent from the timing comparison. At least one full-model quality-and-latency comparison at the same length and with this baseline is needed to judge the practical tradeoff.

  2. The supplied prefill router appears inconsistent with the causal-prefix definition in Section 4.1. In misa_ops_fp8.py, misa_mqa_logits pools the gathered key array into complete blocks and uses ceil(query_end / block_size) as the routing endpoint. The pooling kernel in block_ops_fp8.py has no per-query endpoint. An earlier query inside a block can therefore select heads using later keys from that block; masking the subsequent token scores does not undo the head-selection decision. I found the same boundary rule in the supplied two-stage BF16 path. Unless request boundaries are aligned or handled elsewhere, pooling can also combine keys from adjacent packed requests. This is a finding from static inspection, not a reproduced GPU failure or evidence of inflated benchmark scores. Nevertheless, it calls into question whether the supplied code implements the stated method and whether prefill chunking can change its behavior.

Questions for authors
  1. Can you provide a full-model comparison of both quality and TTFT at a common context length for DSA, the two MISA variants, and IndexCache? A 128K timing comparison would already help connect the existing results; extending quality evaluation to 256K is another way to resolve this.
  2. How does prefill routing exclude future keys within a partial block and keys from adjacent requests? Please identify the handling I have missed, or show prefix-invariance checks and the quality/timing impact of a correction. Which BF16/FP8 path, chunking configuration, and hardware produced the reported results?
  3. Which RULER subtasks account for the single-stage loss at 128K, and does refinement consistently recover those failures? The per-task results would help determine whether the average hides a specific capability loss relevant to the proposed tradeoff.
5 comments

Answer for Q1

We agree that quality and latency should be compared at the same context length, rather than pairing speedups at longer contexts with average quality at shorter lengths. We added TTFT measurements for the full DeepSeek-V3.2 model at 128K, 256K, 512K, and 1M; all methods use the same test settings with a maximum request concurrency of 4, and parentheses indicate speedups relative to DSA. At 128K, DSA, MISA, HISA, and IndexCache 1/2 take 51.89 s, 47.94 s, 45.68 s, and 42.64 s, respectively. MISA primarily reduces scoring computation while retaining token-level selection, rather than using HISA's block-level filtering. On 128K RULER in the original manuscript, MISA's 85.44 is higher than HISA's 55.79 but lower than IndexCache 1/2's 89.80, while MISA† improves the score to 91.19 through reranking with all heads. These results reflect quality–speed trade-offs. The table below presents the completed TTFT comparison.

Mean TTFT (seconds)128K256K512K1M
MISA47.94 (1.08 ×)112.23 (1.21 ×)299.36 (1.39 ×)925.04 (1.52 ×)
DSA51.89135.34414.911410.09
HISA45.68 (1.14 ×)95.77 (1.41 ×)209.99 (1.98 ×)494.99 (2.85 ×)
IndexCache 1/242.64 (1.22 ×)101.77 (1.33 ×)279.78 (1.48 ×)874.05 (1.61 ×)

To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K and 512K. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model; MISA and MISA† retain this mechanism. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.

The results show that both HISA and MISA exhibit substantial score drops at longer contexts, but MISA† recovers most of MISA's score loss relative to the base model through second-stage reranking with all heads. At 256K and 512K, MISA† scores 79.87 and 77.44, respectively, both substantially higher than HISA's 59.03 and 42.55. In particular, MISA† outperforms HISA by 34.89 percentage points at 512K (77.44 vs 42.55).

Method256K512K
IndexCache (base model)95.4193.18
HISA B12859.0342.55
MISA50.5828.46
MISA†79.8777.44

Answer for Q3

We added results for all 13 RULER subtasks at 128K, with MISA and MISA† using the corrected implementation. The largest losses of single-stage MISA relative to DSA occur on NIAH-MK2, NIAH-MK1, QA-1, and QA-2, with drops of 14, 9, 6, and 5 points, respectively. This confirms that the average score masks more substantial losses on multi-key retrieval and question-answering tasks. MISA†'s second-stage reranking with all heads matches or exceeds MISA on all 13 subtasks: NIAH-MK2 recovers from 86 to 100, matching DSA; CWE improves from 86.90 to 95.10, and QA-1 from 75 to 85, both exceeding the corresponding DSA scores in this table. However, NIAH-MK1 and QA-2 remain 5 and 4 points below DSA, respectively. Thus, reranking consistently improves or maintains the single-stage results in this evaluation, but does not fully recover the losses on every task: it can only rerank candidates retained in the first stage and cannot retrieve tokens that have already been omitted.

SubtaskNative DSA (reused baseline)MISAMISA†
NIAH-S1100.00100.00100.00
NIAH-S2100.0099.00100.00
NIAH-S3100.00100.00100.00
NIAH-MK196.0087.0091.00
NIAH-MK2100.0086.00100.00
NIAH-MK3100.0099.00100.00
NIAH-MQ100.0099.75100.00
NIAH-MV100.0099.50100.00
CWE89.0086.9095.10
FWE94.6795.0095.00
VT99.8099.80100.00
QA-181.0075.0085.00
QA-266.0061.0062.00

Answer for Q2

We acknowledge that the routing boundary handling in the original implementation was flawed and inconsistent with the causal-prefix definition in the paper: an earlier query within a block could use future keys from the same block for head selection, and keys from adjacent requests could also be included when request boundaries were not aligned. Applying a causal mask to token scores afterward cannot undo the resulting head-selection bias. This was an implementation oversight on our part. We have corrected the code and uploaded the updated code implementation in the supplementary materials. All additional experiments during the rebuttal used the corrected implementation. We report component runtimes, end-to-end TTFT, and the completed 128K RULER evaluation below.

For speed, the block-level head-scoring runtime at 256K increases from 0.45 ms to 1.00 ms, an increase of approximately 121.30%. However, the absolute increase is only 0.55 ms, and this component accounts for just 2.41% of the total indexer runtime before the correction. The total indexer runtime increases from 18.68 ms to 19.18 ms, an increase of 0.49 ms (2.65%). In the full-model HTTP test of DeepSeek-V3.2 at 128K, mean TTFT increases from 45.44 s to 47.94 s (a 5.50% increase), while median TTFT increases from 53.11 s to 56.09 s. Thus, the correction does introduce additional overhead, but the increases in total indexer runtime and end-to-end TTFT are both modest in these tests.

End-to-end TTFT comparison at 128K (s):

TTFTMISA (Original)MISA (Corrected)Corrected − original
Mean45.4447.94+2.50 (+5.50%)
Median53.1156.09+2.98

Runtime comparison before and after the correction at 256K (ms):

ComponentDSAMISA (Original)MISA (Corrected)
Routing: block mean pooling—0.060.06
Routing: block-level head scoring—0.451.00
Routing: head selection and Q/weight gathering—0.120.08
Token logits35.6713.0213.01
Token Top-K (vLLM, k=2048)4.724.434.43
Other0.020.600.60
Total runtime40.4118.6819.18

For downstream quality, we re-evaluated DeepSeek-V3.2 on RULER at 128K: the original implementation scores 90.76, and the corrected implementation scores 91.38, an increase of 0.62 percentage points. The additional experiments use a different GPU model from the original manuscript, so we compare the original and corrected implementations within this new evaluation rather than directly combining these scores with those in the manuscript. Together with the timing measurements above, the before-and-after comparisons indicate that the boundary-handling issue in the original implementation had a limited impact on the overall results of these tests: after correction, quality improves slightly, while total component runtime and end-to-end TTFT change only modestly.

Method128K RULER
MISA (Original)90.76
MISA (Corrected)91.38

Thank you for the detailed rebuttal response. Most of my concerns are addressed. While I'm curious:

  1. Though MISA† recovers better than MISA for longer sequence lengths, the score is still substantially lower than the base model, so how would you conclude that this is good enough?
  2. For the speedups disclosed in the answer to Q1, it seems it is more significant as the sequence length increase. So is everyone's implementation optimal? Or do the results have mixed engineering effort with idea contribution?

Thank you again for the substantial evaluations of speedups and score evaluations for longer contexts, additional evaluations at RULER subtasks, and updated code implementation and evaluation results.

In summary, I think this is a good paper, with effort invested in the rebuttal. But I choose to maintain my score, as there were flaws detected, as admitted by the authors. But my attitude toward this paper is positive.

Reviewer_niZyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MISA is a training-free, drop-in change to the DeepSeek Sparse Attention (DSA) indexer that treats the HI=64H_I=64 indexer heads as experts. For each query, a router scores every head against block-pooled indexing keys (B=1024B=1024), keeps the top h=8h=8 heads, and only those heads score every prefix token. This cuts the per-query cost from O(HIL)O(H_I L) to O(hL+HIM)O(hL + H_I M). A two-stage variant, MISA†, uses the routed heads to pick k′=8192k'=8192 candidates and then re-ranks them with all 64 heads to keep k=2048k=2048. On DeepSeek-V3.2:

  • RULER, averaged over 4K–128K: MISA† scores 94.94 and MISA 93.73, vs. 95.62 for DSA and 82.54 for HISA.
  • NIAH: MISA† matches DSA.
  • Mean TTFT: 1.42× faster for MISA on the full model at 256K, and 1.57× on a pruned 8-layer model at 512K.
Critical strengths
  • Training-free, with real end-to-end speedups at long context. MISA needs no new parameters or fine-tuning and keeps the downstream sparse-attention interface unchanged, so it can be deployed directly on existing DSA models. The speedups are measured end to end on the full DeepSeek-V3.2 (1.27–1.42× at 256K), not only at the kernel level. The component breakdown (Table 3) and the concurrency sweep make clear where the gains come from. Practical, measurable deployment value is the main argument for acceptance.
Critical weaknesses
  • Quality degrades with length, and quality and speed are measured at different lengths.
    • At 128K, single-stage MISA drops 8.2 RULER points (85.44 vs. 93.65), and MISA† drops 2.5 (91.19).
    • The gap grows steadily with context length.
    • The headline 1.42× speedup belongs to the variant with the largest quality loss. MISA† gives only 1.27× on the full model.
    • All speedups are measured at 256K–512K, but quality is only measured up to 128K. Accuracy is therefore unknown in exactly the regime the efficiency claims target, and the trend suggests it is worse there.
  • The evaluation is too narrow to support the quality–efficiency claim.
    • Quality is measured only on synthetic retrieval (RULER and NIAH) with a single model. There are no realistic long-context benchmarks (e.g., LongBench v2) and no reasoning or long-horizon agentic workloads.
    • All ablations are judged by eye from mostly saturated NIAH heatmaps.
    • The efficiency tables compare only against DSA, while IndexCache½ matches MISA† on quality (94.84 vs. 94.94). Without TTFT for IndexCache and HISA, there is no quality–latency Pareto comparison.
    • The router-block-size sweep shows accuracy barely changes even at B=LB=L, i.e., with a single global pooled key. This suggests the prefix-dependent router may be unnecessary, which undercuts the claimed motivation that routing must consult the prefix (Appendix E).
    • The key observation that "relevant heads change slowly along the prefix" is never measured directly.
Questions for authors
  1. What are RULER and NIAH scores for MISA and MISA† at 256K and 512K, the lengths where speedups are reported?
  2. What are mean TTFT values for HISA and IndexCache under the same setup, so that methods can be compared on a quality–latency Pareto?
  3. Given that B=LB=L works nearly as well, how does a router that reads only the query (no prefix) compare? Is the prefix-dependent design actually needed?
  4. Why does Top-k time increase for MISA (1.18s → 1.76s at 256K) when the final kk is unchanged?
  5. How does MISA perform on non-synthetic benchmarks (LongBench v2) and agentic benchmarks, and on a second DSA-style model (e.g., GLM-5 or DeepSeek-V4)?
9 comments

We thank the reviewer for the detailed assessment of long-context quality, quality–latency trade-offs, the role of prefix-dependent routing, Top-K costs, and evaluation breadth.

Answer for Q1

To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K and 512K. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model; MISA and MISA† retain this mechanism. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.

The results show that both HISA and MISA exhibit substantial score drops at longer contexts, but MISA† recovers most of MISA's score loss relative to the base model through second-stage reranking with all heads. At 256K and 512K, MISA† scores 79.87 and 77.44, respectively, both substantially higher than HISA's 59.03 and 42.55. In particular, MISA† outperforms HISA by 34.89 at 512K (77.44 vs 42.55).

Method256K512K
IndexCache (base model)95.4193.18
HISA B12859.0342.55
MISA50.5828.46
MISA†79.8777.44

Answer for Q2

We added a comparison of mean TTFT for DSA, MISA, HISA, and IndexCache 1/2 on the full DeepSeek-V3.2 model under the same test settings, with a maximum request concurrency of 4. Parentheses in the table indicate speedups relative to DSA, calculated by dividing DSA's TTFT by that of each method. MISA achieves 1.08–1.52 × speedups from 128K to 1M, but is slower than HISA and IndexCache 1/2. MISA primarily reduces scoring computation along the head dimension and does not target index-cache reduction. Unlike HISA, which first filters blocks, MISA retains token-level selection, using finer-grained scoring to preserve quality. On 128K RULER in the original manuscript, MISA scores 85.44, higher than HISA's 55.79 but lower than IndexCache 1/2's 89.80; MISA† improves the score to 91.19 through second-stage reranking with all heads. Thus, the available results show that the MISA family offers different quality–speed trade-offs, rather than outperforming all baselines on both metrics.

Mean TTFT (seconds)128K256K512K1M
MISA47.94 (1.08 ×)112.23 (1.21 ×)299.36 (1.39 ×)925.04 (1.52 ×)
DSA51.89135.34414.911410.09
HISA45.68 (1.14 ×)95.77 (1.41 ×)209.99 (1.98 ×)494.99 (2.85 ×)
IndexCache 1/242.64 (1.22 ×)101.77 (1.33 ×)279.78 (1.48 ×)874.05 (1.61 ×)

Answer for Q3

B=L still uses prefix information and is not equivalent to query-only routing: it pools the prefix into a single global key and scores heads by matching the query against that key. This ablates the spatial resolution of prefix information, not whether the prefix is used. Figure 7 in Appendix E already includes a prefix-free comparison: Figure 7(b) scores heads using only the L2 norm of each query head and shows widespread retrieval failures in the NIAH heatmaps at 64K and 128K, whereas the prefix-pooled scoring in Figure 7(c) recovers DSA's retrieval accuracy. Thus, the block-size experiment shows that head selection can tolerate a coarse prefix summary; it does not imply that prefix information can be removed altogether. The routing-score ablation demonstrates that retaining query-to-prefix matching information remains necessary.

Answer for Q4

Although DSA and MISA both select 2048 tokens in the original results, they use different Top-K implementations: DSA uses the kernel provided by vLLM, whereas MISA uses a kernel ported from SGLang. Thus, the difference between 1.18 s and 1.76 s at 256K includes implementation effects and does not imply that MISA inherently incurs additional Top-K overhead at the same selection budget. To control for this factor, we conducted additional component-level tests using the same vLLM Top-K kernel as DSA for MISA and the second stage of MISA†. The first stage of MISA† selects 8192 candidates and cannot directly use this vLLM Top-K kernel, so it still uses an external Top-K kernel. The measurement settings also differ: the original results were obtained from profiling end-to-end TTFT experiments and report cumulative component time per sample, whereas the new tests run the indexer directly on synthetic data with key lengths of 256K/512K and a fixed query length of 8K, reporting component times in milliseconds.

In the new tests, the Top-K times for DSA and MISA are 4.74 ms and 4.43 ms at 256K, and 9.53 ms and 8.68 ms at 512K, respectively. With the same implementation, their Top-K times are similar, with the remaining small differences arising from kernel scheduling, and MISA no longer exhibits the additional overhead seen in the original table. The main gains come from logits computation, which decreases from 35.41 ms to 12.98 ms at 256K and from 75.99 ms to 26.37 ms at 512K; the corresponding total indexer times decrease from 40.19/85.55 ms to 19.13/36.97 ms. We have updated the “Component latency breakdown” section of the revised MISA manuscript with the component comparison using the same Top-K implementation for DSA and MISA, separate timings for both stages of MISA†, and the measurement settings, and marked the section in blue.

Indexer component runtimes: key length 256K, query length 8K (ms)

componentDSAMISAMISA†
routing-1.121.12
logits35.4112.9813.07
Top-K (stage 1 for MISA†)4.744.438.26
logits (stage 2)--1.97
Top-K (stage 2)--0.31
other0.030.600.54
indexer total40.1919.1325.28

Indexer component runtimes: key length 512K, query length 8K (ms)

componentDSAMISAMISA†
routing-1.281.28
logits75.9926.3726.31
Top-K (stage 1 for MISA†)9.538.6814.71
logits (stage 2)--2.16
Top-K (stage 2)--0.33
other0.030.630.57
indexer total85.5536.9745.36

Answer for Q5

We added experiments on LongBench using DeepSeek-V3.2, extending the evaluation beyond synthetic retrieval to real-world long-context tasks. As shown in the table, MISA† achieves the highest average score, 50.95, among the lossy acceleration methods compared in this evaluation, only 0.10 points below DSA's 51.05, closely matching DSA's quality. Single-stage MISA scores 50.85, also exceeding the 50.78 achieved by both HISA and IndexCache 1/4, and trailing IndexCache 1/2's 50.93 by just 0.08 points. These results provide quality evidence beyond synthetic retrieval, showing that MISA remains competitive on real-world long-context tasks, while second-stage reranking with all heads further narrows the gap to DSA.

MethodSQAMQASumFSSynCodeAvg.
DSA50.8952.6622.1162.2469.8348.5651.05
IndexCache 1/249.9852.2422.2162.2670.1748.7450.93
IndexCache 1/449.7251.8122.1762.2570.0048.7450.78
HISA49.1751.9622.1361.6270.8348.9950.78
MISA50.8351.6222.3161.7770.0048.5450.85
MISA†50.9152.2722.0462.1069.8348.4550.95

In addition, we added RULER experiments on a second model, GLM-5.2, at 4K–128K. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model. MISA and MISA† retain this mechanism, and HISA, MISA, and MISA† follow the experimental settings in the paper. Single-stage MISA achieves an average score of 87.87, below HISA's 90.45, whereas MISA† reaches 97.08. This is only 0.49 points below the base model's 97.57 and 6.63 points above HISA. These results show that second-stage reranking with all heads is particularly important for this model. The LongBench and GLM-5.2 experiments add evidence on non-synthetic tasks and cross-model generalization, respectively.

The table below reports RULER results on GLM-5.2. Avg. is the equally weighted mean over the six context lengths, rounded to two decimal places.

Method4K8K16K32K64K128KAvg.
IndexCache (base model)97.9197.8097.6797.8397.3196.8797.57
HISA98.1498.1896.9791.5582.6275.2190.45
MISA98.0297.3997.1596.1375.5762.9487.87
MISA†98.1797.9597.5497.4696.9694.3997.08

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.