acceptodds
Under review as a conference paper at ICLR 2027

Linear Indexer: Understanding and Accelerating DeepSeek Sparse Attention

Abstract

DeepSeek's Lightning Indexer scores every query against all visible keys, incurring quadratic prefill cost. In our DeepSeek-V4-Flash profile at 256K context, it accounts for % of prefill GPU busy time. Surprisingly, we find that removing its head-wise ReLUs largely preserves retrieval. This observation yields **Linear Indexer**, a training-free method that folds the 64 weighted query heads exactly into one before the global scan, reducing 64 query–key dot products per key to one. The resulting linear score directly selects keys for attention; for higher retrieval recall, the original scorer can rerank an enlarged linear shortlist. Our analysis links the high recall to learned coupling between stable query–key signs and head weights, a pattern observed across domains, context lengths, checkpoints, and Indexer architectures. On V4-Flash, Linear Indexer maintains performance comparable to the original model on LongBench-v2 and RULER at context lengths up to 256K. At 256K on NVIDIA H20 GPUs, direct mode achieves a indexer speedup and a speedup in end-to-end prefill time to first token (TTFT).

Primary area:
foundation or frontier models, including LLMs
Keywords:
DeepSeek Sparse Attention, Compressed Sparse Attention, Lightning Indexer, long-context inference, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PbE1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Linear Indexer, which removes the ReLU nonlinearity and aggregates the DSA query heads before computation to reduce the computational cost of the DSA indexer. The Linear Indexer serves as a filtering mechanism for the subsequent DSA computation by providing a filtered set of KV indices. The authors also provide interesting observations about the DSA indexer that justify its linearization and suggest that the Linear Indexer can serve as a good proxy for the original DSA indexer.

The proposed Linear Indexer achieves a significant speedup over the DSA indexer while largely preserving downstream task accuracy. Although the proposed method demonstrates substantial efficiency gains while maintaining accuracy, the current manuscript does not provide sufficient direct comparisons against other baseline methods mentioned in the paper.

Critical strengths
  1. Simple and performant design: The proposed method exploits the sign stability of the learned DSA indexer and introduces a simple algebraic simplification without requiring additional training. Its direct mode achieves a 10.4× indexer speedup and a 1.39× prefill speedup at a 256K context length. Negligible or no accuracy degradation is observed on the evaluated RULER and LongBench v2 benchmarks.

  2. Design supported by empirical observations: The authors provide interesting observations about the behavior of the DSA indexer, particularly the finding that certain DSA indexer outputs exhibit consistent signs across different inputs. Based on these observations, the authors provide empirical justification for the proposed Linear Indexer.

Critical weaknesses
  1. Incomplete comparison against baseline methods: Although the paper directly compares the proposed method against MISA and HISA for KV retrieval, it does not provide comparisons of downstream task accuracy, indexer speedup, or end-to-end prefill speedup against these methods.

  2. Limited decoding latency improvement: The demonstrated efficiency gains of the Linear Indexer are primarily focused on prefill, and the authors acknowledge this limitation. However, this limits its practical benefits for workloads where decoding accounts for a substantial portion of the total latency.

Questions for authors
  1. Is the Linear Indexer used during decoding? Given the design of the Linear Indexer, it is reasonable to expect that decoding latency would not improve significantly. However, if the Linear Indexer is also adopted during decoding, can we expect it to preserve generation accuracy?

  2. Applicability to QSA: In the analysis in Section 3, the authors argue that sign dominance is a learned property of the DSA indexer and that the input-dependent weight term further reinforces this sign consistency, making the output scores predominantly either positive or negative. Considering that Qwen Sparse Attention (QSA) shares many design similarities with DSA, would similar observations hold for QSA, and could the Linear Indexer also be applied to QSA?

5 comments
AuthorsOpenReview ↗

Response to Reviewer PbE1 — W1: Downstream quality, indexer latency, and end-to-end comparisons with MISA and HISA

We thank the reviewer for motivating these additional comparisons. The downstream-quality and TTFT results are now included in Sections 5.4–5.5, with experimental settings in Section 5.1 and Appendix G, and detailed downstream results in Appendix H.4.

Our added DeepSeek-V3.2 experiments show that direct Linear Indexer achieves comparable downstream quality to Exact DSA on LongBench-v2 and RULER, higher aggregate scores than HISA/MISA, and the lowest TTFT at all three tested context lengths. At 128K, it achieves a 1.73×1.73\times TTFT speedup over Exact DSA. These results complement the indexer latency comparisons already reported in Table 10 of the original submission (Table 18 in the revised manuscript).

We evaluate DeepSeek-V3.2, integrating the released HISA/MISA kernels into a common inference framework. All methods retain the native final K=2048K=2048 budget and replace indexing only during prefill; decoding uses Exact DSA. Direct Linear Indexer operates as a training-free, drop-in scorer replacement, without refinement or layer-specific budget tuning.

Downstream quality. Since the DeepSeek-V3.2 checkpoint specifies a maximum context length of 160K tokens, all methods are evaluated on the same 316 LongBench-v2 examples with input lengths up to 128K. RULER is evaluated at 128K on the same 50 examples for each of its 13 tasks, yielding 650 examples per method.

MethodLongBench-v2 correct / totalLongBench-v2 accuracyRULER mean score at 128K
Exact DSA152 / 31648.10%88.92%
HISA, B=8192B=8192133 / 31642.09%59.17%
MISA, h=8h=8134 / 31642.41%81.37%
Linear direct, B=2048B=2048153 / 31648.42%88.19%

Indexer latency. On the 64K V3.2 capture, direct Linear Indexer takes 2.51 ms per layer, compared with 18.26 ms for Exact DSA, 11.45 ms for HISA (B=8192B=8192), and 10.14 ms for MISA (h=8h=8). The corresponding speedups over Exact DSA are 7.26×7.26\times, 1.60×1.60\times, and 1.80×1.80\times, respectively. These measurements include final Top-2048 selection; the full protocol and budget sweep are provided in Appendix H.2.

End-to-end prefill latency. We measure served TTFT on eight NVIDIA H20 GPUs with TP8, a 2,048-token prefill chunk, and piecewise CUDA graphs. Inputs and serving settings are matched across methods. Results are medians of three timing trials after two warmups.

Method32K TTFT64K TTFT128K TTFT128K speedup
Exact DSA10.45 s25.27 s69.49 s1.00×1.00\times
HISA, B=8192B=819211.03 s22.48 s45.70 s1.52×1.52\times
MISA, h=8h=89.79 s21.76 s52.18 s1.33×1.33\times
Linear direct, B=2048B=20488.80 s18.34 s40.08 s1.73×1.73\times
AuthorsOpenReview ↗

Response to Reviewer PbE1 — W2: Practical benefits beyond prefill latency

We thank the reviewer for highlighting this limitation. Linear Indexer achieves a 1.38×1.38\times speedup in end to end latency at 256K context under four concurrent requests, including 512-token generation. This measures the time to complete generation, rather than TTFT alone.

We use the same serving settings as in the paper, with four concurrent requests, each generating 512 tokens. Under continuous batching and chunked prefill, decoding tokens can be processed in the same execution batches as other requests’ prefill chunks, allowing prefill acceleration to benefit complete-request latency.

Input contextExactDirect Linear Indexer (B=512B=512)E2E Speedup
64K27.98 s25.60 s1.09×
128K60.18 s50.23 s1.20×
256K149.81 s108.88 s1.38×

The results demonstrate benefits for concurrent long-input workloads, while gains remain workload-dependent and may be smaller when decoding dominates total latency.

Complementarity with decoding acceleration. Head folding reduces scoring computation and is complementary to key compression and candidate restriction, as illustrated by its applicability to DSA, CSA, and CSA2. It can therefore be combined with methods targeting memory traffic during decoding.

Implications for training. Our findings also motivate training indexers directly with the folded linear score. Indexer training processes many query positions over long sequences, making query–key scoring a natural target for acceleration. The same factorization reduces scoring arithmetic in both forward and backward passes, offering a route to lower long-context indexer-training cost. We leave native training and its quality–efficiency tradeoff to future work.

AuthorsOpenReview ↗

Response to Reviewer PbE1 — Q1: Use of Linear Indexer during decoding and its impact on generation quality

We thank the reviewer for highlighting this distinction. In our V4 evaluations, Linear Indexer is enabled for both prefill and decoding. Under this configuration, it maintains downstream quality comparable to Exact CSA on LongBench-v2 and RULER through 256K context.

In our newly added V3.2 comparisons, Linear Indexer is applied only during prefill, while decoding retains Exact DSA. It likewise achieves comparable downstream quality: 48.42% versus 48.10% on LongBench-v2, and 88.19% versus 88.92% on RULER at 128K. A supplementary V3.2 run enabling replacement during both prefill and decoding also matches Exact DSA’s aggregate LongBench-v2 accuracy at 48.10% (152/316).

These results are encouraging for generation quality in the evaluated settings.

Reviewer_MSQgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Linear Indexer, a training-free approximation to DeepSeek’s Lightning Indexer. Removing the head-wise ReLUs allows 64 weighted query heads to be folded into one before scanning the keys. An optional refinement stage reranks a larger shortlist using the original scorer. The paper connects its effectiveness to learned activation signs and head weights.

Critical strengths
  1. Simple and useful idea. Removing ReLU enables a substantial reduction in scoring computation without retraining.
  2. Retrieval, attention fidelity, downstream quality, and end-to-end latency measurements provide a convincing evaluation.
  3. Weight permutation and sign randomization controls support the importance of learned head–weight pairing.
Critical weaknesses
  1. Lower scoring cost does not always improve latency. On V3.2 at 64K, refinement with B=8192 takes 18.57 ms, versus 18.26 ms for exact DSA. Comparisons with competing methods lack matched-quality end-to-end measurements.
  2. Cross-model validation remains incomplete. Downstream quality and served TTFT are evaluated only on V4-Flash.
  3. The mechanism explanation is partial. Approximately 56% of V4 heads have mixed activation signs, but their contribution to retrieval errors is not clearly isolated. The proposed statistics also do not explain the difficult V4.1 layers.
  4. RULER is largely saturated, while the direct-mode LongBench-v2 confidence interval still allows a 2.7 percentage-point accuracy drop.
Questions for authors
  1. Can you compare Linear Indexer, HISA, and MISA at matched downstream quality, using actual latency and TTFT under the same serving configuration?
  2. What happens when Linear Indexer replaces both the candidate-source and reindex layers in V4.1? Does refinement resolve the low recall at L24?
  3. How much retrieval error comes from mixed-sign heads? Can the proposed mechanism statistics predict which layers need refinement?
  4. Can you report LongBench-v2 quality differences by context length and task category, with confidence intervals?
10 comments
AuthorsOpenReview ↗

Response to Reviewer MSQg— W1 Part1: Scoring cost versus measured latency

We thank the reviewer for highlighting the distinction between reduced scoring cost and actual latency, and for prompting us to clarify the selection and refinement overhead.

The reported B=8192B=8192 latency reflects additional proposal-selection and reranking costs, which can offset the savings in global scoring.

Direct mode uses vLLM’s optimized top_k_per_row_prefill kernel. For the V3.2 refinement measurements, first-stage proposal selection uses torch.topk(..., sorted=False), following the released two-stage MISA implementation. At B=8192/16384B=8192/16384, this also avoids exceeding the vLLM kernel’s default per-block shared-memory allowance. Nonlinear reranking is followed by final Top-2048 selection using the optimized vLLM kernel.

Head folding is independent of the selection backend, making optimized large-budget selection a complementary route to further acceleration. Our normalized-compute comparison measures scoring arithmetic, while the latency measurements include these selection and reranking costs.

AuthorsOpenReview ↗

Response to Reviewer MSQg— W1 Part2-W2 / Q1: Cross-model validation and end-to-end quality–latency comparisons

We thank the reviewer for motivating this end-to-end comparison. The new TTFT and quality results are now included in Sections 5.4–5.5, with experimental settings in Section 5.1 and Appendix G, and detailed downstream results in Appendix H.4.

Our added DeepSeek-V3.2 experiments show that direct Linear Indexer achieves comparable downstream quality to Exact DSA on LongBench-v2 (48.42% vs. 48.10%) and RULER (88.19% vs. 88.92%), together with the lowest TTFT among the tested configurations and a 1.73×1.73\times speedup at 128K context. These results use a training-free, drop-in scorer replacement, without refinement or layer-specific budget tuning.

End-to-end comparison on DeepSeek-V3.2. We compare Exact DSA, HISA, MISA, and direct Linear Indexer on DeepSeek-V3.2, reusing the released HISA/MISA kernels within a common inference framework. All methods retain the native final K=2048K=2048 budget. The replacements operate only during prefill; decoding uses Exact DSA.

All methods are evaluated on the same 316 LongBench-v2 examples with input lengths up to 128K, and on the same 650 RULER examples at 128K, covering 13 tasks with 50 examples each.

MethodLongBench-v2 correct / totalLongBench-v2 accuracyRULER mean score at 128K
Exact DSA152 / 31648.10%88.92%
HISA, B=8192B=8192133 / 31642.09%59.17%
MISA, h=8h=8134 / 31642.41%81.37%
Linear direct, B=2048B=2048153 / 31648.42%88.19%

We measure served TTFT on eight NVIDIA H20 GPUs with TP8, a 2,048-token prefill chunk, and piecewise CUDA graphs. Inputs and serving settings are matched across methods; timings are medians of three trials after two warmups.

Method32K TTFT64K TTFT128K TTFT128K speedup
Exact DSA10.45 s25.27 s69.49 s1.00×1.00\times
HISA, B=8192B=819211.03 s22.48 s45.70 s1.52×1.52\times
MISA, h=8h=89.79 s21.76 s52.18 s1.33×1.33\times
Linear direct, B=2048B=20488.80 s18.34 s40.08 s1.73×1.73\times

Thus, direct Linear Indexer provides measured acceleration at quality comparable to Exact DSA, and achieves both higher aggregate quality and lower TTFT than the tested HISA and MISA configurations.

AuthorsOpenReview ↗

Response to Reviewer MSQg — W3 / Q3 Part1: Contribution of mixed-sign heads

We thank the reviewer for motivating this analysis. It is now included in Appendix D.3, with a corresponding reference in Section 3.5.

Our selective interventions show that mixed-sign (MID) heads contribute to retrieval errors: removing their ReLUs alone reduces Top-512 recall by 11.99 percentage points. We focused on PD/ND heads to make the explanation clearer. Removing ReLU from PD heads has little effect, while removing it from ND heads produces the largest single-group recall loss.

On the 12-domain, 64K V4 captures, we sample 32 queries per file across all 21 CSA layers. Holding captured queries, keys, and weights fixed, we remove ReLU from one head group at a time while leaving the remaining heads unchanged:

Heads with ReLU removedTop-512 recall against Exact Indexer
None100.00%
PD only99.18%
ND only83.70%
MID only88.01%
All heads80.95%

These are group-wise intervention effects, not an additive decomposition of the total recall loss, since the groups jointly determine the ranking.

The learned weight–sign pairing also extends to MID heads: 64.37% have mean weights opposing their majority activation sign, compared with 54.34% under random within-group pairing. Thus, the pairing pattern is not restricted to the predominantly positive and negative heads studied in the main analysis.

Reviewer_x76kOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a strong "linear retrieval redundancy" in trained Lightning Indexers used by DeepSeek sparse attention (DSA, CSA, CSA2): removing the head-wise ReLU preserves most of the original Top-K set. Based on this, the authors propose the Linear Indexer, a training-free method that drops ReLU and folds 64 weighted query heads into a single query vector before the global scan, reducing per-key scoring from O(Hd)O(Hd) to O(d)O(d). It offers a direct mode (B=KB=K) and a refined mode (B>KB>K, linear proposal + exact re-scoring). Experiments on DeepSeek-V4-Flash, V3.2, and V4.1-Flash show high candidate recall, preserved attention mass and downstream quality (LongBench-v2/RULER up to 256K), indexer speedups up to 10.4×, and prefill TTFT speedups of 1.33–1.39× at 256K. A mechanistic analysis—toy model, cone-shaped key geometry, sign-stable heads, opposite weight signs, correlated PD/ND contributions—supports the finding via weight-permutation and random-query interventions.

Critical strengths
  1. Counter-intuitive, well-validated finding. Removing head-wise ReLU from a trained Lightning Indexer largely preserves retrieval across three indexer designs, multiple checkpoints, domains, and 4K–256K contexts. Direct linear recall (0.809 V4 CSA, 0.732 V3.2 DSA, 0.691 V4.1 CSA2) is consistent and non-trivial.

  2. Mechanistic analysis is the strongest contribution. The three-layer explanation—toy-model preference for the signed linear component, cone-shaped keys inducing PD/ND heads (23.1%/21.1%, 99.7% sign retention), learned weights opposing dominant signs (Awp=0.818A_{\mathrm{wp}}=0.818) with positively correlated PD/ND contributions (r=0.46r=0.46, 28% Top-512 overlap vs ~4.1% chance)—is coherent and supported by the right interventions. It goes well beyond "it works empirically."

  3. Honest evaluation. Reports recall, attention mass, output cosine, and downstream benchmarks with paired bootstrap CIs; compares against MISA/HISA under matched compute; documents limitations (post hoc explanation, DeepSeek-only, no decoding speedup, unoptimized kernels).

Critical weaknesses
  1. Conceptual overlap with linear attention is not addressed. The identity ∑hwh(qh⊤k)=(∑hwhqh)⊤k\sum_h w_h(q_h^\top k)=(\sum_h w_h q_h)^\top k is the same "commute sum and dot product" trick underlying linear/kernelized attention (Performer, Linear Transformer, RWKV, Kimi-style hybrids). The paper cites none of this and does not distinguish itself.

  2. Long-context coverage is narrow for token-level DSA. Mechanistic and retrieval analyses are at 64K (pilot) and 4K–64K (Diverse); 256K is only used for downstream RULER/LongBench-v2 and TTFT. Crucially, V3.2 token-level DSA is evaluated only up to 64K. Since V3.2 selects K=2048K=2048 from ~64K token-level candidates (vs. V4's 512 from ~16K compressed entries), and Table 3 shows direct recall degrading with length (0.933 at 4K → 0.809 at 64K), the linear-redundancy claim at 128K–256K for token-level DSA is not directly evidenced. The "up to 256K" framing in the contributions overstates what is shown for DSA. Either add V3.2 results at 128K/256K or explicitly scope the claim.

  3. All experiments are conducted on DeepSeek-family models (V4-Flash, V3.2, V4.1-Flash). Whether this method can be adapted to a more general setting, which is unknown.

Questions for authors

Do you have evidence on when during pretraining PD/ND structure and opposite weight-sign alignment emerge? Is there an early signature predicting whether linear retrieval will work for a new indexer?

7 comments
AuthorsOpenReview ↗

Response to Reviewer x76k — W1: Connection to Linear Attention

We thank the reviewer for highlighting the algebraic connection. Reassociation is well established in linear/kernelized attention. We have added these references, and clarified the distinction in Appendix A (Related Work).

Linear attention aggregates historical key–value contributions across tokens within each head; Linear Indexer instead aggregates weighted query vectors across heads, retaining a separate score for every candidate key. The former computes attention outputs from an aggregated state, whereas the latter selects Top-KK key indices and leaves the main attention operator unchanged. Both nevertheless exploit the associativity of matrix multiplication.

AspectStandard kernelized linear attentionLinear Indexer
Aggregation axisAcross tokens, within each headAcross heads, for each query
Target moduleMain attentionSparse-attention indexer
Historical informationKV contributions combined into a state per headIndividual keys retained
Top-KK selectionNot directly available from the aggregated state aloneExplicit per-key scores enable selection
Prefill scalingLinear in context lengthFull-scan scoring remains quadratic

In Linear Indexer, “linear” refers to the score being linear in the key after removing ReLU, rather than linear complexity in context length.

AuthorsOpenReview ↗

Response to Reviewer x76k — W2: Long-context coverage for token-level DSA.

We have added 128K retrieval analysis and downstream-quality and TTFT evaluations for token-level DSA, extending the evidence beyond the original 64K range. Mean direct recall decreases only modestly, from 73.23% at 64K to 72.30% at 128K, while refinement substantially improves recall. On DeepSeek-V3.2, direct Linear Indexer achieves comparable LongBench-v2 and RULER performance to Exact DSA and a 1.73×1.73\times TTFT speedup at 128K context. These results are now included in the revised manuscript, and the 256K results are explicitly scoped to DeepSeek-V4-Flash.

Retrieval and mechanism analysis at 128K. Since the DeepSeek-V3.2 checkpoint specifies a maximum context length of 160K tokens, we extend the capture analysis to 128K, covering all 61 indexer layers and 12 domains at the native K=2048K=2048 budget.

Following the original appendix protocol, we analyze 32 sampled queries per capture file, yielding 23,424 query–layer records across 732 files. Mean direct recall is 72.30%, compared with 73.23% in the original 64K captures. Learned weight–polarity anti-alignment and PD–ND co-ranking remain strong at 0.883 and 0.549, respectively.

Mechanism statistic64K128K
RdirectR_{\mathrm{direct}}0.73230.7230
CconeC_{\mathrm{cone}}0.999970.99994
ρa,p\rho_{a,p}0.87510.7135
PPD/NDP_{\mathrm{PD/ND}}0.62000.5710
AwpA_{\mathrm{wp}}0.83520.8834
ρPD,ND\rho_{\mathrm{PD,ND}}+0.4810+0.5494

Retrieval at long visible prefixes. Because a 128K capture also contains queries with shorter visible prefixes, we additionally report retrieval results restricted to genuinely long prefixes:

Linear Indexer settingVisible prefix >64KVisible prefix >96K
Direct, B=2048B=204868.23%67.70%
Refined, B=4096B=409686.19%85.58%
Refined, B=8192B=819294.56%94.08%
Refined, B=16384B=1638497.93%97.62%

These subsets contain 5,856 and 2,928 query–layer records, respectively. These results demonstrate high-recall linear proposals for token-level DSA beyond 64K: refinement substantially improves average recall, which declines only modestly as visible prefixes grow longer.

LongBench-v2 and RULER quality. We additionally evaluate DeepSeek-V3.2 with direct Linear Indexer, B=K=2048B=K=2048. All methods use the same 316 LongBench-v2 examples with input lengths up to 128K. For RULER, we evaluate all 13 tasks at 128K with 50 shared examples per task, yielding 650 examples per method. Each replacement is applied only during prefill; decoding retains Exact DSA.

MethodLongBench-v2 correct / totalLongBench-v2 accuracy(%)RULER mean score at 128K(%)
Exact DSA152 / 31648.1088.92
HISA, B=8192B=8192133 / 31642.0959.17
MISA, h=8h=8134 / 31642.4181.37
Linear direct, B=2048B=2048153 / 31648.4288.19

Linear Indexer achieves comparable aggregate performance to Exact DSA on both benchmarks while outperforming the tested HISA and MISA configurations. Its mean RULER score is only 0.73% below Exact DSA.

TTFT. On eight NVIDIA H20 GPUs, we measure median TTFT over three trials after two warmup trials, using matched inputs, a 2,048-token prefill chunk, and piecewise CUDA graphs.

Method32K TTFT64K TTFT128K TTFT128K speedup
Exact DSA10.45 s25.27 s69.49 s1.00×1.00\times
HISA, B=8192B=819211.03 s22.48 s45.70 s1.52×1.52\times
MISA, h=8h=89.79 s21.76 s52.18 s1.33×1.33\times
Linear direct, B=2048B=20488.80 s18.34 s40.08 s1.73×1.73\times

Linear Indexer achieves the lowest TTFT at all three tested lengths, with speedup increasing from 1.19×1.19\times at 32K to 1.73×1.73\times at 128K.

We thank the reviewer for prompting this broader evaluation. The retrieval analysis is included in Appendix E.3, and the TTFT and downstream-quality results are reported in Sections 5.4–5.5, with detailed task results in Appendix H.4. The revised abstract and contributions explicitly distinguish the 128K token-level DSA evidence from the 256K V4-Flash results.

AuthorsOpenReview ↗

Response to Reviewer x76k— W3: Generalization beyond DeepSeek-family models

Our new experiments on Qwen3.8-Flash-Next show that linear retrieval redundancy also appears in Qwen Sparse Attention (QSA): direct retrieval recovers 85.02% of the native Top-512, increasing to 98.22% with refinement at B=2048B=2048. We thank the reviewer for motivating this cross-family evaluation. These results are now included in Appendix E.6, with capture details in Appendix C and corresponding updates to the Introduction, Section 3.5, and contributions.

Compared with the DSA-style indexers in GLM and Hy, QSA provides a more distinct architectural test: it uses only four heads and no query-dependent head weights. We evaluate all 12 QSA layers across 12 domains at 64K context.

Proposal budget BB (blocks)Recall of native Top-512
512, direct85.02%
1,024, refined94.65%
2,048, refined98.22%
4,096, refined99.64%

All budgets use the same 12,528 query–layer records with more than 4,096 visible blocks; each block represents four tokens. Results are consistent across domains, with B=2048B=2048 recall ranging from 97.53% to 98.84%.

The highest-scoring candidates are particularly well preserved: direct linear Top-512 retrieval retains 97.36% of the native Top-32, while 75.3% of its missed Top-512 blocks lie in ranks 257–512.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

We thank the reviewers and the Area Chair for their constructive feedback. We have revised the manuscript as follows:

Changes are highlighted in blue. For newly added tables, only the captions are highlighted.

  • End-to-end baseline comparisons: Added latency and downstream-quality comparisons with HISA and MISA on DeepSeek-V3.2, including LongBench-v2 and RULER results (Sections 5.1, 5.4–5.5 and Appendix H).
  • Expanded analysis: Added mixed-sign head interventions (Appendix D.3), token-level DSA captures at 128K, joint source–reindex replacement and refinement, layer-wise retrieval diagnostics, and cross-family validation on QSA (Appendix E.3–E.6).
  • Main-text updates: Updated Section 3.5 to reference the additional mixed-sign analysis and summarize the expanded generalization evidence. Added QSA to the second paragraph of the Introduction.
  • Related work and experimental details: Clarified the distinction from linear attention in Appendix A and updated capture protocols and implementation details in Appendices C and G.

The abstract, contributions, and limitations have also been updated to reflect the expanded evidence and its scope.

Dear Area Chair, Senior Area Chairs, and Program Chairs,

Thank you for your time and consideration. As the discussion period draws to a close, we summarize the main additions made in response to the reviewers’ feedback.

1. Downstream quality and end-to-end comparisons with HISA and MISA. We added DeepSeek-V3.2 evaluations under matched inputs and serving settings. Direct Linear Indexer achieves LongBench-v2 accuracy comparable to Exact DSA (48.42% vs. 48.10%) and comparable RULER performance at 128K (88.19% vs. 88.92%), while outperforming the tested HISA and MISA configurations on both benchmarks. It also achieves the lowest TTFT among the tested configurations at 32K, 64K, and 128K, reaching a 1.73× TTFT speedup over Exact DSA at 128K.

2. Broader context-length and cross-family evidence. We extended token-level DSA retrieval and mechanism analyses to 128K and explicitly distinguished these results from the 256K V4-Flash evaluations. We also added experiments on Qwen Sparse Attention (QSA), whose indexer has only four heads and no query-dependent head weights. Direct linear retrieval recovers 85.02% of the native Top-512, increasing to 98.22% with refinement at B=2048. These results extend the retrieval evidence beyond the DeepSeek family.

3. More detailed mechanism analysis. Selective ReLU interventions now isolate the effects of positive-, negative-, and mixed-sign heads. Additional layer-wise analyses distinguish global score agreement from Top-K overlap and examine V4.1’s difficult L24 layer. In frozen-capture replay, refining both candidate-source and reindex selection raises L24 recall to 98.01%. Our mechanism analysis, supplemented by an alignment measure, predicts layer-wise score correlation well and provides predictive value for recall, which also depends on the score distribution near the Top-KK cutoff.

4. More granular quality evaluation and clearer scope. We provided LongBench-v2 breakdowns by context length, task category, and difficulty, with paired confidence intervals, alongside detailed V3.2 RULER results. We also expanded Related Work to clarify the distinction from linear attention and clarified the scope of the efficiency claims.

The revised manuscript incorporates the principal additions, with changes highlighted in blue.

Reviewer x76k has explicitly maintained their positive score and recommendation for acceptance after reading our responses. Reviewer MSQg has also maintained their positive score; we have responded to their follow-up comments with further clarifications. Reviewer PbE1 also gave a positive overall rating, and we have provided additional experiments and analyses in response to their main concerns and questions.

We appreciate all reviewers’ constructive feedback and remain available to clarify any outstanding concerns. Thank you again for considering our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.