acceptodds
Under review as a conference paper at ICLR 2027

Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention Language Models

14 views

Abstract

Sparse attention reduces the compute and memory cost of long-context inference, but already-trained sparse attention models may lose substantial accuracy at their deployed selection budgets. We localize where this loss occurs. Using exact at- tention mass selection as an oracle diagnostic per token and per head, we locate the multi-key retrieval loss in the InfLLM-V2 family to three or four of its 64 key-value heads, the same heads in four checkpoints of one backbone. These blind heads attend to the needle, yet their selector misses it likely because the mean-pooled key summary scored by the selector dilutes single decisive token. A retrieval-head score ranks heads whose selector already finds the needle above the blind heads. Once the heads are found, the loss can be repaired without training most simply by using full attention in them, costing the least for a single sequence. An alternative is a scorer ranking blocks with each query head’s full softmax over a 4- bit copy of the keys while keeping the shipped budget and reading an eighth of the bytes that full attention reads in blind heads. On prompts held out from every design we test, the two repairs recover 31.4 and 30.1, respectively, out of the 33.2 points on RULER MK3 multi-key retrieval that InfLLM-V2-8B loses, and 41.0 and 41.1 of 41.8 on MiniCPM4.1-8B at 64K. On four other selector designs, the scorer recovers most of the loss by acting in many or all of their heads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long-context inference, block selection, attention heads, retrieval heads, needle-in-a-haystack, KV cache, training-free repair, large language models, diagnosis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vACXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper diagnoses retrieval failures in deployed sparse-attention checkpoints by replacing their block selections with exact attention-mass selections at the same budget. A calibration score combining needle-recall shortfall and needle attention mass identifies a few “blind” heads in the InfLLM-V2 family. Full attention in these heads, or selection using token-resolution 4-bit keys, recovers much of the lost retrieval accuracy. The paper supports the diagnosis with interventions, a pooling-based explanation, held-out evaluation draws, additional selector families, and measurements of the repairs' costs and limitations.

The growing adoption of sparse attention in models such as DeepSeek-V4.1-Flash and GLM-5 makes understanding its failure modes particularly relevant. The paper provides valuable insight into how selection errors in a small number of heads can cause substantial retrieval failures, and how targeted repairs can recover the lost accuracy.

Critical strengths
  1. The diagnostic contribution is valuable and well isolated. Exact-mass selection holds the sparse budget fixed, while interventions on selected tokens and heads separate where the loss occurs. Table 1 and Appendices B–C include particularly informative controls: repairing other heads, giving random heads full attention, and scanning individual heads. This is stronger evidence than an attention visualization or a correlation alone.

  2. The proposed repairs recover most of the retrieval accuracy lost to sparse attention in the studied InfLLM-V2 family by modifying only a few heads, without retraining. These substantial gains are consistently supported by held-out evaluations.

Critical weaknesses
  1. The few-head conclusion has a meaningful scope limitation. Its strongest evidence comes from related MiniCPM/InfLLM-V2 checkpoints and retrieval tasks with look-alike distractors. I'm not sure if the same finding would hold for other sparse attention models, such as those using DSA, or if repairing only a few heads would bring similar gains.
Questions for authors
  1. Given the common-word and QA regressions, which repair would the authors recommend for an unknown mixture of workloads? A concise decision rule grounded in the existing results would make the practical takeaway more useful.

  2. Could the authors add experiments on another sparse attention model, such as one using DSA, to assess whether the few-head finding and the benefits of the proposed repairs generalize?

7 comments
AuthorsOpenReview ↗

Response to Reviewer vACX (1/3): a decision rule for an unknown mixture of workloads

We thank the reviewer for the careful and constructive review, and for two questions that improved the paper.

Q1. A decision rule for an unknown mixture of workloads.

Thank you for this very practical suggestion. The rule: for an unknown mix, use the 4-bit repair at the decode tokens of the three heads flagged once on synthetic multi-key prompts, with no calibration on the unknown workload. Across 17 task types, its worst change is −0.9-0.9 points, it beats the shipped selector in all 10,000 random weightings of the 17 tasks, and it recovers nearly two thirds of the multi-key loss. Use the 4-bit repair at the question tokens as well only when the workload is known to be retrieval-heavy, since they cost up to 4 points on open-domain question answering. On held-out seeds, InfLLM-V2-8B at 32K; repair = the 4-bit repair (blocks selected from a 4-bit copy of the keys):

FixMK3CWEPopQANQHotpotQA
Shipped64.644.641.244.741.5
Repair, decode tokens85.644.541.045.242.2
Repair, + question tokens95.044.637.244.241.2
Repair, every token94.838.937.843.742.3
Full attention, flagged heads96.033.138.044.843.5
Repair, decode, 16 common-word heads85.150.840.043.342.2
Repair, + question, 16 common-word heads94.252.437.241.541.7
Dense97.951.539.542.542.5
  • Aggregation-heavy workloads. The 16 heads calibrated on common-word prompts contain the three flagged heads, and the repair in them recovers common word extraction while keeping the multi-key gain (last two repair rows of the table above).
  • Mixtures of these tasks need no calibration of their own. On nine workloads, one uniform and eight random mixtures of three to five tasks, our head-count rule (rank heads by how much fixing each one helps, keep the fewest that close 90% of the gap), calibrated on the workload's own prompts, keeps at most five heads if there is some count that reaches 90% and never loses more than 0.3 points. It does not improve on the three flagged heads, which do as well or better in 11 of 14 cases. Whether the repair includes the question tokens matters more than which heads it uses.

Gain over the shipped selector in points for each repair when it uses the workload-calibrated heads (those our head-count rule selects on each workload's own calibration prompts) and when it uses the original three flagged heads. Gap is dense minus shipped on the workload's evaluation prompts and SE is its standard error. (*) means no head count closed 90% of the gap, so the rule kept the count that scored best on calibration. 0 means the workload's calibration prompts show no gap, so the rule keeps no heads. (--) is the shipped selector:

Workload (tasks, largest weight)Gap (SE)Decode repair: number of workload-calibrated headsDecode repair: gain on workload-calibrated headsDecode repair: gain on original three flagged heads+ Question repair: number of workload-calibrated heads+ Question repair: gain on workload-calibrated heads+ Question repair: gain on original three flagged heads
W0: all 17 tasks, equal+5.2+5.2 (0.4)3+2.1+2.1+2.5+2.54+3.7+3.7+3.7+3.7
W1: 3 tasks, LongFuncEval .86+3.9+3.9 (1.4)7*+1.3+1.30.03*+0.7+0.7+1.2+1.2
W2: 5 tasks, QA 1 .46+2.4+2.4 (0.8)22*−0.3-0.30.01+0.5+0.5−0.1-0.1
W3: 5 tasks, HotpotQA .37+0.9+0.9 (0.6)1−0.1-0.1+0.4+0.410.0+0.3+0.3
W4: 4 tasks, Single 3 .45+0.6+0.6 (0.5)0--+0.6+0.60--+1.2+1.2
W5: 4 tasks, Single 1 .64+0.6+0.6 (0.2)1*+0.1+0.1+0.4+0.45+0.3+0.3+0.4+0.4
W6: 4 tasks, Multi-value .52+5.5+5.5 (0.8)1+2.2+2.2+2.4+2.41+2.4+2.4+4.0+4.0
W7: 4 tasks, MK2 .66+9.1+9.1 (1.6)37*+2.5+2.5+2.4+2.42+7.6+7.6+8.1+8.1
W8: 5 tasks, Single 2 .34−0.6-0.6 (0.4)0--0.00--−1.0-1.0
AuthorsOpenReview ↗

Response to Reviewer vACX (2/3): the rule across unknown mixtures of 17 task types

Q1 (continued). The rule across unknown mixtures of 17 task types.

  • Tested on unknown mixtures. We ran every fix on 17 task types (the 12 RULER tasks, HELMET's PopQA, NQ, HotpotQA and JSON key-value retrieval, and LongFuncEval), every task's arms on one GPU type. A workload's accuracy is the weighted mean of its tasks', so the worst case for a fix is a single task, and a random workload draws its weights uniformly from the simplex. Only the decode-token repair never loses more than a point on any task, and it gains 2.5 points on average. Every fix that rewrites prompt positions gains about one point more on average and pays for it on one kind of task: the question-token variants on open-domain question answering (−4.0-4.0 on PopQA, where the shipped selector is ahead of dense), and full attention in the flagged heads, a 4×4\times selection budget in the three flagged layers and the every-token repair on common word extraction (−12.5-12.5, −9.6-9.6 and −5.5-5.5).

Across the 17 task types, change against the shipped selector in points. RULER tasks use one seed of 200 prompts, so CWE differs slightly from the held-out seeds of the previous comment. Largest regret: the most the fix falls below the best fix on any one task. Beats shipped: share of 10,000 random mixtures in which the fix is ahead, with weights drawn uniformly over the 17 tasks or over the three benchmarks and then their tasks. 5th percentile: of the change over the task-weighted mixtures. CWE / FWE: common / frequent words extraction:

FixWorst change (task)Largest regret (task)Mean changeBeats shipped, % of mixtures (task / benchmark weights)5th percentile
Full attention, flagged heads−12.5-12.5 (CWE)19.7 (CWE)+3.7+3.797.3 / 98.8+0.5+0.5
4×4\times budget, flagged layers−9.6-9.6 (CWE)16.8 (CWE)+3.8+3.898.5 / 99.0+0.8+0.8
Repair, every token−5.5-5.5 (CWE)12.6 (CWE)+3.6+3.698.9 / 98.3+0.7+0.7
Repair, + question tokens−4.0-4.0 (PopQA)5.8 (CWE)+3.7+3.799.8 / 98.0+1.1+1.1
Repair, decode tokens−0.9-0.9 (multi-value)9.5 (MK2)+2.5+2.5100.0 / 100.0+0.7+0.7
Repair, decode, 16 common-word heads−1.7-1.7 (FWE)9.0 (MK2)+2.7+2.799.8 / 92.0+0.7+0.7
Repair, + question, 16 common-word heads−4.0-4.0 (PopQA)4.0 (PopQA)+3.9+3.999.6 / 97.0+1.1+1.1
AuthorsOpenReview ↗

Response to Reviewer vACX (3/3): other sparse attention models, including DSA

Q2. Another sparse attention model, such as one using DSA.

Thank you; we agree this is the natural next question. In short: on other block selectors a few heads carry the loss, with 4 to 22 heads closing 88 to 99% of the gap on six checkpoints of three designs. DeepSeek-V3.2 itself and a second token-level selector of the DSA kind show no multi-key loss for a head-level repair to fix, which is what the paper's account predicts.

  • Other block selectors (table below). With a rule that sets the head count per checkpoint, introduced after submission (on the submission's four InfLLM-V2 checkpoints it ranks the same three blind heads first), full attention in the flagged heads closes 88 to 99% of the gap between the sparse and the dense model on SeerAttention-R over four backbones (Qwen3-4B, 8B and 14B, and DeepSeek-R1-Distill-Qwen-14B), on Dash-8B and on NSA-8B, while as many random heads recover little except on Dash-8B. MoBA is the exception: it scores long chunks by their mean key in every query head, and its loss spreads over about 40% of them.

MK3 at 32K (16K for Dash-8B and NSA-8B; RULER's second single-needle task for MoBA), mean of five evaluation seeds of 200 prompts. Heads are key-value heads, query heads for MoBA:

ModelSelectorHeads flaggedDenseSparseFull attention, flagged heads4-bit repair, flagged headsRetrieval heads, same countRandom heads, same count
Qwen3-8BSeerAttention-R18 of 28898.729.897.797.369.431.4
Qwen3-4BSeerAttention-R9 of 28897.516.687.888.956.317.5
Qwen3-14BSeerAttention-R5 of 32099.754.596.998.164.154.6
R1-Distill-Qwen-14BSeerAttention-R22 of 38485.617.979.173.942.921.2
Dash-8BDashAttention13 of 6499.466.698.092.080.991.0
NSA-8BNSA4 of 6498.114.195.294.568.213.5
MoBA, Qwen3-8BMoBA513 of 115294.421.189.675.885.950.5
MoBA, Qwen3-4BMoBA433 of 115285.715.280.074.187.625.2
  • A model using DSA. We ran DeepSeek-V3.2 itself (685B, in a build with 4-bit feed-forward weights, served in SGLang on four H200s). With its shipped sparse attention, which selects 2,048 tokens for each query, it scores 100.0 on multi-key retrieval at 32K, 64K and 128K, against 100.0, 100.0 and 98.0 with every token selected (one seed of 200 prompts at each length). LongCat-Flash-Lite-Sparse (69B), whose attention extends DSA's lightning indexer, shows the same on five seeds of 200 prompts: 99.4 at 32K and 99.3 at 64K with its shipped selector, against 99.5 and 99.3 with every token selected. In both models, a probe of the serving engine confirms what each arm reads. Models of this size are beyond the single-GPU instrumentation our head-level probe needs, so these runs compare the shipped selection with full selection end to end.
  • Why. The failure we localize comes from scoring blocks by mean-pooled keys (Proposition 1). A token-level indexer does not pool, and its selection is shared by the heads of a layer, so a head that is blind in our sense cannot arise in the same form.
Reviewer_3s1rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors find that most multi-key retrieval failures in the InfLLM-V2 family of models come from a few attention heads selecting the wrong blocks. Identifying these specific heads and giving them full attention, or selecting blocks using more expressive but still cheap 4-bit keys, recovers most of the lost accuracy without any training.

Critical strengths
  • The authors localize recall failures to a few heads whose selectors miss blocks containing the answer tokens. They also show that better block selection can recover accuracy without increasing the block budget
  • Above diagnosis leads to simple and training-free fixes evaluated across multiple backbones and sparse-attention designs. Full attention in the specific heads and block selection using even 4-bit keys recovers most of the lost accuracy
Critical weaknesses
  • the finding that almost all the recall loss lies in three or four heads is strongest within the MiniCPM4-based InfLLM-V2 family -- other settings require repairing more heads or have substantial errors. The repairs also hurt some tasks: full attention in those few specific heads reduces common-word extraction accuracy.
Questions for authors
  • What explains the decrease in performance on common-word extraction and some question-answering tasks?
  • On models where repairing the identified heads does not recover most of the lost accuracy, are failures spread across many heads, or does the calibration procedure miss important heads?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3s1r (1/2): why common word extraction and some QA tasks drop

We thank the reviewer for the insightful summary and the two precise questions, both of which led to new analyses in the revision.

Q1. What explains the decrease on common word extraction and some question answering tasks?

Thank you for asking this. It led us to a clearer account of the performance drop. In short: the common-word loss sits in heads beyond the three that our multi-key calibration flags, and rewriting the flagged heads' selections across the prompt disturbs aggregation. Calibrating on common-word prompts instead finds 16 heads whose repair recovers common word extraction and multi-key retrieval together.

Held-out seeds, InfLLM-V2-8B at 32K; repair = the 4-bit repair (blocks selected from a 4-bit copy of the keys):

FixMK3CWEPopQANQHotpotQA
Shipped64.644.641.244.741.5
Repair, decode tokens85.644.541.045.242.2
Repair, + question tokens95.044.637.244.241.2
Repair, every token94.838.937.843.742.3
Full attention, flagged heads96.033.138.044.843.5
Repair, decode, 16 common-word heads85.150.840.043.342.2
Repair, + question, 16 common-word heads94.252.437.241.541.7
Dense97.951.539.542.542.5
  • The evidence. On the four seeds held out from every design choice (800 prompts per arm), the 4-bit repair at the decode tokens of those 16 heads scores 50.8 on common word extraction (shipped sparse 44.6, dense 51.5) while multi-key retrieval stays at 85.1, the level of the three-head repair in our paper. With the repair also at the question tokens, it scores 52.4 and 94.2 (table above). The 16 common-word heads contain the three flagged heads. The same holds on MiniCPM4.1 at 64K, whose own 16 common-word heads give 36.4 against 23.2 shipped sparse and 43.1 dense.
  • Why the fixes in the three flagged heads cost on common word extraction. The more of the prompt a fix changes in those heads, the more it costs there (Table 14). On the held-out seeds, against the shipped selector's 44.6 on CWE, the every-token repair scores 38.9 (−5.7-5.7) and full attention in the flagged heads scores 33.1 (−11.5-11.5), while the decode-only and question-token variants score 44.5 and 44.6.
  • Question answering. On HELMET's PopQA and NQ, the shipped selector is ahead of the dense model itself (41.2 against 39.5 and 44.7 against 42.5 respectively), so these tasks do not reward recovering dense attention, and the fixes that rewrite prompt tokens cost up to 4 points there. The decode-only repair costs 0.2 points on PopQA and nothing on NQ and HotpotQA.
AuthorsOpenReview ↗

Response to Reviewer 3s1r (2/2): spread across heads, or heads the calibration missed

Q2. On models where repairing the identified heads does not recover most of the loss, are failures spread across many heads, or does the calibration miss heads?

Thank you for this important question. In short: both happen, and they can be told apart. When each checkpoint is probed on its own heads, a few heads carry the loss on every block selector we ran except MoBA, where the loss is truly spread.

MK3 at 32K (16K for Dash-8B and NSA-8B; RULER's second single-needle task for MoBA), mean of five evaluation seeds of 200 prompts. Heads are key-value heads for all models except for MoBA which uses query heads:

ModelSelectorHeads flaggedDenseSparseFull attention, flagged heads4-bit repair, flagged headsRetrieval heads, same countRandom heads, same count
Qwen3-8BSeerAttention-R18 of 28898.729.897.797.369.431.4
Qwen3-4BSeerAttention-R9 of 28897.516.687.888.956.317.5
Qwen3-14BSeerAttention-R5 of 32099.754.596.998.164.154.6
R1-Distill-Qwen-14BSeerAttention-R22 of 38485.617.979.173.942.921.2
Dash-8BDashAttention13 of 6499.466.698.092.080.991.0
NSA-8BNSA4 of 6498.114.195.294.568.213.5
MoBA, Qwen3-8BMoBA513 of 115294.421.189.675.885.950.5
MoBA, Qwen3-4BMoBA433 of 115285.715.280.074.187.625.2
  • NSA-8B: heads carried over, not calibrated. Our submitted paper applied the InfLLM-V2 family's three flagged heads without probing NSA-8B. Our submitted probe (fixed threshold), run on NSA-8B itself, flags 11 heads, including the three heads we carried over, and the repair in those 11 recovers all of the loss. On four held-out seeds at 16K, it scores 97.2, against 14.2 shipped sparse and 97.8 dense.
  • Dash-8B: ranked correctly, cut too early. Our submitted probe ranks Dash-8B's heads correctly but its fixed threshold stops at three heads. On four held-out seeds, the repair at the decode tokens of its top 10 heads scores 92.8, level with the same repair in every head (92.9), while the repair in ten random heads scores 71.4 (shipped sparse 67.4, dense 99.4).
  • One general rule for the heads count. For both NSA-8B and Dash-8B, our submitted calibration probe flagged heads under a fixed threshold. After submission, we replaced it with a rule that sets the count per checkpoint, where the count is decided as the fewest top-ranked heads that close 90% of the gap on calibration prompts. Heads are ranked by how much fixing each one helps on those prompts. On the four InfLLM-V2 checkpoints in our submission (MiniCPM4-8B backbone), the new rule ranks the same three blind heads first and keeps two to four heads, so the submitted results with those heads still hold. The final version will present this new rule in the method section. The table above uses it: on five evaluation seeds, full attention in the heads flagged by the new rule closes 88 to 99% of the gap on SeerAttention-R over four backbones (5 to 22 of 288 to 384 heads), Dash-8B (13 of 64) and NSA-8B (4 of 64), while as many random heads recover little except on Dash-8B.
  • MoBA: truly spread. MoBA scores long chunks by their mean key in every query head. The rule keeps about 40% of its query heads, and as many generic retrieval heads recover as much or more.

We agree that the evidence in the submission is strongest within the MiniCPM4-8B lineage, and the revised version will state both new findings: heads that the submitted probe missed on NSA-8B and Dash-8B, and a truly spread loss on MoBA.

thanks a lot for your response! however, i'd like to maintain my positive score.

the reason i'm not increasing the score is because this method will require tuning on different tasks to acheive the best performance for that task -- e.g. to get best perf. on QA, one needs to calibrate on the QA task

Thank you very much for reading our responses and for keeping your positive score! We would like to clarify one point, since it may change how the method reads: it does not need calibration per task.

The heads are calibrated once per checkpoint, on synthetic multi-key prompts, and then used on every task. With those three flagged heads, the 4-bit repair at the decode tokens never loses more than a point on any of 17 task types (worst change −0.9-0.9, on multi-value retrieval), gains 2.5 points on average, recovers nearly two thirds of the multi-key loss and is ahead of the shipped selector in all 10,000 random mixtures of the tasks (Section 7, Choosing a fix, and Appendix E, Table R7 of the revised PDF).

Question answering in particular needs no QA calibration. On HELMET's PopQA and NQ, the shipped sparse selector is already ahead of the dense model (41.2 against 39.5 and 44.7 against 42.5), so there is no loss to recover there. The same three-head repair stays level on them: 41.0, 45.2 and 42.2 on PopQA, NQ and HotpotQA, against 41.2, 44.7 and 41.5 for the shipped selector. Calibrating the heads on each workload's own prompts instead did not improve on the three flagged heads in 11 of 14 cases across nine workloads, two of them led by question answering (table in our response to Reviewer vACX, 1/3).

Task calibration is an optional refinement for even better performance. When a deployer knows the workload and it has a gap of its own, calibrating on it can do better still. On common word extraction, the 16 heads calibrated on that task lift it from 44.6 to 50.8 (dense 51.5) while keeping the multi-key gain, where the three flagged heads leave it level (44.5).

We will state this in the Method section of the final version: one calibration per checkpoint is the method, and calibration on a known workload is an optional step on top of it.

Thank you again for your time and thoughtful review! If this addresses your concern about per-task tuning, we would be grateful if you would consider raising your score, and we are happy to continue the discussion on anything that remains.

Reviewer_i7xBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a fix for sparse attention methods which pool block of queries and keys in the selection process. The fix requires a calibration phase to identify which heads are missing retrieval tasks the most and then doing an intervention on those heads to fix their shortcomings.

Critical strengths
  • Pooling the queries and the keys before scoring loses a lot of precision that the exponential on individual dot products provides in standard attention.

  • The authors did a good job of finding the weakness and deriving a fix for a specific shortcoming of pooled sparse attention methods.

Critical weaknesses
  • One major component which is missing is latency measurements on both of the proposed interventions. the interventions necessitate more computation than the baseline model and therefore will likely slow down the overall latency.

    • Anyone looking to employ this method would be very interested to know the added latency, as this directly affects whether the utility of the underlying sparse attention mechanism has been affected.
  • The evaluated tasks lean heavily towards RULER evaluation. However, RULER is a highly synthetic task. It would be interesting to see if the same failed retrieval pattern could be seen in other long real-world context tasks such as MRCR or AA-LCR.

Questions for authors
  • Can you add evaluations of before and after the interventions on MRCR and AA-LCR?

  • Can you add latency measurements of both interventions agains the original and dense models to see what the overall effect is?

5 comments
AuthorsOpenReview ↗

Response to Reviewer i7xB (1/3): latency against the original and dense models

We thank the reviewer for the insightful comments and for two concrete, practical requests. We ran both.

Q2 / W1. Latency of both interventions against the original and dense models.

Thank you for pressing on this. We agree that the comparison with the dense model, and in a production engine, was missing. The submission compared only with the shipped sparse model in App F. For InfLLM-V2-8B, at 128K on one H100 at batch size one, the time to first token is 17.53 s for the dense model, 7.57 s for the sparse model, and 8.68 s for the sparse model with full attention in the flagged heads. The fix thus costs 1.11 s and keeps 89% of the sparse model's saving. Per output token, the sparse model with our fix takes 19.9 ms, against 19.3 ms for the base sparse model and 20.9 ms for the dense model. These are measured on CPM.cu, the inference engine OpenBMB ships for its sparse checkpoints. The other lengths are below and the revised Appendix F will include these numbers.

CPM.cu, one H100, batch size one; fix = full attention in the flagged heads (three on InfLLM-V2-8B, four on MiniCPM4.1):

ContextTime to first token: dense / sparse / fixFix keeps of sparse's savingms per output token: dense / sparse / fix
InfLLM-V2, 32K1.79 / 1.50 / 1.54 s84%10.9 / 12.1 / 12.2
InfLLM-V2, 64K5.39 / 3.32 / 3.56 s88%14.4 / 14.0 / 14.4
InfLLM-V2, 128K17.53 / 7.57 / 8.68 s89%20.9 / 19.3 / 19.9
MiniCPM4.1, 64K5.40 / 3.33 / 3.66 s84%14.3 / 13.5 / 13.9
  • The second intervention, the 4-bit repair. It is not implemented in CPM.cu or SGLang, so we time it in the released attention kernels for InfLLM-V2 and MiniCPM4.1 in the variant that repairs the selection at every prompt token. At 128K, its prefill attention takes 7.13 s, against 5.61 s for the sparse model, 6.23 s for sparse with full attention in the flagged heads and 13.04 s for the dense model. This higher prefill cost at the shipped token selection budget is why full attention in the flagged heads is our default fix. The variant that repairs only the decode tokens leaves the prefill as shipped, apart from a 4-bit copy of the flagged heads' keys, and costs the same per decode step (last column of the attention-kernel table below).
  • The engine. vLLM and SGLang serve InfLLM-V2-8B in dense mode only (per its model card), so its sparse mode runs in CPM.cu, which OpenBMB ships for it. We added the fix in the CPM.cu engine as a per-head exception in the block mask, where it recovers 30 of the 32 points the sparse selector loses on 50 MK3 prompts at 32K. On an A40 where decoding is bound by reading the cache, the sparse model decodes 16-44% faster than dense and the fix keeps 98% of that saving.
  • Batched serving (MiniCPM4.1 in SGLang's new sparse backend). At 1 to 64 concurrent requests, the fix costs 1.3-2.7% of output throughput. In this engine, the base sparse model itself serves fewer tokens per second than the dense one at 32K and 64K because its prefill is slower, and at one request its decoding is also slower than dense.

MiniCPM4.1-8B in SGLang, one H100, 256 output tokens, fix = full attention in the flagged heads:

ContextRequestsTokens per s: dense / sparse / fixms between tokens: dense / sparse / fix
32K171.1 / 45.5 / 44.88.6 / 12.7 / 13.0
32K16156.1 / 97.0 / 95.814.3 / 16.1 / 16.6
32K64164.4 / 101.9 / 100.626.3 / 21.8 / 23.3
64.5K142.8 / 29.8 / 29.09.0 / 14.3 / 14.8
64.5K1663.7 / 49.2 / 47.919.5 / 18.3 / 19.2
64.5K6464.2 / 49.9 / 48.629.1 / 20.5 / 21.7
  • Decoding many sequences (using released attention kernels). With 64 sequences, sparse attention takes 10.4 to 16.1 ms per decode step against 22.6 to 89.1 ms dense from 32K to 128K, and sparse with full attention in the flagged heads keeps 91 to 94% of that saving.

Attention kernels, InfLLM-V2-8B, all 32 layers, one H100 (decode: ms per step, prefill: s); fix = full attention in the three flagged heads:

ContextPhase, sequencesDenseSparseFix4-bit repair, every token
32KDecode, 10.88.68.78.8
32KDecode, 166.19.29.59.5
32KDecode, 6422.610.411.511.0
32KPrefill, 10.74 s0.94 s0.98 s1.06 s
64KDecode, 11.310.510.610.7
64KDecode, 1611.611.011.611.4
64KDecode, 6444.712.314.413.3
64KPrefill, 12.98 s2.25 s2.40 s2.67 s
128KDecode, 12.114.214.314.4
128KDecode, 1622.714.615.715.2
128KDecode, 6489.116.120.317.9
128KPrefill, 113.04 s5.61 s6.23 s7.13 s

At one sequence, the release's sparse kernels are slower than dense attention. At 16 sequences, they are slower only at 32K and faster at other context lengths.

AuthorsOpenReview ↗

Response to Reviewer i7xB (2/3): latency in the releases' HuggingFace code and on a second selector family

Q2 / W1. Latency of both interventions against the original and dense models.

  • HuggingFace. In the release's HuggingFace code for InfLLM-V2, sparse decoding is slower than dense at every length (~18 ms of attention per token at 32K against under 1 ms) for the shipped model and both fixes. This is because sparse selection runs as many small kernels.

The release's HuggingFace code, InfLLM-V2-8B, one H100, batch size one; fix = full attention in the three flagged heads, repair = the every-token 4-bit repair in them:

ContextTime to first token: dense / sparse / fix / repairms per output token: dense / sparse / fix / repair
32K1.86 / 2.00 / 2.04 / 2.13 s29.6 / 47.2 / 47.1 / 48.8
64K5.28 / 4.39 / 4.52 / 4.80 s33.2 / 55.9 / 56.6 / 58.6
128K17.14 / 9.74 / 10.39 / 11.03 s42.0 / 67.2 / 67.2 / 68.9

End to end. We measure the latency on another selector family, SeerAttention-R, timed end to end in its release's HuggingFace code on one H100 (128 output tokens; prefill is dense in every arm, so the time to first token is the same):

Model (flagged heads)ContextSequencesms per output token: dense / sparse / fixFix keeps of sparse's saving
Qwen3-8B (18 heads)32K138.1 / 57.7 / 59.0sparse slower than dense
Qwen3-8B (18 heads)64K136.8 / 47.7 / 49.8sparse slower than dense
Qwen3-8B (18 heads)128K160.5 / 53.7 / 54.786%
Qwen3-8B (18 heads)32K8105.7 / 91.9 / 93.092%
Qwen3-14B (5 heads)32K132.7 / 50.0 / 50.7sparse slower than dense
Qwen3-14B (5 heads)64K145.0 / 54.1 / 56.2sparse slower than dense
Qwen3-14B (5 heads)128K171.5 / 63.8 / 64.097%
Qwen3-14B (5 heads)32K470.1 / 65.7 / 65.994%

At one sequence and context up to 64K, the SeerAttention-R gate's per-layer scoring adds more host time than its sparse attention saves, so this code decodes slower than dense. Where decoding is GPU-bound (128K, or several sequences at 32K), full attention in the heads our calibration rule flags on synthetic multi-key prompts keeps 86-97% of the sparse model's saving.

Attention only. To remove the release's per-token Python overhead, we also time the attention itself with SeerAttention-R's own decode kernel on an H100 (one decode step over Qwen3-8B's 36 layers, no other parts of the model; fix = full attention in the 18 flagged heads):

ContextSequencesDecode attention, ms: dense / sparse / fixFix keeps of sparse's saving
32K12.1 / 1.1 / 1.740%
32K1625.1 / 6.1 / 8.090%
32K64122.7 / 17.0 / 23.594%
64K13.7 / 1.1 / 1.871%
64K1649.9 / 6.2 / 9.692%
64K64248.3 / 15.6 / 28.594%
128K16.7 / 1.1 / 2.084%
128K1699.8 / 6.1 / 12.793%
128K64494.7 / 24.6 / 49.995%

In short, in every end-to-end timing on one H100, across InfLLM-V2-8B, MiniCPM4.1-8B and SeerAttention-R on Qwen3-8B and Qwen3-14B, full attention in the flagged heads adds up to 7% to the sparse model's time per output token. It adds 2-15% to the time to first token of InfLLM-V2 and MiniCPM4.1, the only models whose prefill it changes, while keeping 84-89% of the sparse model's saving there in CPM.cu.

AuthorsOpenReview ↗

Response to Reviewer i7xB (3/3): MRCR, AA-LCR and other real long-context tasks

Q1 / W2. Real long-context tasks: MRCR and AA-LCR before and after.

Thank you for this suggestion. We agree RULER is synthetic and we ran both benchmarks before and after our fixes. On MRCR with MiniCPM4.1, the sparse model shows no deficit: 19.9 sparse vs 19.6 dense at 16-32K, and 12.8 vs 10.1 at 32-64K. On MRCR, every fix on MiniCPM4.1 stays within two points of the sparse model's score. On Qwen3-8B, a stronger dense model (36.0), SeerAttention-R's sparse gates trail it by 6.8 points over all 593 prompts (29.2). Full attention and the 4-bit repair in the 18 heads that our rule flags on synthetic multi-key prompts both reach 33.2, recovering ~4 of those 6.8 points (standard error 0.8), while three random sets of 18 heads recover only 0.1 on average. No MRCR data was used to choose the heads. On AA-LCR, MiniCPM4.1 and our second backbone (InfLLM-V2 continued from Qwen3-8B, App C) answer only 3 and 6 of 100 questions even as dense models. Qwen3-14B with SeerAttention-R answers 21 dense and 14 with the sparse gates, and 18 and 16 respectively with full attention and the 4-bit repair in the rule's five heads. But with 100 questions, none of these differences exceeds two standard errors.

MiniCPM4.1PromptsDenseSparseFull attention in flagged heads4-bit repair, decode tokens4-bit repair, + question tokens4-bit repair, every token
MRCR 16-32K29719.619.920.718.818.218.7
MRCR 32-64K20910.112.812.213.213.812.8
OOLONG80026.724.926.225.227.225.6
AA-LCR100354442
MRCR, SeerAttention-RPromptsDenseSparseOur rule's heads: full attention / repairRandom heads, full attention
Qwen3-8B (18 of 288 heads)59336.029.233.2 / 33.229.3
Qwen3-14B (5 of 320 heads)59335.232.732.7 / 33.333.0
  • MRCR on MiniCPM4.1. The sparse model does not trail the dense one at either length. Split by 2, 4 and 8 needles in MRCR prompts, it is never more than one point below dense, and dense itself scores under 30 in every cell.
  • MRCR on SeerAttention-R. Qwen3-8B and Qwen3-14B, the backbones of SeerAttention-R's decode-time gates, are the strongest dense models our fixes apply to (MRCR's 8K to 32K bins, all 593 prompts). Our rule ranks heads by how much fixing each one helps on multi-key prompts and keeps the fewest top heads that close 90% of the gap between the sparse and the dense model on those prompts. On Qwen3-14B, the sparse model loses only 2.5 points and neither fix in the rule's five heads recovers them (full attention 32.7, 4-bit repair 33.3, sparse 32.7, five random heads 33.0), so on Qwen3-14B this loss does not sit in the heads that multi-key prompts flag.
  • What MRCR measures at this scale. The dense models mostly reproduce the most recent instance of the repeated request whichever one is asked: Qwen3-14B scores 79.3 when the last instance is asked and 17.3 otherwise (Qwen3-8B: 76.5 and 19.5, MiniCPM4.1: 46.5 and 9.2). On Qwen3-8B and Qwen3-14B, the sparse model's loss is largest on prompts that ask for the last instance, and so is the recovery with our fix on Qwen3-8B.
  • AA-LCR. We grade the models' answers on AA-LCR as the benchmark's original version specifies, where Qwen3-235B marks each answer correct or incorrect against the reference. AA-LCR documents run 99K to 209K tokens, beyond MiniCPM4.1's 64K window (run untruncated), so even the dense model answers only 3 of 100 questions. On the paper's second backbone, the same InfLLM-V2 selector continued from Qwen3-8B with a 256K window, the dense model answers 6 of 100 with thinking off and 1 of 100 with thinking on. The released Qwen3-8B itself answers 12 of 100 with thinking off. On the released Qwen3-14B with SeerAttention-R's gates (both with the YaRN scaling their model cards give for long inputs), the dense model answers 21, the sparse gates 14, and full attention and the 4-bit repair in the rule's five heads answer 18 and 16 respectively. A paired difference over 100 questions has a standard error of 3 to 4 points, so the benchmark resolves neither the loss nor a recovery.
  • Where the failure does appear outside RULER. On HELMET's JSON key-value retrieval with InfLLM-V2-8B, the sparse model scores 71.5, the dense model 92.5, and sparse with full attention in the flagged heads 93.0. On LongFuncEval, whose questions retrieve one record among hundreds of near-identical API responses, both fixes recover a half to three quarters of the loss (Table 4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank all three reviewers for their careful reading and insightful comments. We have uploaded a revised PDF with new experiments and each reply below includes the tables it relies on, so it can be read on its own. Changed or added text and tables are shown in blue, and all table and appendix numbers cited in the reviews still point to the same content.

While answering the reviews, we also rechecked the appendix and corrected three statements, in App E, G and H. Each is marked in blue. App G (NSA-8B) is explained in our reply to Reviewer 3s1r.

We look forward to hearing your thoughts and would be happy to address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.