- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper diagnoses retrieval failures in deployed sparse-attention checkpoints by replacing their block selections with exact attention-mass selections at the same budget. A calibration score combining needle-recall shortfall and needle attention mass identifies a few “blind” heads in the InfLLM-V2 family. Full attention in these heads, or selection using token-resolution 4-bit keys, recovers much of the lost retrieval accuracy. The paper supports the diagnosis with interventions, a pooling-based explanation, held-out evaluation draws, additional selector families, and measurements of the repairs' costs and limitations.
The growing adoption of sparse attention in models such as DeepSeek-V4.1-Flash and GLM-5 makes understanding its failure modes particularly relevant. The paper provides valuable insight into how selection errors in a small number of heads can cause substantial retrieval failures, and how targeted repairs can recover the lost accuracy.
Critical strengths
-
The diagnostic contribution is valuable and well isolated. Exact-mass selection holds the sparse budget fixed, while interventions on selected tokens and heads separate where the loss occurs. Table 1 and Appendices B–C include particularly informative controls: repairing other heads, giving random heads full attention, and scanning individual heads. This is stronger evidence than an attention visualization or a correlation alone.
-
The proposed repairs recover most of the retrieval accuracy lost to sparse attention in the studied InfLLM-V2 family by modifying only a few heads, without retraining. These substantial gains are consistently supported by held-out evaluations.
Critical weaknesses
- The few-head conclusion has a meaningful scope limitation. Its strongest evidence comes from related MiniCPM/InfLLM-V2 checkpoints and retrieval tasks with look-alike distractors. I'm not sure if the same finding would hold for other sparse attention models, such as those using DSA, or if repairing only a few heads would bring similar gains.
Questions for authors
-
Given the common-word and QA regressions, which repair would the authors recommend for an unknown mixture of workloads? A concise decision rule grounded in the existing results would make the practical takeaway more useful.
-
Could the authors add experiments on another sparse attention model, such as one using DSA, to assess whether the few-head finding and the benefits of the proposed repairs generalize?
7 comments
Response to Reviewer vACX (2/3): the rule across unknown mixtures of 17 task types
Q1 (continued). The rule across unknown mixtures of 17 task types.
- Tested on unknown mixtures. We ran every fix on 17 task types (the 12 RULER tasks, HELMET's PopQA, NQ, HotpotQA and JSON key-value retrieval, and LongFuncEval), every task's arms on one GPU type. A workload's accuracy is the weighted mean of its tasks', so the worst case for a fix is a single task, and a random workload draws its weights uniformly from the simplex. Only the decode-token repair never loses more than a point on any task, and it gains 2.5 points on average. Every fix that rewrites prompt positions gains about one point more on average and pays for it on one kind of task: the question-token variants on open-domain question answering ( on PopQA, where the shipped selector is ahead of dense), and full attention in the flagged heads, a selection budget in the three flagged layers and the every-token repair on common word extraction (, and ).
Across the 17 task types, change against the shipped selector in points. RULER tasks use one seed of 200 prompts, so CWE differs slightly from the held-out seeds of the previous comment. Largest regret: the most the fix falls below the best fix on any one task. Beats shipped: share of 10,000 random mixtures in which the fix is ahead, with weights drawn uniformly over the 17 tasks or over the three benchmarks and then their tasks. 5th percentile: of the change over the task-weighted mixtures. CWE / FWE: common / frequent words extraction:
| Fix | Worst change (task) | Largest regret (task) | Mean change | Beats shipped, % of mixtures (task / benchmark weights) | 5th percentile |
|---|---|---|---|---|---|
| Full attention, flagged heads | (CWE) | 19.7 (CWE) | 97.3 / 98.8 | ||
| budget, flagged layers | (CWE) | 16.8 (CWE) | 98.5 / 99.0 | ||
| Repair, every token | (CWE) | 12.6 (CWE) | 98.9 / 98.3 | ||
| Repair, + question tokens | (PopQA) | 5.8 (CWE) | 99.8 / 98.0 | ||
| Repair, decode tokens | (multi-value) | 9.5 (MK2) | 100.0 / 100.0 | ||
| Repair, decode, 16 common-word heads | (FWE) | 9.0 (MK2) | 99.8 / 92.0 | ||
| Repair, + question, 16 common-word heads | (PopQA) | 4.0 (PopQA) | 99.6 / 97.0 |
Response to Reviewer vACX (3/3): other sparse attention models, including DSA
Q2. Another sparse attention model, such as one using DSA.
Thank you; we agree this is the natural next question. In short: on other block selectors a few heads carry the loss, with 4 to 22 heads closing 88 to 99% of the gap on six checkpoints of three designs. DeepSeek-V3.2 itself and a second token-level selector of the DSA kind show no multi-key loss for a head-level repair to fix, which is what the paper's account predicts.
- Other block selectors (table below). With a rule that sets the head count per checkpoint, introduced after submission (on the submission's four InfLLM-V2 checkpoints it ranks the same three blind heads first), full attention in the flagged heads closes 88 to 99% of the gap between the sparse and the dense model on SeerAttention-R over four backbones (Qwen3-4B, 8B and 14B, and DeepSeek-R1-Distill-Qwen-14B), on Dash-8B and on NSA-8B, while as many random heads recover little except on Dash-8B. MoBA is the exception: it scores long chunks by their mean key in every query head, and its loss spreads over about 40% of them.
MK3 at 32K (16K for Dash-8B and NSA-8B; RULER's second single-needle task for MoBA), mean of five evaluation seeds of 200 prompts. Heads are key-value heads, query heads for MoBA:
| Model | Selector | Heads flagged | Dense | Sparse | Full attention, flagged heads | 4-bit repair, flagged heads | Retrieval heads, same count | Random heads, same count |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | SeerAttention-R | 18 of 288 | 98.7 | 29.8 | 97.7 | 97.3 | 69.4 | 31.4 |
| Qwen3-4B | SeerAttention-R | 9 of 288 | 97.5 | 16.6 | 87.8 | 88.9 | 56.3 | 17.5 |
| Qwen3-14B | SeerAttention-R | 5 of 320 | 99.7 | 54.5 | 96.9 | 98.1 | 64.1 | 54.6 |
| R1-Distill-Qwen-14B | SeerAttention-R | 22 of 384 | 85.6 | 17.9 | 79.1 | 73.9 | 42.9 | 21.2 |
| Dash-8B | DashAttention | 13 of 64 | 99.4 | 66.6 | 98.0 | 92.0 | 80.9 | 91.0 |
| NSA-8B | NSA | 4 of 64 | 98.1 | 14.1 | 95.2 | 94.5 | 68.2 | 13.5 |
| MoBA, Qwen3-8B | MoBA | 513 of 1152 | 94.4 | 21.1 | 89.6 | 75.8 | 85.9 | 50.5 |
| MoBA, Qwen3-4B | MoBA | 433 of 1152 | 85.7 | 15.2 | 80.0 | 74.1 | 87.6 | 25.2 |
- A model using DSA. We ran DeepSeek-V3.2 itself (685B, in a build with 4-bit feed-forward weights, served in SGLang on four H200s). With its shipped sparse attention, which selects 2,048 tokens for each query, it scores 100.0 on multi-key retrieval at 32K, 64K and 128K, against 100.0, 100.0 and 98.0 with every token selected (one seed of 200 prompts at each length). LongCat-Flash-Lite-Sparse (69B), whose attention extends DSA's lightning indexer, shows the same on five seeds of 200 prompts: 99.4 at 32K and 99.3 at 64K with its shipped selector, against 99.5 and 99.3 with every token selected. In both models, a probe of the serving engine confirms what each arm reads. Models of this size are beyond the single-GPU instrumentation our head-level probe needs, so these runs compare the shipped selection with full selection end to end.
- Why. The failure we localize comes from scoring blocks by mean-pooled keys (Proposition 1). A token-level indexer does not pool, and its selection is shared by the heads of a layer, so a head that is blind in our sense cannot arise in the same form.
Response to Reviewer vACX (1/3): a decision rule for an unknown mixture of workloads
We thank the reviewer for the careful and constructive review, and for two questions that improved the paper.
Q1. A decision rule for an unknown mixture of workloads.
Thank you for this very practical suggestion. The rule: for an unknown mix, use the 4-bit repair at the decode tokens of the three heads flagged once on synthetic multi-key prompts, with no calibration on the unknown workload. Across 17 task types, its worst change is −0.9 points, it beats the shipped selector in all 10,000 random weightings of the 17 tasks, and it recovers nearly two thirds of the multi-key loss. Use the 4-bit repair at the question tokens as well only when the workload is known to be retrieval-heavy, since they cost up to 4 points on open-domain question answering. On held-out seeds, InfLLM-V2-8B at 32K; repair = the 4-bit repair (blocks selected from a 4-bit copy of the keys):
Gain over the shipped selector in points for each repair when it uses the workload-calibrated heads (those our head-count rule selects on each workload's own calibration prompts) and when it uses the original three flagged heads. Gap is dense minus shipped on the workload's evaluation prompts and SE is its standard error. (*) means no head count closed 90% of the gap, so the rule kept the count that scored best on calibration. 0 means the workload's calibration prompts show no gap, so the rule keeps no heads. (--) is the shipped selector: