- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper develops a sparse attention indexer, namely ResMiSA, which supports the sparse attention mechanism in DeepSeek. ResMiSA reformulates the weighted ReLU-based score as a shared term and nonlinear residuals. Accordingly, it evaluates the shared term to preserve the effect of all heads, while only a small subset of residual heads is dynamically selected using upper bounds derived from legal-key interval extrema. Experiments including the training-free evaluations, indexes warmup training, and end-to-end TTFT measurements demonstrates the effectiveness of the method.
Critical strengths
-
The proposed method is well-motivated and mathematically straightforward. The decomposition is derived from a simple mathematical identity, providing an intuitive and theoretically sound formulation for reducing indexer computation while preserving the linear contributions of all heads.
-
The experimental evaluation is multifaceted. ResMiSA is evaluated in terms of both quantitative task performance and attention mass recall, providing empirical evidence of its effectiveness and insights into the underlying mechanism.
Critical weaknesses
-
The novelty is relatively limited, with the contributions over baselines not sufficiently distinguished. ResMiSA appears to be an incremental extension of MISA, primarily replacing the 8 selected scoring heads with 1 aggregated shared head and 7 residual ones. However, the efficiency gains are solely compared with DSA and LiteTopK, raising concerns that the reported speedups may largely stem from the head sparsification already introduced by MISA, rather than ResMiSA itself. Regarding the methodology, while a modified head selection criterion is introduced, there seems to be limited theoretical justification, e.g. it remains unclear why larger estimated residual upper bounds indicate greater head importance, as their correlation with actual attention distributions is not established. Besides, Sec.4.3 largely builds upon existing techniques, i.e. LiteTopK. Given the promising empirical effectiveness of the method, I encourage the authors to strengthen its theoretical contributions and more clearly distinguish its novelty from prior work.
-
The experimental setting contains several inconsistencies. Although the paper includes analyses on GLM-5, both downstream accuracy and inference efficiency are evaluated only on DeepSeek-V3.2, leaving the generalizability of ResMiSA across different models insufficiently validated. Another concern is the mismatch between efficiency and quality evaluations: the paper reports substantial acceleration at 1M context length, whereas downstream task quality is evaluated only up to 128K. Additional long-context evaluations, e.g. Needle-In-A-Haystack retrieval tests at comparable context lengths, would help establish whether the reported efficiency gains can be achieved without significant degradation in downstream performance.
-
I consider the manuscript exhibits noticeable signs of AI-assisted writing and presentation, including the formatting and organization of mathematical derivations, the structure and presentation of experimental results, and the design of Figure 1. This undermines the technical contributions of the manuscript. Therefore, I encourage the authors to carefully and manually revise the manuscript, particularly its technical exposition, logical organization, and visual presentation.
Questions for authors
-
What is the purpose of the zero-rate analysis in Sec. 5.4? The reported results only compare scalar activation sparsity between positive- and negative-ReLU responses, without reflecting their contributions to the aggregated indexing effects. To my understanding, while negative-ReLU responses exhibit a higher zero rate, ResMiSA also selects fewer residual experts (7) than the positive-ReLU counterpart in MISA (8). Therefore, a direct comparison of their zero rates seems insufficient, as the higher activation sparsity may be offset by the smaller number of selected experts, potentially resulting in comparable overall information loss.
-
Since DSA already utilizes all indexing heads, it does not suffer from the missing gradient paths associated with head selection in MISA. Therefore, the slightly higher training-batch recall of ResMiSA over DSA in Sec. 5.6 does not directly demonstrate the benefit of preserving gradient paths. Why is MISA, which directly motivates this design, omitted from the training comparison?
-
Why does ResMiSA underperform MISA$^\dagger$ at shorter context positions but outperform it at longer positions in Fig.3(a)? Does this suggest that ResMiSA offers limited advantages for relatively short-context tasks?
13 comments
Answer for W3
We accept this criticism. The paper discloses the use of AI assistance. AI was also used to help draft some passages, figures, and tables, but all figures, tables, mathematical derivations, and manuscript content have been manually reviewed by the authors, who take full responsibility for all content. We will further revise the technical wording and logical organization manually, and check the derivations and experimental presentation to make the method and supporting evidence clearer and more accurate.
Answer for Q1
We agree that the greater sparsity of the negative ReLU branch does not by itself establish lower overall information loss after head selection, but motivates using this branch for sparse corrections. ResMiSA does not merely replace eight positive-branch heads with seven residual heads; it also exactly preserves the linear contributions of all heads through a shared expert. The 1+7 configuration matches MISA's eight-channel budget for final per-key scoring, whereas 1+8 would add another scoring channel. Information preservation therefore cannot be assessed solely by comparing the number of residual heads.
The overall effect is directly evaluated by the attention mass recall experiments in Section 5.3: across ten SWE-bench examples and all 61 layers, ResMiSA's 1+7 configuration preserves more dense-attention probability mass than eight-head MISA. This metric uses the optimal mass captured by dense-attention top-K selection under the same token budget as its reference, rather than measuring set overlap with the tokens selected by DSA. Thus, the existing results support that the complete shared-term-plus-residual design reduces attention mass loss under the same final scoring budget, and show that using seven residual heads does not offset this overall advantage. This conclusion follows from selection using the aggregated scores, not from the zero-rate statistics alone.
Answer for Q2
We have added the indexer pretraining and attention-fitting comparison, including the MISA′ variants, to Section 5.6, “Indexer Warmup and Attention Fitting,” of the revised ResMiSA manuscript and marked the section in blue. DSA indeed does not lose gradient paths due to head selection, but using all heads does not mean that every query–key pair can propagate gradients through each head's score: when the dot product is negative, the positive ReLU branch blocks this path. ResMiSA's shared linear term preserves a direct scoring contribution from every head, so a head can still receive gradients through the shared term even when its residual is not selected. Thus, this design not only avoids the loss of direct gradient paths for unselected heads in MISA, but also changes the gradient structure under sparse scoring.
The additional experiments freeze the same dots-3-densemla (30B-A3B) backbone and train only the indexers. All methods use the same data, 512K sequences, 400 training steps, learning rate, and trainable parameter count, and select 2048 tokens. MISA′ uses the same upper-bound routing as ResMiSA but directly aggregates the selected heads' positive-ReLU scores without constructing a shared expert, providing a comparison to assess the shared–residual design. As shown below, at matched per-key scoring budgets of four or eight channels, ResMiSA achieves lower indexer cross-entropy loss and higher training-batch attention mass recall than DSA, whereas MISA′ shows the opposite trend. With eight channels, ResMiSA (1+7) reduces loss by 0.0101 and improves recall by 0.1394 relative to DSA; MISA′ (8 heads) instead increases loss by 0.0127 and reduces recall by 0.1734. The shared-only configuration, ResMiSA (1+0), also does not outperform DSA. These results therefore support combining an all-head shared term with sparse residual corrections, rather than attributing the gains solely to gradient paths based on slightly higher recall than DSA.
The table reports means over steps 351–400. All Δ values are relative to DSA; recall is reported in %, and Δ recall in percentage points. Recall is measured on training-batch queries: within each attention head, the probability mass retained by the selected tokens is divided by the mass captured by that head's dense top-K selection under the same budget, and these ratios are then averaged across attention heads.
| Method | loss | loss | recall | recall |
|---|---|---|---|---|
| DSA | 7.2327 | 0.0000 | 86.2112 | 0.0000 |
| MISA′ (4 heads) | 7.2670 | 0.0343 | 85.7857 | -0.4255 |
| MISA′ (8 heads) | 7.2454 | 0.0127 | 86.0377 | -0.1734 |
| ResMiSA (1+0) | 7.2706 | 0.0380 | 85.7876 | -0.4236 |
| ResMiSA (1+3) | 7.2299 | -0.0028 | 86.2743 | 0.0631 |
| ResMiSA (1+7) | 7.2226 | -0.0101 | 86.3506 | 0.1394 |
Answer for Q3
To clarify, in Figure 3(a), ResMiSA achieves higher attention mass recall than single-stage MISA at every context position; the method that outperforms ResMiSA at earlier query positions is two-stage MISA†, not MISA. MISA† first selects 8,192 candidates using eight heads, then reranks them using all 64 heads to retain 2,048 tokens. ResMiSA instead scores tokens directly with its 1+7 configuration, without additional full-head reranking. When the history is short, the 8,192 candidates can cover a large fraction, or even all, of the valid tokens, giving full-head reranking more opportunities to correct ranking errors from the first stage. This is consistent with MISA†'s early advantage.
As the query position advances and the valid history grows, the fixed-size candidate set covers a smaller fraction of the history, and reranking cannot recover high-attention-mass tokens omitted by the first stage. ResMiSA, in contrast, preserves the linear contributions of all heads over all valid keys and applies the selected residual corrections. ResMiSA surpasses MISA† beyond approximately 23K in the figure, consistent with this mechanism. Thus, the figure shows that additional full-head reranking remains beneficial for shorter histories, but that for longer histories, ResMiSA's attention mass recall even exceeds that of MISA† with its two-stage reranking; it does not indicate that ResMiSA's advantage over single-stage MISA is limited to long histories.
We thank the reviewer for the detailed comments on novelty, the theoretical motivation for residual routing, experimental consistency, gradient paths, and the clarity of the manuscript.
Answer for W1
Compared with MISA, ResMiSA makes three contributions: (1) A shared linear term exactly preserves the linear contributions of all heads, discarding only unselected nonlinear residuals rather than entire head responses. (2) Upper-bound routing selects heads using a consistent bound on residual magnitude, avoiding the suppression of large residuals on a small number of tokens by mean pooling. (3) A shared gradient path retains a direct gradient path for heads whose residuals are not selected, whereas ReLU blocks gradients for negative dot-product responses in DSA. LiteTopK reduces memory usage and memory-access overhead by reducing intermediate score storage, while ResMiSA reduces scoring computation through a shared term and sparse residuals. The two methods optimize memory and computation, respectively, making them orthogonal and complementary; combining them yields greater speedups.
Regarding the relationship between the bound and importance, the theoretical basis in Section 4.2 is that under a fixed residual budget, selecting the heads with the largest bounds minimizes a separable error upper bound on the omitted residuals, rather than proving that these heads necessarily make the largest actual attention contributions. The same rule is applied to each head over the same valid-key range, with absolute head weights used to express the bounds on the index-score scale. In contrast, mean pooling can underestimate large local residuals because of cancellation between positive and negative values. Appendix D.1 provides empirical support: with the 1+7 configuration fixed and only the routing criterion changed, upper-bound routing improves the average RULER score by 0.17 percentage points and the score at 128K by 0.44 percentage points.
Regarding inference speed, we added a direct comparison between MISA and ResMiSA, both combined with LiteTopK; the experimental settings and results are provided in our response to Reviewer YDYC's Q2. The additional results show that ResMiSA maintains practical inference costs close to those of MISA while achieving the quality gains discussed above, without substantial additional overhead from routing or the shared expert.
Answer for W2
We added RULER experiments on GLM-5.2 at 4K–128K, extending the original DeepSeek-V3.2 evaluation to a second model. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model; HISA, MISA, and MISA† follow the experimental settings in the paper. ResMiSA achieves an average score of 97.45, the highest among the acceleration methods compared here, only 0.12 percentage points below the base model's 97.57. Its scores at 64K and 128K are 97.20 and 96.72, exceeding MISA† by 0.24 and 2.33 percentage points, respectively. These results add cross-model quality evidence.
The table below reports RULER results on GLM-5.2. Avg. is the equally weighted mean over the six context lengths, rounded to two decimal places.
To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K, 512K, and 1M. We used GLM-5.2 because DeepSeek-V3.2 does not support the 1M-context quality evaluation conducted here. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.
The results show that ResMiSA achieves the highest score among the lossy sparse attention methods compared here at all three context lengths. HISA and MISA both exhibit substantial score drops at 256K–1M, and MISA†, which uses two-stage reranking, also degrades substantially at 1M. In contrast, ResMiSA consistently maintains task quality close to the base model, scoring 95.49, 92.65, and 83.71 at 256K, 512K, and 1M, respectively, compared with the base model's 95.41, 93.18, and 82.60.