acceptodds
Under review as a conference paper at ICLR 2027

ResMiSA: Residual Mixture of Sparse Attention for Long-Context LLM Inference

Abstract

Multi-head token scoring in the DeepSeek Sparse Attention (DSA) indexer remains costly for long-context inference. MISA reduces this cost by activating a subset of indexer heads as scoring experts, but omits the remaining heads' contributions and their direct gradient paths through the indexing score during training. Inspired by residual connections and shared experts, we introduce ResMiSA, which uses to decompose each head's response into an identity branch and a nonlinear residual branch. Because all indexer heads share the same key, their weighted identity branches merge exactly into one always-active shared expert that requires a single dot product per key. Routed residual experts supply nonlinear corrections, while the shared path retains a direct gradient path through every head. Applied without additional training to DeepSeek-V3.2 across 4K–128K contexts, one shared expert and seven residual experts replace 64-head scoring, with RULER scores only 0.06 percentage points below DSA on average and 0.21 points below at 128K. In frozen-backbone indexer warmup, ResMiSA achieves slightly lower attention-fitting loss and higher attention mass recall on training batches than DSA, using only or as many per-key scoring channels at the same trainable parameter count and training steps. Combined with LiteTopK, ResMiSA achieves a end-to-end time-to-first-token (TTFT) speedup over DSA at 1M context.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attenion, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_795zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a sparse attention indexer, namely ResMiSA, which supports the sparse attention mechanism in DeepSeek. ResMiSA reformulates the weighted ReLU-based score as a shared term and nonlinear residuals. Accordingly, it evaluates the shared term to preserve the effect of all heads, while only a small subset of residual heads is dynamically selected using upper bounds derived from legal-key interval extrema. Experiments including the training-free evaluations, indexes warmup training, and end-to-end TTFT measurements demonstrates the effectiveness of the method.

Critical strengths
  1. The proposed method is well-motivated and mathematically straightforward. The decomposition is derived from a simple mathematical identity, providing an intuitive and theoretically sound formulation for reducing indexer computation while preserving the linear contributions of all heads.

  2. The experimental evaluation is multifaceted. ResMiSA is evaluated in terms of both quantitative task performance and attention mass recall, providing empirical evidence of its effectiveness and insights into the underlying mechanism.

Critical weaknesses
  1. The novelty is relatively limited, with the contributions over baselines not sufficiently distinguished. ResMiSA appears to be an incremental extension of MISA, primarily replacing the 8 selected scoring heads with 1 aggregated shared head and 7 residual ones. However, the efficiency gains are solely compared with DSA and LiteTopK, raising concerns that the reported speedups may largely stem from the head sparsification already introduced by MISA, rather than ResMiSA itself. Regarding the methodology, while a modified head selection criterion is introduced, there seems to be limited theoretical justification, e.g. it remains unclear why larger estimated residual upper bounds indicate greater head importance, as their correlation with actual attention distributions is not established. Besides, Sec.4.3 largely builds upon existing techniques, i.e. LiteTopK. Given the promising empirical effectiveness of the method, I encourage the authors to strengthen its theoretical contributions and more clearly distinguish its novelty from prior work.

  2. The experimental setting contains several inconsistencies. Although the paper includes analyses on GLM-5, both downstream accuracy and inference efficiency are evaluated only on DeepSeek-V3.2, leaving the generalizability of ResMiSA across different models insufficiently validated. Another concern is the mismatch between efficiency and quality evaluations: the paper reports substantial acceleration at 1M context length, whereas downstream task quality is evaluated only up to 128K. Additional long-context evaluations, e.g. Needle-In-A-Haystack retrieval tests at comparable context lengths, would help establish whether the reported efficiency gains can be achieved without significant degradation in downstream performance.

  3. I consider the manuscript exhibits noticeable signs of AI-assisted writing and presentation, including the formatting and organization of mathematical derivations, the structure and presentation of experimental results, and the design of Figure 1. This undermines the technical contributions of the manuscript. Therefore, I encourage the authors to carefully and manually revise the manuscript, particularly its technical exposition, logical organization, and visual presentation.

Questions for authors
  1. What is the purpose of the zero-rate analysis in Sec. 5.4? The reported results only compare scalar activation sparsity between positive- and negative-ReLU responses, without reflecting their contributions to the aggregated indexing effects. To my understanding, while negative-ReLU responses exhibit a higher zero rate, ResMiSA also selects fewer residual experts (7) than the positive-ReLU counterpart in MISA (8). Therefore, a direct comparison of their zero rates seems insufficient, as the higher activation sparsity may be offset by the smaller number of selected experts, potentially resulting in comparable overall information loss.

  2. Since DSA already utilizes all indexing heads, it does not suffer from the missing gradient paths associated with head selection in MISA. Therefore, the slightly higher training-batch recall of ResMiSA over DSA in Sec. 5.6 does not directly demonstrate the benefit of preserving gradient paths. Why is MISA, which directly motivates this design, omitted from the training comparison?

  3. Why does ResMiSA underperform MISA$^\dagger$ at shorter context positions but outperform it at longer positions in Fig.3(a)? Does this suggest that ResMiSA offers limited advantages for relatively short-context tasks?

13 comments

We thank the reviewer for the detailed comments on novelty, the theoretical motivation for residual routing, experimental consistency, gradient paths, and the clarity of the manuscript.

Answer for W1

Compared with MISA, ResMiSA makes three contributions: (1) A shared linear term exactly preserves the linear contributions of all heads, discarding only unselected nonlinear residuals rather than entire head responses. (2) Upper-bound routing selects heads using a consistent bound on residual magnitude, avoiding the suppression of large residuals on a small number of tokens by mean pooling. (3) A shared gradient path retains a direct gradient path for heads whose residuals are not selected, whereas ReLU blocks gradients for negative dot-product responses in DSA. LiteTopK reduces memory usage and memory-access overhead by reducing intermediate score storage, while ResMiSA reduces scoring computation through a shared term and sparse residuals. The two methods optimize memory and computation, respectively, making them orthogonal and complementary; combining them yields greater speedups.

Regarding the relationship between the bound and importance, the theoretical basis in Section 4.2 is that under a fixed residual budget, selecting the heads with the largest bounds minimizes a separable error upper bound on the omitted residuals, rather than proving that these heads necessarily make the largest actual attention contributions. The same rule is applied to each head over the same valid-key range, with absolute head weights used to express the bounds on the index-score scale. In contrast, mean pooling can underestimate large local residuals because of cancellation between positive and negative values. Appendix D.1 provides empirical support: with the 1+7 configuration fixed and only the routing criterion changed, upper-bound routing improves the average RULER score by 0.17 percentage points and the score at 128K by 0.44 percentage points.

Regarding inference speed, we added a direct comparison between MISA and ResMiSA, both combined with LiteTopK; the experimental settings and results are provided in our response to Reviewer YDYC's Q2. The additional results show that ResMiSA maintains practical inference costs close to those of MISA while achieving the quality gains discussed above, without substantial additional overhead from routing or the shared expert.

Answer for W2

We added RULER experiments on GLM-5.2 at 4K–128K, extending the original DeepSeek-V3.2 evaluation to a second model. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model; HISA, MISA, and MISA† follow the experimental settings in the paper. ResMiSA achieves an average score of 97.45, the highest among the acceleration methods compared here, only 0.12 percentage points below the base model's 97.57. Its scores at 64K and 128K are 97.20 and 96.72, exceeding MISA† by 0.24 and 2.33 percentage points, respectively. These results add cross-model quality evidence.

The table below reports RULER results on GLM-5.2. Avg. is the equally weighted mean over the six context lengths, rounded to two decimal places.

Method4K8K16K32K64K128KAvg.
IndexCache (base model)97.9197.8097.6797.8397.3196.8797.57
HISA98.1498.1896.9791.5582.6275.2190.45
MISA98.0297.3997.1596.1375.5762.9487.87
MISA†98.1797.9597.5497.4696.9694.3997.08
ResMiSA97.8497.9597.5197.4697.2096.7297.45

To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K, 512K, and 1M. We used GLM-5.2 because DeepSeek-V3.2 does not support the 1M-context quality evaluation conducted here. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.

The results show that ResMiSA achieves the highest score among the lossy sparse attention methods compared here at all three context lengths. HISA and MISA both exhibit substantial score drops at 256K–1M, and MISA†, which uses two-stage reranking, also degrades substantially at 1M. In contrast, ResMiSA consistently maintains task quality close to the base model, scoring 95.49, 92.65, and 83.71 at 256K, 512K, and 1M, respectively, compared with the base model's 95.41, 93.18, and 82.60.

Method256K512K1M
IndexCache (base model)95.4193.1882.60
HISA59.0342.5526.54
MISA50.5828.466.54
MISA†79.8777.4424.10
ResMiSA95.4992.6583.71

Answer for W3

We accept this criticism. The paper discloses the use of AI assistance. AI was also used to help draft some passages, figures, and tables, but all figures, tables, mathematical derivations, and manuscript content have been manually reviewed by the authors, who take full responsibility for all content. We will further revise the technical wording and logical organization manually, and check the derivations and experimental presentation to make the method and supporting evidence clearer and more accurate.

Answer for Q1

We agree that the greater sparsity of the negative ReLU branch does not by itself establish lower overall information loss after head selection, but motivates using this branch for sparse corrections. ResMiSA does not merely replace eight positive-branch heads with seven residual heads; it also exactly preserves the linear contributions of all heads through a shared expert. The 1+7 configuration matches MISA's eight-channel budget for final per-key scoring, whereas 1+8 would add another scoring channel. Information preservation therefore cannot be assessed solely by comparing the number of residual heads.

The overall effect is directly evaluated by the attention mass recall experiments in Section 5.3: across ten SWE-bench examples and all 61 layers, ResMiSA's 1+7 configuration preserves more dense-attention probability mass than eight-head MISA. This metric uses the optimal mass captured by dense-attention top-K selection under the same token budget as its reference, rather than measuring set overlap with the tokens selected by DSA. Thus, the existing results support that the complete shared-term-plus-residual design reduces attention mass loss under the same final scoring budget, and show that using seven residual heads does not offset this overall advantage. This conclusion follows from selection using the aggregated scores, not from the zero-rate statistics alone.

Answer for Q2

We have added the indexer pretraining and attention-fitting comparison, including the MISA′ variants, to Section 5.6, “Indexer Warmup and Attention Fitting,” of the revised ResMiSA manuscript and marked the section in blue. DSA indeed does not lose gradient paths due to head selection, but using all heads does not mean that every query–key pair can propagate gradients through each head's score: when the dot product is negative, the positive ReLU branch blocks this path. ResMiSA's shared linear term preserves a direct scoring contribution from every head, so a head can still receive gradients through the shared term even when its residual is not selected. Thus, this design not only avoids the loss of direct gradient paths for unselected heads in MISA, but also changes the gradient structure under sparse scoring.

The additional experiments freeze the same dots-3-densemla (30B-A3B) backbone and train only the indexers. All methods use the same data, 512K sequences, 400 training steps, learning rate, and trainable parameter count, and select 2048 tokens. MISA′ uses the same upper-bound routing as ResMiSA but directly aggregates the selected heads' positive-ReLU scores without constructing a shared expert, providing a comparison to assess the shared–residual design. As shown below, at matched per-key scoring budgets of four or eight channels, ResMiSA achieves lower indexer cross-entropy loss and higher training-batch attention mass recall than DSA, whereas MISA′ shows the opposite trend. With eight channels, ResMiSA (1+7) reduces loss by 0.0101 and improves recall by 0.1394 relative to DSA; MISA′ (8 heads) instead increases loss by 0.0127 and reduces recall by 0.1734. The shared-only configuration, ResMiSA (1+0), also does not outperform DSA. These results therefore support combining an all-head shared term with sparse residual corrections, rather than attributing the gains solely to gradient paths based on slightly higher recall than DSA.

The table reports means over steps 351–400. All Δ values are relative to DSA; recall is reported in %, and Δ recall in percentage points. Recall is measured on training-batch queries: within each attention head, the probability mass retained by the selected tokens is divided by the mass captured by that head's dense top-K selection under the same budget, and these ratios are then averaged across attention heads.

MethodlossΔ\Delta lossrecallΔ\Delta recall
DSA7.23270.000086.21120.0000
MISA′ (4 heads)7.26700.034385.7857-0.4255
MISA′ (8 heads)7.24540.012786.0377-0.1734
ResMiSA (1+0)7.27060.038085.7876-0.4236
ResMiSA (1+3)7.2299-0.002886.27430.0631
ResMiSA (1+7)7.2226-0.010186.35060.1394

Answer for Q3

To clarify, in Figure 3(a), ResMiSA achieves higher attention mass recall than single-stage MISA at every context position; the method that outperforms ResMiSA at earlier query positions is two-stage MISA†, not MISA. MISA† first selects 8,192 candidates using eight heads, then reranks them using all 64 heads to retain 2,048 tokens. ResMiSA instead scores tokens directly with its 1+7 configuration, without additional full-head reranking. When the history is short, the 8,192 candidates can cover a large fraction, or even all, of the valid tokens, giving full-head reranking more opportunities to correct ranking errors from the first stage. This is consistent with MISA†'s early advantage.

As the query position advances and the valid history grows, the fixed-size candidate set covers a smaller fraction of the history, and reranking cannot recover high-attention-mass tokens omitted by the first stage. ResMiSA, in contrast, preserves the linear contributions of all heads over all valid keys and applies the selected residual corrections. ResMiSA surpasses MISA† beyond approximately 23K in the figure, consistent with this mechanism. Thus, the figure shows that additional full-head reranking remains beneficial for shorter histories, but that for longer histories, ResMiSA's attention mass recall even exceeds that of MISA† with its two-stage reranking; it does not indicate that ResMiSA's advantage over single-stage MISA is limited to long histories.

Reviewer_YDYCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ResMiSA, a residual-based sparse indexer for accelerating long-context LLM inference. Instead of selecting a subset of the original indexer heads and discarding all contributions from unselected heads, ResMiSA decomposes each head's ReLU score into a linear term and a nonlinear residual. Since all heads share the same indexing keys, the linear terms can be combined into a single always-active shared expert, while a small number of residual experts are selected dynamically for each query. The authors further introduce a routing criterion based on coordinate-wise extrema of legal keys to bound omitted residual contributions, and implement the method in a fused CUDA scoring and candidate-filtering pipeline integrated with LiteTopK.

The evaluation applies ResMiSA to DeepSeek-V3.2 without additional training and compares it against DSA, MISA, MISA's two-stage variant, and other sparse-indexing baselines. With one shared expert and seven residual experts, ResMiSA uses eight per-key scoring channels instead of DSA's 64. It reports an average RULER score of 95.56, compared with 95.62 for DSA, and end-to-end time-to-first-token speedups of 2.43× over DSA and 2.19× over LiteTopK at a 1M-token context. A separate frozen-backbone indexer-warmup experiment reports slightly lower attention-fitting loss and higher training-batch attention recall with fewer scoring channels.

The main contribution is a score-level residual decomposition combined with a conservative, query-dependent routing rule and a performance-oriented implementation. The results suggest a potentially useful inference optimization. However, the quality evaluation does not extend to the 1M context length at which the largest latency gains are reported, and further evidence is needed to establish the robustness of the quality–latency trade-off.

Critical strengths
  1. The key idea is mathematically grounded: the identity ReLU(x) = x + ReLU(−x) allows the linear contributions from all indexer heads to be combined into a single shared expert, leaving only the nonlinear residuals to be routed sparsely. The proposed upper-bound routing criterion also has a clear analytical motivation: it minimizes a separable upper bound on omitted residual contributions for a fixed routing budget. This is more substantive than simply proposing another heuristic for selecting a subset of heads.

  2. The reported 2.43× TTFT speedup over DSA and 2.19× over LiteTopK at 1M context is practically relevant, and the gains persist across the tested concurrency settings. Meanwhile, the average RULER score over 4K–128K is close to that of DSA. The paper also includes attention-mass recall, routing ablations, and component-level timing, providing evidence beyond a single aggregate accuracy metric. Together, these results make the proposed optimization worth considering for long-context inference systems.

Critical weaknesses
  1. The paper reports task-quality results only from 4K to 128K, whereas its headline 2.43× TTFT speedup is measured at 1M tokens. This leaves an important gap: the paper does not establish that ResMiSA preserves downstream task quality at 1M, where the computational savings are most substantial and the omitted residuals could have different effects. Attention-mass recall and score reconstruction provide useful diagnostics, but neither guarantees preservation of downstream task performance. Evaluation on long-context tasks at or near 1M tokens would materially strengthen the central claim.

  2. The proposed decomposition and routing rule are technically motivated, but the practical contribution depends on whether they provide a robust advantage over existing sparse-indexing methods under comparable conditions. The paper compares RULER quality against MISA and its two-stage variant, but the end-to-end latency comparison focuses on DSA and LiteTopK. It therefore remains unclear how ResMiSA compares with MISA in actual inference latency, including the overhead of residual routing and shared-expert construction. Moreover, the RULER evaluation uses 100 examples per subtask at each context length, while the reported latency measurements use a small number of repeated requests without uncertainty estimates. More direct, controlled comparisons would make the magnitude and robustness of the claimed advantage easier to assess.

Questions for authors
  1. Can the authors provide downstream task-quality results at 1M context, or explain why such an evaluation is infeasible? Since the largest reported latency gain occurs at this length, evidence that the quality–latency trade-off remains acceptable there would materially affect my assessment.

  2. Can the authors report end-to-end TTFT and indexer component timings for MISA under the same hardware, serving configuration, context lengths, and concurrency settings as ResMiSA? In particular, how much of ResMiSA's advantage remains after accounting for the cost of computing routing bounds and constructing the shared expert?

  3. Do the authors have repeated runs, confidence intervals, or per-task results that establish the stability of the small RULER gap between ResMiSA and DSA? Given that the average difference is only 0.06 percentage points, it would be useful to understand whether the reported ordering is robust to evaluation sampling and other sources of variation.

4 comments

We thank the reviewer for emphasizing downstream quality at 1M context, controlled latency comparisons with MISA, and the statistical robustness of small quality differences.

Answer for Q1

To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K, 512K, and 1M. We used GLM-5.2 because DeepSeek-V3.2 does not support the 1M-context quality evaluation conducted here. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.

The results show that ResMiSA achieves the highest score among the lossy sparse attention methods compared here at all three context lengths. HISA and MISA both exhibit substantial score drops at 256K–1M, and MISA†, which uses two-stage reranking, also degrades substantially at 1M. In contrast, ResMiSA consistently maintains task quality close to the base model, scoring 95.49, 92.65, and 83.71 at 256K, 512K, and 1M, respectively, compared with the base model's 95.41, 93.18, and 82.60.

Method256K512K1M
IndexCache (base model)95.4193.1882.60
HISA59.0342.5526.54
MISA50.5828.466.54
MISA†79.8777.4424.10
ResMiSA95.4992.6583.71

Answer for Q2

We added a direct comparison between MISA and ResMiSA, both combined with LiteTopK, using a maximum request concurrency of 4, with 10 warm-up requests and 10 measured requests. During the 512K TTFT test, we used a profiler to record cumulative component runtimes per request and per GPU. At 128K, 256K, and 512K, ResMiSA's mean TTFT is 0.22 s, 0.07 s, and 3.96 s lower than MISA's, respectively, indicating similar overall runtimes. The component analysis shows that the min/max statistics used for upper-bound routing are more expensive than MISA's mean pooling: routing time increases from 1.67 s to 2.40 s (a 43.88% increase). However, routing accounts for only a small fraction of the full indexer runtime, and the total indexer runtime increases only from 25.17 s to 25.23 s (a 0.23% increase) when all execution overhead is included. These additional results therefore support that ResMiSA maintains practical inference costs close to those of MISA while delivering the previously reported quality gains, rather than achieving these gains at the cost of substantial additional routing or shared-expert overhead.

TTFT:

Mean TTFT (seconds)128K256K512K
MISA + LiteTopK40.9390.20217.15
ResMiSA + LiteTopK40.7190.14213.19

Component runtimes:

Component at 512K (seconds)MISA + LiteTopKResMiSA + LiteTopK
Routing1.672.40
Fused logits + TopK23.1522.56
Standalone token TopK0.010.01
Other indexer overhead0.340.25
Total indexer runtime25.1725.23

Answer for Q3

A full RULER evaluation is time-consuming, and we have not yet completed repeated runs or obtained confidence intervals to report. To further examine whether this quality performance generalizes to another model, we added RULER experiments on GLM-5.2 at 4K–128K. HISA, MISA, and MISA† follow the experimental settings in the paper. ResMiSA achieves an average score of 97.45, only 0.12 below the base model's 97.57, and the highest among the acceleration methods compared here. This provides additional empirical support for ResMiSA maintaining task quality close to the base model across different models.

The table below reports RULER results on GLM-5.2. Avg. is the equally weighted mean over the six context lengths, rounded to two decimal places.

Method4K8K16K32K64K128KAvg.
IndexCache (base model)97.9197.8097.6797.8397.3196.8797.57
HISA98.1498.1896.9791.5582.6275.2190.45
MISA98.0297.3997.1596.1375.5762.9487.87
MISA†98.1797.9597.5497.4696.9694.3997.08
ResMiSA97.8497.9597.5197.4697.2096.7297.45

With approximately 57 hours remaining in the author–reviewer discussion period, we would like to kindly follow up on our responses to your concerns about long-context quality, controlled latency comparisons, and the robustness of quality differences. We have added RULER evaluations on GLM-5.2 at 256K, 512K, and 1M, as well as direct TTFT comparisons and component-level timing analyses with both MISA and ResMiSA using LiteTopK. In addition, we have provided cross-model quality results on GLM-5.2 at 4K–128K. Thank you for taking the time to review these additions. If your schedule permits, we would greatly appreciate your feedback, particularly on any issues that still need clarification, so that we can address them further during the discussion period.

As the author–reviewer discussion period draws to a close, we would greatly appreciate your feedback on whether our responses have addressed your concerns. If so, would you consider raising your rating?

Dear Reviewer,

Thank you for your comments. As the discussion period is drawing to a close, we would appreciate it if you could reconsider your rating if our responses have satisfactorily addressed your concerns.

Thank you for your time and consideration.

Reviewer_rLuVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper provide a simple, approach to reudce the complexity of the Deepseek DSA. It splits each head’s score into two parts. The linear parts of all heads collapse into one query-dependent vector.

Critical strengths
  • Simple idea easily adoptable and requires no training

  • Provide a good fused kernel implementation for easy deployment

Critical weaknesses
  • Evaluation over one model deepseek, can you add GLM

  • Seconds, limited applicability on DSA

  • Quality value not provided for 1 Million, which is promised speedup

Questions for authors

See weakness.

2 comments

We thank the reviewer for highlighting cross-model generalization, architectural applicability, and the need to align long-context quality evaluation with the reported acceleration.

Answer for W1

We added RULER experiments on GLM-5.2 at 4K–128K, extending the original DeepSeek-V3.2 evaluation to a second model. GLM-5.2 uses IndexCache for cross-layer index reuse, so IndexCache in the table denotes the base model rather than an additional compression baseline; HISA, MISA, and MISA† follow the experimental settings in the paper. ResMiSA achieves an average score of 97.45, the highest among the acceleration methods compared here, only 0.12 below the base model's 97.57. At 64K and 128K, it scores 97.20 and 96.72, exceeding two-stage MISA† by 0.24 and 2.33, respectively. These results support that ResMiSA maintains task quality close to the base model on a second model.

The table below reports RULER results on GLM-5.2. Avg. is the equally weighted mean over the six context lengths, rounded to two decimal places.

Method4K8K16K32K64K128KAvg.
IndexCache (base model)97.9197.8097.6797.8397.3196.8797.57
HISA98.1498.1896.9791.5582.6275.2190.45
MISA98.0297.3997.1596.1375.5762.9487.87
MISA†98.1797.9597.5497.4696.9694.3997.08
ResMiSA97.8497.9597.5197.4697.2096.7297.45

Answer for W2

We agree that ResMiSA is currently designed for the DSA indexer, rather than as a general-purpose acceleration method for all attention architectures. However, this focus does not limit its practical relevance to a single model: DSA and its variants have been adopted across multiple mainstream open-source model families, including Hy4 preview, DeepSeek-V4.1 Flash, GLM-5.1/5.2/5.3, Kimi K3.1 Pro, Meituan's LongCat-2.5, and StepFun's Step 5 preview. Notably, while Kimi K3 did not use DSA, K3.1 Pro adopted this architectural approach. This adoption across model families and successive model versions indicates that DSA has become an important direction for long-context architectures in mainstream open-source models, making reductions in its indexing overhead of broad and continuing practical relevance.

ResMiSA specifically addresses the multi-head indexing bottleneck in this architectural paradigm by reducing per-key scoring overhead through a shared linear term and sparse residual corrections. We therefore believe that ResMiSA should be viewed as a targeted optimization for a widely adopted and evolving class of architectures, rather than a technique tailored to a single model.

Answer for W3

To evaluate task quality at longer contexts, we added RULER evaluations on GLM-5.2 at 256K, 512K, and 1M. We used GLM-5.2 because DeepSeek-V3.2 does not support the 1M-context quality evaluation conducted here. Given the time required for a full evaluation, we evaluated 10 samples for each of the 13 RULER subtasks.

The results show that ResMiSA achieves the highest score among the lossy sparse attention methods compared here at all three context lengths. HISA and MISA both exhibit substantial score drops at 256K–1M, and MISA†, which uses two-stage reranking, also degrades substantially at 1M. In contrast, ResMiSA consistently maintains task quality close to the base model, scoring 95.49, 92.65, and 83.71 at 256K, 512K, and 1M, respectively, compared with the base model's 95.41, 93.18, and 82.60.

Method256K512K1M
IndexCache (base model)95.4193.1882.60
HISA59.0342.5526.54
MISA50.5828.466.54
MISA†79.8777.4424.10
ResMiSA95.4992.6583.71

With approximately 57 hours remaining in the author–reviewer discussion period, we would like to kindly follow up on our responses to your concerns about cross-model generalization, architectural applicability, and long-context task quality. We have added RULER experiments on GLM-5.2 at 4K–128K, along with supplementary evaluations at 256K, 512K, and 1M, showing that ResMiSA maintains task quality close to the base model. We have also clarified the method's scope as an optimization for DSA-style indexers and its practical relevance beyond a single model. Thank you for taking the time to review our work. If your schedule permits, we would greatly appreciate your feedback on these clarifications and additional results, and we would be happy to address any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.