acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining

Abstract

Mixture of Block Attention (MoBA) reduces long-context computation by selecting key-value (KV) blocks for each query. FlashMoBA implements MoBA efficiently but retains a fixed Top- budget and query-centric schedule. Fixed Top- applies one budget across different score distributions and couples routing to task granularity; query-range ownership fragments K/V reuse when several ranges select the same column. We present LightningMoBA, which co-designs selection and execution through dynamic routing, KV-centric ownership, and runtime pipelining. Dynamic routing thresholds MoBA's per-query block scores and directly emits variable-cardinality sparse metadata. KV-centric ownership groups selected rows by (query head, KV block) column, splits high-fanout columns into bounded tasks, and reuses each task's loaded KV tile. On Hopper, pipelining overlaps regular K/V transfers and irregular query gathers with Tensor Core computation. Together these mechanisms decouple retention from GPU scheduling granularity. At matched Top- retention across five checkpoints, LightningMoBA accelerates end-to-end generation by to while raising mean RULER score from to . In controlled two-model replays, complete device-path speedup grows from to at Top- to to at Top-, while routing and CSC construction reach to at Top-. The code of LightningMoBA is available at https://anonymous.4open.science/r/LightningMoba-A090

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
sparse attention, long-context language models, efficient inference, GPU kernel optimization, block sparsity, dynamic sparsity, attention kernel, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CkhROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LightningMoBA, which replaces MoBA’s fixed Top-k routing with per-query threshold selection. It groups work by KV block, splits high-fanout columns, and pipelines query gathers with attention computation on Hopper. The block-scoring function and softmax over retained tokens are preserved. On the RULER suite across five checkpoints, the reported speedups are 1.20-1.60x. Quality improves for three models and drops for two.

Critical strengths

The new kernel accelerates the attention by 16.9-33.8% while computing the same selected attention pairs. This is a significant efficiency improvement.

Loading queries while other work runs makes the tested query-gather kernel 2.53-2.65x faster compared to loading them one step at a time. Profiling also shows fewer stalls waiting for memory.

Critical weaknesses

All five models lose quality at 64K. Table 10 increases the threshold router’s budget without showing whether increasing Top-k gives similar quality gains.

End-to-end gains shrink sharply with length: Qwen3-4B goes from 2.41x at 4K to 1.13x at 128K. The full attention path also takes 15.8 ms at 32K, versus 13.0 ms for dense FA3 in Figure 5.

Questions for authors

How did you check the budget match in the full evaluation?

Can you show both routers’ quality-latency curves using the same final kernel?

Which stages explain the smaller end-to-end gain at 128K?

2 comments
AuthorsOpenReview ↗

Response to Reviewer CkhR

We thank the reviewer for recognizing the efficiency improvements from KV-centric execution and query-gather pipelining. We conducted additional experiments to verify retention-budget matching and isolate the contribution of dynamic routing under the same optimized backend. We address the three questions below.

Q1. Verification of Matched Retention Budgets

We calibrate threshold parameters on held-out samples to match the average retention budget of fixed Top-k routing, rather than enforcing an identical number of retained blocks per query. The thresholds remain fixed throughout the full evaluation.

We additionally measured the realized number of retained attention tokens per query on the complete Qwen3-4B RULER evaluation set across all six context lengths (4K–128K), including compulsory sink and local blocks.

As reported in Appendix C.1, Table 11, dynamic routing's average retained-token counts differ from fixed Top-16 by only −1.01% to +0.88%. These measurements confirm closely matched average sparsity across the evaluated lengths, providing a consistent basis for quality and latency comparisons.

Q2. Quality–Latency Comparison Using the Same Optimized Backend

To isolate routing from execution optimizations, we added end-to-end comparisons between fixed Top-k and dynamic threshold routing at budgets of 16, 32, and 64. Both strategies use the identical LightningMoBA execution backend, including KV-centric execution, fanout-aware splitting, and runtime pipelining, with the same RULER samples and evaluation settings.

The results in Appendix C.1, Table 12 and Figure 6 cover two models, three context lengths (32K, 64K, 128K), and three retention budgets:

  • Top-64: Dynamic routing simultaneously improves accuracy and latency in all six configurations, achieving +0.85 to +5.95 RULER points and 1.16–1.56× end-to-end speedups over fixed Top-k.
  • Top-32: Dynamic routing reduces latency in all six configurations, achieving 1.10–1.26× speedups, while accuracy changes depend on the model and context length.
  • 64K: For Qwen3-4B, the dynamic-minus-fixed accuracy difference improves from −6.43 points at Top-16 to +4.73 at Top-64, with a 1.23× speedup at Top-64. For GQA-30B-A3B, the difference improves from −2.48 to +0.85 points, with a 1.35× speedup.

These controlled results establish that dynamic routing independently improves the quality–latency trade-off, particularly at larger retention budgets, without attributing KV-centric execution or pipelining gains to routing.

We also analyzed the 64K quality degradation in Appendix C.2, Tables 13–16, identifying variable tracking (VT) as a major contributor. Two complementary experiments further clarify how quality can be improved:

  • Retention budget (Table 15): Increasing the budget from Top-16 to Top-64 improves the overall LightningMoBA-minus-FlashMoBA accuracy difference in all four tested model–length configurations.
  • Block granularity (Table 16): Under matched average retained-token budgets, reducing block size from 128 to 8 improves VT accuracy by 10.20–23.80 points and overall RULER accuracy by 3.03–7.73 points across both models at 64K and 128K, with 2.25–2.48× higher end-to-end latency.

Together, these findings demonstrate two ways to improve long-context quality: retaining more evidence or using finer-grained scoring under the same token budget, with an explicit quality–efficiency trade-off.

Q3. Why Does the End-to-End Speedup Decrease at Longer Context Lengths?

LightningMoBA optimizes sparse attention during prefill, while other model operations, tensor-parallel communication, and dense decoding remain unchanged. Their costs limit how much attention-level acceleration translates into full-model speedup.

Figure 5 confirms substantial long-context benefits in a controlled single-rank setting: at 128K, the complete sparse-attention path decreases from 135.1 ms to 83.2 ms, a 1.62× speedup over FlashMoBA. The end-to-end evaluation instead uses TP=8, introducing cross-rank communication and synchronization.

Appendix B.3.2, Table 9 reveals significant TP-rank workload imbalance at 128K. Selected query-block pair counts and K1 attention latencies have mean max-to-mean ratios of 1.88× and 1.86×, respectively. In 428 of 432 request-layer groups, the rank with the most selected pairs is also the slowest K1 rank.

Fanout-aware splitting balances work within each rank but cannot redistribute work across TP ranks. Consequently, synchronization remains sensitive to the slowest rank.

Thus, LightningMoBA retains strong sparse-attention acceleration at 128K, while unchanged full-model costs and TP execution overheads, potentially amplified by cross-rank imbalance, constrain the end-to-end speedup.

AuthorsOpenReview ↗

Follow-up on Additional Experiments and Reviewer Concerns

We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. As the discussion period approaches its conclusion, we would like to briefly follow up on our detailed responses to the individual reviews and the additional experiments presented in the revised manuscript.

We have addressed the major concerns through controlled same-backend comparisons (Table 12), fine-grained RULER analyses (Tables 13–14), retention-budget and block-granularity studies (Tables 15–16), and broader baseline evaluations (Table 17).

We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. We remain available to clarify any outstanding questions during the discussion period.

Thank you again for your consideration and for evaluating the complete experimental evidence and system-level contributions of LightningMoBA.

Reviewer_EeovOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Following the Mixture of Block Attention (MoBA), and FlashMoBA, this manuscript presented LightningMoBA indicating a threshold based KV block selection and a KV-centric execution with hardware level pipeline optimization. Based on the experiment, the proposed method can provide a quick inference and comparable accuracy.

Critical strengths
  • Introduction of algorithm MoBA in section 3.1 is clear.
  • provide a cuda level optimization.
Critical weaknesses
  • contributions:
    • dynamic routing: this is a tiny adjustment from picking blocks from top-k to blocks that have score above a threshold.
    • Since the sparse nature of attention weight, only few token will have larger weight, and most of them will be tiny weights. For top-k cases, if K is large, the remained kv-caches must contain the kv-pair that correspond to small attention weights. The threshold method, however, remove all those kv-pair with small weights, result in a smaller set of kv-pairs. Therefore, the speedup is mainly due to computing with smaller set of kv-pairs. Table 3 do support this argument. when K = 16, the threshold method provide a small speedup, while k=64 the threshold method will provide a significant large speed up.
    • Evaluation: see below. The comparison is quite limit, and hard to show the strengths.
  • Writing: hard to read. Especially when introducing other algorithms (excluding subsection 3.1 2nd paragraph), the logic flow is hard to follow.
  • Experiments:
    • The RULER bench is a collection of many sub-benchmarks, such as, (NIAH Single, NIAH multiValue, NIAH MultiQuery, ...). Table 1 and the rest of the manual script just report with a single value, which may not provide clear comparison under different settings.
    • Comparison are mainly against FlashMoBA. Although, MoBA is relative old, it should also be included in the comparison.
    • There are various other KV optimization algorithms, only compare with MoBA family of algorithm may increase bias.
    • Table 1, averaging results under different length can not provide much reference, since the variance is too large.
Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer Eeov

We thank the reviewer for recognizing the MoBA introduction and CUDA optimizations. We address the main concerns below.

1. Contributions Beyond Threshold Selection

We do not claim thresholding as a new primitive. LightningMoBA instead co-designs adaptive routing, sparse workload organization, and hardware-efficient execution. Query-relative thresholding on MoBA's original block scores enables variable retention and direct compact CSC construction without ordered Top-k maintenance. KV-centric ownership, fanout-aware splitting, and an SM90 pipeline then combine KV reuse, bounded work units, TMA K/V transfers, irregular Q gathering, and Tensor Core computation.

Controlled ablations isolate these contributions: KV-centric ownership reduces attention latency by 16.9–33.8% (Table 4); splitting mitigates long-tail workloads (Table 7); and pipelined Q gathering achieves 2.53–2.65× speedups over serial gathering in the same kernel (Table 5). Table 2 reports cumulative gains.

2. Speedups Are Not Simply Due to Fewer Retained KV Pairs

Table 3 matches the realized number of valid query-block pairs within 0.05% between fixed Top-k and threshold routing. Thus, its higher-budget routing speedups are not caused by selecting fewer pairs. FlashMoBA maintains ordered Top-k candidates, whereas threshold routing uses score comparisons and directly constructs CSC. Including CSC construction, the matched-pair speedups are 3.15–4.13× at Top-32 and 6.99–11.25× at Top-64 (Table 3).

Separate measurements over the full Qwen3-4B RULER suite confirm that dynamic routing's average retained-token budget differs from fixed Top-16 by only −1.01% to +0.88% across 4K–128K (Appendix C.1, Table 11).

We also isolate routing from kernel differences by using the identical optimized LightningMoBA backend for fixed and dynamic routing. At Top-64, dynamic routing improves accuracy (+0.85 to +5.95 points) and end-to-end latency (1.16–1.56× speedups) in all six configurations (Table 12, Figure 6).

3. Fine-grained RULER Evaluation

We added all 13 RULER subtask results for five models at 64K and 128K (Appendix C.2, Tables 13–14). At 64K, variable tracking (VT) declines by 25.77 points on average, accounting for 55.3% of the 13-task macro-average reduction. At 128K, its average decline narrows to 8.10 points, revealing task-dependent quality effects.

We investigate two complementary mitigation strategies:

Retention budget (Table 15). Increasing the reference budget from Top-16 to Top-64 improves the overall LightningMoBA-minus-FlashMoBA score difference in all four model–length configurations; at Top-64, the difference is positive in every case. At 64K it improves from −5.67 to +4.73 points on Qwen3-4B and −1.41 to +0.85 on GQA-30B-A3B. The Qwen3-4B VT gap narrows from −36.96 to −0.40.

Block granularity (Table 16). Under matched average retained-token budgets, reducing block size from 128 to 8 improves VT by 10.20–23.80 points and overall RULER by 3.03–7.73 points across both models at 64K/128K, with 2.25–2.48× higher end-to-end latency. Finer blocks can make local evidence more distinguishable under mean-pooled scoring, at increased execution cost.

4. Interpretation Across Context Lengths

Table 1 reports each of six lengths (4K–128K) separately for all five models; its Average row is only a compact summary. Table 10 adds 128K density-dependent operating points, while Table 12 and Figure 6 show same-backend, matched-reference-budget quality–latency comparisons at 32K, 64K, and 128K.

5. Additional Baselines

We added original MoBA and FlexPrefill, a training-free sparse prefill method outside the MoBA family, on Qwen3-4B and GQA-30B-A3B at all six lengths (Appendix C.3, Table 17). LightningMoBA has the lowest end-to-end generation time in all 12 configurations, with 1.84–3.78× speedups over MoBA and 1.13–3.91× over FlexPrefill. At 128K, its speedups over FlexPrefill are 1.13× and 1.49×, respectively.

FlexPrefill obtains higher RULER scores in most configurations, at longer generation times.

6. Presentation and Clarity

We clarify how routing selects KV blocks, KV-centric execution organizes the selected work, and runtime pipelining executes it on Hopper (Sections 3.2–3.4). We also distinguish device-path, full-model end-to-end, and same-backend routing timing scopes. We hope these revisions address the reviewer's concerns.

AuthorsOpenReview ↗

Follow-up on Additional Experiments and Reviewer Concerns

We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. As the discussion period approaches its conclusion, we would like to briefly follow up on our detailed responses to the individual reviews and the additional experiments presented in the revised manuscript.

We have addressed the major concerns through controlled same-backend comparisons (Table 12), fine-grained RULER analyses (Tables 13–14), retention-budget and block-granularity studies (Tables 15–16), and broader baseline evaluations (Table 17).

We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. We remain available to clarify any outstanding questions during the discussion period.

Thank you again for your consideration and for evaluating the complete experimental evidence and system-level contributions of LightningMoBA.

Reviewer_dkZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LightningMoBA, a GPU-optimized sparse attention system that improves Mixture of Block Attention (MoBA) by jointly redesigning how attention blocks are selected, how GPU work is organized, and how data movement overlaps computation.

Critical strengths
  1. The systems design is coherent. The paper provides credible evidence that its KV-centric execution and query-gather pipeline accelerate MoBA. The paper explains the tradeoff between reuse and parallelism explicitly.

  2. Several ablations are well targeted. The ownership experiment uses identical routing metadata and tensors, while the separate splitting and pipeline experiments provide stronger controls than a simple comparison between two complete implementations.

Critical weaknesses
  1. The accuracy improvements are inconsistent across context lengths and models. While the average RULER score improves slightly, two of the five models experience lower average accuracy, and all five models show accuracy degradation at 64K. The paper does not sufficiently explain this systematic degradation or characterize when adaptive routing is preferable to fixed Top-k selection.

  2. The benefit of dynamic routing is not fully convincing. The ablation results in Table 2 suggest that most improvements come from KV-centric execution and runtime pipelining. At the main Top-16 setting, threshold routing actually increases latency at 128K. This raises the question of how much additional benefit dynamic routing provides beyond the execution optimizations.

Questions for authors
  1. Could the authors provide an end-to-end comparison between fixed Top-k and dynamic routing using the same optimized execution backend to better isolate the benefit of dynamic routing?

  2. Table 1 shows that all five models experience accuracy degradation at 64K, despite improvements at shorter context lengths. Is there a common explanation for this behavior? Could the authors provide a more detailed analysis of the routing patterns or task-level accuracy to explain this degradation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dkZy

We thank the reviewer for the constructive feedback and for recognizing our coherent system design and controlled ablations. We address the independent contribution of dynamic routing and the quality degradation at 64K below.

Q1. End-to-End Comparison Using the Same Optimized Execution Backend

To isolate routing from execution optimizations, we added end-to-end comparisons of fixed Top-k and dynamic threshold routing at reference budgets of 16, 32, and 64. Both use the identical optimized LightningMoBA backend (KV-centric execution, fanout-aware splitting, and runtime pipelining), the same RULER samples and evaluation settings, and calibrated average retention budgets. Results appear in Appendix C.1, Table 12 and Figure 6.

  • Top-64: Dynamic routing improves both accuracy and latency in all six configurations: +0.85 to +5.95 RULER points and 1.16–1.56× end-to-end speedups.
  • Top-32: It reduces latency in all six configurations (1.10–1.26× speedups), with model- and length-dependent accuracy changes.
  • 64K quality–latency trade-off: For Qwen3-4B, the dynamic-minus-fixed accuracy difference improves from −6.43 points at Top-16 to +4.73 at Top-64, with a 1.23× speedup at Top-64. For GQA-30B-A3B, the difference improves from −2.48 to +0.85 points, with a 1.35× speedup at Top-64.

These results also address the reviewer's observation that threshold routing can be slower at Top-16. Its benefit is budget-dependent and becomes more pronounced at larger budgets. Crucially, the execution backend is held constant, so these improvements isolate routing rather than attributing KV-centric execution or pipelining gains to it.

Q2. Analysis of the 64K Quality Degradation

We added a fine-grained analysis of all 13 RULER subtasks across five models at 64K and 128K, together with retention-budget and block-granularity studies (Appendix C.2, Tables 13–16).

Task-level analysis. The 64K quality decrease is concentrated in particular tasks. Variable tracking (VT), RULER's multi-hop tracing task, declines by 25.77 points on average across five models, accounting for approximately 55.3% of the reduction in the 13-task macro-average. Retrieval and aggregation change by only −1.68 and −0.58 points on average. At 128K, the average VT decrease narrows to 8.10 points, indicating task-dependent rather than monotonically worsening quality.

Possible explanation. VT requires recovering multiple intermediate dependencies. Threshold routing may omit blocks with relatively low scores that nevertheless contain necessary intermediate evidence. Mean pooling within large blocks can also dilute localized evidence, reducing its relative score. We examine two complementary ways to mitigate this effect.

Retention-budget sensitivity (Table 15). We compare Top-16, Top-32, and Top-64 reference budgets on Qwen3-4B and GQA-30B-A3B at 64K and 128K, using the original FlashMoBA baseline (distinct from the same-backend study in Table 12). At Top-64, LightningMoBA achieves higher overall RULER scores than FlashMoBA in all four model–length configurations. At 64K, the overall accuracy difference improves from −5.67 to +4.73 points on Qwen3-4B and from −1.41 to +0.85 on GQA-30B-A3B. Qwen3-4B's VT gap narrows from −36.96 to −0.40 points.

Block-granularity sensitivity (Table 16). We further compare LightningMoBA using block sizes of 128 and 8 at 64K and 128K on both models, matching the average retained-token budgets. Reducing the block size increases VT accuracy by 10.20–23.80 points and overall RULER accuracy by 3.03–7.73 points in all four configurations. At 64K, VT improves by 16.80 points on Qwen3-4B and 17.00 points on GQA-30B-A3B. The trade-off is 2.25–2.48× higher end-to-end latency for Block-8, consistent with the larger candidate set and increased routing/execution costs.

Together, these experiments identify two complementary ways to improve long-context quality: retaining more evidence or increasing routing granularity at a matched token budget. The latter yields substantial quality improvements with a quantified efficiency cost, explaining our use of Block-128 as the default quality–efficiency operating point.

We hope these additional controlled comparisons and task-level analyses address the reviewer's concerns.

Thank you for the additional experiments. Table 12 is the key result for me. On the same backend, dynamic routing helps at Top-32 and Top-64, but at Top-16, the operating point of the abstract and Table 1, it is slower in 4 of 6 configurations (0.92–1.17×) and less accurate in 3 of 6. At 64K the degradation comes from the threshold rule itself (Qwen3-4B: fixed 59.84 vs. dynamic 53.41 on the same backend). The Top-16 end-to-end speedup therefore appears to come from the execution backend (1.15–1.53× when comparing Table 12's fixed Top-16 rows with Table 1), not from routing.

AuthorsOpenReview ↗

Clarification on Routing and Execution Contributions

We thank the reviewer for the careful analysis of Table 12. We agree that the Top-16 end-to-end speedups are primarily attributable to execution optimizations. We would like to further clarify the budget-dependent routing efficiency and the observed quality degradation through two complementary analyses.

1. Why does dynamic routing become more efficient at larger budgets?

At Top-16, maintaining a relatively small ordered Top-k candidate set incurs limited overhead. Consequently, replacing Top-k maintenance with threshold-based selection does not consistently improve end-to-end latency. As k increases, however, ordered candidate maintenance becomes more expensive, whereas threshold routing avoids this cost through direct score comparisons and CSC construction.

This trend is supported by Table 3, where routing and CSC construction speedups increase from 0.90–1.33× at Top-16 to 3.15–4.13× at Top-32 and 6.99–11.25× at Top-64, under matched query-block pair counts. The same trend appears in Table 12: dynamic routing accelerates all six Top-32 configurations and simultaneously improves accuracy and latency in all six Top-64 configurations.

Thus, while execution optimizations dominate the Top-16 speedups, dynamic routing provides an increasingly significant independent efficiency contribution at larger retention budgets.

2. Why does accuracy degrade at 64K, and how can it be improved?

The Top-16 quality degradation is consistent with the combined effects of a limited retention budget and relatively coarse block-level scoring. With Block Size 128, mean-pooled key representations may dilute localized but important evidence within a block, reducing its relative routing score. This can be particularly problematic for variable tracking (VT), where recovering multiple intermediate dependencies is essential.

Our additional experiments investigate two complementary approaches:

  • Increasing retention budgets (Table 15): Moving from Top-16 to Top-64 improves the overall LightningMoBA-minus-FlashMoBA accuracy difference across all four evaluated long-context configurations. In the same-backend comparison (Table 12), Qwen3-4B at 64K improves from a −6.43-point gap at Top-16 to a +4.73-point advantage at Top-64.
  • Refining block granularity (Table 16): Under matched average retained-token budgets, reducing Block Size from 128 to 8 improves VT accuracy by 10.20–23.80 points and overall RULER accuracy by 3.03–7.73 points across four configurations, with a 2.25–2.48× end-to-end latency cost.

These results indicate that long-context quality can be improved either by retaining more relevant evidence or by using finer-grained routing to better distinguish localized information, even without increasing the average retained-token budget.

In summary, we agree with the reviewer's assessment that the Top-16 speedups are primarily driven by execution-backend optimizations. At the same time, our additional experiments demonstrate that dynamic routing delivers substantial and independent performance benefits at larger retention budgets, while the quality analyses identify two complementary approaches to improving long-context accuracy.

Together, these findings more clearly distinguish the respective roles of dynamic routing and execution optimizations in LightningMoBA, as well as their complementary contributions to the overall system.

Thank you for the reply and for confirming that the Top-16 speedups come primarily from the execution backend. With this attribution agreed, the paper's lead contribution, dynamic routing, is not supported at the Top-16 operating point on which the abstract and main results are built. Its benefit appears only at budgets of 32 and above on two models. The 64K degradation also remains unexplained: coarse scoring and a limited budget apply equally to fixed Top-k, so they do not account for the 6.43-point gap on the same backend, and the mitigations are partial or cost more than the speedup itself. The execution backend is a solid engineering contribution, but the submission as framed is not yet sufficiently supported. I will keep my rating at 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their constructive feedback and careful evaluation of LightningMoBA. We appreciate their recognition of our system-level optimizations, including KV-centric execution, query-gather pipelining, and controlled performance analyses.

The main concerns center on three aspects: (1) the independent contribution of dynamic routing, (2) quality degradation at long context lengths, particularly 64K, and (3) the breadth and fairness of experimental comparisons. We have substantially strengthened the revised manuscript with additional experiments addressing these concerns.

1. Independent Contributions of Routing and Execution

LightningMoBA's contribution lies in the co-design of adaptive routing, sparse workload organization, and hardware-efficient execution, rather than thresholding alone.

Under matched valid query-block pair counts (within 0.05%), our routing achieves 3.15–4.13× speedups at Top-32 and 6.99–11.25× at Top-64, including CSC construction (Table 3).

Furthermore, controlled end-to-end comparisons using the identical optimized LightningMoBA execution backend demonstrate that dynamic routing independently improves both RULER accuracy and latency. At Top-64, dynamic routing increases accuracy by 0.85–5.95 percentage points while achieving 1.16–1.56× end-to-end speedups across all six evaluated configurations (Appendix C.1, Table 12 and Figure 6).

2. Understanding and Mitigating Long-Context Quality Degradation

We added fine-grained evaluations of all 13 RULER subtasks across five models at 64K and 128K (Tables 13–14). These results identify variable tracking (VT) as a major contributor to the 64K quality degradation, with an average decline of 25.77 percentage points, accounting for approximately 55.3% of the reduction in the 13-task macro-average score.

We further investigate two complementary approaches to improving long-context quality:

  • Increasing retention budgets (Table 15): Raising the reference budget from Top-16 to Top-64 improves the overall accuracy difference between LightningMoBA and FlashMoBA across all four evaluated model–length configurations. At Top-64, LightningMoBA achieves higher overall RULER scores in every case.
  • Refining block granularity (Table 16): Under matched average retained-token budgets, reducing block size from 128 to 8 improves VT accuracy by 10.20–23.80 points and overall RULER accuracy by 3.03–7.73 points across all four configurations. This comes with 2.25–2.48× higher end-to-end latency, establishing a clear quality–efficiency trade-off.

These results show that long-context quality can be improved either by retaining more evidence or by using finer-grained routing to better distinguish relevant information under the same token budget.

3. Broader Baseline Evaluation

We extended our evaluation beyond FlashMoBA to include the original MoBA implementation and FlexPrefill, a training-free sparse prefill method outside the MoBA family.

The additional experiments cover two models across all six context lengths (4K–128K). LightningMoBA achieves the lowest end-to-end generation time in all 12 configurations, delivering 1.84–3.78× speedups over MoBA and 1.13–3.91× over FlexPrefill (Appendix C.3, Table 17).

We also provide realized retention measurements (Table 11) and length-specific quality–latency comparisons to clarify experimental fairness and the contributions of individual design choices.

Summary

Together, these additions provide stronger evidence for LightningMoBA's independent routing and execution contributions, identify practical approaches to mitigating long-context quality degradation, and demonstrate consistent efficiency advantages across a broader set of sparse attention baselines.

We hope the revised manuscript and our detailed responses adequately address the reviewers' concerns, and we welcome further discussion.

AuthorsOpenReview ↗

Additional Clarifications and Summary of Rebuttal Evidence

We thank all reviewers for their thoughtful feedback and continued engagement. We have provided detailed individual responses and substantially expanded the experimental evaluation in the revised manuscript. We would like to summarize the key evidence and clarify how the individual components contribute to LightningMoBA.

1. Independent Contributions of Execution and Routing

We appreciate the discussion regarding the attribution of Top-16 speedups. We agree that the end-to-end acceleration at Top-16 is primarily driven by the optimized execution backend. However, we believe this observation should be considered alongside the broader experimental evidence rather than interpreted as a limitation of the overall system contribution.

LightningMoBA is designed as a co-optimized routing and execution framework, with distinct contributions addressing different bottlenecks.

First, the execution optimizations provide substantial improvements independently of dynamic routing. Under identical sparse patterns, KV-centric ownership reduces attention-stage latency by 16.9–33.8% (Table 4), while query-gather pipelining achieves 2.53–2.65× speedups over serial gathering (Table 5). These results establish the effectiveness of the proposed execution design.

Second, dynamic routing provides an independent, budget-dependent contribution. At Top-16, maintaining a small ordered Top-k candidate set incurs relatively limited overhead. As the retention budget increases, this cost becomes more significant, and the advantage of threshold routing becomes increasingly evident.

Under matched query-block pair counts, routing and CSC construction achieve 3.15–4.13× speedups at Top-32 and 6.99–11.25× at Top-64 (Table 3). More importantly, the same-backend end-to-end comparisons demonstrate latency improvements in all six Top-32 configurations and simultaneous accuracy and latency improvements in all six Top-64 configurations (Table 12 and Figure 6).

Therefore, the limited routing advantage at Top-16 does not negate its independently demonstrated benefits at larger budgets, nor does it diminish the execution contributions. Rather, the controlled experiments clarify the different operating regimes in which the two components provide their strongest benefits.

2. Long-Context Quality and Experimental Scope

We also acknowledge the quality degradation observed at 64K under the Top-16 configuration. The additional task-level analysis identifies variable tracking as a major contributor, accounting for approximately 55.3% of the reduction in the 13-task macro-average score (Tables 13–14).

Our subsequent experiments examine two complementary quality-improvement directions. Increasing the retention budget improves relative accuracy across all four evaluated long-context configurations (Table 15). Under matched average retained-token budgets, reducing block size from 128 to 8 improves VT accuracy by 10.20–23.80 points and overall accuracy by 3.03–7.73 points, with a measured latency trade-off (Table 16).

We do not claim that these experiments eliminate every quality–efficiency trade-off. Instead, they characterize the sensitivity of long-context quality to routing budgets and granularity, providing additional evidence about the strengths and limitations of the design.

We recognize that the same-backend Top-32/64 comparisons currently cover two models. At the same time, the main end-to-end evaluation covers five checkpoints and the full 6,500-example RULER suite, and the extended baseline comparison includes original MoBA and FlexPrefill across 12 model–length configurations (Table 17). These experiments provide complementary evidence at the routing, kernel, and full-model levels.

3. Overall Contribution

We believe the submission should be evaluated as an integrated sparse-attention systems contribution, rather than solely through the independent routing benefit at one retention budget.

The additional experiments distinguish the respective roles of routing and execution: the optimized backend delivers the primary acceleration at Top-16, while dynamic routing provides substantial additional benefits at larger budgets. This budget dependence is an experimentally characterized property of the design, not a claim that every component must improve every operating point.

We sincerely thank all reviewers for their constructive feedback. We have addressed the concerns raised in the individual review threads and provided additional controlled experiments, task-level analyses, and broader baseline comparisons in the revised manuscript.

We would greatly appreciate any further feedback on these additional results and clarifications, particularly whether they address the concerns raised in the original reviews. We hope the complete experimental evidence and our detailed responses will be taken into account in the overall assessment of LightningMoBA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.