- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LightningMoBA, which replaces MoBA’s fixed Top-k routing with per-query threshold selection. It groups work by KV block, splits high-fanout columns, and pipelines query gathers with attention computation on Hopper. The block-scoring function and softmax over retained tokens are preserved. On the RULER suite across five checkpoints, the reported speedups are 1.20-1.60x. Quality improves for three models and drops for two.
Critical strengths
The new kernel accelerates the attention by 16.9-33.8% while computing the same selected attention pairs. This is a significant efficiency improvement.
Loading queries while other work runs makes the tested query-gather kernel 2.53-2.65x faster compared to loading them one step at a time. Profiling also shows fewer stalls waiting for memory.
Critical weaknesses
All five models lose quality at 64K. Table 10 increases the threshold router’s budget without showing whether increasing Top-k gives similar quality gains.
End-to-end gains shrink sharply with length: Qwen3-4B goes from 2.41x at 4K to 1.13x at 128K. The full attention path also takes 15.8 ms at 32K, versus 13.0 ms for dense FA3 in Figure 5.
Questions for authors
How did you check the budget match in the full evaluation?
Can you show both routers’ quality-latency curves using the same final kernel?
Which stages explain the smaller end-to-end gain at 128K?
2 comments
Follow-up on Additional Experiments and Reviewer Concerns
We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. As the discussion period approaches its conclusion, we would like to briefly follow up on our detailed responses to the individual reviews and the additional experiments presented in the revised manuscript.
We have addressed the major concerns through controlled same-backend comparisons (Table 12), fine-grained RULER analyses (Tables 13–14), retention-budget and block-granularity studies (Tables 15–16), and broader baseline evaluations (Table 17).
We would greatly appreciate any final feedback on whether these additional results and clarifications have addressed your original concerns. We remain available to clarify any outstanding questions during the discussion period.
Thank you again for your consideration and for evaluating the complete experimental evidence and system-level contributions of LightningMoBA.
Response to Reviewer CkhR
We thank the reviewer for recognizing the efficiency improvements from KV-centric execution and query-gather pipelining. We conducted additional experiments to verify retention-budget matching and isolate the contribution of dynamic routing under the same optimized backend. We address the three questions below.
Q1. Verification of Matched Retention Budgets
We calibrate threshold parameters on held-out samples to match the average retention budget of fixed Top-k routing, rather than enforcing an identical number of retained blocks per query. The thresholds remain fixed throughout the full evaluation.
We additionally measured the realized number of retained attention tokens per query on the complete Qwen3-4B RULER evaluation set across all six context lengths (4K–128K), including compulsory sink and local blocks.
As reported in Appendix C.1, Table 11, dynamic routing's average retained-token counts differ from fixed Top-16 by only −1.01% to +0.88%. These measurements confirm closely matched average sparsity across the evaluated lengths, providing a consistent basis for quality and latency comparisons.
Q2. Quality–Latency Comparison Using the Same Optimized Backend
To isolate routing from execution optimizations, we added end-to-end comparisons between fixed Top-k and dynamic threshold routing at budgets of 16, 32, and 64. Both strategies use the identical LightningMoBA execution backend, including KV-centric execution, fanout-aware splitting, and runtime pipelining, with the same RULER samples and evaluation settings.
The results in Appendix C.1, Table 12 and Figure 6 cover two models, three context lengths (32K, 64K, 128K), and three retention budgets:
These controlled results establish that dynamic routing independently improves the quality–latency trade-off, particularly at larger retention budgets, without attributing KV-centric execution or pipelining gains to routing.
We also analyzed the 64K quality degradation in Appendix C.2, Tables 13–16, identifying variable tracking (VT) as a major contributor. Two complementary experiments further clarify how quality can be improved:
Together, these findings demonstrate two ways to improve long-context quality: retaining more evidence or using finer-grained scoring under the same token budget, with an explicit quality–efficiency trade-off.
Q3. Why Does the End-to-End Speedup Decrease at Longer Context Lengths?
LightningMoBA optimizes sparse attention during prefill, while other model operations, tensor-parallel communication, and dense decoding remain unchanged. Their costs limit how much attention-level acceleration translates into full-model speedup.
Figure 5 confirms substantial long-context benefits in a controlled single-rank setting: at 128K, the complete sparse-attention path decreases from 135.1 ms to 83.2 ms, a 1.62× speedup over FlashMoBA. The end-to-end evaluation instead uses TP=8, introducing cross-rank communication and synchronization.
Appendix B.3.2, Table 9 reveals significant TP-rank workload imbalance at 128K. Selected query-block pair counts and K1 attention latencies have mean max-to-mean ratios of 1.88× and 1.86×, respectively. In 428 of 432 request-layer groups, the rank with the most selected pairs is also the slowest K1 rank.
Fanout-aware splitting balances work within each rank but cannot redistribute work across TP ranks. Consequently, synchronization remains sensitive to the slowest rank.
Thus, LightningMoBA retains strong sparse-attention acceleration at 128K, while unchanged full-model costs and TP execution overheads, potentially amplified by cross-rank imbalance, constrain the end-to-end speedup.