- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LoGo, a token-level dynamic sparse attention mechanism that combines global- and local-span attentions with per-token adaptive selection. The key techniques include the coupled local and global branches with lightweight transformations between the two, a threshold-based learned gate to decide whether each token should bypass the global branch, and a progressive masking schedule to train this gate. In addition, a Triton-based sparse kernel is implemented to realize the speedup. LoGo has been compared with static inter/intra-layer local-global hybrid schemes, and shows better accuracy at long-context tasks.
Critical strengths
- Pushing forward local-global hybrid attention mechanisms to token-level adaptiveness.
- Showing substantial quality improvements over static baselines on long-context tasks.
- Actual Triton implementation and real runtime performance measurements.
Critical weaknesses
- The proposed mechanism requires substantial training efforts, especially compared to other training-free sparse attention designs. It is unclear whether the training needs to be done per task/dataset, or can be amortized with some generality.
- The evaluated baselines, i.e., intra/inter-layer static, seem to be overly simplified from actual prior works, like TransMamba and Oryx.
- Lack of comparison with more recent sparse attention mechanisms outside local-global hybrids, such as low-precision-based and clustering-based ones.
Questions for authors
-
In your experiments, when doing comparison on each task/dataset, are you training separate instances of LoGo on each specific task/dataset, or a single shared instance is used over all tasks/datasets? How is the generality of your design? In particular, the learned gate is critical for selecting between global and local. If it is trained on one task/dataset but used for another, the performance would likely degrade.
-
The main baselines, static inter/intra-layer-H, seem to be overly simplified from the actual prior works. Could you directly compare to TransMamba, Oryx, and/or CoLT5?
-
Could you quantitatively compare the quality and runtime performance of LoGo with state-of-the-art training-free sparse attention designs? Local-global hybrid is a relatively old paradigm compared to recent ones that uses low-precision computation or cluster retrieval methods, such as Quest, Twilight, RetrievalAttention, etc.
-
A single long-context benchmark RULER is somewhat limited to demonstrate the generality. Could you also evaluate on LongBench or other long-context benchmarks?
4 comments
[2/2] Response to Reviewer BDdZ (W3/Q3 & Q4)
W3 / Q3: Comparison with training-free sparse attention
Thank you for this valuable suggestion. Using the original 128K Transformer checkpoint without further training, we added comparisons against Quest [1] and a post-hoc MoBA-style baseline [2].
(1) Sparsity regimes and setup.
- Quest uses dense prefill and page-sparse decoding, with the first two layers kept dense.
- MoBA (post-hoc) uses top- blocks for sparse prefill and dense decoding. We set the block size to 320 (the smallest size avoiding 64K OOM) and choose top- to target a 50% attention FLOPs ratio.
- LoGo sparsifies attention across training, prefill, and decoding via learned token-level span allocation.
(2) Quality and actual budgets. We evaluate all 14 LongBench-64K tasks and eight RULER-64K tasks, reporting both accuracy and overall attention FLOPs ratio relative to full causal attention (defined in Appendix D.1):
| Method | LongBench-64K ↑ | Overall (%) | RULER-64K ↑ | Overall (%) |
|---|---|---|---|---|
| Transformer | 29.61 | 100.00 | 42.80 | 100.00 |
| LoGo | 29.77 | 56.68 | 52.50 | 57.87 |
| Quest | 29.66 | 98.87 | 43.29 | 99.82 |
| MoBA (post-hoc) | 28.73 | 51.20 | 34.86 | 50.14 |
Quest preserves Transformer quality but retains nearly dense overall budgets due to dense prefill. Post-hoc MoBA reduces prefill FLOPs to ~50% but suffers a sharp drop on RULER. LoGo achieves the highest LongBench and RULER accuracy while cutting overall attention FLOPs to ~58%.
(3) End-to-end runtime. Entries report TTFT / TPOT latency ratios relative to the dense CUDA baseline across Prefill/Decode batch sizes (values below 1.0 indicate speedup):
| Length | P/D batch | LoGo | Quest | MoBA (post-hoc) |
|---|---|---|---|---|
| 8K | 4/8 | 1.297 / 1.193 | 1.134 / 1.385 | 1.337 / 0.999 |
| 16K | 4/8 | 1.185 / 0.943 | 1.129 / 1.182 | 1.541 / 1.002 |
| 32K | 4/10 | 1.036 / 0.723 | 1.116 / 1.051 | 1.990 / 0.999 |
| 64K | 2/4 | 0.889 / 0.744 | 1.086 / 0.929 | 2.508 / 1.000 |
| 128K | 1/1 | 0.749 / 0.886 | 1.096 / 0.885 | 3.388 / 0.998 |
Takeaway: While training-free methods either leave prefill unaccelerated (Quest, with modest decode gains only at 64K–128K) or incur substantial prefill selection overhead and retrieval degradation (post-hoc MoBA), LoGo achieves stronger long-range retrieval via learned routing while reducing both TTFT and TPOT at long contexts (details in Appendix C.3.4, Tables 15–16).
[1] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference.
[2] MoBA: Mixture of Block Attention for Long-Context LLMs.
Q4: LongBench and broader long-context evaluation
Thank you for this suggestion. We have added 14 LongBench-64K tasks, six-domain PPL across 2K–64K, and BABILong-64K evaluations. LoGo achieves comparable performance on LongBench-64K (29.77 vs. 29.61 for Transformer), achieves the lowest PPL in 33/36 domain–length settings (9.29 vs. 9.44 average at 64K), and obtains the best overall BABILong-64K score (34.10 vs. 26.80–32.40). Please see Shared Response 2: Broader Long-Context Evaluation for the full results and manuscript updates.
Follow-up to Reviewer BDdZ
Dear Reviewer BDdZ,
Thank you very much for your helpful suggestions regarding cross-task generality, hybrid/sparse baselines, and broader long-context benchmarks, which have allowed us to provide much stronger empirical support for LoGo. We have addressed all your points with new comparative experiments and uploaded the revised manuscript. We would truly welcome your feedback on our responses and additional results before the discussion window ends.
Sincerely,
The Authors of Submission 1414
Thanks for the new results. I appreciate the added comparison with more baselines. Overall, the cost of the substantial training effort is still a non-negligible concern, especially given the relatively small improvements against the new baselines. I would maintain my original score.
[1/2] Response to Reviewer BDdZ (W1/Q1 & W2/Q2)
We sincerely thank the reviewer for the thoughtful and constructive feedback on training requirements, baseline comparisons, and long-context evaluation. Your suggestions have helped us clarify our training and evaluation protocols and substantially expand our comparative benchmarks, as detailed below.
Thank you for raising these important questions. We confirm that a single shared LoGo checkpoint is used across all evaluation tasks within each stage—no task-specific training, fine-tuning, or threshold tuning is performed.
(1) Single shared checkpoint across all tasks. Similar to standard foundation models and Mixture-of-Experts architectures, LoGo's backbone and router are jointly pretrained once on a broad, general-purpose corpus, followed by general context extension. For evaluation, the exact same stage checkpoint—with its learned per-layer thresholds frozen—is applied directly across benchmarks.
(2) Low training overhead and advantages over training-free sparsity. Training LoGo requires no separate routing stage or task-specific optimization; it simply plugs into standard pretraining pipelines with a single per-step threshold-update hook. Crucially, unlike training-free sparse attention—which offers zero compute savings during pretraining and often suffers from train–inference mismatch—LoGo aligns training and inference behavior while accelerating both forward and backward passes via query-sparse kernels.
(3) Cross-task generality and empirical evidence. While a router trained on a narrow domain could indeed degrade under severe distribution shift, pretraining on a diverse, large-scale corpus enables the gate to learn generalizable token-level allocation patterns. As shown below, a single shared checkpoint transfers cleanly across diverse domains without task-specific adaptation:
Notably, the exact same 128K checkpoint is evaluated across all 30 tasks/domains above (6 PPL domains, 14 LongBench tasks, and 10 BABILong tasks) without any task-specific adaptation, demonstrating strong cross-task generalization.
Thank you for this suggestion. We have added a controlled 1.5B pretraining comparison against TransMamba.
(1) Rationale for static local–global baselines. LoGo investigates token-level attention-span allocation rather than hybridizing distinct sequence-mixing operators. Keeping the softmax attention operator fixed across Inter-/Intra-layer baselines and LoGo isolates static vs. dynamic span allocation as the single controlled variable, whereas models like TransMamba and Oryx simultaneously alter the underlying sequence-mixing mechanism.
(2) Direct TransMamba comparison (1.5B, 100B tokens). Following your suggestion, we adapted TransMamba's sequence-level attention-to-Mamba transition, shared projections, and memory converter (retaining its V9 schedule) to our 1.5B backbone, adjusting its FFN width to align parameter counts (within 0.5% difference):
LoGo outperforms TransMamba across every metric, reducing both WikiText and LAMBADA PPL while gaining +1.49 pp on CSR, +2.49 pp on Recall, and +23.28 pp on RULER-8K.
(3) Why Oryx and CoLT5 are not directly comparable in our setup. Oryx relies on manually specified, task-dependent schedules for switching between sequence mixers, making a fair, unified compute-matched comparison across diverse tasks nontrivial. CoLT5 is designed for encoders and performs non-causal top-k routing, which does not directly transfer to causal decoders without major redesign. Addressing these exact limitations is a primary motivation of LoGo.