acceptodds
Under review as a conference paper at ICLR 2027

LoGo: Token-Level Dynamic Local-Global Attention

Abstract

As context lengths scale, attention increasingly becomes a primary computational bottleneck in large language models. Standard Transformers remain powerful but computationally inefficient, as they allocate the same attention budget to every token regardless of its contextual demand. Existing local-global hybrids provide a more efficient alternative by mixing restricted- and full-context attention, but they typically allocate span statically across layers or heads. To address these limitations, we propose LoGo, a token-level dynamic local-global attention mechanism that uses attention span as a direct proxy for attention budget allocation. Each LoGo layer contains coupled local and global branches: all tokens receive efficient local attention over a restricted context window, while a learned gate activates global attention with full-context access only for tokens requiring long-range information. A threshold-based budget controller maintains a target global ratio without auxiliary losses, and a progressive masking schedule stabilizes training before sparse routing takes effect. We further implement query-sparse Triton kernels that convert reduced global-attention computation into practical speedups. Extensive experiments validate LoGo's effectiveness, showing that it preserves the scaling behavior of full-attention Transformers across model sizes. In controlled comparisons, LoGo improves over the full-attention Transformer and matched-budget static local-global hybrids, with clear gains on long-range retrieval. Analysis further shows that LoGo learns interpretable span allocation patterns. These results suggest that learned token-level span allocation is an effective and scalable way to improve the long-context performance-compute trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Attention, Long-Context Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BDdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LoGo, a token-level dynamic sparse attention mechanism that combines global- and local-span attentions with per-token adaptive selection. The key techniques include the coupled local and global branches with lightweight transformations between the two, a threshold-based learned gate to decide whether each token should bypass the global branch, and a progressive masking schedule to train this gate. In addition, a Triton-based sparse kernel is implemented to realize the speedup. LoGo has been compared with static inter/intra-layer local-global hybrid schemes, and shows better accuracy at long-context tasks.

Critical strengths
  • Pushing forward local-global hybrid attention mechanisms to token-level adaptiveness.
  • Showing substantial quality improvements over static baselines on long-context tasks.
  • Actual Triton implementation and real runtime performance measurements.
Critical weaknesses
  • The proposed mechanism requires substantial training efforts, especially compared to other training-free sparse attention designs. It is unclear whether the training needs to be done per task/dataset, or can be amortized with some generality.
  • The evaluated baselines, i.e., intra/inter-layer static, seem to be overly simplified from actual prior works, like TransMamba and Oryx.
  • Lack of comparison with more recent sparse attention mechanisms outside local-global hybrids, such as low-precision-based and clustering-based ones.
Questions for authors
  1. In your experiments, when doing comparison on each task/dataset, are you training separate instances of LoGo on each specific task/dataset, or a single shared instance is used over all tasks/datasets? How is the generality of your design? In particular, the learned gate is critical for selecting between global and local. If it is trained on one task/dataset but used for another, the performance would likely degrade.

  2. The main baselines, static inter/intra-layer-H, seem to be overly simplified from the actual prior works. Could you directly compare to TransMamba, Oryx, and/or CoLT5?

  3. Could you quantitatively compare the quality and runtime performance of LoGo with state-of-the-art training-free sparse attention designs? Local-global hybrid is a relatively old paradigm compared to recent ones that uses low-precision computation or cluster retrieval methods, such as Quest, Twilight, RetrievalAttention, etc.

  4. A single long-context benchmark RULER is somewhat limited to demonstrate the generality. Could you also evaluate on LongBench or other long-context benchmarks?

4 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer BDdZ (W1/Q1 & W2/Q2)

We sincerely thank the reviewer for the thoughtful and constructive feedback on training requirements, baseline comparisons, and long-context evaluation. Your suggestions have helped us clarify our training and evaluation protocols and substantially expand our comparative benchmarks, as detailed below.

W1 / Q1: Training requirements and cross-task generality

Thank you for raising these important questions. We confirm that a single shared LoGo checkpoint is used across all evaluation tasks within each stage—no task-specific training, fine-tuning, or threshold tuning is performed.

(1) Single shared checkpoint across all tasks. Similar to standard foundation models and Mixture-of-Experts architectures, LoGo's backbone and router are jointly pretrained once on a broad, general-purpose corpus, followed by general context extension. For evaluation, the exact same stage checkpoint—with its learned per-layer thresholds frozen—is applied directly across benchmarks.

(2) Low training overhead and advantages over training-free sparsity. Training LoGo requires no separate routing stage or task-specific optimization; it simply plugs into standard pretraining pipelines with a single per-step threshold-update hook. Crucially, unlike training-free sparse attention—which offers zero compute savings during pretraining and often suffers from train–inference mismatch—LoGo aligns training and inference behavior while accelerating both forward and backward passes via query-sparse kernels.

(3) Cross-task generality and empirical evidence. While a router trained on a narrow domain could indeed degrade under severe distribution shift, pretraining on a diverse, large-scale corpus enables the gate to learn generalizable token-level allocation patterns. As shown below, a single shared checkpoint transfers cleanly across diverse domains without task-specific adaptation:

Shared CheckpointEvaluationTransformerLoGo
8K PTWikiText PPL ↓16.1216.04
8K PTLAMBADA PPL ↓8.367.50
8K PTCSR Avg. (%) ↑55.7056.30
128K CTSix-domain 64K PPL Avg. ↓9.449.29
128K CTLongBench-64K Avg. (%) ↑29.6129.77
128K CTBABILong-64K Avg. (%) ↑26.8034.10

Notably, the exact same 128K checkpoint is evaluated across all 30 tasks/domains above (6 PPL domains, 14 LongBench tasks, and 10 BABILong tasks) without any task-specific adaptation, demonstrating strong cross-task generalization.

W2 / Q2: Comparison with TransMamba, Oryx, and CoLT5

Thank you for this suggestion. We have added a controlled 1.5B pretraining comparison against TransMamba.

(1) Rationale for static local–global baselines. LoGo investigates token-level attention-span allocation rather than hybridizing distinct sequence-mixing operators. Keeping the softmax attention operator fixed across Inter-/Intra-layer baselines and LoGo isolates static vs. dynamic span allocation as the single controlled variable, whereas models like TransMamba and Oryx simultaneously alter the underlying sequence-mixing mechanism.

(2) Direct TransMamba comparison (1.5B, 100B tokens). Following your suggestion, we adapted TransMamba's sequence-level attention-to-Mamba transition, shared projections, and memory converter (retaining its V9 schedule) to our 1.5B backbone, adjusting its FFN width to align parameter counts (within 0.5% difference):

ModelWikiText PPL ↓LAMBADA PPL ↓CSR Avg. (%) ↑Recall Avg. (%) ↑RULER-8K Avg. (%) ↑
TransMamba16.479.3454.8155.7264.69
LoGo16.047.5056.3058.2187.97

LoGo outperforms TransMamba across every metric, reducing both WikiText and LAMBADA PPL while gaining +1.49 pp on CSR, +2.49 pp on Recall, and +23.28 pp on RULER-8K.

(3) Why Oryx and CoLT5 are not directly comparable in our setup. Oryx relies on manually specified, task-dependent schedules for switching between sequence mixers, making a fair, unified compute-matched comparison across diverse tasks nontrivial. CoLT5 is designed for encoders and performs non-causal top-kk routing, which does not directly transfer to causal decoders without major redesign. Addressing these exact limitations is a primary motivation of LoGo.

AuthorsOpenReview ↗

[2/2] Response to Reviewer BDdZ (W3/Q3 & Q4)

W3 / Q3: Comparison with training-free sparse attention

Thank you for this valuable suggestion. Using the original 128K Transformer checkpoint without further training, we added comparisons against Quest [1] and a post-hoc MoBA-style baseline [2].

(1) Sparsity regimes and setup.

  • Quest uses dense prefill and page-sparse decoding, with the first two layers kept dense.
  • MoBA (post-hoc) uses top-kk blocks for sparse prefill and dense decoding. We set the block size to 320 (the smallest size avoiding 64K OOM) and choose top-kk to target a 50% attention FLOPs ratio.
  • LoGo sparsifies attention across training, prefill, and decoding via learned token-level span allocation.

(2) Quality and actual budgets. We evaluate all 14 LongBench-64K tasks and eight RULER-64K tasks, reporting both accuracy and overall attention FLOPs ratio RfR_f relative to full causal attention (defined in Appendix D.1):

MethodLongBench-64K ↑Overall RfR_f (%)RULER-64K ↑Overall RfR_f (%)
Transformer29.61100.0042.80100.00
LoGo29.7756.6852.5057.87
Quest29.6698.8743.2999.82
MoBA (post-hoc)28.7351.2034.8650.14

Quest preserves Transformer quality but retains nearly dense overall budgets due to dense prefill. Post-hoc MoBA reduces prefill FLOPs to ~50% but suffers a sharp drop on RULER. LoGo achieves the highest LongBench and RULER accuracy while cutting overall attention FLOPs to ~58%.

(3) End-to-end runtime. Entries report TTFT / TPOT latency ratios relative to the dense CUDA baseline across Prefill/Decode batch sizes (values below 1.0 indicate speedup):

LengthP/D batchLoGoQuestMoBA (post-hoc)
8K4/81.297 / 1.1931.134 / 1.3851.337 / 0.999
16K4/81.185 / 0.9431.129 / 1.1821.541 / 1.002
32K4/101.036 / 0.7231.116 / 1.0511.990 / 0.999
64K2/40.889 / 0.7441.086 / 0.9292.508 / 1.000
128K1/10.749 / 0.8861.096 / 0.8853.388 / 0.998

Takeaway: While training-free methods either leave prefill unaccelerated (Quest, with modest decode gains only at 64K–128K) or incur substantial prefill selection overhead and retrieval degradation (post-hoc MoBA), LoGo achieves stronger long-range retrieval via learned routing while reducing both TTFT and TPOT at long contexts (details in Appendix C.3.4, Tables 15–16).

[1] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference.

[2] MoBA: Mixture of Block Attention for Long-Context LLMs.

Q4: LongBench and broader long-context evaluation

Thank you for this suggestion. We have added 14 LongBench-64K tasks, six-domain PPL across 2K–64K, and BABILong-64K evaluations. LoGo achieves comparable performance on LongBench-64K (29.77 vs. 29.61 for Transformer), achieves the lowest PPL in 33/36 domain–length settings (9.29 vs. 9.44 average at 64K), and obtains the best overall BABILong-64K score (34.10 vs. 26.80–32.40). Please see Shared Response 2: Broader Long-Context Evaluation for the full results and manuscript updates.

AuthorsOpenReview ↗

Follow-up to Reviewer BDdZ

Dear Reviewer BDdZ,

Thank you very much for your helpful suggestions regarding cross-task generality, hybrid/sparse baselines, and broader long-context benchmarks, which have allowed us to provide much stronger empirical support for LoGo. We have addressed all your points with new comparative experiments and uploaded the revised manuscript. We would truly welcome your feedback on our responses and additional results before the discussion window ends.

Sincerely,
The Authors of Submission 1414

Thanks for the new results. I appreciate the added comparison with more baselines. Overall, the cost of the substantial training effort is still a non-negligible concern, especially given the relatively small improvements against the new baselines. I would maintain my original score.

Reviewer_CRbcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoGo, a local–global token-sparse attention mechanism designed as an efficient replacement for full quadratic attention. The paper shows that LoGo attention scales well with respect to both model size and training data scale (up to 3.3B parameters and 100B tokens).

Critical strengths

The critical strength of this paper is its extensive set of large-scale experiments. However, in its current state, the paper does not sufficiently demonstrate the strength of this research, as the training-related information is not disclosed in enough detail.

Critical weaknesses

I find the critical weaknesses to lie in two aspects: (1) the model architecture is not particularly novel, and (2) the training-related information (training dynamics, dataset, training loop, and infrastructure) is insufficiently reported.

I strongly suggest that the authors provide the following information:

  • Training dynamics records (loss, gradients, and validation metrics)
  • Infrastructure setup and practical experiences
  • Dataset configurations
  • Optimizer setups

Regarding the architecture, I believe the model should be compared with prior dynamic token sparsity methods, such as:

  • Token Merging
  • Sparse Token Transformer with Attention Back Tracking
Questions for authors

Please refer to the Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CRbc

We sincerely thank the reviewer for the thoughtful feedback on novelty, training details, and related work. Your suggestions have meaningfully helped us sharpen LoGo's positioning and expand the training information, which we address point by point below.

W1: Architectural novelty

Thank you for encouraging us to clarify LoGo's architectural contribution. While acknowledging prior conditional-attention work, we clarify that LoGo's contribution lies in the joint design and validation of token-level span allocation to improve the performance–compute trade-off in pretrained decoder-only LLMs. Please see Shared Response 1: Novelty and Positioning for our research question, differences from prior work, and supporting experimental evidence.

W1 / W2: Training details and reproducibility

Thank you for highlighting the need for fuller training records and implementation details. We have included loss and pre-clipping gradient-norm curves for all four models across the three training stages, and collected the requested infrastructure, dataset, and optimizer configurations. The observed runs converge stably, with LoGo achieving lower late-stage training loss. Please see Shared Response 3: Training Details and Reproducibility for these details and the additional FineWeb-Edu results.

W3: Comparison with prior dynamic token sparsity methods

Thank you for pointing out these relevant works. We acknowledge Token Merging [1] and Sparse Token Transformer with Attention Back Tracking [2] as important contributions to token-level adaptive computation, and apologize for omitting them. We have added these references and clarified their relationship to LoGo in Appendix A.

(1) Relationship to token merging and pruning. Token Merging combines similar tokens in ViTs, while Attention Back Tracking prunes tokens in vision and language encoders. These methods permanently reduce the tokens processed by subsequent layers. LoGo instead learns token-level attention-span allocation in decoder-only LLMs, selecting which queries execute global attention.

(2) Why direct comparisons are structurally nontrivial. Token merging and pruning are designed for non-causal encoders where dropping or combining hidden states does not break autoregressive next-token prediction. Adapting them to causal decoder LLMs requires non-trivial redesigns—such as preserving strict causal dependencies, defining how merged or pruned tokens map to persistent KV caches, and building custom kernels. We will make these structural distinctions explicit in the revision while acknowledging both works.

(3) Added decoder-compatible sparsity baselines. To broaden our empirical comparisons within the causal LLM paradigm, we evaluated Quest [3] and a post-hoc MoBA-style baseline (block size 320) [4] on our 128K checkpoint across 14 LongBench-64K tasks and 8 RULER-64K tasks, reporting both accuracy and overall attention FLOPs ratio RfR_f relative to full causal attention (defined in Appendix D.1):

MethodLongBench-64K ↑Overall RfR_f (%)RULER-64K ↑Overall RfR_f (%)
Transformer29.61100.0042.80100.00
LoGo29.7756.6852.5057.87
Quest29.6698.8743.2999.82
MoBA (post-hoc)28.7351.2034.8650.14

LoGo outperforms both baselines on RULER-64K (+9.21 pp over Quest and +17.64 pp over post-hoc MoBA) while cutting overall attention FLOPs to ~58%. Detailed breakdowns and end-to-end runtime comparisons are provided in Appendix C.3.4 (Tables 15–16).

[1] Token Merging: Your ViT But Faster.

[2] Sparse Token Transformer with Attention Back Tracking.

[3] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference.

[4] MoBA: Mixture of Block Attention for Long-Context LLMs.

AuthorsOpenReview ↗

Follow-up to Reviewer CRbc

Dear Reviewer CRbc,

We sincerely appreciate your constructive comments on LoGo's positioning, training details, and dynamic-sparsity baselines. Your feedback has helped us make the experimental setup much more transparent. We have provided detailed clarifications, full training records, and additional experiments in our responses and revised PDF. With the discussion deadline approaching, we would be grateful for your perspective on our updates and welcome any further questions.

Warm regards,
The Authors of Submission 1414

Reviewer_MT9mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents LoGo, a token-level dynamic local-global attention mechanism for long-context large language models. The core idea treats attention span as a computational budget: all tokens perform low-cost sliding-window local attention, while a learnable gate activates full global attention only for tokens that require long-range information. LoGo employs lightweight transformations to enable parameter sharing and functional differentiation between the local and global branches. It maintains the fraction of global activations around a target budget via an adaptive threshold and uses progressive masking to mitigate routing instability in early training stages. To translate theoretical FLOPs savings into practical efficiency, the authors implement a Triton sparse attention kernel that computes only for selected queries while preserving the full KV context. Experiments span model sizes from 200M to 3.3B parameters, and comparisons with full attention and static local-global hybrid architectures are conducted on the 1.5B model.

Critical strengths
  1. The method is well-designed and amenable to practical implementation. LoGo casts attention span into a token-level dynamic computational budget. With lightweight branch decoupling, threshold-based budget control, progressive masking, and a query-sparse Triton kernel, it forms an end-to-end pipeline spanning model design to efficient implementation. Meanwhile, it retains dense KV, allowing selected queries to precisely access the full context.

  2. The experiments are conducted at a substantial scale, with clear benefits for long-context settings. The authors validate the scalability of the approach across 200M–3.3B parameter models and compare against both full-attention Transformers and static hybrid architectures under matched compute budgets. LoGo maintains competitive performance on language modeling and delivers notable, consistent improvements on long-context retrieval tasks ranging from 32K to 128K context lengths.

Critical weaknesses
  1. The paper controls the compute budget via the proportion of global queries. However, queries at different positions incur different computational costs under causal attention; the true FLOPs correspond to the position-weighted sum of selected queries. Consequently, an “identical global ratio” does not necessarily guarantee “identical attention FLOPs”, which weakens the conclusion of fair comparison against static hybrid baselines.
  2. Efficiency experiments only evaluate the attention operator under batch size 1. Training throughput, prefill latency, decoding speed, peak memory usage, or serving throughput of the full model are not reported. The proposed Triton implementation outperforms the mature CUDA FlashAttention only under high sparsity rates. It therefore remains unproven whether theoretical FLOPs savings translate into practical gains for large-model training or deployment.
  3. The paper lacks comparisons against random routing, fixed-interval routing, position-heuristic routing, or intra-batch top-k routing. It is thus unclear whether performance gains stem from the learned token-wise span allocation, or merely from the base architecture of “local attention plus partial global queries”.
  4. Some improvements on language modeling and commonsense reasoning are modest and may fall within training variance. The long-context advantage is mainly observed on synthetic needle-style RULER benchmarks, with insufficient validation on real long-document understanding, long-form generation, and repository-level code tasks.
  5. Token-level conditional computation, light-heavy attention branches, and dynamic routing have all appeared in prior work. LoGo’s contribution is closer to a systematic integration and engineering realization tailored for decoder-only long-context models. In addition, the paper uses non-public internal corpora and does not fully specify the positional encoding scheme for 32K/128K context extension, hardware setup, inference threshold calibration, and complete implementation configurations, raising barriers for reproducibility by the community.
Questions for authors
  1. The paper approximates the compute budget using the fraction of global queries, yet queries at different positions have varying costs under causal attention. Could the authors report the true position-weighted attention FLOPs and re-run comparisons among all methods under strictly matched true FLOPs?
  2. Could the authors provide full-model efficiency metrics, including training throughput, prefill latency, batched decoding throughput and peak memory usage, rather than only attention kernel latency measured at batch size 1?
  3. How much improvement does LoGo achieve over random routing, fixed-interval routing, or intra-batch top-k routing? These experiments are critical to verifying that the learned token-level span allocation is indeed effective.
  4. How is the routing threshold determined during inference? Does the global activation ratio and actual compute budget remain stable across shifts in domain, sequence length and batch size?
6 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer MT9m (W1/Q1)

We sincerely thank the reviewer for the thoughtful review and constructive feedback on compute accounting, efficiency, and evaluation. We have added new experiments and thoroughly revised the manuscript to address each point, as detailed below.

W1 / Q1: Position-weighted attention FLOPs and fair comparisons

Thank you for highlighting this important distinction. We agree that equal global-query ratios do not imply equal causal attention FLOPs. We have revised the manuscript to distinguish the query-count budget from the position-weighted FLOPs budget and provide the following evidence.

(1) Query-count and position-weighted budgets. We distinguish

\qquad R_f^{(\ell)}=\frac{\sum_{(b,t)\in\mathcal B}z_{b,t}^{(\ell)}a_{b,t}}{\sum_{(b,t)\in\mathcal B}a_{b,t}},$$ where $\mathcal B$ contains valid query positions and $a_{b,t}$ is the number of causally visible keys. $R_c$ measures the global-query fraction, while $R_f$ measures the attention FLOPs ratio relative to full causal attention. **With the original thresholds, mean $R_c/R_f$ is 42.83%/45.96% for PPL, 17.32%/18.81% for CSR (Table 2), and 36.59%/42.05% for short recall (Table 3)**—all below the static hybrids' 50%. **RULER has mean $R_c/R_f$ of 54.37%/56.16% (32K CT) and 56.56%/57.55% (128K CT)**. Because the router is trained to target $R_c\approx0.5$ on the training distribution, inference budgets can vary according to each task's demand for global context. We have incorporated both ratios into Tables 2–4, with per-task breakdowns in Appendix D (Tables 21–23). **(2) RULER reevaluation under a FLOPs cap.** We freeze model weights and raise all layer thresholds by a shared offset ($0.04$ and $0.05$ for the 32K and 128K checkpoints, respectively), selected on an independent RULER calibration set using FLOPs budgets alone. The target is to constrain $R_f\leq50\%$, averaged over tasks and evaluated lengths. | Model | 32K CT Avg. ↑ | $R_f$ (%) | 128K CT Avg. ↑ | $R_f$ (%) | |---|---|---|---|---| | Inter-layer hybrid | 80.2 | 50.00 | 58.5 | 50.00 | | Intra-layer hybrid | 75.8 | 50.00 | 47.6 | 50.00 | | LoGo, recalibrated | **81.90** | **49.23** | **63.16** | **48.52** | **Despite post-hoc threshold adjustment, LoGo still surpasses both static hybrids under the FLOPs cap.** Full results, including the uncalibrated LoGo and Transformer baselines, are reported in revised Table 4. **(3) FLOPs-based training control.** To directly control FLOPs during training, we simply replace $R_c$ with $R_f$ in the existing threshold update: $$b_{\mathrm{th}}^{(\ell)}\leftarrow b_{\mathrm{th}}^{(\ell)} +\gamma\operatorname{sign}\left(R_f^{(\ell)}-\rho^{(\ell)}\right).$$ We validate this at the third point of our scaling study (900M, 15B tokens), without changing the training recipe. Evaluated on the same held-out training slice, **the realized FLOPs ratio drops from 56.22% to 49.57%, while RULER-8K improves by +10.28 pp** with PPL, CSR, and recall remaining comparable to the count-controlled model. FLOPs-based LoGo also improves overall performance over the Transformer baseline: | Model | Training-data $R_c/R_f$ (%) | Train loss ↓ | WikiText PPL ↓ | LAMBADA PPL ↓ | CSR (%) ↑ | Recall (%) ↑ | RULER-8K (%) ↑ | |---|---|---|---|---|---|---|---| | Transformer | 100/100 | 2.294 | 22.04 | 17.72 | 47.63 | 48.91 | 51.38 | | LoGo: count controller | 49.61/56.22 | 2.286 | 21.65 | 16.43 | 48.66 | 49.65 | 55.16 | | LoGo: FLOPs controller | **42.73/49.57** | 2.289 | 21.83 | 16.62 | 48.35 | 48.49 | 65.44 | **(4) Evidence from the existing ratio sweep (Table 19).** The 0.25-ratio checkpoint achieves 73.2 on RULER-32K at 29.87% realized FLOPs, outperforming the 50% inter-layer hybrid (66.5) at nearly half the attention FLOPs, further confirming its retrieval advantage at a lower compute budget. **Together, these results strengthen LoGo's performance–compute advantage at comparable or lower attention FLOPs budgets.**
AuthorsOpenReview ↗

[2/3] Response to Reviewer MT9m (W2/Q2 & W3/Q3)

W2 / Q2: Full-model efficiency

Thank you for this valuable suggestion. We have added full-model benchmarks on 1.5B models (A100-80GB GPUs) measuring training throughput, TTFT, batched decoding throughput/TPOT, and peak memory, extending the operator-level results in Figure 2.

(1) End-to-end throughput and latency. The table below reports relative gains of LoGo (at 50% and 25% global-attention FLOPs budgets) over dense baselines, formatted as Triton / CUDA baselines (absolute metrics in Appendix B.2, Figure 4). All metrics represent speedup ratios (>1×>1\times indicates faster execution). The final column reports the decode peak-memory ratio, which remains invariant across budgets and baselines:

LengthTrain (50%)Train (25%)TTFT (50%)TTFT (25%)Decode (50%)Decode (25%)Decode Mem.
8K1.01 / 0.811.09 / 0.870.89 / 0.810.94 / 0.871.31 / 1.311.80 / 1.811.07×
16K1.17 / 0.811.36 / 0.941.02 / 0.881.13 / 0.981.70 / 1.342.47 / 1.961.06×
32K1.39 / 0.821.78 / 1.051.21 / 1.001.44 / 1.183.18 / 1.384.56 / 1.981.05×
64K1.62 / 0.852.33 / 1.221.46 / 1.151.90 / 1.516.48 / 1.349.09 / 1.881.04×
128K1.83 / 0.872.95 / 1.401.74 / 1.322.57 / 1.9518.65 / 1.1223.86 / 1.441.04×

Setup: Training and TTFT use our Triton query-sparse kernel. Batched decoding uses a CUDA kernel built on the same FlashAttention cache API as the CUDA baseline. TTFT and decoding use the largest common batch size per length within 80GB HBM (full protocols in Appendix B.2).

  • Training & Prefill: LoGo-25% surpasses the dense CUDA baseline from 32K onward in both training throughput (up to 1.40× at 128K) and TTFT (up to 1.95×). Both budgets show monotonically growing speedups over dense Triton across all lengths.
  • Batched Decoding: Under the unified CUDA backend, LoGo achieves 1.31–1.38× (50% budget) and 1.80–1.98× (25% budget) decoding throughput over the CUDA baseline across 8K–64K.

(2) Decode peak memory. Peak decode memory increases by only 4–7% over the dense baseline and remains invariant across sparsity budgets, as LoGo preserves the full KV cache for global queries.

(3) Isolating algorithmic gains from backend differences. Within the same Triton backend, full-model training speedups reach 1.83× / 2.95× at 128K (aligning with the 1.99× operator-level speedup at 64K in Figure 2). Comparing sparse Triton against dense CUDA conflates algorithmic sparsity with low-level kernel optimization, while same-backend comparisons isolate the true algorithmic gain. Moreover, our CUDA decoding results confirm that query-sparsity speedups are backend-agnostic, while a dedicated CUDA training/prefill kernel would unlock further end-to-end gains.

These results demonstrate practical full-model speedups in long-context prefill and batched decoding, alongside clear training gains at higher sparsity, with minimal memory overhead.

W3 / Q3: Learned routing versus alternative routing strategies

Thank you for suggesting these router ablations. We trained random and fixed-interval routing controls from scratch within the exact same LoGo architecture, preserving the local/global branches, branch parameterization, and learned output fusion.

(1) Routing controls. At the 1.5B scale, both controls follow the same 100B-token, 8K pretraining recipe. Random independently selects each query with probability 0.5, while Periodic selects every other token with alternating parity across layers. Both use content-independent selection and start with the same 20% dense warmup before switching to their prescribed routing.

(2) Results. We evaluate PPL, CSR, short-context recall, and eight RULER-8K tasks:

RouterTrain Loss ↓Wiki PPL ↓LAMBADA PPL ↓CSR (%) ↑Recall (%) ↑RULER-8K (%) ↑
LoGo (learned)2.11216.047.5056.3058.2187.97
Periodic2.12016.287.9355.8957.5784.81
Random2.12616.477.8855.8656.7779.52

LoGo's learned router outperforms Periodic by +3.16 pp and Random by +8.45 pp on RULER-8K, while consistently achieving lower PPL and higher CSR and recall (per-task breakdowns in Appendix C.4.1, Table 17), confirming the advantage of content-aware, learned span allocation over fixed or random routing.

AuthorsOpenReview ↗

[3/3] Response to Reviewer MT9m (W4, W5 & Q4)

W4: Training variance and broader long-context evaluation

Thank you for raising these points regarding evaluation reliability and breadth.

(1) Consistent language-modeling gains. Across all five scaling comparison sizes (200M to 3.3B), LoGo reliably reduces training loss by 0.004–0.012 and improves both WikiText and LAMBADA PPL. Crucially, these gains persist under a larger TPP (100B-token pretraining) and after 32K context extension across both PPL and CSR. This cross-scale and cross-stage consistency confirms that the improvements reflect genuine modeling gains rather than training variance.

(2) Broader long-context evaluation. We have added six-domain PPL across 2K–64K, 14 LongBench-64K tasks, and BABILong-64K evaluations. LoGo achieves the lowest PPL in 33/36 domain–length settings and the best overall BABILong score, while remaining comparable on LongBench. Please see Shared Response 2: Broader Long-Context Evaluation for the full results and manuscript updates.

Together, these results confirm that LoGo delivers robust language-modeling gains, alongside strong performance across both diagnostic retrieval and broader long-context benchmarks.

W5: Novelty, positioning, and reproducibility

Thank you for encouraging us to clarify LoGo's positioning and reproducibility details.

(1) Novelty and positioning. We sincerely acknowledge prior work on conditional computation and dynamic routing. LoGo's contribution is the joint design and validation of token-level span allocation for pretrained decoder-only LLMs, improving the performance–compute trade-off with practical speedups. Please see Shared Response 1: Novelty and Positioning for the research question, design contributions, and supporting evidence.

(2) Reproducibility and implementation details. We clarify the hardware, training configuration, RoPE setup, and inference thresholds in Shared Response 3: Training Details and Reproducibility. We also added a 200M/5B-token FineWeb-Edu experiment, where LoGo reduces WikiText/LAMBADA PPL (38.96 vs. 41.22 and 125.03 vs. 158.75) and improves average CSR (41.65% vs. 40.90%) over the Transformer, confirming its effectiveness on public English corpus.

Q4: Inference thresholds and budget stability

Thank you for asking about inference thresholding and routing stability across distribution shifts. We clarify how thresholds are set and how realized budgets behave across domains, sequence lengths, and batch sizes.

(1) Inference thresholds are inherited from training and kept fixed. The gate is trained jointly with the language model, and each layer's threshold is updated during training through budget feedback, analogous to auxiliary-loss-free load balancing [1]. Default inference loads these per-layer thresholds directly without task-specific tuning.

(2) The realized budget is input-dependent, rather than a fixed 50% inference ratio. Using the original 128K CT checkpoint and thresholds, we measure the global-query count and position-weighted FLOPs ratios (Rc/RfR_c/R_f) across four domains from 1K to 64K:

Input lengthWikiTextPG19GSM8KCodeParrotAvg.
1K42.90% / 47.78%40.00% / 44.03%23.47% / 25.19%42.09% / 46.92%37.12% / 40.98%
4K50.59% / 52.83%48.57% / 51.44%27.76% / 29.09%52.33% / 55.93%44.82% / 47.32%
8K53.07% / 55.11%51.68% / 53.98%28.79% / 29.68%56.32% / 59.31%47.46% / 49.52%
16K54.13% / 55.11%53.33% / 54.76%29.97% / 30.83%57.09% / 57.98%48.63% / 49.67%
32K54.86% / 55.47%55.26% / 56.66%31.07% / 31.89%57.43% / 58.06%49.65% / 50.52%
64K55.62% / 56.08%56.65% / 57.67%32.17% / 32.98%57.20% / 57.22%50.41% / 50.99%
  • Length stability: The four-domain average RfR_f remains between 49.52% and 50.99% across 8K–64K, while shorter inputs receive less global attention.
  • Domain variation: Budgets differ across input domains. At 64K, Wiki/PG19/CodeParrot use 56.08%/57.67%/57.22% FLOPs, versus 32.98% for GSM8K. The training target therefore does not enforce an identical budget on every test distribution.

(3) Realized budgets are invariant to batch size and co-batched inputs. Gating operates token-wise without cross-request budget allocation. On Wiki at 1K and 16K, changing the batch size from 1 to 4 shifts both RcR_c and RfR_f by less than 0.003 pp (due solely to floating-point reduction order). At a fixed batch shape, replacing co-batched sequences leaves the target sequence's routing unchanged.

[1] Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Shared Response 1: Novelty and Positioning

This comment addresses the shared concerns raised by Reviewers MT9m (W5) and CRbc (W1) regarding architectural novelty.

We sincerely acknowledge prior work on conditional computation and dynamic attention routing, including CoLT5 [1] and SMA [2]. We discuss these works and their differences from LoGo in Appendix A. LoGo's contribution lies in the joint design and validation of token-level attention-span allocation for pretrained decoder-only LLMs.

(1) Research question and our answer. An open question is whether learned token-level span allocation can offer a better performance–compute trade-off than static local–global hybrids and dense Transformers at pretrained LLM scale, considering both model quality and practical efficiency. Our answer is LoGo, which uses attention span as a direct proxy for attention budget and learns where to allocate global attention under a controlled budget. Every token attends locally, while selected queries additionally attend to the full causal prefix.

(2) Joint design for effective span allocation. Realizing this trade-off requires effective branch learning, stable routing, budget control, and efficient execution. LoGo therefore combines lightweight shared-projection decoupling and ContextNorm-stabilized gated fusion to support branch specialization and balanced fusion, an auxiliary-loss-free threshold controller to regulate global computation, progressive masking to establish branch learning before sparse routing, and query-sparse kernels to translate sparsity into execution savings. These designs jointly make learned span allocation trainable, budget-controlled, and practically efficient, with their contributions examined in our ablations and efficiency experiments.

(3) Evidence for the performance–compute trade-off. Across 200M–3.3B models, controlled 100B-token pretraining, and context extension to 128K, LoGo preserves Transformer scaling behavior and improves the performance–compute trade-off over dense Transformers and static hybrids, particularly on long-range retrieval. Our revised comparisons account for actual attention FLOPs (Appendix D). Moreover, full-model benchmarks demonstrate end-to-end speedups (Appendix B.2, Figure 4). Learned routing also outperforms fixed-interval and random routing, significantly improving RULER, alongside lower PPL and higher CSR and recall (Appendix C.4.1, Table 17).

We sincerely hope these findings provide useful insights into learned attention-budget allocation for the community and clarify LoGo's contribution to making this approach effective at pretrained LLM scale.

[1] CoLT5: Faster Long-Range Transformers with Conditional Computation.

[2] Sparse Modular Activation for Efficient Sequence Modeling.

AuthorsOpenReview ↗

Shared Response 2: Broader Long-Context Evaluation

This comment addresses the shared concerns raised by Reviewers MT9m (W4) and BDdZ (Q4) regarding evaluation beyond RULER.

(1) Why RULER as a primary long-context benchmark. RULER specifically stress-tests long-range retrieval (especially multi-needle) with high diagnostic sensitivity. In contrast, broader long-context understanding/QA benchmarks often compress architectural differences: across all four 128K models, LongBench-64K scores span less than 1.5 pp (even Intra-layer hybrid, which drops to 34.00 on RULER-64K, scores highest on LongBench), whereas RULER-64K spans 18.5 pp. This motivates using RULER as a primary diagnostic while complementing it with broader evaluations.

(2) Broader long-context evaluation. Following the GDN paper [1], we evaluate the 128K checkpoints on six-domain PPL across 2K–64K, 14 LongBench-64K tasks [2], and BABILong-64K (qa1–qa10) [3]. Per-task results and measured attention budgets are reported in Appendix C.3.1 (Tables 11–12).

BenchmarkTransformerInterIntraLoGo
6-domain avg PPL-64K ↓9.449.389.399.29
RULER-64K ↑42.8044.9034.0052.50
LongBench-64K ↑29.6129.2430.4629.77
BABILong-64K ↑26.8032.4028.1034.10
  • Long-context PPL: LoGo achieves the lowest PPL in 33/36 domain–length settings, including all six domains at 64K (Appendix C.3.1, Figure 5).
  • BABILong: LoGo achieves the best overall score (34.10 vs. 26.80–32.40).
  • LongBench: LoGo's performance is comparable to the Transformer and static hybrids.

Across these four benchmarks, LoGo achieves the best overall scores on six-domain PPL, RULER, and BABILong, while remaining comparable on LongBench. Its advantage is particularly pronounced on RULER, supporting the effectiveness of learned span allocation for long-range retrieval.

[1] Gated Delta Networks: Improving Mamba2 with Delta Rule.

[2] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding.

[3] BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack.

AuthorsOpenReview ↗

Shared Response 3: Training Details and Reproducibility

This comment addresses the shared concerns raised by Reviewers MT9m (W5) and CRbc (Weaknesses) regarding training records, implementation details, and reproducibility.

Below, we summarize the requested training configurations and additional public-corpus validation. We have updated the manuscript accordingly and included our model and query-sparse kernel code in the supplementary material to ensure transparency and reproducibility.

(1) Training dynamics. We have included training-loss and pre-clipping gradient-norm curves for Transformer, Inter, Intra, and LoGo across 8K pretraining, 32K context extension, and 128K context extension (Appendix C.3.3, Figure 6). All runs exhibit smooth loss convergence and bounded gradient norms, and LoGo achieves lower late-stage training loss across all three stages. Rather than running periodic validation during training, we evaluated the final stage checkpoints on held-out PPL and downstream benchmarks (reported in the main evaluation tables).

(2) Infrastructure and practical experience. All controlled 1.5B runs use 64 A100-80GB GPUs, FSDP1, BF16 computation with FP32 gradient reduction, and activation checkpointing. LoGo reuses the standard data-loading, optimizer, and checkpointing pipelines, requiring only the LoGo attention module and one budget-controller update per optimizer step. Routing statistics are aggregated across gradient accumulation and the data-parallel group to update thresholds. During query compaction, kernels preserve original token positions and packed-document boundaries (Appendix B.1).

(3) Dataset configuration and public-data validation. All four models share identical stage-specific splits of our proprietary bilingual (Chinese/English) corpus and document-aware dynamic sequence packing (up to 8K/32K/128K). Attention masks and next-token losses respect document boundaries. To evaluate LoGo on public data, we added a 200M/5B-token FineWeb-Edu experiment using the same 200M scaling recipe:

CorpusModelWiki PPL ↓LAMBADA PPL ↓CSR (%) ↑
FineWeb-EduTransformer41.22158.7540.90
FineWeb-EduLoGo38.96125.0341.65
Original corpusTransformer43.3792.2339.25
Original corpusLoGo42.9989.6840.37

On FineWeb-Edu, LoGo also reduces WikiText/LAMBADA PPL and improves average CSR, supporting the transfer of its gains to a public English pretraining corpus.

(4) Optimizer and positional encoding. All controlled 1.5B models use AdamW (β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, ϵ=10−8\epsilon=10^{-8}, weight decay 0.10.1), with global gradient-norm clipping at 1.01.0. Gate and branch parameters share the backbone optimizer. Layer thresholds are non-gradient buffers updated by the budget controller once per optimizer step. Across all three stages, these models use RoPE with θ=500,000\theta=500,000 without additional position-scaling schemes.

(5) Inference thresholds. Default evaluation uses the fixed per-layer thresholds saved at the end of training, without task-specific tuning. For the separately reported RULER FLOPs-matched comparisons, a shared threshold offset is calibrated on an independent held-out set using FLOPs budgets alone (Appendix D.3).

AuthorsOpenReview ↗

Summary of Revisions and Additional Results

Dear Area Chair and Reviewers,

We sincerely thank the reviewers and Area Chair for their time and constructive feedback. We have uploaded a revised manuscript, with changes highlighted in blue. For ease of reference, we summarize the main revisions and new results below:

  • Actual attention FLOPs and budget control [Reviewer MT9m]. We distinguish query-count ratios (RcR_c) from position-weighted attention FLOPs ratios (RfR_f) and report measured budgets across all main tables. Under a calibrated FLOPs cap (Rf≤50%R_f \le 50\%), LoGo retains higher average RULER scores than both static hybrids. We also validate direct FLOPs-based budget control during training (Section 4.2, Tables 2–4, and Appendix D).
  • End-to-end efficiency [Reviewer MT9m]. We add full-model (1.5B) benchmarks covering training throughput, prefill TTFT, batched decoding TPOT, and peak memory across 8K–128K contexts, demonstrating practical speedups at long contexts (up to 1.95× prefill and 1.98× decoding speedup over dense CUDA at the 25% budget) and isolating algorithmic gains from backend differences (Appendix B.2, Figure 4).
  • Additional baselines and routing ablations [Reviewers MT9m, CRbc, BDdZ]. We add controlled comparisons against TransMamba (outperforming it across PPL, CSR, recall, and RULER-8K), Quest, and a post-hoc MoBA-style baseline (outperforming both on RULER-64K at 58% attention FLOPs ratio), together with Periodic and Random routing controls. These results strengthen the evidence for learned span allocation (Appendices C.3.4 and C.4.1).
  • Broader evaluation and reproducibility [Reviewers MT9m, CRbc, BDdZ]. We add six-domain long-context perplexity (lowest in 33/36 settings), LongBench-64K, BABILong-64K (best overall average), public-corpus validation on FineWeb-Edu, full-stage loss and gradient-norm curves, and budget-stability analyses across domains, lengths, and batch sizes, while expanding training configurations and related-work discussions (Appendices A, C.1, C.3, and D.5).

Detailed point-by-point responses are provided in our reviewer-specific replies and Shared Responses 1–3. We remain available to clarify any outstanding questions.

Best regards,
The Authors of Submission 1414

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.