acceptodds
Under review as a conference paper at ICLR 2027

Post-Boundary Bridge: Must Local Attention Go Global Between Global Layers?

Abstract

Hybrid Transformers interleave local attention with periodic Full layers to cut long-context cost. Recent work on sparse and local attention reveals a gap between theoretical cross-layer reach and effective information propagation. This raises a key question: when periodic Full layers already provide global communication, how should local computation be allocated? We argue that local layers need not maximize cross-layer relay range; they can instead emphasize dense local modeling and timely nearby exchange. Post-Boundary Bridge (PBB) realizes this view by preserving dense causal attention within blocks while adding selective cross-block connections under a single softmax, with no extra parameters or routing. In controlled 200M experiments, PBB+Full matches SWA+Full in perplexity and improves the main source-retrieval probe; score-matched controls show that allocation, not only budget size, matters. In larger-scale transfer, PBB+Full stays close to Full attention in perplexity on 350M dense and 2B MoE models, and improves exact entity–attribute retrieval by 14.02 percentage points on the 2B MoE model. At 512K context, it achieves 3.32× prefill and 2.11× cached-decoding throughput over Full attention. These results support allocating local computation to targeted nearby exchange rather than maximum relay range.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, hybrid Transformers, efficient language models, attention kernels, KV cache

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fUXuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how local attention should be allocated in hybrid Transformers where periodic full-attention layers already provide global communication. The authors propose Post-Boundary Bridge, which combines dense causal attention within fixed blocks with selective cross-block connections immediately after block boundaries. The method introduces no additional learnable parameters and is accompanied by Flash-PBB. The paper evaluates PBB on 200M dense models with controlled architectural comparisons, followed by 350M dense and 2B MoE models.

Critical strengths
  1. The comparison between purely local and hybrid attention is informative. At 200M, SWA outperforms PBB when all layers are local, whereas PBB+Full achieves higher accuracy on the primary source-retrieval probe than SWA+Full while maintaining comparable perplexity.
  2. The paper connects the attention structure to an exact cached-decoding implementation using bounded ring-buffer storage. The authors provide numerical correctness checks, kernel-level profiling, and whole-model throughput measurements. The experiments across dense and MoE architectures also suggest that the proposed hybrid is practically viable.
Critical weaknesses
  1. The main methodological contribution appears to be the application and analysis of an existing local attention pattern within a periodic-Full architecture, rather than a fundamentally new attention mechanism.
  2. The most substantial retrieval improvements are obtained on controlled, four-choice tasks with fixed templates and exact source–query wording. Their relationship to realistic long-context reasoning and retrieval remains uncertain.
Questions for authors
  1. Given the equivalence between the default PBB configuration and the 1D local attention formulation in Parmar et al. (2018), what is the essential methodological novelty beyond combining this existing attention pattern with periodic Full layers?
  2. Can the authors provide comparisons against an approximately score-matched SWA baseline, such as SWA with a window of 96 or 97 tokens, and examine different bridge widths or Full-layer frequencies?
  3. How well do the observed source-retrieval improvements transfer to more realistic tasks involving paraphrased queries, multiple relevant entities, conflicting attributes?
3 comments

The revised manuscript adds the score-matched SWA-97+Full baseline and narrow-bridge PBB (b=128, h=48)+Full ablation at 200M, extends SWA-128+Full to 350M and 2B MoE, and clarifies our methodological and evaluation scope.

1. Score-matched SWA-97+Full and bridge-width ablation (Q2; Sections 5.2-5.3, 6, Tables 2-3; Appendix D.8). Under the 3:1 hybrid schedule at 200M (table below), shrinking SWA's window from 128 to 97 (within 0.45% of default PBB's local scores) improves SWA+Full's PPL (21.29 to 21.24) and main probe (76.50% to 80.98%), so competitive hybrid modeling does not require SWA-128's larger relay range. SWA-97+Full has slightly lower PPL and higher needle accuracy, whereas PBB (h=64)+Full leads on main probe, source gain GG, entity-attribute accuracy, and LM-6. Narrowing PBB's bridge to h=48h=48 removes 14.36% of PBB's local scores (14.74% below SWA-97; 35.26% below SWA-128) while keeping 21.26 PPL and outperforming SWA-97+Full on main probe, GG (2.565 nats), entity-attribute accuracy (+2.31 pp), and LM-6, with lower needle accuracy (56.37% vs. 57.96%). Table 3 extends SWA-128+Full to 350M and 2B MoE, where PBB+Full has the higher LM-6 at both scales and +2.34 points entity-attribute accuracy at 2B, while SWA+Full leads some retrieval columns and has slightly lower 2B PPL (14.29 vs. 14.35).

2. Methodological contribution (W1, Q1; Sections 2, 5, 7, 8; Appendix A.3). We do not claim the default mask as new; Section 8 states its coincidence with Parmar et al. (2018) at b=2hb=2h and h=bh=b. The contribution is what this geometry establishes between periodic Full layers. First, we separate global input coverage from next-layer boundary exchange (Equation 4) and test the distinction: the stronger pure-local graph becomes the weaker hybrid component on the main probe (Figure 3), and equal-arithmetic Random-Edge and WideBlock-192 controls show that edge placement matters at fixed budget. Second, independent bb and hh give a family beyond the uniform-chunk formula; its h=48h=48 member (48/80-token query regions) is the lowest-budget hybrid in Table 2 at competitive quality. Third, the same regular support yields Flash-PBB, an exact fixed-ring decoding kernel with measured core and whole-model gains over the tested Flash-SWA path (8.91x and +7.1% at batch 64, default h=64h=64).

3. Retrieval scope vs. natural-text and downstream evidence (W2, Q3; Sections 5.4, 6; Appendices D.3, D.10). Our synthetic probes use single-source exact wording and 4-label rotation to isolate source sensitivity. Beyond templates, pure PBB reduces natural-text NLL over Random-Edge by 0.1235 nats at repaired boundary phases (vs. 0.0028 elsewhere; Figure 4a), PBB+Full lowers mean NLL across all 128 phases vs. Full (Figure 4b) while matching SWA+Full under context truncation (Appendix D.10), and 2B PBB+Full leads Full (+0.925 pp, 95% CI [0.194,1.655][0.194, 1.655]) and SWA+Full (+0.79 pp) on LM-6. These measurements do not test paraphrased queries, multiple relevant entities, or conflicting attributes, and we do not claim transfer to those settings; the paper presents the exact-wording tasks as controlled measurements of source use (Section 4; Appendix D.8).

200M 3:1 Hybrid (or Control)Scores/head/layer (k)Test PPL ↓\downarrowMain Probe (% / GG) ↑\uparrowNeedle (%) ↑\uparrowEntity (% / GG) ↑\uparrowLM-6 (%) ↑\uparrow
SWA-128+Full1,040.421.2976.50 / 2.30553.6561.18 / 1.91145.82
SWA-97+Full (near-equal budget)790.021.2480.98 / 2.42657.9657.14 / 1.66445.96
Random-Edge+Full (equal budget)786.421.6175.68 / 1.93452.4659.46 / 1.71746.02
WideBlock-192+Full (equal budget)786.421.6780.57 / 2.33655.9554.05 / 1.33245.92
PBB (b=128, h=64)+Full786.421.2781.31 / 2.54355.8058.07 / 1.97546.13
PBB (b=128, h=48)+Full (narrow)673.521.2681.13 / 2.56556.3759.45 / 1.88046.20

Scores are per query head and per local layer at 8K. All rows are 200M 3:1 hybrids under the matched seed-13 training recipe.

I appreciate the authors' effort on the additional ablation experiments, which address several of the concerns and strengthen the paper. However, the performance of the proposed PBB stays close to the baselines while the overall novelty remains limited. Therefore, I will maintain my rating of 2.

We would like to clarify a fundamental misunderstanding regarding the novelty and objective of our work.

The primary contribution of PBB is neither to introduce an entirely new attention mechanism nor to achieve state-of-the-art benchmark scores. Instead, our work challenges a common design assumption in hybrid attention architectures: when global attention layers are already present, must local attention layers also be designed to propagate information globally?

Our results suggest that this is not necessarily the case. Rather than explicitly extending the global receptive field of local attention layers, PBB focuses on efficient information exchange across local boundaries. This design achieves performance comparable to, and in several evaluations better than, widely adopted attention baselines, while providing substantial computational and GPU efficiency benefits.

The reviewer argues that the performance improvements over the baselines are relatively small. However, this criticism overlooks the central objective of our study. Our goal is not to substantially outperform Full Attention in absolute accuracy, but to demonstrate that comparable or even superior performance can be achieved at a significantly lower computational cost.

Furthermore, the fact that our baselines are widely adopted makes this finding more meaningful, not less. If a more computationally efficient architecture can match or outperform established methods, this provides evidence that the additional global information propagation often expected from local attention layers may not always be necessary in hybrid architectures.

This raises a fundamental question: Should the novelty and scientific contribution of a research paper be judged solely by whether it achieves state-of-the-art benchmark performance? If so, does demonstrating that a more computationally efficient architecture can preserve or even improve upon the performance of widely adopted baselines have limited scientific value?

We respectfully disagree with such an evaluation criterion. Scientific novelty should be assessed based on the research question addressed, the architectural insights established, and the experimental evidence supporting those insights—not merely the magnitude of benchmark improvements.

We therefore believe that evaluating PBB primarily through its absolute performance gains over the baselines does not adequately reflect the research question or the contribution of this work.

The central finding of PBB is that local attention layers need not independently pursue global information propagation when global layers already provide that capability. Relaxing this requirement can achieve a more favorable balance between model quality and computational efficiency. We believe this architectural insight, supported by our experimental results, constitutes a meaningful contribution regardless of whether PBB achieves state-of-the-art benchmark scores.

Reviewer_7cnoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Post-Boundary Bridge (PBB). It chops into blocks of 128 and let each token attend only within its block. This leads to a cheap SWA-128 (about 24% fewer attention scores) while still giving boundary tokens a direct one-layer connection.

The paper’s key contribution is between coverage and timeliness. A full layer makes every input token reachable, but if a new feature gets computed right after that full layer, block attention can’t pass it across a boundary until the next full layer, up to 4 layers later. PBB, like SWA, passes it in one layer

Critical strengths
  • The paper is well motivated shows how existing LLM structure can be improved.
Critical weaknesses
  • My major concern is writing, it is almost impossible to follow, extrmely long sentences and undefined terms, like what even doees "timely exchange" actually mean

  • How is this different 1D attention in Image Transformer paper - https://arxiv.org/abs/1802.05751

  • SWA+Full in only on 200M model. The 350M and 2B MoE transfer experiments compare only against Full. So “transfer across model families” shows that a hybrid stays near Full, which is already known, not that PBB beats SWA as the local component.

Questions for authors

Look at weakness

1 comment

Our revision rewrites the mechanism in short, defined terms, states the relation to Image Transformer precisely, and adds matched SWA+Full baselines at 350M and 2B.

1. Next-layer boundary exchange (Weakness 1; Sections 2.1-2.2, Equation 4, Figure 2b). A Full layer makes every input token reachable, but later layers keep computing new features, and those features still need a path to their neighbors. We replaced "timely exchange" with global input coverage versus next-layer boundary exchange. Section 2.2 defines structural latency τℓ(s,t)\tau_\ell(s, t) as the minimum number of additional layers for a feature written at ss after layer ℓ\ell to reach tt along an allowed path, and Equation 4 gives its values. In a 3:1 hybrid, a fresh feature at p−1p-1 (before boundary pp) waits up to four layers for the next Full layer under Block+Full to reach pp, whereas PBB delivers it in one layer (τℓ=1\tau_\ell = 1), matching SWA with 24.4% fewer local scores.

2. Difference from Image Transformer (Weakness 2; Sections 5.3, 8; Appendix A.3). PBB coincides with Parmar et al. (2018) at b=2hb=2h (q=m=hq=m=h) and h=bh=b, whereas for 0<h<b0<h<b with b≠2hb\neq 2h its alternating hh and b−hb-h query regions fall outside that uniform-chunk formula. Narrowing hh from 64 to 48 at 200M (48/80-token regions) cuts local scores by 14.36% (14.74% below SWA-97+Full) while keeping 21.26 PPL and leading Table 2 on source gain (2.565 nats) and LM-6 (46.20%). The same regular support gives Flash-PBB an exact fixed ring of max⁡(b,2h)\max(b,2h) slots; at the default width it is 8.91x faster than Flash-SWA in the local core and 7.1% faster for the whole 2B model at batch 64 (Section 7).

3. Matched 350M and 2B SWA+Full baselines (Weakness 3; Sections 4, 6, Table 3; Appendix D.8). With 24.4% fewer local scores than SWA-128+Full, PBB+Full leads on LM-6 at both scales, 350M PPL and main probe (+6.86 pp), and 2B entity-attribute retrieval (+2.34 pp vs. SWA+Full; +14.02 pp vs. Full); SWA+Full leads on 350M needle/entity accuracy and slightly on 2B PPL/probe (table below).

Scale & ModelScores/head/layer (k)Test PPL ↓\downarrowMain Probe (% / GG) ↑\uparrowNeedle (%) ↑\uparrowEntity (% / GG) ↑\uparrowLM-6 (%) ↑\uparrow
350M Full33,558.517.1373.62 / 1.98350.0453.35 / 1.71449.19
350M SWA-128+Full (3:1)1,040.417.4067.93 / 1.86154.0056.96 / 2.10348.69
350M PBB (h=64)+Full (3:1)786.417.3074.79 / 2.33950.8556.03 / 2.13549.55
2B MoE Full33,558.514.2575.62 / 2.32651.7652.44 / 2.06152.02
2B MoE SWA-128+Full (3:1)1,040.414.2978.29 / 2.69756.7364.12 / 2.59052.15
2B MoE PBB (h=64)+Full (3:1)786.414.3577.77 / 2.71456.8866.46 / 2.91752.94

Per local layer for hybrids at 8K (per Full layer for all-Full); 2B PPL uses the matched 1M-target prefix.

Reviewer_4xyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how local attention should be allocated between periodic full-attention layers. PBB combines dense causal attention within blocks with selective connections across block boundaries. Experiments on 200M dense, 350M dense, and 2B MoE models show near-Full perplexity and improvements on controlled retrieval tasks at 8K context. The paper also introduces Flash-PBB and reports efficiency gains at longer sequence lengths.

Critical strengths
  1. The controlled experiments are informative. Comparing pure-local and hybrid schedules, alongside matched-computation controls and position-dependent losses, provides useful evidence that local edge placement matters beyond the attention budget.

  2. The implementation gives the design practical value. Exact ring-cache execution and measured end-to-end gains over the tested SWA backend complement the modeling results, although the full-model improvement is considerably smaller than the kernel-level speedup.

Critical weaknesses
  1. The novelty of the attention pattern is limited. The default configuration used throughout the main experiments coincides with a local-attention pattern from Image Transformer (2018). Allowing independent block and bridge widths generalizes this pattern, but the experiments do not establish the practical value of that generalization. This leaves the empirical findings carrying much of the contribution.

  2. The evidence for choosing PBB over SWA in a hybrid model is incomplete. Quality comparisons with SWA+Full are confined to 200M; the larger models are compared against Full attention. The highlighted 14.02-point retrieval improvement therefore does not establish an advantage over the existing hybrid baseline. Moreover, retrieval gains are measured mainly with a small set of exact-match synthetic templates. I would need stronger evidence against SWA+Full to support acceptance.

Questions for authors

Please refer to Weaknesses.

1 comment

The revised manuscript adds matched 350M and 2B MoE SWA-128+Full baselines, evaluates near-equal-budget SWA-97+Full and narrow-bridge PBB (b=128, h=48)+Full at 200M, and clarifies how local-core speedups translate to whole-model decoding.

1. Matched SWA+Full comparisons at 350M and 2B MoE (Weakness 2; Section 6, Table 3; Appendix D.8). We trained 3:1 SWA-128+Full hybrids at 350M dense (7.08B tokens) and 2B MoE (6.00B tokens) with the same data order, optimizer, and schedule as Full and PBB+Full. With 24.4% fewer local attention scores, PBB+Full has lower PPL than SWA+Full at 350M, stays within 0.5% at 2B, and has the higher LM-6 at both scales. At 350M it also leads on the main probe (+6.86 points) and source gain, while SWA+Full leads on literal-needle and entity-attribute accuracy. At 2B, PBB+Full leads entity-attribute retrieval (66.46% vs. 64.12%; +14.02 points over Full), needle accuracy is close, and SWA+Full is slightly ahead on PPL and main-probe accuracy.

2. Practical value of independent (b,h)(b, h) widths (Weakness 1; Sections 5.2-5.3, 8, Table 2; Appendix A.3). PBB coincides with Parmar et al. (2018) at b=2hb=2h and h=bh=b, whereas for 0<h<b0<h<b with b≠2hb\neq 2h its alternating hh and b−hb-h query regions fall outside that uniform-chunk formula. At 200M (4.11B tokens), narrowing hh from 64 to 48 (48/80-token regions) removes 14.36% of PBB's local scores (14.74% below SWA-97+Full; 35.26% below SWA-128+Full), keeping comparable PPL to SWA-97+Full (21.26 vs. 21.24) while leading on main probe, GG (2.565 nats), entity-attribute accuracy (+2.31 pp), and LM-6, with lower needle accuracy. This is the practical value of decoupling bb and hh: the narrow bridge is the lowest-budget hybrid in Table 2 at competitive quality.

3. From core speedup to whole-model throughput (Section 7, Figure 5; Appendix F.2-F.3). All timings use the default b=128,h=64b=128, h=64. Against the tested FlashAttention 2.8.3 SWA kernel on RTX 4090, the Flash-PBB local core is 8.91x faster at batch 64. The whole-model gain is smaller, rising from +2.3% at batch 1 to +7.1% at batch 64 (6,942 vs. 6,484 tok/s), because local attention is one part of each 2B MoE step alongside five Full and 20 expert layers. Separate profiling of the same attention shape on an RTX 4060 Laptop GPU shows lower register and shared-memory use, zero thread-local memory requests, and 86.0-92.7% fewer global-load sector requests for Flash-PBB. Both 3:1 hybrids share the large 512K savings over all-Full.

Scale & ModelScores/head/layer (k)Test PPL ↓\downarrowMain Probe (% / GG) ↑\uparrowNeedle (%) ↑\uparrowEntity (% / GG) ↑\uparrowLM-6 (%) ↑\uparrow
350M Full33,558.517.1373.62 / 1.98350.0453.35 / 1.71449.19
350M SWA-128+Full (3:1)1,040.417.4067.93 / 1.86154.0056.96 / 2.10348.69
350M PBB (h=64)+Full (3:1)786.417.3074.79 / 2.33950.8556.03 / 2.13549.55
2B MoE Full33,558.514.2575.62 / 2.32651.7652.44 / 2.06152.02
2B MoE SWA-128+Full (3:1)1,040.414.2978.29 / 2.69756.7364.12 / 2.59052.15
2B MoE PBB (h=64)+Full (3:1)786.414.3577.77 / 2.71456.8866.46 / 2.91752.94

Per local layer for hybrids at 8K (per Full layer for all-Full); 2B PPL uses the matched 1M-target prefix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.