- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how local attention should be allocated in hybrid Transformers where periodic full-attention layers already provide global communication. The authors propose Post-Boundary Bridge, which combines dense causal attention within fixed blocks with selective cross-block connections immediately after block boundaries. The method introduces no additional learnable parameters and is accompanied by Flash-PBB. The paper evaluates PBB on 200M dense models with controlled architectural comparisons, followed by 350M dense and 2B MoE models.
Critical strengths
- The comparison between purely local and hybrid attention is informative. At 200M, SWA outperforms PBB when all layers are local, whereas PBB+Full achieves higher accuracy on the primary source-retrieval probe than SWA+Full while maintaining comparable perplexity.
- The paper connects the attention structure to an exact cached-decoding implementation using bounded ring-buffer storage. The authors provide numerical correctness checks, kernel-level profiling, and whole-model throughput measurements. The experiments across dense and MoE architectures also suggest that the proposed hybrid is practically viable.
Critical weaknesses
- The main methodological contribution appears to be the application and analysis of an existing local attention pattern within a periodic-Full architecture, rather than a fundamentally new attention mechanism.
- The most substantial retrieval improvements are obtained on controlled, four-choice tasks with fixed templates and exact source–query wording. Their relationship to realistic long-context reasoning and retrieval remains uncertain.
Questions for authors
- Given the equivalence between the default PBB configuration and the 1D local attention formulation in Parmar et al. (2018), what is the essential methodological novelty beyond combining this existing attention pattern with periodic Full layers?
- Can the authors provide comparisons against an approximately score-matched SWA baseline, such as SWA with a window of 96 or 97 tokens, and examine different bridge widths or Full-layer frequencies?
- How well do the observed source-retrieval improvements transfer to more realistic tasks involving paraphrased queries, multiple relevant entities, conflicting attributes?
3 comments
I appreciate the authors' effort on the additional ablation experiments, which address several of the concerns and strengthen the paper. However, the performance of the proposed PBB stays close to the baselines while the overall novelty remains limited. Therefore, I will maintain my rating of 2.
We would like to clarify a fundamental misunderstanding regarding the novelty and objective of our work.
The primary contribution of PBB is neither to introduce an entirely new attention mechanism nor to achieve state-of-the-art benchmark scores. Instead, our work challenges a common design assumption in hybrid attention architectures: when global attention layers are already present, must local attention layers also be designed to propagate information globally?
Our results suggest that this is not necessarily the case. Rather than explicitly extending the global receptive field of local attention layers, PBB focuses on efficient information exchange across local boundaries. This design achieves performance comparable to, and in several evaluations better than, widely adopted attention baselines, while providing substantial computational and GPU efficiency benefits.
The reviewer argues that the performance improvements over the baselines are relatively small. However, this criticism overlooks the central objective of our study. Our goal is not to substantially outperform Full Attention in absolute accuracy, but to demonstrate that comparable or even superior performance can be achieved at a significantly lower computational cost.
Furthermore, the fact that our baselines are widely adopted makes this finding more meaningful, not less. If a more computationally efficient architecture can match or outperform established methods, this provides evidence that the additional global information propagation often expected from local attention layers may not always be necessary in hybrid architectures.
This raises a fundamental question: Should the novelty and scientific contribution of a research paper be judged solely by whether it achieves state-of-the-art benchmark performance? If so, does demonstrating that a more computationally efficient architecture can preserve or even improve upon the performance of widely adopted baselines have limited scientific value?
We respectfully disagree with such an evaluation criterion. Scientific novelty should be assessed based on the research question addressed, the architectural insights established, and the experimental evidence supporting those insights—not merely the magnitude of benchmark improvements.
We therefore believe that evaluating PBB primarily through its absolute performance gains over the baselines does not adequately reflect the research question or the contribution of this work.
The central finding of PBB is that local attention layers need not independently pursue global information propagation when global layers already provide that capability. Relaxing this requirement can achieve a more favorable balance between model quality and computational efficiency. We believe this architectural insight, supported by our experimental results, constitutes a meaningful contribution regardless of whether PBB achieves state-of-the-art benchmark scores.
The revised manuscript adds the score-matched
SWA-97+Fullbaseline and narrow-bridgePBB (b=128, h=48)+Fullablation at 200M, extendsSWA-128+Fullto 350M and 2B MoE, and clarifies our methodological and evaluation scope.1. Score-matched SWA-97+Full and bridge-width ablation (Q2; Sections 5.2-5.3, 6, Tables 2-3; Appendix D.8). Under the 3:1 hybrid schedule at 200M (table below), shrinking SWA's window from 128 to 97 (within 0.45% of default PBB's local scores) improves
SWA+Full's PPL (21.29 to 21.24) and main probe (76.50% to 80.98%), so competitive hybrid modeling does not require SWA-128's larger relay range.SWA-97+Fullhas slightly lower PPL and higher needle accuracy, whereasPBB (h=64)+Fullleads on main probe, source gain G, entity-attribute accuracy, andLM-6. Narrowing PBB's bridge to h=48 removes 14.36% of PBB's local scores (14.74% belowSWA-97; 35.26% belowSWA-128) while keeping 21.26 PPL and outperformingSWA-97+Fullon main probe, G (2.565 nats), entity-attribute accuracy (+2.31 pp), andLM-6, with lower needle accuracy (56.37% vs. 57.96%). Table 3 extendsSWA-128+Fullto 350M and 2B MoE, wherePBB+Fullhas the higherLM-6at both scales and +2.34 points entity-attribute accuracy at 2B, whileSWA+Fullleads some retrieval columns and has slightly lower 2B PPL (14.29 vs. 14.35).2. Methodological contribution (W1, Q1; Sections 2, 5, 7, 8; Appendix A.3). We do not claim the default mask as new; Section 8 states its coincidence with Parmar et al. (2018) at b=2h and h=b. The contribution is what this geometry establishes between periodic Full layers. First, we separate global input coverage from next-layer boundary exchange (Equation 4) and test the distinction: the stronger pure-local graph becomes the weaker hybrid component on the main probe (Figure 3), and equal-arithmetic
Random-EdgeandWideBlock-192controls show that edge placement matters at fixed budget. Second, independent b and h give a family beyond the uniform-chunk formula; its h=48 member (48/80-token query regions) is the lowest-budget hybrid in Table 2 at competitive quality. Third, the same regular support yieldsFlash-PBB, an exact fixed-ring decoding kernel with measured core and whole-model gains over the testedFlash-SWApath (8.91x and +7.1% at batch 64, default h=64).3. Retrieval scope vs. natural-text and downstream evidence (W2, Q3; Sections 5.4, 6; Appendices D.3, D.10). Our synthetic probes use single-source exact wording and 4-label rotation to isolate source sensitivity. Beyond templates, pure PBB reduces natural-text NLL over
Random-Edgeby 0.1235 nats at repaired boundary phases (vs. 0.0028 elsewhere; Figure 4a),PBB+Fulllowers mean NLL across all 128 phases vs.Full(Figure 4b) while matchingSWA+Fullunder context truncation (Appendix D.10), and 2BPBB+FullleadsFull(+0.925 pp, 95% CI [0.194,1.655]) andSWA+Full(+0.79 pp) onLM-6. These measurements do not test paraphrased queries, multiple relevant entities, or conflicting attributes, and we do not claim transfer to those settings; the paper presents the exact-wording tasks as controlled measurements of source use (Section 4; Appendix D.8).SWA-128+FullSWA-97+Full(near-equal budget)Random-Edge+Full(equal budget)WideBlock-192+Full(equal budget)PBB (b=128, h=64)+FullPBB (b=128, h=48)+Full(narrow)Scores are per query head and per local layer at 8K. All rows are 200M 3:1 hybrids under the matched seed-13 training recipe.