acceptodds
Under review as a conference paper at ICLR 2027

PHBA: Prefix-State Hybrid Block Attention

Abstract

Hybrid architectures combining linear sequence models with softmax attention provide an effective balance between efficient long-context modeling and precise token retrieval. Existing designs such as Native Hybrid Attention (NHA) combine compressed long-term states with sliding-window attention, but their exact attention is restricted to a fixed local window. In this work, we introduce Prefix-State Hybrid Block Attention (PHBA), which replaces local sliding-window attention with top-k block-sparse retrieval and couples each retrieved block with a compact prefix state summarizing its preceding context. The prefix states are constructed by a gated linear recurrence at block boundaries and retrieved together with the corresponding token blocks, allowing the model to combine precise long-range evidence with compressed historical context within a unified layer. We further develop a hardware-aware Triton implementation that streams routed token blocks and prefix states without materializing large intermediate tensors. Experiments show that PHBA improves long-context and retrieval performance over strong linear and hybrid baselines while retaining efficient training and inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Modeling, Hybrid Attention, Block-Sparse Attention, Linear Attention

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_kwYaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes Prefix-State Hybrid Block Attention (PHBA), which stores recurrent key/value states at block boundaries and retrieves these prefix snapshots alongside content-selected historical token blocks. Exact tokens and compressed states participate in a shared softmax. A Triton implementation combines chunkwise state construction with sparse execution that avoids materializing gathered payload tensors. Models trained from scratch at 760M and 1.3B parameters show competitive language modeling and improvements on several retrieval evaluations. The paper also reports LongBench results with inputs truncated to 8K and end-to-end training throughput.

Critical strengths
  1. Pairing retrieved token blocks with snapshots of their preceding context is a coherent extension of sparse retrieval and recurrent memory. Routing, state boundaries, and joint readout are organized around this idea.

  2. The comparisons use models trained from scratch on the same data and context length. At 1.3B, PHBA improves WikiText perplexity and the reported RULER averages over the compared models while remaining competitive on commonsense reasoning. The implementation and end-to-end throughput measurements also support practical feasibility.

Critical weaknesses
  1. PHBA already reads the current block's prefix state, which summarizes all preceding blocks. Its distinctive additional choice is to expose earlier prefix snapshots alongside selected historical token blocks. The comparison with MoBA also introduces recurrent memory, while the comparison with NHA also changes exact-token access from local to content-routed. These comparisons support the complete architecture but do not resolve whether historical snapshots improve upon sparse retrieval augmented with only the current prefix state. The block-size, slot-count, and gate ablations do not isolate this distinction.
Questions for authors
  1. How does full PHBA compare with a variant retaining the same token routing, gate construction, and current-block prefix state, but omitting readout of routed historical prefix states?

  2. The main quality configuration uses C = M = 128 and K = 8 at 8K, whereas Figure 3 evaluates fixed K = 4 or fixed C = 256. How does the reported throughput relate to the configuration producing the main quality results? If it is not already measured, one representative throughput and peak-memory measurement for that configuration, with the benchmark model and batch size specified, would clarify the cost of the demonstrated gains.

6 comments
AuthorsOpenReview ↗

Response to W1/Q1 : Historical prefix-state ablation

We sincerely thank the reviewer for recognizing PHBA's block-aligned design, matched comparisons and practical feasibility. We agree that MoBA/NHA comparisons do not isolate historical snapshots. We have therefore trained the precise W1/Q1 control, plus an extra ablation.

Setup. Full PHBA and A/B were trained independently on the same FineWeb-Edu data with matched optimization: 760M/50B, 8K context, C=M=128, K=8. A (current-state-only) retains Full's routing, gates, recurrent updates, current prefix and exact routed tokens, removing only historical prefix readout. Additional B (routed-states-only) retains current/historical prefix states but omits exact routed tokens. Values: Full/A/B; 16K–64K are zero-shot extrapolations.

Table 1. LM/commonsense (PPL↓; others↑).

MetricFull/A/B
WikiText PPL18.33/19.28/20.28
LAMBADA Acc42.98/40.52/39.38
LAMBADA PPL16.33/18.92/20.17
ARC-Easy66.96/64.35/64.86
ARC-Challenge34.30/32.85/32.51
HellaSwag52.14/50.13/48.76
PIQA70.73/69.80/68.72
WinoGrande55.33/54.54/54.54
Commonsense Avg.55.89/54.33/53.87

Table 2. RULER/NIAH (4K/8K).

Task4K F/A/B8K F/A/B
RULER CWE26.30/21.14/27.623.60/9.02/7.54
RULER FWE19.50/1.80/022.10/0.73/0
RULER HotpotQA20.20/14.40/12.2015.60/9.60/14.20
RULER SQuAD24.40/9.80/14.0014.60/5.60/9.80
RULER VT1.00/9.04/0.922.00/10.36/0
RULER Avg.18.30/11.24/10.9511.60/7.06/6.31
NIAH Single-Key 199.40/99.40/97.4020.40/21.40/21.80
NIAH Single-Key 297.40/93.80/80.8042.80/38.40/48.00
NIAH Single-Key 367.40/73.20/15.0011.40/22.40/2.00
NIAH Multi-Key 178.00/57.00/38.0044.00/37.20/25.20
NIAH Multi-Key 243.20/5.20/0.205.60/0.80/0
NIAH Multi-Key 318.80/3.40/00/2.20/0
NIAH Extra MultiQuery41.70/32.05/31.5522.20/17.10/19.95
NIAH Extra MultiValue46.00/28.80/29.9025.00/20.30/18.50
NIAH Avg.61.50/49.11/36.6121.40/19.98/16.93

Table 3. RULER/NIAH (16K–64K).

Task16K F/A/B32K F/A/B64K F/A/B
RULER CWE0.10/0.10/00/0/00/0.06/0
RULER HotpotQA12.80/11.40/0.4010.00/8.00/06.40/5.00/0
RULER SQuAD8.80/7.80/06.90/6.60/05.50/1.20/0
RULER Avg.7.20/6.43/0.135.60/4.87/04.00/2.09/0
NIAH Multi-Key 110.00/10.00/02.20/1.80/03.20/2.80/0
NIAH Multi-Key 20/0.60/00/0/00/0/0
NIAH Multi-Key 30.20/0.10/00/0/00/0/0
NIAH MultiQuery4.80/4.50/01.20/0.70/00.30/0/0
NIAH MultiValue7.40/5.50/01.90/1.75/02.40/2.40/0
NIAH Avg.4.50/4.14/01.10/0.85/01.20/1.04/0

Table 4. LongBench (11 tasks).

TaskFull/A/B
Single-Doc: MultiFieldQA14.82/13.74/12.26
Single-Doc: Qasper6.54/6.62/4.75
Multi-Doc: HotpotQA6.03/6.07/5.95
Multi-Doc: 2WikiMultihopQA8.84/9.11/8.22
Summarization: GovReport8.77/9.81/4.80
Summarization: MultiNews8.74/11.35/5.28
Summarization: SAMSum19.06/17.91/19.03
Few-shot: TREC38.67/31.50/32.50
Few-shot: TriviaQA24.19/13.67/14.55
Code: LCC12.95/19.19/19.49
Code: RepoBench-P12.90/18.99/20.22
Avg.14.68/14.36/13.37

Analysis and response to W1/Q1. (1) Direct evidence: Full exceeds A on WikiText, LAMBADA, commonsense, all RULER/NIAH averages, and LongBench (e.g., RULER-8K 11.60 vs. 7.06; NIAH-4K 61.50 vs. 49.11). A retains the same routed tokens, gates and current prefix; only historical snapshot readout differs. This isolates the reviewer's precise concern without confounding MoBA/NHA architectural differences.

(2) Why historical snapshots help: The fixed-capacity current prefix summarizes all past tokens; later recurrent updates may dilute or overwrite earlier details. By contrast, each historical snapshot captures a block's preceding context before those later updates, potentially offering richer, temporally aligned information for retrieved exact tokens. This helps explain why historical states add value even when the current state already summarizes the full preceding history.

(3) Complementary evidence: B retains historical snapshots but discards the exact tokens whose content determined routing. Its lower RULER/NIAH and LongBench averages, especially at long lengths, show that preceding context alone cannot substitute for the selected block's contents. Together, these controlled ablations support PHBA's joint readout of retrieved exact tokens and block-aligned historical context. We appreciate this constructive suggestion.

AuthorsOpenReview ↗

Response to Q2 : Throughput and peak memory under the main quality configuration

We thank the reviewer for pointing out the difference between the main quality configuration and Fig. 3. We therefore conducted additional measurements covering the exact configuration used for our main quality results.

Experimental protocol. All models are approximately 1.3B parameters and run on a single NVIDIA H20 GPU, without activation checkpointing (AC). Each step processes 16,384 tokens (2K×8, 4K×4, 8K×2, or 16K×1). Following Fig. 3, timing covers complete training computation, including forward, backward, and parameter updates, excluding input preparation and data loading. We use 20 warm-up steps and 50 measured steps, with three independent runs. Throughput is averaged across runs; peak allocated/reserved memory is averaged over the three per-run peaks. All triplets below are PHBA/MoBA/Transformer.

Main quality configuration: C=M=128, K=8, 8K×2 (batch size 2).

ModelThroughput (tok/s) ↑Peak allocated (GiB) ↓Peak reserved (GiB) ↓
PHBA9,85862.8965.56
MoBA9,51060.6363.60
Transformer10,36744.0547.07

Under the exact main-quality configuration, PHBA achieves 3.7% higher throughput than MoBA, despite using 2.26 GiB (3.7%) more peak allocated memory and 1.96 GiB (3.1%) more reserved memory. Its throughput is within 4.9% of Transformer. Despite introducing additional prefix states and the corresponding attention computations, PHBA maintains comparable and slightly higher training throughput than MoBA. We attribute this low overhead to our carefully optimized Triton operators, including chunkwise prefix-state construction, metadata-driven token/state access, online-softmax merging, and block-major backward, without materializing sparse gather tensors.

Additional parameter sweeps. We evaluate three strategies: (i) fixed C=128, K=8 without fixing the sparsity ratio; (ii) fixed K=8, increasing C to maintain a 1/8 exact-token budget; and (iii) fixed C=128, increasing K to maintain the same budget. Bold 8K×2 rows correspond to the main-quality configuration. C/K denotes the block size/Top-K for PHBA and MoBA; Transformer uses full attention.

StrategySeq×Batch; C/KThroughput (tok/s)Allocated (GiB)Reserved (GiB)
Fixed C/K2K×8; 128/811042/9983/1378462.89/57.95/44.0566.47/61.18/47.10
4K×4; 128/810440/9682/1245462.90/59.74/44.0567.39/62.13/47.11
8K×2; 128/89858/9510/1036762.89/60.63/44.0565.56/63.60/47.07
16K×1; 128/89280/9468/785862.90/61.08/44.0567.39/64.52/47.03
Fixed K=82K×8; 32/810288.7/7750.4/1382073.02/60.96/44.0575.25/71.47/47.03
4K×4; 64/810367/8908/1241766.27/60.74/44.0569.89/64.97/47.03
8K×2; 128/89861/9518/1036462.88/60.63/44.0564.64/63.27/47.04
16K×1; 256/88709/9242/782461.21/60.57/44.0565.24/63.17/47.11
Fixed C=1282K×8; 128/212814/11935/1381962.31/49.86/44.0564.43/53.65/47.11
4K×4; 128/411678/11020/1244062.52/53.25/44.0564.62/57.92/47.08
8K×2; 128/89831/9496/1037262.89/60.63/44.0565.66/63.65/47.06
16K×1; 128/167269/7481/783563.63/75.34/44.0566.84/79.46/47.07

The three independent parameter sweeps consistently confirm the main-configuration result: PHBA achieves 9.831–9.861K tokens/s, compared with 9.496–9.518K for MoBA at 8K×2. These measurements demonstrate that our efficient Triton operator design keeps PHBA's additional recurrent-state computation affordable, maintaining comparable and slightly higher throughput than MoBA at the cost of additional memory. They also directly quantify the computational trade-offs of the configuration producing our main quality improvements.

Thank you for the response. Most of my concerns have been addressed, and I will maintain my positive score.

Reviewer_Gjv5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PHBA replaces the sliding window branch of NHA with MoBA style top K block routing. The sequence is split into blocks of 128 tokens, and each query attends causally to its current block and to 7 historical blocks chosen by similarity to mean pooled RoPE keys. A GSA style gated slot recurrence with 128 slots is computed chunkwise over the same blocks, and the state at each block boundary is stored as that block's prefix state. Each query also attends to the prefix states of its current block and of every routed block. All token and slot logits share one softmax, merged online in Triton kernels driven by integer route metadata. Models of 760M and 1.3B are trained from scratch at 8K.

Critical strengths
  • Within the 8K training length, PHBA improves over both of its components, MoBA and GSA, on retrieval and language modeling. This is the main evidence that routed exact blocks and slot memory are useful together under one softmax.

  • The prefix states reuse the boundary states that chunkwise GLA already computes, and the kernels work on integer routes without gathering payloads. As a result, the training overhead over MoBA is small.

Critical weaknesses
  • The evaluation setting cannot validate the long context claims. The models are small and trained only at 8K, so RULER and NIAH beyond 8K have limited validity, and LongBench is truncated to the training length. It remains unclear whether routed blocks and prefix states actually help long context modeling.

  • The novelty is limited. PHBA combines NHA's joint softmax over tokens and slot memory with MoBA's block routing. The main new modeling element, attaching prefix states to routed blocks, is not isolated by any ablation.

  • The inference efficiency claims are not supported. Prefix states are stored for every block, and in the default setting their total size matches the KV cache itself, so they add substantial memory on top of the full KV cache that routing already requires. Although the paper claims efficient inference, no inference latency or memory is measured.

Questions for authors
  • What window size does NHA use in the main comparison, and what block size and number of blocks does MoBA use?

  • Can you provide results with larger models or with training at a longer context? In that setting the baselines would also handle long context reasonably, so RULER and NIAH could meaningfully separate PHBA from the other methods.

  • Can you add a variant that keeps only the current block prefix state, and a MoBA variant whose budget matches PHBA's total attended entries?

  • Can you report inference memory, prefill latency, and decode throughput against MoBA, NHA, and KDA?

6 comments
AuthorsOpenReview ↗

Response (Q1/W1/Q2/Q3) to Reviewer Gjv5

We thank the reviewer for recognizing the value of our work.

Q1 — Baseline configurations.

For NHA [1], we adopt the best-performing configuration reported in the original paper (Appendix C.4.1, Table 7): a sliding window size of 32 and 64 memory slots, which achieves the highest recall and commonsense performance in its configuration ablation. For MoBA [2], we use the same block size and Top-K setting as PHBA, with C=128 and K=8 (the current block plus seven routed historical blocks), ensuring matched exact-token attention budgets. All six models are approximately 1.3B parameters, trained from scratch on the same FineWeb-Edu data for 100B tokens with an 8K context length.

W1/Q2/Q3 — Model scale, training context, and additional ablations.

We sincerely thank the reviewer for the thoughtful suggestions regarding larger models, longer-context training, and additional controlled comparisons. To put our experimental setting into perspective, we summarize representative efficient-architecture research papers:

Model (venue)AblationsMain comparisonTrain lengthEvaluation
GLA [3] (ICML'24)340M/15B1.3B/100B2K~20K extrap.
DeltaNet [4] (NIPS'24)340M/15B1.3B/100B2K512 MQAR
GSA [5] (NIPS'24)340M/10B1.3B/100B; 2.7B/100B2K512 MQAR; 2K recall
Comba [6] (NIPS'25)340M/15B1.3B/100B2K2K recall
NHA [1] (ACL'26)340M/15B1.3B/100B2KRULER to 8K
GDN [7] (ICLR'25)400M/15B1.3B/100B4K~20K extrap.
GDN2 [8] (NIPS'26)Gate ablations1.3B/100B4KRULER 1–8K
PHBA (ours)760M/50B1.3B/100B8KRULER/NIAH 4–64K

These studies show that 1.3B/100B is a widely adopted and meaningful evaluation scale in efficient-architecture research. Importantly, our original submission already extends these representative experimental settings in three aspects:

  1. Longer training context: We train at 8K, compared with the commonly adopted 2K–4K contexts.
  2. Larger ablation budget: We conduct architectural ablations at 760M/50B, exceeding the typical 340M/10–15B settings.
  3. More extensive long-context evaluation: Beyond 4K/8K, we additionally evaluate both RULER and NIAH at 16K, 32K, and 64K, reaching 8× our training context. These additional evaluations were already included in our initial submission.

Furthermore, we trained PHBA and all five baselines from scratch on the same FineWeb-Edu data, with matched model scale, 100B training tokens, and 8K context length. Under this controlled setting, PHBA achieves the highest RULER averages at 4K/8K/16K/32K/64K and the highest NIAH averages at 16K/32K/64K among all six models (Table 1, Appendix Table 2, and Fig. 2). Evaluations beyond 8K measure zero-shot length extrapolation without additional training.

We fully appreciate the value of larger models and longer-context training. However, as an academic team focused on efficient architectures rather than foundation-model pretraining, our access to large GPU clusters is limited. Training six 1.3B models from scratch on 100B tokens already required substantial compute. Retraining all models at larger scales or longer contexts under matched conditions exceeds our rebuttal budget.

Nevertheless, we are making every effort to strengthen our empirical evidence through two additional controlled experiments addressing Q3:

  • Current-prefix-only PHBA (760M/50B): Retains historical routed exact tokens and the current-block prefix state, while removing historical routed prefix-state readout. This directly isolates the contribution of block-aligned historical memory.

  • Total-attended-entry-matched MoBA (1.3B/100B): Uses C=256, K=8, providing approximately 2,048 nominal attended token entries, matching PHBA's C=M=128, K=8 configuration (approximately 1,024 token entries + 1,024 prefix-state entries). This examines whether PHBA's improvements arise from its architectural design rather than simply having a larger total attention budget.

We will report both results during rebuttal. We believe our original experiments already provide more extensive ablation, training-context, and long-context evaluation evidence than is customary in comparable efficient-architecture studies, supporting PHBA's effectiveness under rigorous, controlled comparisons. Together with our additional experiments, we hope these efforts address the reviewer's concerns.

References

[1] Du et al., Native Hybrid Attention

[2] Lu et al., MoBA

[3] Yang et al., Gated Linear Attention

[4] Yang et al., Parallelizing Linear Transformers with the Delta Rule

[5] Zhang et al., Gated Slot Attention

[6] Hu et al., Comba

[7] Yang et al., Gated Delta Networks

[8] NVIDIA, Gated DeltaNet-2

[9] Kimi Team, Kimi Linear

AuthorsOpenReview ↗

Response (W2/W3/Q4) to Reviewer Gjv5

W2 — Novelty and architectural contribution.

Advances in efficient architectures often arise from simple yet targeted refinements of established mechanisms that address important limitations while striking a balance between model performance and computational efficiency. Several representative examples illustrate this pattern:

ModelSimple refinement over baselineLimitation addressed
GSA [5]GLA/ABC: Two GLA-style stages for key and value memories, connected by softmax.Improves recall with compact recurrent memory.
GDN [7]DeltaNet: Adds a scalar forgetting gate.Enables adaptive removal of outdated memory.
KDA [9]GDN: Replaces scalar forgetting with channel-wise forgetting.Enables more selective memory retention.
GDN2 [8]KDA: Decouples erase and write into independent gates.Avoids tying memory erasure and writing together.
NHA [1]SWA + GSA-style memory: Combines local exact tokens and recurrent slots under one softmax.Preserves precise recent information alongside compressed long-term context.

These contributions are meaningful not despite their simplicity, but because targeted modifications address concrete architectural limitations, deliver substantial empirical improvements, and are supported by efficient GPU implementations.

PHBA follows this principle while addressing a distinct limitation of existing same-layer hybrids. NHA's fixed sliding window cannot directly retrieve relevant distant tokens, making exact access position-dependent rather than input-adaptive. PHBA addresses this limitation through a simple yet effective refinement: replacing fixed-window attention with dynamic block-sparse retrieval (like MoBA) and pairing each selected block with a gated recurrent prefix state summarizing its preceding context. This enables input-adaptive joint modeling of exact distant-token information and its corresponding historical context, rather than relying on fixed local access and a single recurrent memory.

Despite this straightforward modification, PHBA substantially improves retrieval and long-context understanding over NHA across RULER, NIAH, and LongBench:

MetricNHAPHBA
RULER Avg. (4K/8K/16K/32K/64K)26.4 / 15.1 / 9.3 / 8.3 / 7.027.2 / 23.9 / 10.4 / 16.7 / 9.3
NIAH Avg. (4K/8K/16K/32K/64K)49.5 / 27.4 / 5.7 / 2.9 / 2.362.6 / 29.5 / 11.4 / 3.2 / 2.8
LongBench Avg.14.9118.14

Notably, PHBA improves over NHA by 58.3% on RULER-8K, 101.2% on RULER-32K, 100.0% on NIAH-16K, and 21.7% on LongBench (Table 1 and Appendix Table 2; Fig. 2).

Our Triton implementation further integrates chunkwise state construction, metadata-driven token/state access, online-softmax merging, and block-major backward without materializing sparse gather tensors, maintaining competitive training efficiency. Sec. 3.4 and Appendix A.1 provide the parallel derivations and detailed pseudocode. The code will be released soon.

We therefore believe PHBA represents a well-motivated, effective, and practical contribution to efficient large-model architectures, with clear architectural novelty.

W3/Q4 — Inference efficiency and memory. We thank the reviewer for requesting inference evidence. We benchmarked six ~1.3B architectures on NVIDIA H20 (BF16, batch=1, 32K/64K prompts, 256 cached decode steps). Models run separately on one GPU with fixed-seed random weights. We report medians over five runs after warm-up and maximum peak allocated memory. Implementations follow published inference designs (e.g., MoBA: sparse prefill, full-attention decode); see AXBF Q2 for details.

ModelPrefill s (32K/64K)Decode tok/s (32K/64K)Peak GiB (32K/64K)
Transformer1.80/5.8350.0/50.710.45/18.03
MoBA1.41/3.0145.6/44.915.16/28.61
NHA1.06/2.1032.8/32.57.30/10.81
KDA0.74/1.4534.7/34.94.91/7.29
GSA0.82/1.6241.7/44.45.19/7.69
PHBA1.12/2.5139.8/38.917.44/31.89

PHBA reduces prefill time by 20.8%/16.6% vs. MoBA and 37.8%/56.8% vs. Transformer; its decode latency is 25.10/25.74 ms/token, only 14.6%/15.6% above MoBA.

Memory accounting. We explicitly store and count both full historical token KV and all block-boundary prefix K/V snapshots. At 32K/64K (including 256-token capacity), their allocations are 6.047/12.047 GiB (KV) and 6.070/12.070 GiB (prefix): 1.004×/1.002×. Thus we confirm the reviewer's observation for M=C=128; both caches, recurrent states, workspaces, and weights are included in peak memory. This additional storage supports joint retrieval of exact distant tokens and preceding recurrent context. PHBA ranks first among the six compared models on RULER at 4K–64K and NIAH at 16K–64K, demonstrating a favorable balance between retrieval quality and inference efficiency.

AuthorsOpenReview ↗

Response to Q3: Detailed Experimental Results (Current-prefix-only and routed-states-only)

We thank the reviewer for suggesting these important controls. The MoBA variant matched to PHBA's total attended entries is currently training; we will report its results once available. We have completed the requested current-block prefix-only variant (A) and additionally trained routed-states-only (B) to test whether exact retrieved tokens remain necessary.

Setup. Full/A/B are separately trained 760M/50B models with matched data, optimizer, 8K context, C=M=128, K=8. A retains the current prefix and exact routed tokens, but removes historical routed prefix states. B retains current/historical routed prefix states, but removes routed exact tokens.

Table 1. Language modeling/commonsense.

MetricFull/A/B
WikiText PPL↓18.33/19.28/20.28
LAMBADA Acc↑42.98/40.52/39.38
LAMBADA PPL↓16.33/18.92/20.17
ARC-Easy66.96/64.35/64.86
ARC-Challenge34.30/32.85/32.51
HellaSwag52.14/50.13/48.76
PIQA70.73/69.80/68.72
WinoGrande55.33/54.54/54.54
Commonsense Avg.55.89/54.33/53.87

Table 2. Short-context RULER/NIAH.

Task4K F/A/B8K F/A/B
RULER CWE26.30/21.14/27.623.60/9.02/7.54
RULER FWE19.50/1.80/022.10/0.73/0
RULER HotpotQA20.20/14.40/12.2015.60/9.60/14.20
RULER SQuAD24.40/9.80/14.0014.60/5.60/9.80
RULER VT1.00/9.04/0.922.00/10.36/0
RULER Avg.18.30/11.24/10.9511.60/7.06/6.31
NIAH Single-Key 199.40/99.40/97.4020.40/21.40/21.80
NIAH Single-Key 297.40/93.80/80.8042.80/38.40/48.00
NIAH Single-Key 367.40/73.20/15.0011.40/22.40/2.00
NIAH Multi-Key 178.00/57.00/38.0044.00/37.20/25.20
NIAH Multi-Key 243.20/5.20/0.205.60/0.80/0
NIAH Multi-Key 318.80/3.40/00/2.20/0
NIAH Extra MultiQuery41.70/32.05/31.5522.20/17.10/19.95
NIAH Extra MultiValue46.00/28.80/29.9025.00/20.30/18.50
NIAH Avg.61.50/49.11/36.6121.40/19.98/16.93

Table 3. Long-context RULER/NIAH.

Task16K F/A/B32K F/A/B64K F/A/B
RULER CWE0.10/0.10/00/0/00/0.06/0
RULER HotpotQA12.80/11.40/0.4010.00/8.00/06.40/5.00/0
RULER SQuAD8.80/7.80/06.90/6.60/05.50/1.20/0
RULER Avg.7.20/6.43/0.135.60/4.87/04.00/2.09/0
NIAH Multi-Key 110.00/10.00/02.20/1.80/03.20/2.80/0
NIAH Multi-Key 20/0.60/00/0/00/0/0
NIAH Multi-Key 30.20/0.10/00/0/00/0/0
NIAH MultiQuery4.80/4.50/01.20/0.70/00.30/0/0
NIAH MultiValue7.40/5.50/01.90/1.75/02.40/2.40/0
NIAH Avg.4.50/4.14/01.10/0.85/01.20/1.04/0

Table 4. LongBench (all 11 tasks).

TaskFull/A/B
Single-Doc: MultiFieldQA14.82/13.74/12.26
Single-Doc: Qasper6.54/6.62/4.75
Multi-Doc: HotpotQA6.03/6.07/5.95
Multi-Doc: 2WikiMultihopQA8.84/9.11/8.22
Summarization: GovReport8.77/9.81/4.80
Summarization: MultiNews8.74/11.35/5.28
Summarization: SAMSum19.06/17.91/19.03
Few-shot: TREC38.67/31.50/32.50
Few-shot: TriviaQA24.19/13.67/14.55
Code: LCC12.95/19.19/19.49
Code: RepoBench-P12.90/18.99/20.22
Avg.14.68/14.36/13.37

Analysis:

Analysis.

(1) Historical snapshots matter. Although A retains the current prefix state summarizing all preceding tokens, removing historical routed prefix states consistently degrades retrieval performance. This may be because the current state compresses the entire history into fixed-capacity memory, where successive recurrent updates may gradually dilute or overwrite earlier information. In contrast, each block-aligned historical snapshot preserves the context preceding its corresponding retrieved block before subsequent updates, potentially retaining richer and more relevant historical information. Thus, historical prefix states provide contextual information beyond what the current global memory alone can preserve.

(2) Exact routed tokens matter. B consistently underperforms A on RULER/NIAH averages and LongBench, with particularly severe degradation under long-context extrapolation. Importantly, blocks are selected based on the relevance of their token content, yet B discards those exact tokens and retains only their preceding prefix states. Since these states summarize the context before each retrieved block rather than its actual token content, they cannot fully recover the specific information that motivated retrieval. This explains why historical prefix states cannot effectively replace direct access to exact distant tokens.

(3) Both are complementary. Full PHBA outperforms both ablations by jointly accessing exact retrieved tokens and their corresponding preceding historical context. These results directly support the necessity and effectiveness of PHBA's joint retrieval design.

Reviewer_AXBFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PHBA, a single-layer hybrid of block-sparse attention and gated slot recurrence. The attention has three parts: MoBA-style top-K block routing scores the query against the mean of each block's RoPE keys, A GSA-style gated slot recurrence with M slots is checkpointed at every block boundary, and each query attends under one softmax to two kinds of memory. Models are trained from scratch on FineWeb-Edu at 8K context: 760M models on 50B tokens (ablations) and 1.3B models on 100B tokens (comparisons with Transformer, MoBA, GSA, KDA, and NHA). Evaluation covers LM and commonsense tasks, RULER/NIAH at 4K–8K, zero-shot extrapolation to 16K–64K, LongBench truncated to 8K, and training throughput on an H20.

Critical strengths

S1. There are a sabstantial systematic implemations. The chunkwise boundary recurrence, in-place metadata-driven forward, block-major backward, resets at packing boundaries and log-space cumulative gates are implementated.

S2. The experimental results of in-distrbution are solid. At 1.3B, PHBA has the best LM perplexity and LAMBADA scores and competitive commonsense accuracy. On NIAH at 4K it comes close to the dense Transformer and far exceeds MoBA.

Critical weaknesses

W1: Contribution and position against closely related work is unclear. PHBA is NHA with the sliding window replaced by MoBA routing, plus routed state snapshots. That delta is real, but it should be stated plainly. NSA is the most direct "compressed + selected + local" competitor and should be compared against, not just cited. Related work is a single paragraph inside the experiments section, which is not a well organization.

W2: The paper's thesis is that aligned prefix states attached to routed blocks are what help. The experiments do not separate this from the simpler explanation that adding a global recurrent memory to MoBA helps, which NHA and GSA already suggest.

Questions for authors

Q1. The author should conduct the following key ablations:

  • (a) MoBA + current-block state only, i.e., NHA with MoBA instead of SWA;
  • (b) routed tokens without routed states;
  • (c) routed states without routed tokens.

Q2. What are the decoding latency and peak memory at 32K and 64K for PHBA vs. Transformer, MoBA, NHA, and KDA, given that the prefix states match the KV cache in size?

8 comments
AuthorsOpenReview ↗

Response to Reviewer AXBF

We thank the reviewer for recognizing the value of PHBA and for the constructive feedback.

W1/W2/Q1 — Positioning and ablations. We thank the reviewer for this insightful comment. As outlined in the Abstract and Introduction, unlike NHA, which restricts exact-token access to a fixed sliding window, PHBA dynamically retrieves query-relevant historical blocks (like MoBA) and jointly accesses their exact tokens and aligned prefix-state snapshots, enabling precise long-range retrieval while preserving the preceding context of each retrieved block. To further substantiate this design, we are conducting an NSA comparison (1.3B, 100B training tokens) and targeted PHBA ablations (760M, 50B tokens): MoBA with only the current-block prefix state and routed states without historical routed tokens. Our existing MoBA baseline serves as the routed-token-only comparison. We will share the new results during rebuttal.

Q2 — Decoding latency and peak memory at 32K/64K. We thank the reviewer for highlighting the importance of decoding efficiency and memory overhead. We benchmarked cached inference for Transformer, MoBA, NHA, KDA, GSA, and PHBA on NVIDIA H20 GPUs. All models are approximately 1.3B, with 24 layers, hidden size 2048, 16 heads, and 128 dimensions/head. Each model runs independently on one GPU (BF16, batch size 1); 32K/64K prompts are followed by 256 incremental decode steps. To isolate architectural execution costs, we use fixed-seed (42) randomly initialized weights; the quality results below use our trained models. Under PyTorch 2.9.0/CUDA 12.8, we perform one warm-up and five measured runs per configuration. We report median prefill latency and median synchronized decode wall time per token; peak allocated memory is the maximum across runs, including weights, caches, and workspaces.

Implementations follow the respective architectures: Transformer uses full KV caching; MoBA uses block-sparse prefill (C=128, K=8) and full-attention decode, as in its original inference configuration and FLA; GSA uses 128 recurrent slots; NHA combines 128 slots with a 2048-token sliding window; KDA uses 128-dimensional key/value states and size-4 convolution; PHBA uses C=M=128, K=8, complete historical token KV, and block-aligned prefix snapshots.

ModelPrefill s (32K/64K)Decode ms/tok (32K/64K)Peak GiB (32K/64K)
Transformer1.80 / 5.8320.02 / 19.7310.45 / 18.03
MoBA1.41 / 3.0121.91 / 22.2715.16 / 28.61
NHA1.06 / 2.1030.48 / 30.777.30 / 10.81
KDA0.74 / 1.4528.83 / 28.614.91 / 7.29
GSA0.82 / 1.6223.97 / 22.535.19 / 7.69
PHBA1.12 / 2.5125.10 / 25.7417.44 / 31.89

PHBA reduces prefill latency over Transformer by 37.8%/56.8% and over MoBA by 20.8%/16.6% at 32K/64K. Decoding incurs 14.6%/15.6% more latency than MoBA, while outperforming NHA and KDA in these measurements.

Prefix-state memory accounting. Both PHBA's full token KV cache and historical prefix-state snapshots are explicitly allocated and included in peak memory. For L layers, H heads, head dimension d, cache capacity N, block size C, M slots, and b bytes per element, token KV storage is 2LNHdb; prefix storage is 2L(ceil(N/C)+1)MHdb. For M=C=128, their ratio approaches M/C=1. Measured allocations include 256 generated-token capacity:

PHBA cache32K64K
Full token KV6.047 GiB12.047 GiB
Prefix snapshots6.070 GiB12.070 GiB
Prefix / KV1.004×1.002×

Thus, prefix-state snapshots require approximately the same storage as PHBA's own full KV cache. Both are counted in peak memory, alongside active recurrent states, routing metadata, temporary buffers, and weights.

Quality–efficiency trade-off. On our trained 1.3B models (100B tokens), PHBA achieves the highest RULER average among all six models across 4K–64K:

LengthPHBA RULERRunner-upRelative gain
4K27.2Transformer (26.9)+1.1%
8K23.9KDA (16.7)+43.1%
16K10.4KDA (9.5)+9.5%
32K16.7KDA (8.9)+87.6%
64K9.3GSA/KDA (7.6)+22.4%

PHBA also achieves the highest NIAH average at 16K–64K, ranking second overall at 4K/8K:

LengthPHBA NIAHBest other modelRelative difference
4K62.6‾\underline{62.6}Transformer (67.0)−6.6%
8K29.5‾\underline{29.5}Transformer (44.5)−33.7%
16K11.4KDA (7.4)+54.1%
32K3.2GSA (3.0)+6.7%
64K2.8NHA (2.3)+21.7%

See Table 1 (Parts I/III) for 4K/8K and Appendix Table 2 (Parts I/II) for 16K–64K (also Fig. 2). Models are trained at 8K; longer lengths are zero-shot extrapolation. These gains illustrate the value of jointly retrieving exact historical tokens and their aligned preceding context, alongside the quantified inference costs.

AuthorsOpenReview ↗

Response to W2/Q1: Detailed Experimental Results

We thank the reviewer for proposing these essential controls. All three requested comparisons are available: (A) routed exact tokens + current prefix, no historical prefix readout; (B) current/routed prefixes, no routed tokens. (C) MoBA: routed tokens without recurrent states (original baseline);

Setup. We report two separate scales. Group I (1.3B/100B): original PHBA vs. MoBA, trained independently on FineWeb-Edu at 8K with C=128,K=8 (paper Tables 1–2). Group II (760M/50B): original Full PHBA vs. newly trained A/B, all trained from scratch at 8K with matched data, optimization, C=M=128,K=8, routing, gates and recurrence. Results beyond 8K use zero-shot extrapolation. ↑ higher is better; PPL↓ lower.

Group I — PHBA vs. MoBA (1.3B/100B).

MetricPHBAMoBA
WikiText PPL↓15.2116.22
LAMBADA Acc↑49.2747.22
LAMBADA PPL↓10.8013.04
RULER 4K/8K27.2/23.923.4/10.8
RULER 16K/32K/64K10.4/16.7/9.30.4/0/0
NIAH 4K/8K62.6/29.539.3/23.6
NIAH 16K/32K/64K11.4/3.2/2.80/0/0
LongBench Avg.18.1410.58

Group II — Full/A/B (760M/50B). Every triple below is Full/A/B.

Table 1. Language modeling/commonsense.

MetricFull/A/B
WikiText PPL↓18.33/19.28/20.28
LAMBADA Acc↑42.98/40.52/39.38
LAMBADA PPL↓16.33/18.92/20.17
ARC-Easy66.96/64.35/64.86
ARC-Challenge34.30/32.85/32.51
HellaSwag52.14/50.13/48.76
PIQA70.73/69.80/68.72
WinoGrande55.33/54.54/54.54
Commonsense Avg.55.89/54.33/53.87

Table 2. Short-context RULER/NIAH

Task4K (Full/A/B)8K (Full/A/B)
RULER CWE26.30/21.14/27.623.60/9.02/7.54
RULER FWE19.50/1.80/022.10/0.73/0
RULER HotpotQA20.20/14.40/12.2015.60/9.60/14.20
RULER SQuAD24.40/9.80/14.0014.60/5.60/9.80
RULER VT1.00/9.04/0.922.00/10.36/0
RULER Avg.18.30/11.24/10.9511.60/7.06/6.31
NIAH Single-Key 199.40/99.40/97.4020.40/21.40/21.80
NIAH Single-Key 297.40/93.80/80.8042.80/38.40/48.00
NIAH Single-Key 367.40/73.20/15.0011.40/22.40/2.00
NIAH Multi-Key 178.00/57.00/38.0044.00/37.20/25.20
NIAH Multi-Key 243.20/5.20/0.205.60/0.80/0
NIAH Multi-Key 318.80/3.40/00/2.20/0
NIAH Extra MultiQuery41.70/32.05/31.5522.20/17.10/19.95
NIAH Extra MultiValue46.00/28.80/29.9025.00/20.30/18.50
NIAH Avg.61.50/49.11/36.6121.40/19.98/16.93

Table 3. Long-context RULER/NIAH.

Task16K F/A/B32K F/A/B64K F/A/B
RULER CWE0.10/0.10/00/0/00/0.06/0
RULER HotpotQA12.80/11.40/0.4010.00/8.00/06.40/5.00/0
RULER SQuAD8.80/7.80/06.90/6.60/05.50/1.20/0
RULER Avg.7.20/6.43/0.135.60/4.87/04.00/2.09/0
NIAH Multi-Key 110.00/10.00/02.20/1.80/03.20/2.80/0
NIAH Multi-Key 20/0.60/00/0/00/0/0
NIAH Multi-Key 30.20/0.10/00/0/00/0/0
NIAH MultiQuery4.80/4.50/01.20/0.70/00.30/0/0
NIAH MultiValue7.40/5.50/01.90/1.75/02.40/2.40/0
NIAH Avg.4.50/4.14/01.10/0.85/01.20/1.04/0

Table 4. LongBench (all 11 tasks).

TaskFull/A/B
Single-Doc: MultiFieldQA14.82/13.74/12.26
Single-Doc: Qasper6.54/6.62/4.75
Multi-Doc: HotpotQA6.03/6.07/5.95
Multi-Doc: 2WikiMultihopQA8.84/9.11/8.22
Summarization: GovReport8.77/9.81/4.80
Summarization: MultiNews8.74/11.35/5.28
Summarization: SAMSum19.06/17.91/19.03
Few-shot: TREC38.67/31.50/32.50
Few-shot: TriviaQA24.19/13.67/14.55
Code: LCC12.95/19.19/19.49
Code: RepoBench-P12.90/18.99/20.22
Avg.14.68/14.36/13.37

Analysis. Full PHBA achieves the best overall LM, commonsense, RULER and LongBench averages. (1) Historical snapshots matter: A retains the current prefix summarizing all past tokens, yet removing historical routed prefixes lowers RULER and NIAH. This may be because the current prefix compresses all preceding tokens into fixed-capacity memory, where successive recurrent updates may dilute or overwrite earlier context. Each retrieved block's prefix snapshot instead preserves its preceding context before later updates, providing richer, temporally aligned information for the retrieved exact tokens. (2) Routed exact tokens matter: B underperforms A on RULER/NIAH and LongBench. Blocks are selected for their relevant token content, yet B discards that very content, retaining only the preceding prefix states. This mismatch may explain B's poor long-context retrieval. Thus, exact-token retrieval is essential. (3) Both are complementary: PHBA's gains stem from jointly retrieving exact distant tokens and their preceding historical context, directly validating its design.

AuthorsOpenReview ↗

Response to W1: Completed NSA comparison (1.3B/100B)

We thank the reviewer for recommending NSA as an important baseline. As promised in our previous response, we have completed the NSA comparison, training both NSA and PHBA from scratch at 1.3B parameters, 100B FineWeb-Edu tokens, and 8K context, with the same evaluation protocol.

We reproduce NSA's original published attention configuration [1]: Compression Block=32, Stride=16; Selection Block=64, Top-K=16; Sliding Window=512, retaining all three branches (Compression + Selection + Sliding Window). PHBA uses C=M=128, K=8. All results below are PHBA/NSA (PPL↓; others↑). Tests beyond 8K use zero-shot length extrapolation.

Table 1. Language modeling/commonsense.

MetricPHBA/NSA
WikiText PPL15.21/17.19
LAMBADA Acc49.27/46.56
LAMBADA PPL10.80/13.68
ARC-Easy71.17/69.61
ARC-Challenge38.14/34.39
HellaSwag58.58/54.90
PIQA72.63/71.49
WinoGrande59.83/57.30
Commonsense Avg.60.07/57.54

Table 2. Short-context RULER/NIAH.

Task4K P/N8K P/N
RULER CWE31.0/20.3424.2/1.76
RULER FWE31.3/30.2745.3/29.20
RULER HotpotQA29.0/15.6013.4/17.80
RULER SQuAD43.4/22.2016.2/15.60
RULER VT1.5/3.0420.5/0
RULER Avg.27.2/18.2923.9/12.87
NIAH Single-Key 1100.0/96.8099.4/40.80
NIAH Single-Key 298.6/67.8039.6/25.80
NIAH Single-Key 354.4/28.8023.6/12.40
NIAH Multi-Key 176.0/49.2026.6/29.00
NIAH Multi-Key 245.8/6.400/4.20
NIAH Multi-Key 313.0/3.400/1.00
NIAH Extra MultiQuery55.8/35.6020.5/18.15
NIAH Extra MultiValue59.0/36.5026.1/20.25
NIAH Avg.62.6/40.5629.5/18.95

Table 3. Long-context RULER/NIAH.

Task16K P/N32K P/N64K P/N
RULER CWE5.0/0.0826.0/0.124.0/0.04
RULER HotpotQA14.0/13.6013.0/10.2013.4/10.40
RULER SQuAD12.2/11.0011.0/8.4010.4/7.80
RULER Avg.10.4/8.2316.7/6.249.3/6.08
NIAH Multi-Key 113.6/7.808.2/6.806.2/3.00
NIAH Multi-Key 220.0/1.200/0.400/0.40
NIAH Multi-Key 30/0.200/00/0
NIAH MultiQuery11.5/9.552.1/4.953.0/2.90
NIAH MultiValue11.9/10.805.8/6.654.7/2.65
NIAH Avg.11.4/5.913.2/3.762.8/1.79

Table 4. LongBench (all 11 tasks).

TaskPHBA/NSA
Single-Doc: MultiFieldQA15.79/15.97
Single-Doc: Qasper7.16/5.90
Multi-Doc: HotpotQA7.49/5.86
Multi-Doc: 2WikiMultihopQA9.85/8.49
Summarization: GovReport12.10/7.04
Summarization: MultiNews9.26/14.60
Summarization: SAMSum25.06/29.62
Few-shot: TREC46.83/40.50
Few-shot: TriviaQA26.75/12.63
Code: LCC18.39/20.13
Code: RepoBench-P20.81/20.79
Avg.18.14/16.50

Analysis.

(1) Overall quality. Under matched 1.3B/100B training with 8K context, PHBA outperforms NSA on WikiText PPL, LAMBADA accuracy, all five commonsense tasks, all five RULER averages (4K–64K), four of five NIAH averages, and LongBench average. RULER-8K improves from 12.87 to 23.9, RULER-32K from 6.24 to 16.7, and NIAH-4K from 40.56 to 62.6. NSA leads on NIAH-32K and some LongBench tasks, but PHBA is stronger overall.

(2) Exact-token and state budgets. At 8K, NSA uses 16×64=1,024 selected-token entries plus a 512-token sliding window; these overlap, so their counts cannot be added as unique tokens. PHBA uses 8×128=1,024 exact-token positions (one local and seven routed blocks). For compressed/state-side attention, only representations accessed by each query are counted:

Per-head, 8K contextNSAPHBA
Active K/V pairs≈511 compressed pairs≤8×128=1,024 state slots
K+V scalars (128-d each)≈130,816≤262,144
Relative size1×≤2.00×

Thus, NSA has more nominal exact-token entries, while PHBA allocates up to twice as many active K/V scalars to historical states. These distinct representations are not directly equivalent in information capacity.

(3) Mechanistic distinction. NSA compresses overlapping short windows into KV representations. PHBA instead maintains gated recurrent summaries of the causal prefix preceding each block and pairs each retrieved block's exact tokens with its aligned historical prefix snapshot under a unified softmax. This jointly exposes distant evidence and its preceding context, offering a plausible explanation for PHBA's stronger aggregate results. The completed same-scale comparison supports PHBA's effectiveness against NSA's compression + selection + local attention design. We thank the reviewer for this valuable suggestion.

References

[1] Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. ACL 2025.

Reviewer_UtiQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Prefix-State Hybrid Block Attention (PHBA), a hybrid architecture that combines query-dependent top-K block-sparse attention with gated recurrent prefix states. Each retrieved token block is paired with a compressed representation of its preceding context, and both memory types are integrated through a shared softmax. The authors also develop a Triton implementation with chunkwise state construction and metadata-driven sparse execution.

Critical strengths
  1. The combination of sparse retrieval and recurrent memory is reasonable and shows improvements on several long-context benchmarks.
  2. The paper compares against relevant baselines, including MoBA, GSA, KDA, and NHA.
Critical weaknesses
  1. Limited novelty and unclear contributions. Gating mechanisms and sparse attention have already been extensively studied in prior work, including MoBA, GSA, and DeepSeek-related architectures. The paper does not clearly distinguish its novel contributions from existing techniques.
  2. The experiments are limited to models up to 1.3B parameters, and larger-scale long-context evaluations are missing. Furthermore, the reported 12.8% training-throughput improvement over full attention at 16K is relatively modest for a sparsity-based approach.
Questions for authors
  1. What are the fundamental technical contributions of PHBA beyond combining existing sparse attention and gated recurrent memory mechanisms?
  2. Can the authors evaluate PHBA on larger models with longer context lengths?
  3. Why does PHBA achieve only a 12.8% throughput improvement despite using a 1/8 exact-token attention budget?
9 comments
AuthorsOpenReview ↗

Response to Reviewer UtiQ

We thank the reviewer for recognizing PHBA's value.

W1/Q1. Novelty and Contributions

Efficient architectures advances often arise from simple yet targeted refinements that address existing limitations while improving model effectiveness and computational efficiency:

Sparse attention: SWA uses fixed local windows; MoBA [1] employs block-level top-K, while DSA [2] uses token-level top-K for input-adaptive sparse attention.

Linear attention: GLA [3] = linear attention + adaptive forgetting; GSA [4] = two GLA-like gated memories linked by softmax. DeltaNet [5] = linear attention + targeted delta-rule correction; GDN [6] = Mamba-2-style forgetting + DeltaNet-style correction; KDA [7] = GDN with channel-wise rather than scalar forgetting; GDN2 [8] = KDA with independent channel-wise erase/write gates.

Hybrid attention: NHA [9] = compressed recurrent memory (GSA-like) + fixed-window exact local attention (SWA-like) within the same layer.

Such changes may appear incremental, but they resolve real limitations, improve results, and come with specialized GPU/Triton kernels.

PHBA addresses a limitation of existing same-layer hybrids. NHA [9]'s fixed window cannot directly retrieve relevant distant tokens, making exact access position-dependent rather than input-adaptive. PHBA addresses this limitation by using block-sparse attention (MoBA-like) to select relevant distant token blocks and pairing each with a gated recurrent prefix state (GSA-like) summarizing its preceding context, enabling input-adaptive joint modeling of exact token information and its corresponding historical context. Experiments further show that this intuitive architectural refinement yields consistent improvements over baselines across nearly all evaluated tasks.

Our Triton implementation integrates chunkwise state construction, metadata-driven token/state loading, online-softmax merging, and block-major backward, avoiding materialized gather tensors while maintaining competitive training throughput. Detailed formulations and kernel pseudocode are provided in Sec. 3.4 and Appendix A.1 for reproducibility, and the code will be released.

W2/Q2. Scale and context

Representative experimental settings from well-known efficient-architecture research papers (rather than foundation-model technical reports) are summarized below:

Model (venue)AblationsMain comparisonTrainEvaluation
GLA [3] (ICML'24)340M/15B1.3B/100B2K~20K extrap.
DeltaNet [5] (NIPS'24)340M/15B1.3B/100B2K512 MQAR
GSA [4] (NIPS'24)340M/10B1.3B/100B; 2.7B/100B2K512 MQAR; 2K recall
Comba [10] (NIPS'25)340M/15B1.3B/100B2K2K recall
NHA [9] (ACL'26)340M/15B1.3B/100B2KRULER to 8K
GDN [6] (ICLR'25)400M/15B1.3B/100B4K~20K extrap.
GDN2 [8] (NIPS'26)gate ablations1.3B/100B4KRULER 1–8K
PHBA (ours)760M/50B1.3B/100B8KRULER/NIAH 4–64K

The table shows that 1.3B/100B is a widely used controlled comparison setting in efficient-architecture research, while 4K/8K RULER/NIAH are common long-context evaluation points. We extend prior ablation budgets to 760M/50B and train at 8K context, aiming to examine PHBA as thoroughly as possible with limited hardware. We also go beyond 4K/8K tests by additionally evaluating 16K/32K/64K RULER/NIAH, where PHBA achieves the highest averages among our compared models, further supporting its long-context effectiveness. These longer evaluations are zero-shot extrapolations, not longer-context pretraining.

We trained PHBA and five baselines from scratch on matched data, scale, and 8K context, at substantial GPU cost. Larger-scale training is valuable, but our hardware budget cannot train larger models during rebuttal; we respectfully ask for understanding.

W2/Q3. Throughput

PHBA adopts an intra-layer hybrid architecture: sparse attention retrieves relevant distant token blocks, while gated recurrence provides their preceding historical context. The 1/8 budget counts exact attention tokens only, not total training computation, which also includes projections, MLPs, recurrent state computation, and routing overhead. Thus, sparsity does not translate proportionally into throughput gains. Nevertheless, PHBA achieves 8.79K tokens/s at 16K, 12.8% faster than dense Transformer, with only 6.7–11.9% overhead over MoBA (Fig. 3), offering richer historical context at modest additional cost.

References

[1] Lu et al., MoBA

[2] DeepSeek-AI, DeepSeek-V3.2-Exp

[3] Yang et al., Gated Linear Attention

[4] Zhang et al., Gated Slot Attention

[5] Yang et al., Parallelizing Linear Transformers with the Delta Rule

[6] Yang et al., Gated Delta Networks

[7] Kimi Team, Kimi Linear

[8] NVIDIA, Gated DeltaNet-2

[9] Du et al., Native Hybrid Attention

[10] Hu et al., Comba

Thank you for your rebuttal. However, my concerns about W2 Q2 Q3 are still not solved. Could you please add related experiments to answer my questions directly?

AuthorsOpenReview ↗

Response to Follow-up (W2/Q2) : 3B-Scale Feasibility and Detailed Training Throughput Analysis

We thank the reviewer for the follow-up and fully recognize the value of evaluating PHBA at larger scales and longer training contexts. We actively explored scaling PHBA to 3B parameters at the beginning of the rebuttal period. Our feasibility assessment yielded the following estimates:

Item3B PHBA Feasibility Assessment
Model size3B parameters
Training context8K tokens
Training budget100B tokens
Hardware32× NVIDIA H20 GPUs
Estimated training time~11.5 days
Estimated evaluation time~2 days
Total for PHBA alone~13.5 days
Fair comparisonPHBA + 5 matched-scale baselines

Crucially, a rigorous comparison would also require training and evaluating all five baselines at the corresponding 3B scale under matched conditions. Training PHBA alone at 3B and comparing it against existing 1.3B baselines would not constitute a fair architectural comparison. Completing six such large-scale experiments, let alone retraining them at longer contexts, is unfortunately infeasible within the rebuttal period.

As an academic research team focused on efficient architectures rather than foundation-model pretraining, we do not have the sustained GPU resources available to large industrial laboratories. We have already devoted substantial computational resources to training PHBA and five baselines independently from scratch, each at approximately 1.3B parameters and 100B training tokens, under matched data and training conditions. We sincerely hope the reviewer can appreciate the practical constraints of conducting additional large-scale, rigorously controlled experiments within this limited period.

Importantly, our original submission already provides more extensive experiments than many representative non-foundation-model efficient-architecture studies, as summarized in our previous response:

  • Longer training context: 8K, compared with the commonly adopted 2K–4K settings in GLA, DeltaNet, GSA, GDN, NHA, and related architectures.
  • Larger ablation scale: 760M/50B tokens, compared with the typical 340M/10–15B-token ablations.
  • More extensive long-context evaluation: RULER and NIAH at 4K, 8K, 16K, 32K, and 64K, extending to 8× the training context. These evaluations were already included in our initial submission.

Under this controlled 1.3B/100B setting, PHBA achieves the highest RULER averages at all five evaluated lengths (4K–64K) and the highest NIAH averages at 16K–64K among all six architectures (Table 1, Appendix Table 2, and Fig. 2). Evaluations beyond 8K explicitly measure zero-shot length extrapolation.

We fully agree that larger-scale and longer-context training would provide valuable further evidence. Nevertheless, we believe our extensive, consistently controlled experiments already establish meaningful evidence for PHBA's architectural effectiveness at the evaluated scale, while additional 3B-scale matched training is beyond the computational resources and time available during rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Consensus

Dear PC, SAC and AC,

We thank all reviewers for their constructive engagement during the rebuttal phase. For your convenience, we summarize the key points below. The new experiments and analyses will be incorporated into the revised paper or appendix.

We would like to briefly reiterate PHBA's motivation and core contribution. Existing same-layer hybrid attention architectures such as NHA restrict exact-token access to a fixed local window, preventing input-adaptive retrieval of relevant distant tokens. PHBA addresses this limitation by retrieving query-relevant historical token blocks and pairing each with its aligned gated prefix state, enabling input-adaptive joint modeling of distant exact-token information and its preceding historical context under a unified softmax. An efficient Triton implementation supports this architectural refinement, which achieves consistent improvements over relevant baselines.

The reviewers' feedback recognized several important strengths of the submission:

  • Architectural design: Reviewers kwYa and UtiQ recognized the combination of sparse retrieval and recurrent memory as a coherent and reasonable design. Reviewer Gjv5 highlighted that the improvements over MoBA and GSA support the effectiveness of jointly accessing exact tokens and recurrent memory.

  • GPU implementation: Reviewers AXBF, Gjv5, and kwYa recognized the systematic Triton implementation, efficient reuse of recurrent boundary states, and metadata-driven sparse execution. Reviewer Gjv5 specifically noted the small training overhead over MoBA.

  • Experimental effectiveness: Reviewers AXBF, Gjv5, and kwYa acknowledged PHBA's strong language modeling and retrieval results, including improvements over relevant baselines. Reviewer UtiQ also recognized improvements on several long-context benchmarks.

  • Evaluation rigor: Reviewers UtiQ and kwYa positively noted the relevant baseline comparisons, with kwYa highlighting the controlled from-scratch training on matched data and context lengths.

  • Presentation quality: Reviewers Gjv5 and kwYa rated the paper as clear and easy to follow.

Regarding the rebuttal outcome, we believe the discussion was overall constructive and positive: one reviewer explicitly expressed a positive attitude during rebuttal and reaffirmed a score of 3, two reviewers raised their scores to 3 after rebuttal, and the remaining reviewer thanked us for our response, confirmed having reviewed our rebuttal, and stated that the additional evidence would be considered in the final assessment.

  • Reviewer kwYa initially gave a score of 3, explicitly stated that most concerns had been addressed, and maintained a positive score, with no further concerns raised after rebuttal.

  • Reviewer Gjv5 explicitly confirmed that our additional experiments and clarifications addressed their concerns about evaluation scale, baseline configurations, inference efficiency, and routed prefix-state ablations, recognized that these results support PHBA's design choice, and raised the score to 3.

  • Reviewer AXBF initially gave a score of 2, recognizing the systematic GPU implementation and solid experimental results. During rebuttal, we completed all requested experiments and clarifications, including a matched 1.3B/100B NSA comparison, all three requested ablations, and 32K/64K inference benchmarks. The reviewer thanked us for our rebuttal response, confirmed having reviewed it, and stated that the additional evidence would be considered in the final assessment. In response to their request, we also provided a detailed commitment to incorporating these results and clarifications into the revised manuscript.

  • Reviewer UtiQ appreciated our extensive rebuttal efforts and additional experiments addressing scalability and practical efficiency, and raised the score to 3.

Overall, we believe the major concerns raised by the reviewers have been substantially addressed. We sincerely hope the AC will consider the rebuttal outcomes holistically in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.