- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes Prefix-State Hybrid Block Attention (PHBA), which stores recurrent key/value states at block boundaries and retrieves these prefix snapshots alongside content-selected historical token blocks. Exact tokens and compressed states participate in a shared softmax. A Triton implementation combines chunkwise state construction with sparse execution that avoids materializing gathered payload tensors. Models trained from scratch at 760M and 1.3B parameters show competitive language modeling and improvements on several retrieval evaluations. The paper also reports LongBench results with inputs truncated to 8K and end-to-end training throughput.
Critical strengths
-
Pairing retrieved token blocks with snapshots of their preceding context is a coherent extension of sparse retrieval and recurrent memory. Routing, state boundaries, and joint readout are organized around this idea.
-
The comparisons use models trained from scratch on the same data and context length. At 1.3B, PHBA improves WikiText perplexity and the reported RULER averages over the compared models while remaining competitive on commonsense reasoning. The implementation and end-to-end throughput measurements also support practical feasibility.
Critical weaknesses
- PHBA already reads the current block's prefix state, which summarizes all preceding blocks. Its distinctive additional choice is to expose earlier prefix snapshots alongside selected historical token blocks. The comparison with MoBA also introduces recurrent memory, while the comparison with NHA also changes exact-token access from local to content-routed. These comparisons support the complete architecture but do not resolve whether historical snapshots improve upon sparse retrieval augmented with only the current prefix state. The block-size, slot-count, and gate ablations do not isolate this distinction.
Questions for authors
-
How does full PHBA compare with a variant retaining the same token routing, gate construction, and current-block prefix state, but omitting readout of routed historical prefix states?
-
The main quality configuration uses C = M = 128 and K = 8 at 8K, whereas Figure 3 evaluates fixed K = 4 or fixed C = 256. How does the reported throughput relate to the configuration producing the main quality results? If it is not already measured, one representative throughput and peak-memory measurement for that configuration, with the benchmark model and batch size specified, would clarify the cost of the demonstrated gains.
6 comments
Response to Q2 : Throughput and peak memory under the main quality configuration
We thank the reviewer for pointing out the difference between the main quality configuration and Fig. 3. We therefore conducted additional measurements covering the exact configuration used for our main quality results.
Experimental protocol. All models are approximately 1.3B parameters and run on a single NVIDIA H20 GPU, without activation checkpointing (AC). Each step processes 16,384 tokens (2K×8, 4K×4, 8K×2, or 16K×1). Following Fig. 3, timing covers complete training computation, including forward, backward, and parameter updates, excluding input preparation and data loading. We use 20 warm-up steps and 50 measured steps, with three independent runs. Throughput is averaged across runs; peak allocated/reserved memory is averaged over the three per-run peaks. All triplets below are PHBA/MoBA/Transformer.
Main quality configuration: C=M=128, K=8, 8K×2 (batch size 2).
| Model | Throughput (tok/s) ↑ | Peak allocated (GiB) ↓ | Peak reserved (GiB) ↓ |
|---|---|---|---|
| PHBA | 9,858 | 62.89 | 65.56 |
| MoBA | 9,510 | 60.63 | 63.60 |
| Transformer | 10,367 | 44.05 | 47.07 |
Under the exact main-quality configuration, PHBA achieves 3.7% higher throughput than MoBA, despite using 2.26 GiB (3.7%) more peak allocated memory and 1.96 GiB (3.1%) more reserved memory. Its throughput is within 4.9% of Transformer. Despite introducing additional prefix states and the corresponding attention computations, PHBA maintains comparable and slightly higher training throughput than MoBA. We attribute this low overhead to our carefully optimized Triton operators, including chunkwise prefix-state construction, metadata-driven token/state access, online-softmax merging, and block-major backward, without materializing sparse gather tensors.
Additional parameter sweeps. We evaluate three strategies: (i) fixed C=128, K=8 without fixing the sparsity ratio; (ii) fixed K=8, increasing C to maintain a 1/8 exact-token budget; and (iii) fixed C=128, increasing K to maintain the same budget. Bold 8K×2 rows correspond to the main-quality configuration. C/K denotes the block size/Top-K for PHBA and MoBA; Transformer uses full attention.
| Strategy | Seq×Batch; C/K | Throughput (tok/s) | Allocated (GiB) | Reserved (GiB) |
|---|---|---|---|---|
| Fixed C/K | 2K×8; 128/8 | 11042/9983/13784 | 62.89/57.95/44.05 | 66.47/61.18/47.10 |
| 4K×4; 128/8 | 10440/9682/12454 | 62.90/59.74/44.05 | 67.39/62.13/47.11 | |
| 8K×2; 128/8 | 9858/9510/10367 | 62.89/60.63/44.05 | 65.56/63.60/47.07 | |
| 16K×1; 128/8 | 9280/9468/7858 | 62.90/61.08/44.05 | 67.39/64.52/47.03 | |
| Fixed K=8 | 2K×8; 32/8 | 10288.7/7750.4/13820 | 73.02/60.96/44.05 | 75.25/71.47/47.03 |
| 4K×4; 64/8 | 10367/8908/12417 | 66.27/60.74/44.05 | 69.89/64.97/47.03 | |
| 8K×2; 128/8 | 9861/9518/10364 | 62.88/60.63/44.05 | 64.64/63.27/47.04 | |
| 16K×1; 256/8 | 8709/9242/7824 | 61.21/60.57/44.05 | 65.24/63.17/47.11 | |
| Fixed C=128 | 2K×8; 128/2 | 12814/11935/13819 | 62.31/49.86/44.05 | 64.43/53.65/47.11 |
| 4K×4; 128/4 | 11678/11020/12440 | 62.52/53.25/44.05 | 64.62/57.92/47.08 | |
| 8K×2; 128/8 | 9831/9496/10372 | 62.89/60.63/44.05 | 65.66/63.65/47.06 | |
| 16K×1; 128/16 | 7269/7481/7835 | 63.63/75.34/44.05 | 66.84/79.46/47.07 |
The three independent parameter sweeps consistently confirm the main-configuration result: PHBA achieves 9.831–9.861K tokens/s, compared with 9.496–9.518K for MoBA at 8K×2. These measurements demonstrate that our efficient Triton operator design keeps PHBA's additional recurrent-state computation affordable, maintaining comparable and slightly higher throughput than MoBA at the cost of additional memory. They also directly quantify the computational trade-offs of the configuration producing our main quality improvements.
Thank you for the response. Most of my concerns have been addressed, and I will maintain my positive score.
Response to W1/Q1 : Historical prefix-state ablation
We sincerely thank the reviewer for recognizing PHBA's block-aligned design, matched comparisons and practical feasibility. We agree that MoBA/NHA comparisons do not isolate historical snapshots. We have therefore trained the precise W1/Q1 control, plus an extra ablation.
Setup. Full PHBA and A/B were trained independently on the same FineWeb-Edu data with matched optimization: 760M/50B, 8K context, C=M=128, K=8. A (current-state-only) retains Full's routing, gates, recurrent updates, current prefix and exact routed tokens, removing only historical prefix readout. Additional B (routed-states-only) retains current/historical prefix states but omits exact routed tokens. Values: Full/A/B; 16K–64K are zero-shot extrapolations.
Table 1. LM/commonsense (PPL↓; others↑).
Table 2. RULER/NIAH (4K/8K).
Table 3. RULER/NIAH (16K–64K).
Table 4. LongBench (11 tasks).
Analysis and response to W1/Q1. (1) Direct evidence: Full exceeds A on WikiText, LAMBADA, commonsense, all RULER/NIAH averages, and LongBench (e.g., RULER-8K 11.60 vs. 7.06; NIAH-4K 61.50 vs. 49.11). A retains the same routed tokens, gates and current prefix; only historical snapshot readout differs. This isolates the reviewer's precise concern without confounding MoBA/NHA architectural differences.
(2) Why historical snapshots help: The fixed-capacity current prefix summarizes all past tokens; later recurrent updates may dilute or overwrite earlier details. By contrast, each historical snapshot captures a block's preceding context before those later updates, potentially offering richer, temporally aligned information for retrieved exact tokens. This helps explain why historical states add value even when the current state already summarizes the full preceding history.
(3) Complementary evidence: B retains historical snapshots but discards the exact tokens whose content determined routing. Its lower RULER/NIAH and LongBench averages, especially at long lengths, show that preceding context alone cannot substitute for the selected block's contents. Together, these controlled ablations support PHBA's joint readout of retrieved exact tokens and block-aligned historical context. We appreciate this constructive suggestion.