- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HiRouter, a sparse attention mechanism that uses learned hierarchical routing to retrieve candidate keys for attention. It jointly trains query/key representations and routing centroids with entropy-based objectives that encourage confident assignments and balanced bucket occupancy. A hierarchical beam search selects candidate buckets, and custom Triton kernels support efficient execution. Experiments evaluate language modeling, downstream tasks, long-context retrieval, and runtime against full attention and efficient sequence models.
Critical strengths
HiRouter jointly learns hierarchical routing and attention representations, directly addressing retrieval quality and bucket balance within a unified training objective. Its custom Triton implementation also demonstrates practical efficiency gains on long sequences.
Critical weaknesses
Q1. Could the authors report long-context Needle-in-a-Haystack results at the 1B-parameter scale, alongside relevant baselines, at least NSA, under matched training and evaluation settings? This would help establish whether the gains in Table 5 persist at a larger model scale.
Q2. Could the authors provide training scaling curves comparing HiRouter with full attention and, especially, NSA under matched model sizes and training settings?
Q3. Could the authors provide an apples-to-apples comparison of end-to-end training time and cost across different model sizes against full attention and NSA? Please report throughput and GPU-hours under matched hardware, context lengths, batch sizes, and training-token budgets, together with the cost of reaching comparable validation loss.
Questions for authors
See weaknesses.
2 comments
Official Comment by Reviewer U1WU
Thank you for addressing my concerns. I will maintain my score of 2.
Q1. Long-context NIAH at larger scale
Setting. We ran this evaluation on pre-trained Qwen3-4B and Qwen3-8B as some checkpoints of Tables 14–15 are no longer available and due to some computational limitations during the discussion period. All models share frozen weights and prompts, and we train only the router in HiRouter. Any difference comes from the keys that each query reads. NSA needs pretraining from scratch, so its comparison stays at 410M (Tables 5 and 30).
Accuracy is the mean ± SE over four RULER needle-retrieval tasks with 50 prompts each. In all sparse runs, each query also attends to its local segment, and the final 256 prompt rows use full attention. Qwen3-8B stops at 32K, because its native position range ends at 40,960 tokens.
On both backbones, HiRouter stays within one point of full attention up to 16K. The gap then grows to 3.2 and 3.6 points at 32K, and to 7.9 points at 64K on the 4B backbone. In return, prefill is 1.3 times faster at 32K on the 8B backbone and 2.2 times faster at 64K on the 4B backbone. Moreover, HiRouter scores 4.3 points above MoBA at 32K on the 4B backbone, whereas the 2.3-point difference on the 8B backbone is within noise.
Q2. Training scaling curves
Matched training setup. We compare at 50M, 125M, and 360M scales on WikiText-103. Only the attention mechanism differs. We report mean ± standard deviation of test perplexity over two seeds.
HiRouter consistently outperforms NSA and only trails full attention at 50M parameters. We further check 125M training trajectory and see that HiRouter reaches lower perplexity earlier. Its advantage is most pronounced at 33M tokens and gradually narrows as the other methods converge.
Overall, the matched experiments show faster early progress per training token at 125M, followed by comparable final performance to full attention and NSA at the two larger scales. We have added the scaling curves into the revised paper (Figure 5).
Q3. End-to-end training time and cost
Matched training cost. Complete training-step throughput was measured on one H100 using
bf16, 2,048-token sequences, 8 sequences per micro-batch, with gradient accumulation of 32. HiRouter's training path uses fused GPU kernels while preserving the model architecture and objective. GPU-hours for longer runs are projected from the measured throughput.HiRouter matches NSA throughput at 50M and is 19% and 16% faster at 125M and 360M, respectively. Full attention remains fastest at 2K, where sparse routing overhead offsets its attention-computation savings.
Cost to comparable perplexity. The 125M convergence results in Q2 provide a checkpoint-based comparison. Using a perplexity threshold of 25, full attention reaches this at 328M tokens, while both NSA and HiRouter do so at 197M tokens. Combined with measured throughput, these correspond to projected costs of 0.44, 0.39, and 0.33 GPU-hours, respectively.
Thus, HiRouter shows a lower estimated cost of training, despite lower throughput than full attention. The estimates use test-perplexity checkpoints, with the exact crossing times and corresponding validation-loss costs requiring finer-grained training records.