acceptodds
Under review as a conference paper at ICLR 2027

Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head

136 views

Abstract

Scaling attention parameters can improve language model quality, but retaining full token histories makes additional heads costly at long contexts. Furthermore, since attention retrieves and combines contextual information, parameter scaling should also support longer contexts. We therefore ask whether attention parameter scaling can directly enable efficient and effective context scaling. We introduce NAMOH, an architecture-native sparse attention mechanism that activates of heads per token. Each head retains only its assigned tokens and performs causal attention within this subsequence. Head selection thus jointly determines active parameters and available context without scanning the full history. Under balanced assignments, increasing at fixed shortens head histories and reduces per-token key-value (KV) access without increasing total KV storage. We further support head-relative rotary position embeddings to shorten positional spans within routed subsequences, aiming to mitigate position-induced attention noise. Experiments show that NAMOH can outperform fully activated models with the same total parameters, while enabling more efficient long-context inference than smaller dense models with matched active parameter counts. It remains compatible with GQA and existing sparse attention mechanisms. We hope this work offers a new path for scaling attention, with parameter scaling directly enabling context scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention scaling, sparse attention, mixture-of-head

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Hu9ZOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors examine the problem of scaling parameters in the classical transformer mechanism. Usually the researchers examine the scalability of the FFN part due to amount of its parameters. However, the authors focuses on the scaling attention part of the model.

The authors discuss the problems in the given field and propose their own mechanism of attention parameters aggregation. They propose the upgrade of the classical Mixture of Heads (MoH) and Sparse Attention mechanism (SA) and call it NAMOH.

The general idea of the approach is a mix of the several ideas like: token-to-end routing (Jin et al., 2024), sparsification, routing balance, etc. As a result they provide a model that reduces number of KV parameters from HTHT (storage) and HTHT (activation) in MHA to KTKT (storage) and approximately KT2/HKT^2/H.

The authors provide diagnostics of the NAMOH for different sets of parameters and cases (benchmark comparison, LongBench quality, Needle in haystack quality, etc.). They train from scratch several models, based on the aggregation principle (MoH, GQA, SA, etc.), size (from 0.6B to 1.2B) and other parameters (SA fractions, NAMOH K parameter values, etc.). As a result they discovered:

  1. Quality under a reduced KV budget. Models are trained from scratch and evaluated on 10 benchmarks. With 1/2–1/4 of the MHA KV cache and 1/4–1/16 of the KV read per token, NAMOH stays on par with MHA‑32 (average 41.13 and 40.47 vs. 39.73). At the same budget it clearly beats GQA (+1.7 and +2.3), MHA with fewer heads, and sparse attention. The gap is about +5 against SA‑32, which has the same total parameters and the same KV read per token. At the 1/8 budget the advantage largely disappears.

  2. Long context and efficiency. On LongBench at 32k context, NAMOH‑32A8 scores 32.73 vs. 32.14 for MHA‑32 and 30.90 for GQA with the same KV size, while storing only 1/4 of the KV cache. At 256k context, time per output token is about 5× lower than MHA‑32 and about 3× lower than GQA with the same KV size. Time to first token is also lower at all context lengths.

  3. Ablations. Head‑relative RoPE, which numbers positions within each head, beats global RoPE on LongBench, and the gap grows with context length: +1.0, +1.5 and +1.7 at 8k, 16k and 32k. The best result in the paper (42.93) comes from Switch‑style load balancing plus 4 shared sliding‑window heads, but this uses extra KV budget. How often each head is used depends on the task, which points to head specialization.

Overall, the authors provide a solid paper with very applicable method.

Critical strengths
  1. Simple and well-examined idea. The authors provide the step-by-step solution based on natural extension of MoH ideas. Unlike mixture-of-heads they provide the both memory and compute saving ideas. Also the cost analysis shows that the NAMOH is frequently theoretically better than alternatives.

  2. Well observed experiments. The section contains consecutive examination of the major questions: context dependence, benchmark qualities, dependence on the positional encoding, etc. All of the main experiments show the better quality (and moreover lower storage size) of the NAMOH method compared with GQA, MHA and SA.

Critical weaknesses

Despite the quality of the paper there are at least two weaknesses:

  1. The quality of the experiments is strange. Many of the baselines remain close to noise. F.e. MMLU and GSMK8 are close to the zero or random answer. Some of the baselines remain the same with the increase of the parameters GQA -KV 4/8. It would be nice to add several seeds and CI for the given analysis.

  2. Despite there is a comparison with classical SOTA methods, there are no additional examination of the close methods. F.e. MoSA (P.Piekos et al., 2025) or SwitchHead (R.Csordas et al.,2023). I understand that those methods sometimes aren't comparable with yours, but I think it is important to mention them because they are theoretically closer to the idea you proposed.

Questions for authors
  1. Have you considered closest baselines (mentioned before f.e.)? If so, can you provide the analysis of comparison.

  2. Despite you provided the analysis of the head-usage. It would be nice to theoretically examine what will be if your balancing doesn't work. As far as I understood in the worst case scenario it shouldn't be better than MoH in some cases, but will require proper training procedure.

  3. Based on the Figure 4, the main attention mass goes to the end of the context. Can you give a quantitative measure, such as attention mass on the needle vs. the distractors, or accuracy on NIAH or RULER?

  4. See Weaknesses 1.

Reviewer_GLCDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NAMOH, where each token selects a few attention heads, and each head only stores and attends to the tokens assigned to it. This connects head selection with context sparsity, reducing KV access under balanced routing. The authors also use positions within each head’s token sequence for RoPE. Experiments show competitive model quality and faster long-context inference compared with dense and sparse attention baselines.

Critical strengths
  1. Extending the MoE idea to jointly route attention heads and their KV histories is interesting and insightful.
  2. The downstream results also look promising, showing competitive or better task performance alongside improved long-context inference efficiency.
Critical weaknesses
  1. The experiments mostly fix the total number of heads at 32 and vary the active head count. A comparison that fixes KK and increases HH would better support the claim that adding attention parameters improves context scaling.
  2. A query cannot directly access a historical token within a layer unless they share an active head. This introduces a causal constraint absent from token-wise MLP expert routing: historical tokens must be assigned to heads before future queries are known, yet those queries must select overlapping heads to retrieve them directly. The paper does not measure how often relevant information is excluded by routing. Long-context quality is also only evaluated up to 32K, while the speed tests extend to 256K.
Questions for authors

See weaknesses.

Reviewer_wRAHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NAMOH, a native sparse attention architecture that activates different attention heads for each query. Routing determines both the active projections and the head-specific KV histories available for attention. Under balanced routing, increasing the total head count at a fixed active head count reduces KV access without increasing total KV storage. The method also introduces head-relative RoPE to shorten positional spans within routed subsequences.

Critical strengths
  1. The idea is interesting. The method reduces active attention heads and KV cache by design.
  2. The experiments include models trained from scratch under the same token budget, with comparisons against MHA, GQA, and block-selection sparse attention.
Critical weaknesses
  1. The scaling claim is not validated. The main experiments fix the total head count at 32 and vary the active head count. Experiments that increase the total head count while keeping the active head count fixed are needed to demonstrate that parameter scaling improves the quality-efficiency trade-off.
  2. The long-context evaluation is limited to LongBench tasks up to 32K, while latency measurements extend to 256K. Harder benchmarks such as RULER are missing.
  3. The MHA, GQA, and SA baselines are modified with head gating and load balancing. These controls might help isolate routed context sparsity, but unmodified baselines are also needed to show improvements over the standard architectures.
Questions for authors

please see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.