- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors examine the problem of scaling parameters in the classical transformer mechanism. Usually the researchers examine the scalability of the FFN part due to amount of its parameters. However, the authors focuses on the scaling attention part of the model.
The authors discuss the problems in the given field and propose their own mechanism of attention parameters aggregation. They propose the upgrade of the classical Mixture of Heads (MoH) and Sparse Attention mechanism (SA) and call it NAMOH.
The general idea of the approach is a mix of the several ideas like: token-to-end routing (Jin et al., 2024), sparsification, routing balance, etc. As a result they provide a model that reduces number of KV parameters from (storage) and (activation) in MHA to (storage) and approximately .
The authors provide diagnostics of the NAMOH for different sets of parameters and cases (benchmark comparison, LongBench quality, Needle in haystack quality, etc.). They train from scratch several models, based on the aggregation principle (MoH, GQA, SA, etc.), size (from 0.6B to 1.2B) and other parameters (SA fractions, NAMOH K parameter values, etc.). As a result they discovered:
-
Quality under a reduced KV budget. Models are trained from scratch and evaluated on 10 benchmarks. With 1/2–1/4 of the MHA KV cache and 1/4–1/16 of the KV read per token, NAMOH stays on par with MHA‑32 (average 41.13 and 40.47 vs. 39.73). At the same budget it clearly beats GQA (+1.7 and +2.3), MHA with fewer heads, and sparse attention. The gap is about +5 against SA‑32, which has the same total parameters and the same KV read per token. At the 1/8 budget the advantage largely disappears.
-
Long context and efficiency. On LongBench at 32k context, NAMOH‑32A8 scores 32.73 vs. 32.14 for MHA‑32 and 30.90 for GQA with the same KV size, while storing only 1/4 of the KV cache. At 256k context, time per output token is about 5× lower than MHA‑32 and about 3× lower than GQA with the same KV size. Time to first token is also lower at all context lengths.
-
Ablations. Head‑relative RoPE, which numbers positions within each head, beats global RoPE on LongBench, and the gap grows with context length: +1.0, +1.5 and +1.7 at 8k, 16k and 32k. The best result in the paper (42.93) comes from Switch‑style load balancing plus 4 shared sliding‑window heads, but this uses extra KV budget. How often each head is used depends on the task, which points to head specialization.
Overall, the authors provide a solid paper with very applicable method.
Critical strengths
-
Simple and well-examined idea. The authors provide the step-by-step solution based on natural extension of MoH ideas. Unlike mixture-of-heads they provide the both memory and compute saving ideas. Also the cost analysis shows that the NAMOH is frequently theoretically better than alternatives.
-
Well observed experiments. The section contains consecutive examination of the major questions: context dependence, benchmark qualities, dependence on the positional encoding, etc. All of the main experiments show the better quality (and moreover lower storage size) of the NAMOH method compared with GQA, MHA and SA.
Critical weaknesses
Despite the quality of the paper there are at least two weaknesses:
-
The quality of the experiments is strange. Many of the baselines remain close to noise. F.e. MMLU and GSMK8 are close to the zero or random answer. Some of the baselines remain the same with the increase of the parameters GQA -KV 4/8. It would be nice to add several seeds and CI for the given analysis.
-
Despite there is a comparison with classical SOTA methods, there are no additional examination of the close methods. F.e. MoSA (P.Piekos et al., 2025) or SwitchHead (R.Csordas et al.,2023). I understand that those methods sometimes aren't comparable with yours, but I think it is important to mention them because they are theoretically closer to the idea you proposed.
Questions for authors
-
Have you considered closest baselines (mentioned before f.e.)? If so, can you provide the analysis of comparison.
-
Despite you provided the analysis of the head-usage. It would be nice to theoretically examine what will be if your balancing doesn't work. As far as I understood in the worst case scenario it shouldn't be better than MoH in some cases, but will require proper training procedure.
-
Based on the Figure 4, the main attention mass goes to the end of the context. Can you give a quantitative measure, such as attention mass on the needle vs. the distractors, or accuracy on NIAH or RULER?
-
See Weaknesses 1.