- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper identifies an important issue in QK-norm, which is a industry standard practice.
- The paper then propose a new method called QK-MAP to address the issue of magnitude of vector-length being suppressed, but still serve the purpose of QK-norm, i.e. constraining the for logit growth for stable training.
- Experimental results show the approach achieve better quality across various scales of models.
Critical strengths
- The paper identifies an important issue of the magnitude info loss in industry standard practice that usually neglected in QK-norm.
- It interprets the magnitude in query as reverse temperature that controls the sharpness of attention allocation, and designed novel method to curb attention logit growth with learned projection of bounded sin/cos waves.
- It demonstrates the attention pattern changes: assign less weight to first tokens, and higher evidence SNR.
Critical weaknesses
- Why the method section introduced QUEST, but the experiment section does not compare with QUEST?
- With the learned projection in QK-MAP, it should increase the parameter count of the model, which is not cleared studied in the paper.
- The quality improvement seems to be minor in experiments in terms of AVG8 accuracy
Questions for authors
- Sin/cos frequency waves is one form of function to constrain the magnitude, wonder if the authors considered other function forms that have bounded value ranges to achieve similar goals.
- Figure 2 left: It’s not very clear to me why QK-RSMNorm maps the key norm to around 11.4.
- Figure 6(a): why QK-norm has one dominant key position (non-first-token) that absorbs most attention.
3 comments
Response to Reviewer mVKT (Part 2 / 2)
Q1: Alternative Bounded Functions
The sine/cosine pair is one realization of our broader design principle: controlling feature norms while preserving sensitivity to input magnitude.
We evaluated two non-periodic, fixed-norm mappings:
Sech/Tanh Map:
=[\operatorname{sech}(\theta),\ \tanh(\theta)].$$ **Rational Map**, with $t=\theta/2$: $$\phi_{\mathrm{R}}(\theta) =\left[\frac{1-t^2}{1+t^2},\ \frac{2t}{1+t^2}\right].$$ Both mappings satisfy $\|\phi(\theta)\|_2=1$ and remain sensitive to input magnitude changes without periodicity. | Scale | Mapping | Train CE ↓ | Val. CE ↓ | Avg-8 (%) ↑ | |:---:|:---|---:|---:|---:| | 190M | Sin/Cos | 3.67 | **3.93** | 41.82 | | | Sech/Tanh | **3.66** | 3.98 | 41.10 | | | **Rational** | **3.66** | 3.98 | **42.51** | | 370M | Sin/Cos | 3.10 | **3.45** | 45.94 | | | **Sech/Tanh** | 3.10 | **3.45** | **46.25** | | | Rational | **3.09** | **3.45** | 45.51 | Both non-periodic mappings reproduce the strong training and validation loss improvements of QK-MAP. Rational achieves the highest downstream Avg-8 at 190M (42.51%), while Sech/Tanh achieves the highest at 370M (46.25%). These results extend QK-MAP to a broader family of bounded, magnitude-sensitive query-key representations. We will include them in the revision. --- **Q2: Key Norm under QK-RMSNorm in Figure 2** RMSNorm normalizes the root-mean-square magnitude to approximately one. For the 128-dimensional keys in Figure 2, this gives an Euclidean norm of approximately $\sqrt{128}=11.31$, explaining the observed value around 11.4. We will clarify this in the figure caption. --- **Q3: Dominant Non-First-Key Attention Position in Figure 6(a)** We checked the attention matrix and found that the vertical band corresponds to key position 7, which receives 34.17% of the attention from subsequent queries under QK-Norm. QK-Norm removes key magnitude differences, making attention depend more on query-key directional similarity. In this example, the seventh key attracts attention from many later queries, forming a secondary attention sink. QK-MAP reduces this concentration, leading to a more balanced attention pattern. We will clarify this observation in the revision. --- Thanks again for your time and for these important points; your feedback has been a great help in strengthening our paper.Thank you for addressing the questions. I'll keep the rating.
Response to Reviewer mVKT (Part 1 / 2)
We sincerely thank the reviewer for the thoughtful feedback and for recognizing the importance of preserving magnitude information in attention. Below, we address each point with new comparisons, alternative bounded mappings, and further analyses.
W1: Direct Comparison with QUEST
QUEST [1] retains query magnitude while normalizing keys, thereby removing key-side magnitude information. QK-MAP instead constructs magnitude-sensitive representations for both Q and K through a shared mapping.
Our main pretraining experiments focused on QK-Norm, which is widely adopted in modern LLMs. QUEST primarily evaluates vision applications, with language-modeling experiments covering 44M–151M models. Our original manuscript also compared QUEST in key-side scaling (Section 2.2, Figure 2) and word-replacement experiments (Section 4.5, Table 3), where QK-MAP achieves 13.8%–16.8% lower perplexity than QK-Norm and QUEST at the 5% replacement rate.
We further conducted direct pretraining comparisons with QUEST at 190M and 370M under matched training settings and approximately matched parameter counts:
QK-MAP achieves lower training and validation CE than QUEST at both scales, together with higher downstream accuracy. At 190M and 370M, it improves Avg-8 over QUEST by 1.16 and 1.13 percentage points, respectively. We will incorporate these results into the revision.
W2: Parameter Count Matching
We compensate for the learned QK-MAP projection by slightly reducing the FFN width, keeping total parameter counts closely matched to Standard attention (Section 4.1; Appendix C.1, Tables 4–5).
The parameter difference remains below 0.033% across all five scales. These results demonstrate that QK-MAP's improvements are not simply attributable to additional parameters.
W3: Downstream Performance Improvements
We further evaluated QK-MAP with three independent matched training seeds at 190M.
QK-MAP achieves the highest Avg-8 in every individual seed, with a three-seed mean of 41.82%, compared with 40.10% for Standard, 40.84% for QK-Norm, and 40.66% for QUEST.
At 7B, QK-MAP further improves Avg-8 by 0.89 points over Standard and 1.27 points over QK-Norm (Table 1), ranking first across all 16 late-training checkpoints (Figure 4c). We will include the multi-seed statistics in the revision.
[1] Govindarajan et al. QUEST: A Robust Attention Formulation Using Query-Modulated Spherical Attention. ICLR 2026.