acceptodds
Under review as a conference paper at ICLR 2027

Large Language Models without QK-Normalization

Abstract

Query-Key Normalization (QK-Norm) is widely adopted in modern large language models to control the growth of attention logits and improve training stability. However, query and key norms are not merely sources of instability. The query norm controls the sharpness of the attention distribution, while the key norm determines how strongly a token attracts different queries. These norms therefore carry token-specific information about attention selectivity and token importance. By normalizing both, QK-Norm suppresses this information. This raises a natural question: can we control query-key representations without discarding their sensitivity to magnitude? To address this question, we propose QK-MAP, a Magnitude-Aware Projection for constructing queries and keys. Rather than removing magnitude through normalization, QK-MAP transforms the input representation using multiple learnable frequencies, so that changes in magnitude are encoded as bounded phase patterns. This allows magnitude to be represented through phase and direction rather than vector length alone. Its periodic features have a fixed norm while remaining sensitive to magnitude changes, separating magnitude encoding from norm growth. Multiple frequencies further help distinguish magnitudes and reduce periodic ambiguity. Experiments across model scales from 190M to 7B parameters show that QK-MAP achieves lower training and validation loss than both Standard attention and QK-Norm. Further analyses show that QK-MAP preserves sensitivity to input scale, assigns less attention to the first token, and produces stronger attention contrast for relevant evidence in synthetic long-context tests. These results demonstrate that query-key scale can be effectively controlled without sacrificing token-specific magnitude information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Normalization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mVKTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper identifies an important issue in QK-norm, which is a industry standard practice.
  2. The paper then propose a new method called QK-MAP to address the issue of magnitude of vector-length being suppressed, but still serve the purpose of QK-norm, i.e. constraining the for logit growth for stable training.
  3. Experimental results show the approach achieve better quality across various scales of models.
Critical strengths
  1. The paper identifies an important issue of the magnitude info loss in industry standard practice that usually neglected in QK-norm.
  2. It interprets the magnitude in query as reverse temperature that controls the sharpness of attention allocation, and designed novel method to curb attention logit growth with learned projection of bounded sin/cos waves.
  3. It demonstrates the attention pattern changes: assign less weight to first tokens, and higher evidence SNR.
Critical weaknesses
  1. Why the method section introduced QUEST, but the experiment section does not compare with QUEST?
  2. With the learned projection in QK-MAP, it should increase the parameter count of the model, which is not cleared studied in the paper.
  3. The quality improvement seems to be minor in experiments in terms of AVG8 accuracy
Questions for authors
  1. Sin/cos frequency waves is one form of function to constrain the magnitude, wonder if the authors considered other function forms that have bounded value ranges to achieve similar goals.
  2. Figure 2 left: It’s not very clear to me why QK-RSMNorm maps the key norm to around 11.4.
  3. Figure 6(a): why QK-norm has one dominant key position (non-first-token) that absorbs most attention.
3 comments
AuthorsOpenReview ↗

Response to Reviewer mVKT (Part 1 / 2)

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the importance of preserving magnitude information in attention. Below, we address each point with new comparisons, alternative bounded mappings, and further analyses.


W1: Direct Comparison with QUEST

QUEST [1] retains query magnitude while normalizing keys, thereby removing key-side magnitude information. QK-MAP instead constructs magnitude-sensitive representations for both Q and K through a shared mapping.

Our main pretraining experiments focused on QK-Norm, which is widely adopted in modern LLMs. QUEST primarily evaluates vision applications, with language-modeling experiments covering 44M–151M models. Our original manuscript also compared QUEST in key-side scaling (Section 2.2, Figure 2) and word-replacement experiments (Section 4.5, Table 3), where QK-MAP achieves 13.8%–16.8% lower perplexity than QK-Norm and QUEST at the 5% replacement rate.

We further conducted direct pretraining comparisons with QUEST at 190M and 370M under matched training settings and approximately matched parameter counts:

ScaleVariantTrain CE ↓Val. CE ↓Avg-8 (%) ↑
190MStandard3.854.1440.10
QK-Norm3.764.0540.84
QUEST3.784.0640.66
QK-MAP (Sin/Cos)3.673.9341.82
370MStandard3.193.5445.35
QK-Norm3.173.5245.91
QUEST3.143.5044.81
QK-MAP (Sin/Cos)3.103.4545.94

QK-MAP achieves lower training and validation CE than QUEST at both scales, together with higher downstream accuracy. At 190M and 370M, it improves Avg-8 over QUEST by 1.16 and 1.13 percentage points, respectively. We will incorporate these results into the revision.


W2: Parameter Count Matching

We compensate for the learned QK-MAP projection by slightly reducing the FFN width, keeping total parameter counts closely matched to Standard attention (Section 4.1; Appendix C.1, Tables 4–5).

ScaleStandard ParametersQK-MAP ParametersDifference
190M267,406,080267,419,904+0.0052%
370M473,990,144474,145,792+0.0328%
760M912,311,808912,348,672+0.0040%
1B1,619,077,1201,619,279,872+0.0125%
7B7,298,355,2007,299,600,384+0.0171%

The parameter difference remains below 0.033% across all five scales. These results demonstrate that QK-MAP's improvements are not simply attributable to additional parameters.


W3: Downstream Performance Improvements

We further evaluated QK-MAP with three independent matched training seeds at 190M.

QK-MAP achieves the highest Avg-8 in every individual seed, with a three-seed mean of 41.82%, compared with 40.10% for Standard, 40.84% for QK-Norm, and 40.66% for QUEST.

At 7B, QK-MAP further improves Avg-8 by 0.89 points over Standard and 1.27 points over QK-Norm (Table 1), ranking first across all 16 late-training checkpoints (Figure 4c). We will include the multi-seed statistics in the revision.

[1] Govindarajan et al. QUEST: A Robust Attention Formulation Using Query-Modulated Spherical Attention. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer mVKT (Part 2 / 2)

Q1: Alternative Bounded Functions

The sine/cosine pair is one realization of our broader design principle: controlling feature norms while preserving sensitivity to input magnitude.

We evaluated two non-periodic, fixed-norm mappings:

Sech/Tanh Map:

=[\operatorname{sech}(\theta),\ \tanh(\theta)].$$ **Rational Map**, with $t=\theta/2$: $$\phi_{\mathrm{R}}(\theta) =\left[\frac{1-t^2}{1+t^2},\ \frac{2t}{1+t^2}\right].$$ Both mappings satisfy $\|\phi(\theta)\|_2=1$ and remain sensitive to input magnitude changes without periodicity. | Scale | Mapping | Train CE ↓ | Val. CE ↓ | Avg-8 (%) ↑ | |:---:|:---|---:|---:|---:| | 190M | Sin/Cos | 3.67 | **3.93** | 41.82 | | | Sech/Tanh | **3.66** | 3.98 | 41.10 | | | **Rational** | **3.66** | 3.98 | **42.51** | | 370M | Sin/Cos | 3.10 | **3.45** | 45.94 | | | **Sech/Tanh** | 3.10 | **3.45** | **46.25** | | | Rational | **3.09** | **3.45** | 45.51 | Both non-periodic mappings reproduce the strong training and validation loss improvements of QK-MAP. Rational achieves the highest downstream Avg-8 at 190M (42.51%), while Sech/Tanh achieves the highest at 370M (46.25%). These results extend QK-MAP to a broader family of bounded, magnitude-sensitive query-key representations. We will include them in the revision. --- **Q2: Key Norm under QK-RMSNorm in Figure 2** RMSNorm normalizes the root-mean-square magnitude to approximately one. For the 128-dimensional keys in Figure 2, this gives an Euclidean norm of approximately $\sqrt{128}=11.31$, explaining the observed value around 11.4. We will clarify this in the figure caption. --- **Q3: Dominant Non-First-Key Attention Position in Figure 6(a)** We checked the attention matrix and found that the vertical band corresponds to key position 7, which receives 34.17% of the attention from subsequent queries under QK-Norm. QK-Norm removes key magnitude differences, making attention depend more on query-key directional similarity. In this example, the seventh key attracts attention from many later queries, forming a secondary attention sink. QK-MAP reduces this concentration, leading to a more balanced attention pattern. We will clarify this observation in the revision. --- Thanks again for your time and for these important points; your feedback has been a great help in strengthening our paper.

Thank you for addressing the questions. I'll keep the rating.

Reviewer_4FHTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Summary This paper proposes QK-MAP, an alternative to query–key normalization that aims to control attention scale while retaining sensitivity to token magnitude. It applies a shared learnable sine–cosine feature map before separate query and key projections, leaving the value path unchanged; the default experimental implementation also includes a learned per-dimension residual. Experiments cover five model scales labeled 190M–7B, with lower training and validation cross-entropy at every evaluated scale, although downstream improvements are less consistent.

Critical strengths
  1. Clear motivation and interpretable design. The paper identifies a meaningful tension: QK normalization controls norm growth but suppresses token-specific magnitude information. The proposed periodic representation directly addresses this issue by maintaining a fixed feature norm while remaining sensitive to changes in input magnitude. The connection between the motivation and the mathematical construction is clear.

  2. Well-controlled pretraining comparisons across scales. The authors train separate attention variants under matched data, initialization seeds, data order, optimization settings, and token budgets. They also reduce the QK-MAP feed-forward width to approximately match total parameter counts. These controls strengthen the comparison, while the consistently lower language-modeling loss provides useful evidence of effectiveness across the tested scales.

  3. Informative mechanistic analyses. The norm-shuffling intervention supports the conclusion that QK-MAP uses token-specific projected key magnitudes. Attention diagnostics additionally show reduced first-token attention and stronger evidence-to-distractor attention contrast, providing insight beyond aggregate benchmark scores.

Critical weaknesses
  1. The training budgets limit conclusions about longer-run scaling. The largest model is trained on only 20B tokens. Consequently, the results do not establish whether the advantage persists under substantially longer pretraining or trillion-token budgets. Extending training at one representative scale would help distinguish an early optimization advantage from a persistent improvement in final performance.

  2. The causal explanation for the gains is not fully isolated. Although total parameter counts are approximately matched, QK-MAP changes the nonlinear representation, the depth of the query–key computation, and the allocation of parameters between attention and the FFN. The norm intervention demonstrates dependence on projected Q/K magnitudes, but does not establish that magnitude preservation causes the downstream improvement; the authors explicitly acknowledge the intervention's narrower scope. Moreover, this probe uses the 190M direct-map variant rather than the default residual implementation. A parameter- and compute-matched non-periodic mapping baseline, together with a carefully controlled magnitude-insensitive variant that specifies where magnitude information is removed and controls residual bypasses, would help disentangle these explanations. Such experiments would clarify the contribution of magnitude sensitivity rather than conclusively prove or disprove it in isolation.

  3. Downstream gains are modest and heterogeneous. The default QK-MAP variant underperforms QK-Norm on Avg-8 at 190M and Standard attention at 760M. At 7B, its gains are 0.89 percentage points over Standard and 1.27 points over QK-Norm. The consistent ranking across 16 nearby 7B checkpoints is reassuring, but the average margin over Standard is only 0.45 points. Confidence intervals for key downstream comparisons and independent-seed runs at a smaller scale would better establish the reliability and practical significance of these gains.

  4. The practical efficiency trade-off remains insufficiently quantified. QK-MAP introduces a periodic projection and sine–cosine operations, while reducing FFN width to compensate for the added parameters. Matching parameter counts does not establish comparable computational cost or wall-clock efficiency. Training throughput, inference latency, peak memory, etc. would still increase.

Questions for authors

none

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4FHT (Part 1 / 2)

We sincerely thank the reviewer for the thoughtful feedback and for recognizing our motivation, controlled cross-scale experiments, and mechanistic analyses. We have conducted additional experiments to further strengthen the evidence for QK-MAP, including controlled mapping comparisons, multi-seed evaluations, and practical training efficiency measurements.


W1: Longer-run Persistence

Our 7B experiments demonstrate consistent improvements in training and validation loss over 20B tokens, with QK-MAP achieving the highest downstream Avg-8 across all 16 late-training checkpoints (Figure 4). To directly test whether this advantage persists over a longer horizon, we are extending the 7B training from 20B to 30B tokens and will include the extended trajectories and evaluations in the revision.


W2: Understanding the Source of Improvement

We conducted additional controlled experiments to disentangle the effect of the bounded magnitude-sensitive mapping from changes in projection depth, parameter allocation, and periodicity.

(a) Controlled mapping comparisons.

First, to isolate whether the gains simply come from adding an extra projection layer or reallocating parameters between attention and the FFN, we introduced a Linear Map baseline. It replaces the periodic feature map with a learned full-width linear projection while keeping the same per-dimension residual structure and reduced FFN width.

Second, to test whether our design principle depends specifically on sinusoidal features, we evaluated two non-periodic, fixed-norm mappings under the same architecture:

Sech/Tanh Map:

=[\operatorname{sech}(\theta),\ \tanh(\theta)].$$ **Rational Map**, with $t=\theta/2$: $$\phi_{\mathrm{R}}(\theta) =\left[\frac{1-t^2}{1+t^2},\ \frac{2t}{1+t^2}\right].$$ Both mappings are strictly non-periodic and satisfy $\Vert\phi(\theta)\Vert_2=1$ while retaining sensitivity to input magnitude changes. | Variant (190M) | Train CE ↓ | Val. CE ↓ | Downstream Avg-8 (%) ↑ | |:---|---:|---:|---:| | Standard | 3.85 | 4.14 | 40.10 | | QK-Norm | 3.76 | 4.05 | 40.84 | | Linear Map | 3.80 | 4.13 | 39.65 | | **QK-MAP (Sin/Cos)** | 3.67 | **3.93** | 41.82 | | **QK-MAP (Sech/Tanh)** | **3.66** | 3.98 | 41.10 | | **QK-MAP (Rational)** | **3.66** | 3.98 | **42.51** | All three bounded nonlinear mappings achieve substantially lower training and validation CE than Standard, QK-Norm, and Linear Map. In contrast, adding a parameter-matched linear projection (Linear Map) fails to reproduce these gains (4.13 Val. CE and 39.65% Avg-8), confirming that the improvement is not merely an artifact of extra Q/K depth or parameter reallocation. Furthermore, the strong performance of both Sech/Tanh and Rational (which achieves 42.51% Avg-8) shows that the benefits extend beyond sinusoidal features, directly supporting our core principle of encoding input magnitude into bounded, fixed-norm representations. **(b) Role of the residual bypass and utilization of magnitude information.** Regarding the norm-shuffling intervention (Figure 5; Appendix G.1, Table 10): | Variant (190M) | Q-norm shuffle ΔCE | K-norm shuffle ΔCE | QK-norm shuffle ΔCE | |:---|---:|---:|---:| | Standard | +0.01651 | +0.18816 | +0.22541 | | QK-Norm | +0.00001 | +0.00012 | +0.00013 | | **QK-MAP** | **+0.00617** | **+0.19360** | **+0.20106** | We intentionally performed this intervention on the residual-free direct-map variant (QK-MAP-D) precisely to eliminate any residual bypass and test whether the bounded mapping itself preserves and utilizes magnitude signals. Shuffling key norms increases CE by +0.19360 for QK-MAP-D (comparable to +0.18816 for Standard, vs. +0.00012 for QK-Norm; 95% bootstrap CIs in Table 11), confirming that the mapped representations actively carry token-specific magnitude information for prediction. Moreover, our residual ablations in Appendix E (Table 8) show that this residual-free variant (QK-MAP-D) already achieves the highest downstream Avg-8 at both 370M (46.61%, vs. 45.35% for Standard and 45.91% for QK-Norm) and 760M (50.79%, vs. 50.24% for Standard and 49.61% for QK-Norm). Together with the controlled mapping table above—where Linear Map, Sin/Cos, Sech/Tanh, and Rational share the exact same residual structure—these results show that the performance gains hold both with and without the residual branch and stem from the bounded, magnitude-sensitive mapping.
AuthorsOpenReview ↗

Response to Reviewer 4FHT (Part 2 / 2)

W3: Reliability of Downstream Improvements

To evaluate downstream reliability and account for single-seed variance, we conducted three independent training runs with matched seeds at 190M across five methods (including QUEST and FANformer):

Variant (190M)Train CE ↓Val. CE ↓Downstream Avg-8 (%) ↑
Standard3.85 ± 0.014.14 ± 0.0140.10 ± 0.59
QK-Norm3.76 ± 0.014.05 ± 0.0140.84 ± 0.37
QUEST3.78 ± 0.014.06 ± 0.0140.66 ± 0.86
FANformer3.77 ± 0.014.04 ± 0.0240.67 ± 0.56
QK-MAP3.67 ± 0.013.93 ± 0.0141.82 ± 0.50

Values are reported as mean ± standard deviation across three seeds.

QK-MAP achieves the best training CE, validation CE, and downstream Avg-8 among all five methods, ranking first in downstream accuracy in every individual seed. Averaging over three seeds establishes a clear paired improvement of +1.72 percentage points over Standard (95% CI: [0.54, 2.90]) and +0.98 points over QK-Norm (95% CI: [0.00, 1.95]). At 7B, QK-MAP leads both baselines across all 16 evaluated checkpoints (Figure 4c), providing consistent multi-scale and multi-checkpoint evidence. We will include the multi-seed results and confidence intervals in the revision.


W4: Practical Training Efficiency

We measured actual training efficiency on matched hardware, and QK-MAP achieves the fastest wall-clock time to the target validation loss at both evaluated scales.

ScaleVariantStep time (s) ↓Throughput (k tokens/s, 2 GPUs) ↑Peak memory (GiB/rank) ↓Time to C4 target (min) ↓
190MStandard3.584585.1026.1556.5
QK-Norm3.715564.4927.2844.9
QUEST3.654573.8627.3043.7
FANformer3.644575.4626.7144.2
QK-MAP3.661572.8026.7137.8
370MStandard6.206337.9136.63107.2
QK-Norm6.399327.7238.63110.2
QUEST6.325331.5838.6697.9
FANformer6.355329.9937.6698.4
QK-MAP6.477323.7837.7788.5

QK-MAP reduces the wall-clock time to the target loss by 33.1% at 190M and 17.4% at 370M compared with Standard attention. Its per-step overhead is only 2.1% and 4.4%, respectively, and its peak allocated memory is lower than both QK-Norm and QUEST. Thus, QK-MAP's faster convergence more than compensates for its modest per-step computation.

We will include the training efficiency measurements in the revision.


Thanks again for your time and for these important points; your feedback has been a great help in strengthening our paper.

Thank the authors for the detailed rebuttal. Some of concerns are addressed, and I will keep the score.

Reviewer_5KEUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether attention can control query and key norms while keeping information about vector magnitude. It argues that QK normalization can remove useful magnitude information, including signals related to attention sharpness and how strongly a token attracts attention. The proposed method, QK-MAP, maps the attention input into learned sine and cosine features before producing queries and keys, while leaving the value path unchanged. The paired periodic features have a fixed norm but can still change when input magnitude changes. The main training experiments use a version with a learned residual and a smaller feed-forward layer to keep parameter counts similar. Across five model sizes, from the nominal 190M scale to 7B, the paper reports lower training and validation loss than Standard attention and QK-Norm. It also studies downstream accuracy, norm shuffling, first-token attention, attention to inserted evidence, and word-replacement tests.

Critical strengths
  • Better attention methods could help many language-model applications.
  • Lower loss across five model sizes is encouraging.
  • Keeping the value path unchanged gives a clear design choice that can be tested.
Critical weaknesses
  • FANformer is very close prior work. It already uses learned sine and cosine features before the Q/K/V projections and reports gains across model sizes [1]. The paper needs a direct comparison with it.
  • QK-MAP keeps the original V path and uses a different residual design. These are real differences, but the paper must show why they matter. QUEST also studies how to control norms while keeping query-dependent attention sharpness [2].
  • Actual cost is unclear. Similar parameter counts do not mean similar speed. The paper should report training speed, memory use, and time to reach the same loss. It also needs longer training tests and stronger evidence for the stability of the version used in the main experiments.

References

[1] Dong et al. Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling. NeurIPS 2025.

[2] Govindarajan et al. QUEST: A Robust Attention Formulation Using Query-Modulated Spherical Attention. ICLR 2026.

Questions for authors

refer to weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5KEU (Part 1 / 2)

We sincerely thank the reviewer for the thoughtful feedback. Below, we address each concern with new controlled comparisons, architectural clarifications, and wall-clock efficiency measurements.


W1 & W2: Relation and Direct Comparison to FANformer and QUEST

QK-MAP is motivated by a specific limitation of QK-Norm: controlling attention scale through normalization suppresses token-specific magnitude information. Our goal is to construct Q/K representations that remain sensitive to input magnitude while controlling feature-norm growth.

FANformer [1] and QK-MAP differ in both motivation and attention design. FANformer introduces periodic features to enhance general modeling capacity:

FAN⁡(x)=[cos⁡(Wpx), sin⁡(Wpx), Wˉpx+bˉp],\operatorname{FAN}(x)=[\cos(W_px),\ \sin(W_px),\ \bar W_px+\bar b_p],

FANformer applies these features to the attention projections, including the value path. QK-MAP instead targets attention-logit instability and magnitude information loss in the Q/K dot product. We therefore transform only Q and K while preserving the original linear value path.

Unlike FANformer's concatenated linear branch, QK-MAP uses a zero-initialized, learnable per-dimension residual (Appendix A.3, Eq. 36). Our ablations show that the direct map (QK-MAP-D) already outperforms Standard and QK-Norm at 370M and 760M, while QK-MAP-R starts strictly as a bounded map and gradually blends the pre-normalized input, which proves advantageous at larger scales (Appendix E, Table 8).

QUEST [2] preserves query-dependent attention sharpness by normalizing only keys, thereby removing key-side magnitude information. QK-MAP instead transforms the shared Q/K input through magnitude-sensitive features, preserving sensitivity on both sides of attention.

Importantly, our design principle is not limited to sine and cosine. To examine whether similar benefits can be obtained without periodic features, we additionally evaluated two non-periodic, fixed-norm mappings:

Sech/Tanh Map:

=[\operatorname{sech}(\theta),\ \tanh(\theta)].$$ **Rational Map**, with $t=\theta/2$: $$\phi_{\mathrm{R}}(\theta) =\left[\frac{1-t^2}{1+t^2},\ \frac{2t}{1+t^2}\right].$$ Both mappings are strictly non-periodic and satisfy $\|\phi(\theta)\|_2=1$ while retaining sensitivity to changes in input magnitude. We conducted direct comparisons with FANformer and QUEST alongside these non-periodic maps under matched data order, optimization settings, token budgets, and parameter counts: | Scale | Variant | Train CE ↓ | Val. CE ↓ | Downstream Avg-8 (%) ↑ | |:---:|:---|---:|---:|---:| | 190M | Standard | 3.85 | 4.14 | 40.10 | | | QK-Norm | 3.76 | 4.05 | 40.84 | | | QUEST (K-Norm) | 3.78 | 4.06 | 40.66 | | | FANformer | 3.77 | 4.04 | 40.67 | | | **QK-MAP (Sin/Cos)** | 3.67 | **3.93** | 41.82 | | | **QK-MAP (Sech/Tanh)** | **3.66** | 3.98 | 41.10 | | | **QK-MAP (Rational)** | **3.66** | 3.98 | **42.51** | | 370M | Standard | 3.19 | 3.54 | 45.35 | | | QK-Norm | 3.17 | 3.52 | 45.91 | | | QUEST (K-Norm) | 3.14 | 3.50 | 44.81 | | | FANformer | 3.16 | **3.43** | 45.92 | | | **QK-MAP (Sin/Cos)** | 3.10 | 3.45 | 45.94 | | | **QK-MAP (Sech/Tanh)** | 3.10 | 3.45 | **46.25** | | | **QK-MAP (Rational)** | **3.09** | 3.45 | 45.51 | QK-MAP variants achieve the lowest training CE and the highest downstream average at both scales. Importantly, both Sech/Tanh and Rational maps reproduce the improvements of QK-MAP, demonstrating that periodic features are not essential for obtaining these benefits. These results further support our broader motivation: constructing magnitude-sensitive Q/K representations without relying on uncontrolled feature-norm growth. [1] Dong et al. Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling. NeurIPS 2025. [2] Govindarajan et al. QUEST: A Robust Attention Formulation Using Query-Modulated Spherical Attention. ICLR 2026.
AuthorsOpenReview ↗

Response to Reviewer 5KEU (Part 2 / 2)

W3: Practical Training Cost, Time to Quality, and Stability

We measured training efficiency on matched hardware, and QK-MAP reaches the target validation loss in the shortest wall-clock time across all methods at both evaluated scales, despite a modest per-step overhead.

ScaleVariantStep time (s) ↓Throughput (k tokens/s, 2 GPUs) ↑Peak memory (GiB/rank) ↓Time to C4 target (min) ↓
190MStandard3.584585.1026.1556.5
QK-Norm3.715564.4927.2844.9
QUEST (K-Norm)3.654573.8627.3043.7
FANformer3.644575.4626.7144.2
QK-MAP (Sin/Cos)3.661572.8026.7137.8
370MStandard6.206337.9136.63107.2
QK-Norm6.399327.7238.63110.2
QUEST (K-Norm)6.325331.5838.6697.9
FANformer6.355329.9937.6698.4
QK-MAP (Sin/Cos)6.477323.7837.7788.5

Compared with Standard attention, QK-MAP incurs only a 2.1% and 4.4% per-step overhead at 190M and 370M, while using less peak memory than both QK-Norm and QUEST. Because of its faster convergence, QK-MAP reduces the wall-clock time to the target loss by 33.1% at 190M and 17.4% at 370M relative to Standard attention, and by 10%–20% relative to QK-Norm, QUEST, and FANformer.

Regarding stability and longer training, QK-MAP trains smoothly at 7B over 20B tokens without loss spikes (Figure 4). We are also continuing training from the 7B checkpoint and will include the extended results in the revision.


Thanks again for your time and for these important points; your feedback has been a great help in strengthening our paper.

Thanks the authors for their detailed rebuttal. My concerns have been addressed, and I will raise my score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary to AC

Dear Area Chair,

We sincerely thank you for overseeing our submission and the reviewers for their valuable comments. To help your assessment, we summarize below how we addressed the main concerns during the rebuttal.


1. What QK-MAP actually does (why it differs from FANformer and QUEST).

The main idea of QK-MAP is to preserve input magnitude information using a bounded, scale-sensitive mapping, instead of throwing magnitude away with normalization or letting norms grow unchecked. Unlike QUEST, which only normalizes keys, or FANformer, which puts periodic features across Q, K, and V, QK-MAP only transforms the shared input to Q and K and leaves V unchanged. In our additional experiments at 190M and 370M, QK-MAP gets lower loss than both QUEST and FANformer. To test how broadly our idea applies, we also designed two non-periodic bounded maps (Sech/Tanh and Rational). Both variant perform well, while just adding a linear projection alone at 190M does not give the same gain. This shows that the improvement comes from bounded magnitude sensitivity itself, not just periodicity or an extra linear projection.

2. Reliability across seeds and longer training at 7B.

At 190M, we ran three matched seeds for five methods. QK-MAP ranked first on the 8-task average in every seed, with a mean accuracy of 41.82%, compared with 40.84% for QK-Norm and 40.10% for Standard attention. For longer training, we extended the 7B model from 20B to nearly 30B tokens, and QK-MAP still maintained a lower final validation CE than QK-Norm (2.84 vs. 2.85), which is a meaningful margin in pretraining and shows that the gap holds up with longer training.

3. Speed, memory, and time to reach the same loss.

We measured step time, throughput, peak memory, and wall-clock time to reach the same C4 validation loss, along with single-GPU inference at 190M. Standard attention is slightly faster per step, but QK-MAP reaches the target loss in 33% less time at 190M and 17% less time at 370M than Standard, while using less peak memory than QK-Norm and QUEST. Overall, QK-MAP improves practical training efficiency.


We thank the reviewers for suggesting these comparisons, which helped clarify the benefits and cost of QK-MAP. We also thank the Area Chair for overseeing the discussion and considering our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.