acceptodds
Under review as a conference paper at ICLR 2027

Attention Regulates Resolution, Not Sharpness: A Diffraction Limit for the Softmax Scale

Abstract

While the canonical softmax scale preserves variance at initialization, it fails to capture the operating regime of trained attention. With normalized queries and keys, logits factor exactly into a cosine similarity and an explicit scale , where the canonical choice corresponds to setting . Because the learned geometry can spread its cosines to offset an imposed scale, the scale and the geometry are redundant; we sweep on a logarithmic grid to examine how autoregressive training resolves this redundancy. Training absorbs much of the imposed scale into the learned geometry, yielding a rank correlation of across a 32-fold sweep, and conserves a logit margin rather than a scale. Indeed, eight public models with learned scales ranging from 11 to 73 maintain nearly the same margin. At the optimum, this margin sits at or just past the diffraction limit, defined as the threshold at which a single key claims half the attention weight against equal competitors, a property we formalize and verify in Lean 4. Thus, while the scale dictates sharpness, the margin normalized by the diffraction limit dictates resolution, and next-token prediction regulates resolution rather than sharpness. Empirically, the loss-optimal scale sits at from 25M to 393M block parameters, across context lengths from 512 to 2048 tokens and vocabularies from 6 to 32,768 tokens, on text and RNA, with one slight exception. At this fixed scale, 64-dimensional heads nearly match 128-dimensional heads, and learned global, per-layer, or per-head scales fail to outperform the constant value. Degradation under the canonical grows quadratically with its log-distance from this optimum, a distance set by the head dimension alone, and steepens as the context lengthens. These findings show that comparisons across head sizes under the canonical scale conflate a systematic scale artifact with architectural capacity, and that freezing decouples attention resolution from head size.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention, softmax scale, query-key normalization, transformers, language modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mZQ5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies attention-logit scaling in gain-free QK-normalized transformers and its confounding effects on comparisons across head dimensions and model shapes. Controlled sweeps across model sizes, context lengths, and text/RNA settings identify a near-optimal scale around 16 in most tested settings, with fixed scaling reducing performance gaps attributed to architecture. The paper also examines training-induced angular-margin compensation and introduces a margin normalized by ln(T-1), motivated by an equal-competitor softmax threshold.

Critical strengths
  1. Controlled evidence for scale effects: Matched architectures, training budgets, and three paired seeds support the principal findings, including an approximately 0.0094 improvement at head dimension 64.
  2. Quantifying an approximately 2.2x amplification of the wide–deep performance gap by default scaling demonstrates why architecture comparisons should control attention scale.
  3. Comparisons between retraining, inference-only temperature adjustment, and global, layerwise, and headwise learnable scales help distinguish representation adaptation from simple temperature calibration.
Critical weaknesses
  1. The general half-mass threshold requires a competing-logit dispersion correction, while top-key mass near 0.29 and 96–98% margin retention after key shuffling weaken the proposed task-relevant interpretation. The normalized margin should remain an empirical diagnostic unless corrected-threshold and null-control analyses establish its connection to prediction loss.
  2. Nonuniform channelwise gains change directions and introduce input-dependent norms, so public-checkpoint effective scales are not automatically comparable with the gain-free global scale.
Questions for authors

See weaknesses.

Reviewer_9VMbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the attention-logit scale in autoregressive transformers with qk-norm. The authors argue that a scale independent of the head dimension DD can outperform the conventional choice s=Ds=\sqrt{D}. They pretrain small transformers from scratch with different fixed scales and find that s=16s=16 achieves the lowest or nearly lowest prediction loss across their tested configurations. To interpret this finding, they introduce a simplified model based on the threshold required for one informative key to receive half the attention weight when other competing keys have equal smaller logits. They observe that training partially compensates for changes in the scale. They also test their theory on pretrained OLMo-2 and Qwen3.

Critical strengths

None

Critical weaknesses

While I appreciate the authors' effort to motivate a constant attention-logit scale, the presentation is difficult to follow and requires substantial revision. The idea of developing a theory by assuming a constant logit gap has been repeatedly rediscovered in different settings under different names, which is unsurprising. The training experiments are limited to relatively small models, leaving the applicability of the findings at larger scales uncertain. The open-source models evaluated in the paper are also not up to date, and it does not resolve this concern on training.

Questions for authors

I don't have any question.

Reviewer_YhafOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed to use a fixed scalar ss to replace the conventional D\sqrt{D} rule attention with qk-normalization. The authors sweep a wide range of ss and report that the loss optimum is generally s=16s=16, rather than D\sqrt D.

Critical strengths
  1. The main empirical question is precise and important. Once QK normalization removes norm variation, setting the effective temperature to D\sqrt D is not obviously justified.

  2. The finding that same approximate optimum appearing across size, shape, context, tokenization, and two modalities is interesting.

Critical weaknesses

The proposed theoretic interpretation does not lead to s=16s=16. The paper establishes three observations: i) the trained geometry changes when ss changes; ii) the optimal models have a certain cosine gap; iii) their normalized margin lies near a softmax threshold. However, it does not derive why optimization selects that cosine gap or why the loss optimum should occur at the half-mass threshold. The paper explicitly leaves “why training pins the gap” as an open conjecture.

Questions for authors

Does s=16s=16 remain optimal at longer context, such as 8K, 32K or even longer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.