- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies attention-logit scaling in gain-free QK-normalized transformers and its confounding effects on comparisons across head dimensions and model shapes. Controlled sweeps across model sizes, context lengths, and text/RNA settings identify a near-optimal scale around 16 in most tested settings, with fixed scaling reducing performance gaps attributed to architecture. The paper also examines training-induced angular-margin compensation and introduces a margin normalized by ln(T-1), motivated by an equal-competitor softmax threshold.
Critical strengths
- Controlled evidence for scale effects: Matched architectures, training budgets, and three paired seeds support the principal findings, including an approximately 0.0094 improvement at head dimension 64.
- Quantifying an approximately 2.2x amplification of the wide–deep performance gap by default scaling demonstrates why architecture comparisons should control attention scale.
- Comparisons between retraining, inference-only temperature adjustment, and global, layerwise, and headwise learnable scales help distinguish representation adaptation from simple temperature calibration.
Critical weaknesses
- The general half-mass threshold requires a competing-logit dispersion correction, while top-key mass near 0.29 and 96–98% margin retention after key shuffling weaken the proposed task-relevant interpretation. The normalized margin should remain an empirical diagnostic unless corrected-threshold and null-control analyses establish its connection to prediction loss.
- Nonuniform channelwise gains change directions and introduce input-dependent norms, so public-checkpoint effective scales are not automatically comparable with the gain-free global scale.
Questions for authors
See weaknesses.