- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work establishes a critical inverse temperature scale for softmax concentration in the limit of infinitely long sequences, given the attention scores. Derived from the gap counting function, this scale is the maximum entropy-energy ratio between the logarithmic number of tokens within a given gap from the maximum score and the size of this gap. Inverse temperature asymptotically below this scale lead to top-two or rank collapse (vanishing score differences making top-score tokens indistinguishable), and to entropy collapse (softmax concentrates attention on the maximum-score token) asymptotically above this scale. Within scalings that are powers of the logarithm of the sequence length, the critical scale determines the inverse-temperature exponent required to avoid both collapses, that can be decomposed into competitor-count and score-gap contributions. Equicorrelated and block-structured score matrices illustrate how different score configurations can produce different critical scales and exponents. Critical exponents are then computed on empirical attention scores using gap counting, and compared with the ones obtained from the inverse-temperature minimizing perplexity in Qwen7B long-context temperature sweeps, and from the learned inverse-temperature in nanoGPT.
Critical strengths
The article isolates a generic softmax concentration mechanism and provides an implementable and rigorously justified gap-counting procedure to identify temperature-scaling regimes, through a simple and interpretable entropy vs energy competition.
The article validates that, experimentally, on various datasets some langage models satisfy the temperature-scaling relation.
Critical weaknesses
The critical scale is derived, given the attention scores. The article does not describe the attention itself or its particular score structure, that does not seem predictible according to the experiments. The authors acknowledge this limitation in the discussion.
The theory identifies a critical inverse temperature scale to avoid collapse, but this does not establish whether this scale is task-optimal : e.g. the attention may focus on one element. The empirical agreement with perplexity-optimal or learned temperatures is therefore not directly explained.
Questions for authors
For clarification, in the qwen7B experiment, evaluated with gap-counting changes between the 16k and 32k settings. How should the dependance of the critical exponent on the sequence length be interpreted ? Should one expect the exponent to converge to a single value in the limit of large context-length, or is this viewed as a consequence of changes in the underlying score structure with increasing context length ? Do the exponents found through optimal perplexity converge with increasing context length ?
3 comments
I thank the authors for their answers and tests on the case of single-element retrieval. The added experiment does support better predictability, but as convergence of the estimated exponents with increasing sequence length is not directly tested, I keep my score of weak accept.
Thank you for your thoughtful review and for considering our responses and additional experiments.
Thank you for your careful review.
Yes, we expect the critical exponent to converge to a single value for a fixed model and text distribution. We interpret the 16K–32K difference as finite-context variation on the approach to stable score scaling. Anson et al. (2025) motivate preserving access to information across linguistic scales, while Garnier-Brun et al. (2025) show how hierarchical dependencies appear in learned attention in a controlled sequence model. Our hypothesis is that such statistical structure persists in longer contexts, yielding stable scaling in the scores.
Our expectation is yes. We expect stabilization if avoiding collapse remains a principal constraint on performance. The grid optima were 2.5 at 16K and 2.0 at 32K, but their PPL values were close at both lengths. Our response to waZB reports an additional experiment with a fixed exponent ≈2.1, estimated from attention scores in contexts up to 12K. Its PPL differed from the original grid minimum by less than 0.09% on both PG19 and Proof-Pile 2 at 16K and 32K. Please see our response to waZB for the calibration method and experimental protocol. These results support using a single score-derived exponent at longer contexts.
We understand this comment as questioning whether avoiding entropy collapse is necessarily beneficial for tasks where focusing on one element may be useful.
Prior work shows that inverse-temperature scaling laws motivated by language modeling also benefit retrieval. SSMax adopts the logarithmic dependence observed during language-model training and reports improved seven-digit number retrieval and increased attention to the number span (Nakanishi, 2025, §§2.1, 3.3–3.4). YaRN fits its temperature-scaling law to perplexity minima and reports improved passkey retrieval in models using that law (Peng et al., 2024, §§3.3, 4.3).
We also tested this on single-element retrieval, where concentrating attention on the target seems intuitively useful.
We evaluated Qwen-7B-Chat on RULER single-element retrieval with essay backgrounds at 16K and 32K, using 20 examples per length. Each example requires retrieving a single seven-digit number. We used exponent 2, which gave the best PPL at 32K in the paper, and the score-derived exponent of approximately 2.1 reported in our response to waZB, without tuning either for retrieval. All layers and heads shared the same inverse-temperature multiplier, normalized at 8K.
Exponents 2 and ≈2.1 matched logarithmic scaling at 16K and gave more correct answers than exponents 1 and 3 at 32K, differing from each other by only one answer.
Across 14 successful examples at 32K with exponent 2, mean attention entropy across all heads in the 31 intermediate layers was 4.36 nats when generating the first digit of the answer. This average is well above the near-zero entropy of attention concentrated on a single token. One head nevertheless had a mean maximum attention weight of 0.986. Concentration differs across heads because their score structures differ, even under a shared inverse temperature. Thus, shared inverse-temperature scaling and strong concentration in individual heads coexisted.
Our experiments show close agreement between the critical inverse-temperature scaling for avoiding collapse and the PPL-optimal scaling. Next-token prediction, evaluated by PPL, is the pretraining task underlying downstream capabilities. We therefore expect this context-length scaling law to carry over to downstream tasks, as our retrieval results also suggest. For task adaptation, we envisage fine-tuning parameters such as head-specific temperatures while retaining this scaling law.
References: