acceptodds
Under review as a conference paper at ICLR 2027

A model-agnostic scaling rule behind the critical inverse temperature in softmax self-attention

Abstract

Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest different inverse-temperature laws for the context length , ranging from to and . We provide a general theory showing that the critical scale is determined by the gap-counting function of each attention row, without specifying the details of the model structure. Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale and prove that it gives the critical inverse-temperature scale for softmax concentration: below this scale, the top competitors remain unseparated, whereas above it, the attention entropy collapses. This framework provides a model-agnostic, higher-level explanation of how scaling laws vary with , and yields a direct diagnostic for attention-score families, from idealized theoretical models to practical transformers.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-attention, Critical inverse-temperature scaling, Softmax concentration, Long-context transformers, Gap-counting framework

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_MUoZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work establishes a critical inverse temperature scale for softmax concentration in the limit of infinitely long sequences, given the attention scores. Derived from the gap counting function, this scale is the maximum entropy-energy ratio between the logarithmic number of tokens within a given gap from the maximum score and the size of this gap. Inverse temperature asymptotically below this scale lead to top-two or rank collapse (vanishing score differences making top-score tokens indistinguishable), and to entropy collapse (softmax concentrates attention on the maximum-score token) asymptotically above this scale. Within scalings that are powers of the logarithm of the sequence length, the critical scale determines the inverse-temperature exponent required to avoid both collapses, that can be decomposed into competitor-count and score-gap contributions. Equicorrelated and block-structured score matrices illustrate how different score configurations can produce different critical scales and exponents. Critical exponents are then computed on empirical attention scores using gap counting, and compared with the ones obtained from the inverse-temperature minimizing perplexity in Qwen7B long-context temperature sweeps, and from the learned inverse-temperature in nanoGPT.

Critical strengths

The article isolates a generic softmax concentration mechanism and provides an implementable and rigorously justified gap-counting procedure to identify temperature-scaling regimes, through a simple and interpretable entropy vs energy competition.

The article validates that, experimentally, on various datasets some langage models satisfy the temperature-scaling relation.

Critical weaknesses

The critical scale is derived, given the attention scores. The article does not describe the attention itself or its particular score structure, that does not seem predictible according to the experiments. The authors acknowledge this limitation in the discussion.

The theory identifies a critical inverse temperature scale to avoid collapse, but this does not establish whether this scale is task-optimal : e.g. the attention may focus on one element. The empirical agreement with perplexity-optimal or learned temperatures is therefore not directly explained.

Questions for authors

For clarification, in the qwen7B experiment, ξ^Λ\hat \xi_\Lambda evaluated with gap-counting changes between the 16k and 32k settings. How should the dependance of the critical exponent on the sequence length be interpreted ? Should one expect the exponent to converge to a single value in the limit of large context-length, or is this viewed as a consequence of changes in the underlying score structure with increasing context length ? Do the exponents found through optimal perplexity converge with increasing context length ?

3 comments

Thank you for your careful review.

For clarification, in the qwen7B experiment, ξ^Λ\hat \xi_\Lambda evaluated with gap-counting changes between the 16k and 32k settings. How should the dependance of the critical exponent on the sequence length be interpreted ? Should one expect the exponent to converge to a single value in the limit of large context-length, or is this viewed as a consequence of changes in the underlying score structure with increasing context length ?

Yes, we expect the critical exponent to converge to a single value for a fixed model and text distribution. We interpret the 16K–32K difference as finite-context variation on the approach to stable score scaling. Anson et al. (2025) motivate preserving access to information across linguistic scales, while Garnier-Brun et al. (2025) show how hierarchical dependencies appear in learned attention in a controlled sequence model. Our hypothesis is that such statistical structure persists in longer contexts, yielding stable scaling in the scores.

Do the exponents found through optimal perplexity converge with increasing context length ?

Our expectation is yes. We expect stabilization if avoiding collapse remains a principal constraint on performance. The grid optima were 2.5 at 16K and 2.0 at 32K, but their PPL values were close at both lengths. Our response to waZB reports an additional experiment with a fixed exponent ≈2.1, estimated from attention scores in contexts up to 12K. Its PPL differed from the original grid minimum by less than 0.09% on both PG19 and Proof-Pile 2 at 16K and 32K. Please see our response to waZB for the calibration method and experimental protocol. These results support using a single score-derived exponent at longer contexts.

The theory identifies a critical inverse temperature scale to avoid collapse, but this does not establish whether this scale is task-optimal : e.g. the attention may focus on one element. The empirical agreement with perplexity-optimal or learned temperatures is therefore not directly explained.

We understand this comment as questioning whether avoiding entropy collapse is necessarily beneficial for tasks where focusing on one element may be useful.

Prior work shows that inverse-temperature scaling laws motivated by language modeling also benefit retrieval. SSMax adopts the logarithmic dependence observed during language-model training and reports improved seven-digit number retrieval and increased attention to the number span (Nakanishi, 2025, §§2.1, 3.3–3.4). YaRN fits its temperature-scaling law to perplexity minima and reports improved passkey retrieval in models using that law (Peng et al., 2024, §§3.3, 4.3).

We also tested this on single-element retrieval, where concentrating attention on the target seems intuitively useful.

We evaluated Qwen-7B-Chat on RULER single-element retrieval with essay backgrounds at 16K and 32K, using 20 examples per length. Each example requires retrieving a single seven-digit number. We used exponent 2, which gave the best PPL at 32K in the paper, and the score-derived exponent of approximately 2.1 reported in our response to waZB, without tuning either for retrieval. All layers and heads shared the same inverse-temperature multiplier, normalized at 8K.

Context lengthLog scaling (exponent 1)PPL-based exponent 2Score-based exponent ≈2.1Exponent 3
16K20/2020/2020/2017/20
32K12/2016/2015/2011/20

Exponents 2 and ≈2.1 matched logarithmic scaling at 16K and gave more correct answers than exponents 1 and 3 at 32K, differing from each other by only one answer.

Across 14 successful examples at 32K with exponent 2, mean attention entropy across all heads in the 31 intermediate layers was 4.36 nats when generating the first digit of the answer. This average is well above the near-zero entropy of attention concentrated on a single token. One head nevertheless had a mean maximum attention weight of 0.986. Concentration differs across heads because their score structures differ, even under a shared inverse temperature. Thus, shared inverse-temperature scaling and strong concentration in individual heads coexisted.

Our experiments show close agreement between the critical inverse-temperature scaling for avoiding collapse and the PPL-optimal scaling. Next-token prediction, evaluated by PPL, is the pretraining task underlying downstream capabilities. We therefore expect this context-length scaling law to carry over to downstream tasks, as our retrieval results also suggest. For task adaptation, we envisage fine-tuning parameters such as head-specific temperatures while retaining this scaling law.

References:

  • Nakanishi, K. M. (2025). Scalable-Softmax Is Superior for Attention.
  • Peng, B., Quesnelle, J., Fan, H., and Shippole, E. (2024). YaRN: Efficient Context Window Extension of Large Language Models.

I thank the authors for their answers and tests on the case of single-element retrieval. The added experiment does support better predictability, but as convergence of the estimated exponents with increasing sequence length is not directly tested, I keep my score of weak accept.

Thank you for your thoughtful review and for considering our responses and additional experiments.

Reviewer_mVx7OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers the inverse temperature scaling problem for a single attention row. It intends to provide a model agnostic theory to identify the appropriate scaling based on the gap-counting function, which counts the number of competitors within a small neighborhood of the maximum score. It connects the counting function's upper-tail accumulation scale to the critical inverse-temperature scale that maintains attention's selectivity. The authors support their theory with experiments on temperature scaling during inference in Qwen-7B-Chat and temperature parameters learned during training in GPT-124M.

Critical strengths

The paper is well written and easy to follow. Its gap-counting formulation provides a general framework for understanding how attention score gaps determine the critical inverse temperature scaling. The theoretical analysis appears sound and is complemented by experiments examining both inference-time temperature scaling in pretrained large languge models and training-time learning of temperature multipliers.

Critical weaknesses

Although the appendices describe the experimental setup, I could not find a public link to the modified model code or analysis scripts. Providing the codes would improve the reproducibility of the experimental results.

Questions for authors
  1. Could the authors provide their modified model code and the scripts needed to reproduce the empirical results? Given the scale of these experiments, independently reimplementing and reproducing them would require substantial time and computational resources.

  2. Beyond the use of polylogarithmic inverse temperature scaling in existing models, what motivates restricting the empirical investigation to this family of scaling laws? Since the empirical fits cover only a finite range of sequence lengths, could other scaling laws, such as polynomial scaling in sequence length, yield comparable or even better performance improvements? I would appreciate the authors' comments on this direction.

3 comments

Thank you for your careful review and for recognizing the generality of our theory and its empirical validation.

Could the authors provide their modified model code and the scripts needed to reproduce the empirical results? Given the scale of these experiments, independently reimplementing and reproducing them would require substantial time and computational resources.

We have now attached an anonymous ZIP containing the modified model implementations and the training, evaluation, and analysis scripts for the main experiments. The central change is a position-dependent inverse-temperature factor in the attention computation. In Qwen, we make the exponent of the existing logarithmic scaling adjustable; in nanoGPT, we learn the position-dependent factors.

The package explains the modifications and provides experiment configurations and instructions for preparing data and running the experiments. It also includes saved numerical data for regenerating the curves in Figure 1 and the aggregate results in Table 2. You can inspect the modifications directly and reproduce these analyses from the saved data without rerunning GPU training or inference.

Please also see the paper's Reproducibility statement. Appendix A documents the experimental setup, including model and dataset identifiers and implementation changes in A.5. Appendix B describes temperature fitting, and Appendices F and G describe score acquisition and gap counting for nanoGPT and Qwen, respectively. Section 4.1 explains the aggregation and statistical estimation for Table 2.

Beyond the use of polylogarithmic inverse temperature scaling in existing models, what motivates restricting the empirical investigation to this family of scaling laws?

We focused on polylogarithmic scaling to explain existing inverse-temperature scaling rules through the score structure. This family includes scaling laws derived in theoretical analyses as well as those used in practical models. We used (log⁡n)ξ(\log n)^\xi to express these rules in a common form, then compared the exponents estimated from scores with those obtained from PPL evaluations or learned temperatures. Our general gap-counting theory does not assume this logarithmic form: it determines the critical scale from the score structure.

Since the empirical fits cover only a finite range of sequence lengths, could other scaling laws, such as polynomial scaling in sequence length, yield comparable or even better performance improvements? I would appreciate the authors' comments on this direction.

Yes, other scaling families could yield comparable or better performance. In nanoGPT, the position-dependent inverse temperature is learned without imposing a logarithmic form during training. The fitted logarithmic exponents and the critical-scale exponents estimated from scores are both below one (Table 2), providing a consistent indication of slow growth.

In a toy model with random keys on a sphere, Bruno et al. (2026) derive a critical inverse-temperature scale that grows as a power of context length. Their result is consistent with our framework, in which the appropriate growth law depends on the score structure.

Over a finite range of context lengths, logarithmic and polynomial scaling can give very similar inverse-temperature values. For example, consider the following two schedules, normalized to 1 at 8192 tokens:

\left(\frac{n}{8192}\right)^{1/\log8192},$$ Both schedules also have the same slope at 8192 tokens. The inverse-temperature values differ by about 1% at 32K tokens and only about 6% even at 256K. Comparing PPL beyond 256K tokens would require substantial computation, leaving open which scaling family yields lower PPL at those lengths. Reference: Bruno et al. (2026). *Scaling Limits of Long-Context Transformers*. arXiv:2605.08505.

Thank you for the response and for providing the codes. I encourage the authors to incorporate the clarifications on alternative scaling laws into the manuscript. The authors have addressed my questions, and I'm therefore raising my score.

Thank you for your careful review and for taking the time to consider our response. When we looked more closely at the result of Bruno et al., we found a more direct connection to our gap-counting framework than we first expected. We have added this discussion to Appendix H and updated the reproducibility statement to point readers to the accompanying code and data. Thank you again for your helpful suggestion!

Reviewer_bMFKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a model-agnostic theoretical framework to study the effect of different choices of the inverse temperature in self-attention. They mathematically analyze the critical scale of the inverse temperature and point out that the gap-counting function is a key quantity to determine it.

Critical strengths

This paper is solid in theory. They carefully study when the top-two/rank/entropy collapse will occur and how it is related to the scale of the gap-counting function. Their results provide a high-level framework which helps us better understand how different practical inverse-temperature rules work and whether they are optimal.

Critical weaknesses

The main weakness is that their theoretical results may not be directly used in practice. Are there any suggestions on how the key quantities (ΛnΛ_n,αnα_n,ΔnΔ_n) could be used to guide practical choice of the inverse temperature?

Questions for authors
  1. The theoretical bounds analyze a single deterministic vector of attention logits. How does this row-wise framework extend to multi-head self-attention where individual heads specialize (e.g., induction heads requiring sharp attention vs. summary heads requiring diffuse attention)?

  2. Staying within the non-degenerate regime (between rank collapse and entropy collapse) guarantees information capacity, but does tuning β\beta strictly to match Λn\Lambda_n lead to better performance on downstream tasks like long-document retrieval or complex reasoning?

1 comment

Thank you for your thoughtful review and for recognizing our theoretical framework for understanding practical inverse-temperature scaling rules.

The theoretical bounds analyze a single deterministic vector of attention logits. How does this row-wise framework extend to multi-head self-attention where individual heads specialize (e.g., induction heads requiring sharp attention vs. summary heads requiring diffuse attention)?

Yes. The theory applies to each attention row in each head, with a critical scale determined by that row's score structure. In our nanoGPT experiments, the context-length dependence βn\beta_n is shared across layers and heads, with separately learned scales and biases: βn,ℓ,h=sℓ,hβn+bℓ,h\beta_{n,\ell,h}=s_{\ell,h}\beta_n+b_{\ell,h}. Each head can adjust attention concentration through its QK scores and temperature parameters, so shared length dependence is compatible with different degrees of concentration.

Across the 12 layers trained on OpenWebText, the within-layer mean of sℓ,hs_{\ell,h} ranges from 0.97 to 3.30. Thus, different temperature adjustments are learned across both layers and heads despite the shared length dependence.

Staying within the non-degenerate regime (between rank collapse and entropy collapse) guarantees information capacity, but does tuning β\beta strictly to match Λn\Lambda_n lead to better performance on downstream tasks like long-document retrieval or complex reasoning?

For retrieval, yes: we observed an improvement in our 32K retrieval experiment. The score-derived exponent, applied without retrieval-specific tuning, gave more correct answers than logarithmic scaling. For details, see our response to Reviewer MUoZ on "whether this scale is task-optimal". SSMax also reports retrieval improvements from temperature scaling (Nakanishi, 2025). For complex reasoning, we expect benefits but have not yet tested them.

Scaling the query at position nn to βnqn\beta_n q_n adjusts the weights assigned to candidate keys according to the context length available to that query. The scale Λn\Lambda_n guides temperature selection to avoid losing the weight difference between the top two candidates or concentrating all attention on one candidate. When these collapses impair information readout, we expect this adjustment to improve retrieval or reasoning performance.

Are there any suggestions on how the key quantities (ΛnΛ_n,αnα_n,ΔnΔ_n) could be used to guide practical choice of the inverse temperature?

Yes. We suggest estimating the asymptotic growth exponent of Λn\Lambda_n from scores on target data as a starting point for choosing the length dependence of inverse temperature at finite lengths. Fitting Λn\Lambda_n measured at several context lengths to (log⁡n)ξ(\log n)^\xi provides a candidate exponent without gradients or next-token labels.

For an example, see §1, "Extrapolation and practical relevance," of our response to Reviewer waZB. In Qwen, fixing the exponent estimated from short-context Λn\Lambda_n measurements gave PPL close to the grid-search minimum at longer contexts.

For further task adaptation, nearby exponents and scale factors can be compared using validation PPL or task metrics. As in YaRN's PPL-based temperature calibration (Peng et al., 2024), measured performance can guide this refinement.

With limited adaptation data or updates, one could fix the estimated exponent ξ\xi and fine-tune only sℓ,hs_{\ell,h} in βn,ℓ,h=sℓ,h(log⁡n)ξ\beta_{n,\ell,h}=s_{\ell,h}(\log n)^\xi. Sharing each scale across positions lets an update at one position adjust temperatures elsewhere, potentially improving adaptation with limited data.


Parameter summary: GPT-124M trained on OpenWebText with learned absolute positional embeddings and context length 1024, at 100k training steps. Entries are mean ± standard deviation across the 12 heads of each layer.

Layerssbb
13.30 ± 0.842.20 ± 0.86
22.06 ± 0.611.12 ± 0.57
31.82 ± 0.541.11 ± 0.66
41.70 ± 0.340.96 ± 0.42
52.18 ± 2.371.29 ± 2.38
61.31 ± 0.240.29 ± 0.18
71.23 ± 0.300.25 ± 0.20
81.23 ± 0.220.30 ± 0.17
91.04 ± 0.080.13 ± 0.26
101.09 ± 0.060.25 ± 0.20
110.97 ± 0.080.14 ± 0.23
121.05 ± 0.120.14 ± 0.22
Reviewer_waZBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies, given a particular context, the critical power \beta in log(n)^\beta, that avoids different sorts of collapses of the corresponding attention: namely, separating what they call top-two weight collapse from entropy collapse. The exponent depends on how the gaps from the largest attention score, and the number of competing scores within those gaps, change as context length increases. Different models and input sequences can therefore require different temperature-scaling exponents. The experiments show that adjusting the temperature-scaling exponent at inference time improves long-context perplexity, while the position-dependent inverse temperatures learned during training are better described by a different exponent (typically 1/2). These observations come from different models and experimental settings, so they are not directly comparable.

Critical strengths

The critical value introduced in their paper and the corresponding mathematical arguments make clear why the critical value of the exponent depends on the attention scores of a given input. The paper is correct on the technical side, nothing very involved was developed. I cannot judge the mathematical/technical novelty since related quantities were introduced earlier in the statistical physics literature, but I did not find their proposal explicitly. However, technical novelty is not particularly relevant for this paper, whose main contribution is the introduction of this critical value.

Critical weaknesses

The paper provides an interesting statistical quantity, but I feel that the practical consequences are not sufficiently shown or established by the experiments to lean towards acceptance. More arguments/contributions in this direction could help me improve my score. The paper does not really discuss the link with the literature where adaptive temperature is put forward: Softmax is not Enough (for Sharp Size Generalisation) by Velickovic et al. or Selective Attention: Enhancing Transformer through Principled Context Control by Zhang et al.

Questions for authors

Your experiments show a discrepancy between the exponent found on long contexts on Qwen at inference (around 2.) and the exponent found during training (around 1/2). I am curious of the following experiment: does your trained model with a learnt exponent per position still favors a larger exponent for long contexts? (if my understanding is correct: Train your model with the learned vector of temperatures, fit a log n^\beta law, evaluate long-context perplexity) This could suggest that the analytical parametrization in log(n)^\beta is failing. What do you think?

3 comments

Thank you for recognizing our score-dependent critical scale. We address extrapolation and practical relevance (§1), then compare our theory with adaptive-temperature methods (§2).

1. Extrapolation and practical relevance

does your trained model with a learnt exponent per position still favors a larger exponent for long contexts?

Our expectation is no: the exponent ξβ\xi_\beta in βn∝(log⁡n)ξβ\beta_n\propto(\log n)^{\xi_\beta} depends on the model, rather than necessarily increasing with context length. The exponents of about 0.5 and 2 in the paper come from nanoGPT and Qwen, respectively, so they do not demonstrate an increase within one model.

New Qwen experiment. To examine this expectation within one model, we used Qwen: nanoGPT's learned absolute positional embeddings prevent direct extrapolation beyond its training range.

Your proposed approach learns position-wise βn\beta_n and then fits their growth exponent. At context length n=12K, updating position-wise βn\beta_n from all next-token losses with frozen model weights required 95 GB of peak allocated GPU memory without activation checkpointing.

To reduce this memory cost, we estimated the exponent directly from short-context Λn\Lambda_n measurements instead of learning position-wise βn\beta_n. Our theory links the growth of Λn\Lambda_n to that of the critical inverse temperature.

We performed layer-by-layer Λn\Lambda_n calibration, proceeding from the input layer onward. We measured Λn\Lambda_n before inverse-temperature scaling at positions 4K–12K in eight documents, pooling measurements across documents and all processed layers to fit a shared exponent with auxiliary layer-wise coefficients. After each update, we reran all processed layers from the input with the updated exponent and coefficients before measuring the next layer. This procedure uses neither next-token loss nor backpropagation, with a measured peak memory of 19 GB.

For long-context evaluation, we fixed ξβ=ξ^Λ\xi_\beta=\hat\xi_\Lambda in

βn=max⁡(1,log⁡nlog⁡ntrain)ξβ.\beta_n=\max\left(1,\frac{\log n}{\log n_{\mathrm{train}}}\right)^{\xi_\beta}.

Here, nn and ntrainn_{\mathrm{train}} are the per-token and training context lengths.

Extrapolation results and practical relevance.

the practical consequences are not sufficiently shown or established by the experiments

To test whether the short-context exponent remains useful at longer contexts, we evaluated PPL using the paper's documents and sliding-window protocol. The paper's PPL-based grid search over exponents 0–3 gave minima at 2–2.5 on PG-19 and Proof-Pile-2 (PP2) at 16K–32K; the PPL difference between 2 and 2.5 was at most 0.56%. Layer-by-layer calibration using Λn\Lambda_n up to 12K yielded about 2.1, within this flat region near the minimum.

Measured PPL (lower is better). Log scaling: exponent 1; PPL-based: PPL-minimizing exponent in Fig. 1; Layer-by-layer: exponent 2.1 from Λn\Lambda_n calibration.

DatasetContextLog scalingPPL-basedLayer-by-layer
PG-1916K13.71913.60313.614
PG-1932K15.74515.15715.154
PP216K2.7632.7542.754
PP232K2.6982.6692.670

The calibrated exponent gave lower PPL than logarithmic scaling and nearly matched the best grid-search PPL. This Qwen example supports our expectation: the short-context exponent remained effective at longer contexts without adjustment. Beyond explaining the temperature scaling observed in the paper, Λn\Lambda_n can thus guide exponent selection without a PPL search.

2. Relation to adaptive-temperature methods

The paper does not really discuss the link with the literature where adaptive temperature is put forward

Veličković et al. [1] propose adaptive temperature scaling based on attention entropy, while Zhang et al. [2] combine a learnable query-dependent term with a logarithmic position-dependent term. Our paper's theory characterizes the critical inverse-temperature scale for general score rows through the cumulative gap counting function NnN_n, showing how the required dependence on context length varies with the score structure. Whereas Zhang et al. analyze a setting with two score levels and a power-law fraction of salient tokens, our theory establishes this relation without assuming a particular score distribution.


Extrapolation experiment: setup

Qwen-7B; training context: 8,192; calibration: eight PG-19 documents, positions 4,096–12,288. Evaluation used unit coefficients and windows of 16,384 and 32,768 tokens, following the PPL protocol of Fig. 1. The PG-19-derived exponent was also applied to PP2.

References

[1] P. Veličković et al. Softmax is not Enough (for Sharp Size Generalisation). ICML, 2025.

[2] X. Zhang et al. Selective Attention: Enhancing Transformer through Principled Context Control. NeurIPS, 2024.

Reviewer_waZBOpenReview ↗

Thank you for your answer

Your additional experiment is a nice addition to the article. Based on practical relevance, I do not change my score.

Thank you for the follow-up and for recognizing the value of the additional experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.