acceptodds
Under review as a conference paper at ICLR 2027

Attention's Gravitational Field: A Power-Law and Linguistic Interpretation of Positional Correlation

Abstract

This paper investigates the underlying principles of positional relationships and encodings in Transformer-based language models. We introduce the Attention Gravitational Field (AGF), which decouples positional encodings from semantic embeddings and models their interaction through a power-law distribution. Drawing on linguistic derivations, we demonstrate that AGF is intrinsically consistent with Newton's law of universal gravitation and his shell theorem, while also aligning with observed learning dynamics and stability curves. Empirically, AGF achieves consistent accuracy improvements over existing relative positional encoding methods on standard benchmarks. Our analysis provides a deeper perspective on the interpretability of attention mechanisms, revealing structural regularities in positional information that bridge linguistics, physics, and model design. This work constitutes a meaningful step toward understanding the fundamental laws underlying Transformer architectures.

Primary area:
interpretability and explainable AI
Keywords:
Newton's law of universal gravitation, shell principle, stability curves, power-law, relative positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3bXLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Motivated by the concepts of distance decay and gravitational fields from linguistics, this paper provides a reinterpretation of power-law relative positional bias. It introduces AGF-S—which incorporates self-position weights, directional parameters, and shell boundaries—along with two multi-shell extensions, AGF-MS. The authors report modest BLEU score improvements across two machine translation tasks and interpret model behavior through the learned positional curves. Since the base AGF model is equivalent to log-KERPLE, the primary methodological contributions lie in the shell-based parameterization and the associated mixing mechanism.

Critical strengths
  1. The relationship between the proposed method and existing families of functions is clear. Section 3.3 explicitly defines the parameter mapping between AGF and log-KERPLE, making it easy to distinguish between reinterpretation and novel design.
  2. Positional bias is amenable to analysis. Components such as position, orientation, shell boundaries, and mixing weights are all explicitly parameterized; Figure 3 illustrates the curves for different attention heads, rendering the design empirically testable.
Critical weaknesses
  1. **Power law explanations mainly come from analogies and have not yet formed a verifiable theory. **PASL of §3 lacks explicit sampling, conditional events, and estimation procedures. It does not follow directly from Shannon's entropy, Zipf's law, or the reliability growth analogy that this distance distribution must be a power law. Although Figure 2 is marked "Actual Data", it does not explain the source of the data and the meaning of the coordinates, nor does it compare the fitting quality of other distributions.
  2. **The learned position prior is not sufficient to verify the syntactic interpretation. ** The objects in Figure 3 are position functions, not actual attention or syntactic dependency metrics after normalization. The concentration of shell radii at 1–2 does not prove that functional attention primarily captures syntactic relations at this distance range, nor does it rule out further dependencies. Ablation differences fixed at self-position weights of 0 or 1 do not exceed 0.1 BLEU, again insufficient to confirm the so-called zero-distance paradox (§§5.2–5.5).
  3. **The small experience gain lacks statistical support. ** The gain is only +0.2/+0.1 BLEU over sinusoidal encoding and +0.3/+0.2 over learnable ALiBi. Neither Table 1 nor text ablation has seed numbers, intervals, or significance tests, so stability is unclear. The +0.3–0.5 highlighted in the introduction does not cover all baseline comparisons.
  4. **Insufficient implementation details. ** Clarification is needed between the integer definition of radius and continuous learning; the handling of parameter initialization, constraints, bidirectional multi-shell implementation, and zero-weight logarithmization are not fully explained. The number of training steps, tokenization, decoding configuration and the complete sacreBLEU signature are also missing. §5.1 points to the experimental setup and §7 is actually the conclusion.
Questions for authors

See Weakness.

Reviewer_C5tTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new relative positional encoding (weighting framework) motivated by linguistic distance decay. AGF is mathematically identical to logarithmic KERPLE, with new features like shell boundaries, directional asymmetry, and separate weights for self-attention. Tests on two translation datasets (WMT16 and WMT17) show that these additions provide minor performance gains over existing baselines (sinusoidal, ALiBi and KERPLE)

Critical strengths
  1. The paper investigate an important question of how relative positional biases should be represented.

  2. experiments where done against relevant baselines such as ALiBi and KERPLE and included ablations tests.

Critical weaknesses
  1. some of the wording are over exaugurated and can be toned down, especially considering to the results (e.g. "This paper represents a novel ,ground breaking...").

  2. The actual results seems marginal and performance improvements are somewhat small (0.1-0.2 BLEU). confidence intervals are missing, its hard to tell if the improvements are noise. please consider adding the confidence interval or any other statistical significance tests.

  3. Minor novelty: AGF is logarithmic KERPLE as the paper itself states.

  4. The theoretical claims are not fully supported (they relay on loose analogies such as Zip's law), the authors need to provide either a clear mathematical proof or a reproducible experiment using real data.

Questions for authors
  • can you provide a clear definition of that is the PASL metric?
Reviewer_udq8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a prior on attention weights, g(d) = 1/(1+d/r)^k. The paper is an interpretation. An equivalent prior (as the authors note) is in Chi et al https://arxiv.org/abs/2205.09921

The authors run some experiments that appear to support the ideas. During review period, the author added more experiments.

The authors claim a connection between their prior and Newton's inverse-square law and Zipf's Law. [These claims were now weakened/withdrawn by the authors in the new version of the manuscript]

In the new version of the manuscript, the authors claim to derive an upper bound on their model based on Shannon Entropy. However, This claim appears to be based on an analogy rather than being a mathematical proof.

In response to my questions, the authors have withdrawn some previous claims.

During review period, the authors have revised the paper substantially. However, the paper still contains unjustified claims.

Critical strengths

The experiments, if expanded to confirm that the prior helps in multiple settings on average rather than hurts in much more diverse settings and with even stronger experiments than prior work such as Chi et al (2022), could improve the state of the art.

Critical weaknesses

In its current form, the theory is essentially not connected to the architectural proposal. There is obviosly no mechanistic connection at all between gravity and attention on a literal level, and the connection to Zipf's Law is similarly not mechanistic. [The claims in the new revised manuscript have been partially withdrawn]

The current version of the manuscript contains an unproven and underspecified claim about an upper bound.

The paper as currently written is quite difficult to read.

Questions for authors
  • Newton's Law and Coloumb's Law are not merely power laws, as you note in your manuscript. Taking the analogy seriously, the k in g(d) = 1/(1+d/r)^k would be related to the dimensionality of the "high-dimensional latent manifolds of transformer models" that you mention. And yet the k's you fit are quite small. Why? If the answer is that the distance is not actually measured on the manifold (that's by itself reasonable), what's the connection to the manifold? [update: the authors have removed claims about Newton's Laws, but not entirely]

  • "Zipf’s Law and Linguistic Economy As early as Zipf’s foundational statistical observations (Zipf, 1935), word frequencies across natural language corpora have been recognized to follow a power-law distribution governed by the principle of least effort. While Zipf’s Law addresses global, corpus-wide lexical distributions, our proposed PASL models localized, structure-specific event descriptions. Despite operating at distinct granularities, both mechanisms share an identical foundation: the minimization of communication and encoding costs. Consequently, it is theoretically grounded to model the spatial decay of PASL via a power-law distribution." This would imply that modelling any aspect of a language model using a power law is theoretically grounded. What is the limiting principle here?

Comments on the paper

2 comments

Hello, the new version of the paper seems very substantially different. It would be helpful if you could produce a point-by-point list of changes so that we can assess (1) whether and what concerns were addressed, what claims were withdrawn, etc and (2) whether this is a completely new paper

Hi,

I think we are asking not so much for a line-by-line diff as for a list of withdrawn old claims and new claims made.

New question:

You claim a "theoretical upper bound".

  1. What is the precise quantity on which you are claiming an upper bound?
  2. What is the proof that the exponential curve is an upper bound? I.e., what is the theoretical justification that that quantity couldn't in principle be super-exponential?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.