acceptodds
Under review as a conference paper at ICLR 2027

Transformers as Random-Lag Autoregressive Models

Abstract

In this paper, we consider a probabilistic generative model approach to explain the Transformer architecture and leads to improved new architectures. We make a minor modification to the classical autoregressive model by replacing its fixed unit lag with a random lag, and prove that the exact posterior mean under the resulting random-lag autoregressive model (RLA) is precisely causal Softmax attention. As a conditional expectation, this attention operator is the unique mean-square-optimal estimator of the latent value readout given the observed prefix. In RLA, all the standard transformer components are endowed with a generative role, such as RoPE, ALiBi positional encodings, and query-key normalization (QK-Norm). By replacing the Gaussian assumption of RLA with a linear directional family on the sphere and again computing the exact posterior mean, we obtain a \em spherical linear attention (SLA). Unlike existing linear attention mechanisms and similar to softmax attention, SLA has normalized and nonnegative attention weights. The experiments show the competitive performance of SLA.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
mixture transition distribution, random lag autoregressive models, spherical linear attention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_P2oiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a random-lag autoregressive (RLA) generative model in which each token is generated conditionally on a randomly selected predecessor. Under Gaussian emissions, the posterior over parent positions recovers causal softmax attention, with RoPE, positional biases, and key normalization arising under additional assumptions. Replacing Gaussian emissions with a linear directional distribution yields Spherical Linear Attention (SLA), with nonnegative normalized weights and a claimed constant-memory recurrence. At 340M parameters and 15B training tokens, SLA achieves competitive language-modeling performance.

Critical strengths
  • Unified probabilistic interpretation. RLA provides a coherent perspective on attention, positional encodings, and gating. The Gaussian posterior and norm-preserving transition derivations appear mathematically valid under their assumptions.
  • Principled linear-attention construction. SLA derives a nonnegative attention kernel from a probabilistic model rather than introducing it heuristically.
  • Promising empirical results. SLA improves the six-task commonsense average over GLA (42.4 vs. 41.5) and achieves better perplexity
Critical weaknesses
  1. Limited novelty. Softmax attention has previously been derived as a Gaussian-mixture posterior (Nguyen et al., 2022), while nonnegative linear attention with positional reweighting is established (Qin et al., 2022). SLA's kernel, 1+⟨q^,k^⟩1+\langle\hat q,\hat k\rangle, is a first-order Taylor approximation. The contribution beyond existing probabilistic interpretations and linear-attention mechanisms is insufficiently established.
  2. Technical inconsistencies. Equations (19) and (21) define different operators. Writing A/BA/B for the strict-past posterior readout, Equation (19) gives (1−η)AB+ηv,(1-\eta)\frac{A}{B}+\eta v, whereas Equation (21) gives A+η(1+c)vB+η(1+c),\frac{A+\eta(1+c)v}{B+\eta(1+c)},
    which are generally unequal. Moreover, the recurrence in Equation (20) appears inconsistent with the lag weights in Equation (73). The paper also does not explain how the orthogonality of Θ\Theta, required for nonnegative weights, is maintained during training. These issues undermine the claimed correspondence between the theoretical operator and its implementation.
  3. Overstated theoretical conclusions. MSE optimality follows from the conditional-expectation identity for the specifically constructed latent readout; it does not establish optimality of softmax attention for language modeling. Furthermore, the flexible lag prior can absorb the key-energy term in Equation (6), making norm preservation and equal key norms sufficient but not necessary for recovering softmax attention. The proposed interpretation is therefore not unique.
  4. Insufficient experimental validation. The evaluation lacks component ablations, measured computational efficiency, uncertainty estimates, and scaling studies. Baseline results are imported from prior work, improvements are modest, and SLA substantially trails softmax attention on aggregate recall performance. It remains unclear whether the gains arise from the proposed formulation or established gating and linear-attention mechanisms.

Nguyen, T. M., et al. (2022). Improving Transformers with Probabilistic Attention Keys. Proceedings of the 39th International Conference on Machine Learning (ICML), PMLR 162, 16595–16621.

Questions for authors
  1. What is the precise theoretical and architectural novelty relative to Nguyen et al. (2022) and existing nonnegative linear-attention methods (Qin et al., 2022)?
  2. Can the authors reconcile Equations (19), (20), (21), and (73), identify which implementation was evaluated, and demonstrate numerical equivalence with the claimed posterior operator? How is orthogonality of Θ\Theta enforced?
  3. Which theoretical assumptions are necessary rather than sufficient? How should the model-conditional MSE optimality result be interpreted beyond the constructed generative model?
  4. Can the authors provide controlled ablations of the kernel, gates, and positional transitions, together with measured efficiency and stronger long-context comparisons?

At minimum, the authors must resolve the mathematical and implementation inconsistencies, substantiate a meaningful contribution beyond prior work, and provide targeted ablations demonstrating that the proposed mechanisms, rather than existing architectural components, explain the observed improvements.

Reviewer_nStoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explains causal softmax attention by showing that it is the best possible guess (the posterior mean) under the following probabilistic process. Each token is produced by picking one earlier token at random (the “parent”), rotating it once per step of distance (all keys are assumed to have same norm), and adding Gaussian noise. Attention guesses which earlier token was the parent and reports that parent’s expected value.

For example, at token #4 suppose the candidates #1, #2, #3 get scores (log-prior + <q,k>) of −1.5, 1.0 and 1.0. The softmax probabilities are then 0.04, 0.48 and 0.48. With values 10, 20, 30, the output is ~24.4, which is exactly what a standard attention head computes.

Within this generative framework:

  • <q,k> measures how well a candidate explains the current token
  • ALiBi and forget gates act as the prior over distance
  • RoPE is the length-preserving rotation applied per step
  • key normalization cancels a key-norm penalty that would otherwise appear in the score.

Authors show that swapping the Gaussian for a "linear directional" distribution gives Spherical Linear Attention (SLA). SLA's weights are prior_j x (1 + cosine-sim(q, rotated k_j)) (matching 1st-order approximation exp(<q,k>) ~ 1+<q,k>), which can be computed as an RNN similar to linear attention variants. Authors support their claims with proofs. Experiments consists of a 340M-parameter run on 15B tokens, compared on 6 commonsense tasks and 3 recall tasks against baseline numbers copied from the GLA paper.

Critical strengths
  1. The framework proposed in the paper acts as a unifying simple vocabulary for many attention components. The derivations are short and the framing is useful for reasoning about design choices in Transformers.
  2. SLA is simple and has a recurrent form similar to linear attention. Its weights are non-negative and normalized and it uses constant memory. Reported numbers are promising: Wiki perplexity 26.05 vs 28.65 for GLA, and a 6-task average of 42.4 vs 41.5.
Critical weaknesses
  1. The experimental gains are modest and not reliable. Baseline numbers are copied from GLA paper that trained with LR 3e-4, while SLA uses a higher LR of 1e-2. The 0.9-point average gain on benchmarks is modest. No comparison with DeltaNet, Gated DeltaNet, Mamba-2, no speed/memory measurements. SLA is also close to existing designs: it is essentially normalized linear attention with a decay gate. Forget gates for softmax attention also appear in the Forgetting Transformer (ICLR 2025).

  2. The main theorem is largely known, and its optimality claim holds by construction. Treating attention weights as the posterior of a Gaussian mixture was shown before by Movellan & Gabbur (2020), Gabbur et al (NeurIPS 2021) and Nguyen et al (ICML 2022), not cited. Any posterior mean is optimal under its own model. Here the model's parameters are simply set equal to the network's weights rather than fitted (Eq 16), so the theory places no restriction on trained networks. Because the prior may depend on past tokens, the key-norm penalty can also be absorbed into it. Attention without key normalization therefore fits the story too, which weakens the claims that key normalization is "necessary" and that RoPE is "forced". Likewise, the self-weight is defined to match attention (Eq 40), even though the model never allows a token to be its own parent.

  3. The implemented SLA does not match the theory. In Eq 19, the update gate is the share of the output taken from the current token. In the recurrent form used in practice (Eq 21), the update gate instead weights an unnormalized self-score, so the actual share is different.

Questions for authors
  1. Since the prior may depend on past tokens, can the key-norm term in Eq 6 be absorbed into it, making attention without key normalization an exact posterior too? If so, what does the framework rule out?

  2. Were the reported results produced with Eq 19 or Eq 21? What is c_nn?

  3. Could you train SLA and GLA (ideally also Gated DeltaNet and Mamba-2) with the same recipe and add ablations?

  4. Discuss how this work relates to the citations included in weaknesses section above.

Reviewer_Dfb6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a generative model ("Random-lag autoregressive") in which each token representation is generated from a single earlier position (its latent "parent"), considering a lag prior and transitions as Θlag\Theta^\text{lag}. Under a Gaussian likelihood, identic key norms, and isometric transitions, the posterior over the parent recovers the softmax attention, with the lag prior acting as an additive positional bias and the transition enacting RoPE. The authors build a value readout with a self-weight that makes causal attention the posterior mean. Replacing the Gaussian likelihood with a linear directional family yields spherical linear attention (SLA), whose weights are linear in score and admit a constant-memory recurrence. SLA is evaluated on language pretraining and retrieval tasks.

Critical strengths

The derivation of the posterior provides a unified interpretation query-key scoring, lag priors, and positional transformations. Changing the likelihood family to obtain a linear-attention variant is an interesting use of the framework.

Critical weaknesses
  • Motivation and intuition. Overall, for a paper aiming for explanatory power, it is quite hard to follow and gives limited human intuition on why attention should be modeled as inferring a single parent position. Under this model, a head that aggregates many tokens appears to be highly uncertain about which single token is the parent; such an insight should be more thoroughly discussed and illustrated with trained models. Furthermore, any normalized nonnegative weighting can be written as a posterior. Therefore, the notion of "optimality" is only relative to the specific settings described by the chosen model, making the explanations quite circular.

  • Prior work. Two closely related works are not cited. Xu et al. (ICML 2015, "Show, Attend, and Tell", §4) treats the attention location as a latent variable and recovers soft attention. Nguyen et al. (ICML 2022, “Improving Transformers with Probabilistic Attention Keys”, §2.1) model the query as drawn from a Gaussian mixture centered on keys and also recovers the softmax attention scores, under assumptions of normalized queries and keys and uniform prior. The paper should discuss these works and states what RLA adds.

  • Experimental support. The baseline numbers are copied from the GLA paper and SLA results are from a single run (no seeds). Architecture details are not given, and the learning rate (1e-2) is unusually high and differs from the hyperparameters of the baselines. There are no ablations (eg: choice of kernel, rotations and gates): the claim that gates "also improve softmax attention" is unsupported. The results are mixed; and unlike in the remark SLA does not have the best LMB in the subquadratic family: Mamba 39.66 is lower than SLA 41.15. The constant-memory recurrent form is highlighted but the experimental tests and measures have not been reported. Some results being incorrectly bolded or not highlighted raises concerns about the reliability and integrity of the reported results.

Questions for authors

How does your generative model and your interpretation of attention differ from Xu et al. and Nguyen et al.? Can causal attention that also includes current token be derived from a unified mixture model, rather than introduced post-hoc via the self-weight? How well is the likelihood model supported by actual models? For instance, does the likelihood of a layer's input activations increase during training? Could you make the experiments more conclusive and significant? For instance, by reporting several seeds with variance, giving full architecture details, and reporting throughput and memory for the recurrent form? Could you run ablations to identify the role of the different suggested mechanisms (key normalization, score kernel, transitions, gates)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.