- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a random-lag autoregressive (RLA) generative model in which each token is generated conditionally on a randomly selected predecessor. Under Gaussian emissions, the posterior over parent positions recovers causal softmax attention, with RoPE, positional biases, and key normalization arising under additional assumptions. Replacing Gaussian emissions with a linear directional distribution yields Spherical Linear Attention (SLA), with nonnegative normalized weights and a claimed constant-memory recurrence. At 340M parameters and 15B training tokens, SLA achieves competitive language-modeling performance.
Critical strengths
- Unified probabilistic interpretation. RLA provides a coherent perspective on attention, positional encodings, and gating. The Gaussian posterior and norm-preserving transition derivations appear mathematically valid under their assumptions.
- Principled linear-attention construction. SLA derives a nonnegative attention kernel from a probabilistic model rather than introducing it heuristically.
- Promising empirical results. SLA improves the six-task commonsense average over GLA (42.4 vs. 41.5) and achieves better perplexity
Critical weaknesses
- Limited novelty. Softmax attention has previously been derived as a Gaussian-mixture posterior (Nguyen et al., 2022), while nonnegative linear attention with positional reweighting is established (Qin et al., 2022). SLA's kernel, , is a first-order Taylor approximation. The contribution beyond existing probabilistic interpretations and linear-attention mechanisms is insufficiently established.
- Technical inconsistencies. Equations (19) and (21) define different operators. Writing for the strict-past posterior readout, Equation (19) gives whereas Equation (21) gives
which are generally unequal. Moreover, the recurrence in Equation (20) appears inconsistent with the lag weights in Equation (73). The paper also does not explain how the orthogonality of , required for nonnegative weights, is maintained during training. These issues undermine the claimed correspondence between the theoretical operator and its implementation. - Overstated theoretical conclusions. MSE optimality follows from the conditional-expectation identity for the specifically constructed latent readout; it does not establish optimality of softmax attention for language modeling. Furthermore, the flexible lag prior can absorb the key-energy term in Equation (6), making norm preservation and equal key norms sufficient but not necessary for recovering softmax attention. The proposed interpretation is therefore not unique.
- Insufficient experimental validation. The evaluation lacks component ablations, measured computational efficiency, uncertainty estimates, and scaling studies. Baseline results are imported from prior work, improvements are modest, and SLA substantially trails softmax attention on aggregate recall performance. It remains unclear whether the gains arise from the proposed formulation or established gating and linear-attention mechanisms.
Nguyen, T. M., et al. (2022). Improving Transformers with Probabilistic Attention Keys. Proceedings of the 39th International Conference on Machine Learning (ICML), PMLR 162, 16595–16621.
Questions for authors
- What is the precise theoretical and architectural novelty relative to Nguyen et al. (2022) and existing nonnegative linear-attention methods (Qin et al., 2022)?
- Can the authors reconcile Equations (19), (20), (21), and (73), identify which implementation was evaluated, and demonstrate numerical equivalence with the claimed posterior operator? How is orthogonality of enforced?
- Which theoretical assumptions are necessary rather than sufficient? How should the model-conditional MSE optimality result be interpreted beyond the constructed generative model?
- Can the authors provide controlled ablations of the kernel, gates, and positional transitions, together with measured efficiency and stronger long-context comparisons?
At minimum, the authors must resolve the mathematical and implementation inconsistencies, substantiate a meaningful contribution beyond prior work, and provide targeted ablations demonstrating that the proposed mechanisms, rather than existing architectural components, explain the observed improvements.