- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SPLA, which replaces the (λI + KᵀK)⁻¹ query preconditioning used in MesaNet/GKA with a learned polynomial of a gated, Frobenius-normalized Gram matrix of keys. The authors derive chunk-wise forward and backward passes, implement a kernel, and evaluate at 340M and 1.3B scale.
Critical strengths
The method is simple, noticeably cheaper than GKA, and clearly improves over its GLA backbone (though the GLA numbers come from released weights; see also the head-dim caveat below). The explicit backward derivation and kernel are useful for people working on efficient linear attention.
Critical weaknesses
My main concern is novelty. Once the coefficients are learned, the layer is essentially a0·q + a1·qH̃ + a2·qH̃², and the Taylor motivation does little work, the series for (λ + x)^(−θ) converges only for |x| < λ, so normalization guarantees convergence only if λ > 1, which is never specified. The results also don't really support the claims. For example, at 1.3B, SPLA is behind GDN on CSR and behind GKA on recall, the differences are under a point, and no variance across seeds is reported. The paper emphasizes long-range memory, but everything is trained and evaluated at about 2048 tokens, so long-context evaluation (NIAH, RULER, extrapolation) seems necessary. SPLA also uses head dim 128 vs. 256 for most baselines, and the Muon result is reported only for SPLA, so it says little without baselines.
Questions for authors
Table 3's four-term row matches the main 340M results exactly, while Section 3.2 describes three terms. Which variant is used in the main tables? What is λ? How does SPLA do at matched head dimension? Are there any results beyond the training length?