acceptodds
Withdrawn from ICLR 2027

Spectral Polynomial Linear Attention (SPLA): polynomial singular-value maps for gated linear models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Linear models have gained widespread attention as a viable alternative to reduce the cost of full-attention Transformers. However, due to their fixed-size state space, the spectral behavior of the state space directly determines their performance. Recent work has proposed that their dominant components cause the effective rank of the state to collapse, making it difficult to maintain long-range memory capacity. We draw on the ideas of the Muon optimizer and focus on recent work such as MesaNet, finding that it essentially regulates the key-value matrix of all past states. We argue that a broader singular value spectrum of the keys and values may provide the model with more choices. Based on this, we propose Spectral Polynomial Linear Attention (SPLA). The architecture we propose provides an alternative way of using the state of linear models through high-order polynomial approximation function transformations. We further conduct experiments on different tasks, and the results show that our method effectively preserves the long-term recall capacity of linear models while maintaining good inference speed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, optimization theory, SVD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5rjHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPLA, which replaces the (λI + KᵀK)⁻¹ query preconditioning used in MesaNet/GKA with a learned polynomial of a gated, Frobenius-normalized Gram matrix of keys. The authors derive chunk-wise forward and backward passes, implement a kernel, and evaluate at 340M and 1.3B scale.

Critical strengths

The method is simple, noticeably cheaper than GKA, and clearly improves over its GLA backbone (though the GLA numbers come from released weights; see also the head-dim caveat below). The explicit backward derivation and kernel are useful for people working on efficient linear attention.

Critical weaknesses

My main concern is novelty. Once the coefficients are learned, the layer is essentially a0·q + a1·qH̃ + a2·qH̃², and the Taylor motivation does little work, the series for (λ + x)^(−θ) converges only for |x| < λ, so normalization guarantees convergence only if λ > 1, which is never specified. The results also don't really support the claims. For example, at 1.3B, SPLA is behind GDN on CSR and behind GKA on recall, the differences are under a point, and no variance across seeds is reported. The paper emphasizes long-range memory, but everything is trained and evaluated at about 2048 tokens, so long-context evaluation (NIAH, RULER, extrapolation) seems necessary. SPLA also uses head dim 128 vs. 256 for most baselines, and the Muon result is reported only for SPLA, so it says little without baselines.

Questions for authors

Table 3's four-term row matches the main 340M results exactly, while Section 3.2 describes three terms. Which variant is used in the main tables? What is λ? How does SPLA do at matched head dimension? Are there any results beyond the training length?

Reviewer_bJwkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the preconditioner derived from MesaNet style test-time regression reconstruction error by approximating a generalized variant (eq 9) with a three-term Taylor expansion with learnable coefficients. The authors introduce a chunk-based parallel forward and backward pass algorithm to calculate the target recurrence. At the 1B scale, the SPLA architecture falls short of the Gated DeltaNet baseline by ~0.8 percentage point on average for common-sense reasoning but achieving a better perplexity on WikiText (but not Lambada). On retrieval-based tasks, SPLA performs better than GDN but performs slightly worse than GKA on average. SPLA also requires more compute than GDN (leading to higher latency) but is quicker than GKA.

Critical strengths

I found the overall approximation of the preconditioner intuitive and an interesting alternative to current inverse solver methods, e.g., ones used in GKA. The experimental setup is at a scale which gives reasonable signal about the model, and both language modeling and recall tasks help give a useful picture of the strengths and weaknesses of SPLA. While the actual layer itself might not be a strict improvement upon prior baselines, i.e., GDN, the approximation of the inverse and its potential stability may prove helpful for the community.

Critical weaknesses

It does seem like the overall layer lags a bit behind in language modeling but has strengths in retrieval. However, the field has generally settled on using hybrid models as a way to supplement the retrieval-weaknesses of linear models. How does SPLA perform in this paradigm? Do the retrieval gains still hold over current baselines, and how do the language modeling performances stack?

The latency and computation cost of SPLA is higher than that of GDN despite not being strictly better than it. It seems like SPLA sits in the middle of GDN and GKA as an efficiency-performance tradeoff. Are there any other potential benefits of using SPLA over baselines, i.e., better length extrapolation, long-context modeling, etc? On the other hand, would reducing the base three-term expansion to two-terms improve latency and cost without degrading overall performance? It seems like currently, the overall long-term memory motivation is not directly addressed via the current experiments.

Questions for authors

How does SPLA fare in a hybrid-style architecture? For instance, 3:1 linear to global attention regime.

Reporting the heldout test loss along with the WikiText and Lambada perplexity would be helpful.

Do the 1.3B experiments with SPLA use the three or four term expansion? The 340M experiments in table 6 seem to reflect the four term numbers (table 3).

Please also clarify how the latency numbers were measured.

Reviewer_5obhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spectral Polynomial Linear Attention (SPLA), which preconditions queries using a learned polynomial of a gated key Gram matrix before applying a GLA-type method. Motivated by a spectral interpretation of regression-based linear models, the method learns different transformations across heads and layers. The authors derive efficient chunk-wise forward and backward computations and evaluate 340M and 1.3B models on language modeling, commonsense reasoning, and recall tasks. SPLA improves over GLA and approaches GKA’s quality at lower reported single-layer latency, while results relative to GatedDeltaNet are mixed.

Critical strengths
  • The spectral interpretation offers a useful perspective on regression-based linear attention and motivates learning different transformations across heads and layers.
  • The method provides chunk-wise forward and backward computations that avoid storing a full Gram matrix at every token.
  • SPLA improves quality over GLA and approaches GKA’s performance with lower reported single-layer latency, offering a potentially useful quality–cost trade-off.
Critical weaknesses
  • W1: The practical advance over strong alternatives is limited. At 1.3B, SPLA trails GatedDeltaNet on commonsense average (50.93 versus 51.76) and improves recall average by only 0.50 points. These mixed gains leave the significance of the new formulation unclear.
  • W2: The evaluation does not convincingly establish that spectral balancing causes the recall gains. A capacity-matched control is missing to separate this effect from changes in recurrent state and head configuration.
  • W3: Learning the coefficients gives small improvements on some metrics and worsens others. Without results from repeated training runs, it is unclear whether these differences are consistent.
  • W4: The paper should specify the fixed Taylor baseline’s λ and θ, how the query-interpolation gate is computed and initialized, and whether the main experiments and timings use three or four polynomial terms
  • W5: Single-layer timings do not establish meaningful end-to-end savings, although efficiency is a principal justification relative to GKA.
  • W6: Evaluation stops at 1.3B, with detailed ablations at 340M. Whether the modest gains and computational trade-offs persist at larger scales remains unestablished.
Questions for authors
  1. Are the negative responses in Figure 2 intentional? How do they relate to the positive inverse-power function motivating the method?
  2. How were the representative heads in Figures 2–3 selected, and how common are these behaviors across the model?
  3. Does learned query gating improve over a fixed mixture or always using the preconditioned query?
  4. Could you clarify how the small additive ε in the Frobenius normalization is handled in the backward derivation and implementation? Is its effect omitted in the displayed formulas for simplicity?

Minor comments:

  • Briefly define MesaNet in the abstract and distinguish its existing formulation from this paper’s spectral interpretation, so readers can identify the contribution without already knowing the prior work.
  • Give a high-level computational overview in the main text and move detailed chunk-wise algebra to the appendix. The current presentation makes the core mechanism and its benefits harder to follow.
  • Correct the Qwen reference: the cited original Qwen3 report describes conventional attention, whereas Qwen3-Next uses a GatedDeltaNet–attention hybrid. The current citation therefore does not support the adoption claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.