acceptodds
Under review as a conference paper at ICLR 2027

FasterWeight-MLP Attention: Sequence Modeling from a Neural-Network-Native Perspective

Abstract

A causal attention layer computes its scores with a network whose weights the context instantiates: a softmax head is a two-layer network whose hidden units and readout come from the prefix. We take that network as the design object and ask two questions: what may it read, and how much learned capacity does it carry? A query position and a key position split the causal history into three regions: the prefix the query sees, the prefix the key sees, and the interval between them. FasterWeight-MLP Attention (FW-MLP) reads each region with a small learned network, a fast MLP, and multiplies the three outputs into one score. Reading the interval by subtracting continuous prefix statistics leaves one exact form, a running additive sum, and with it FW-MLP evaluates the score exactly by a recurrence linear in sequence length. Linear attention, retention, and gated decays are exact special cases of the resulting score class, and delta-rule transports and the continuous scores on compact domains lie in its uniform closure. Under a matched attention-side parameter budget, FW-MLP attains the best mean rank among 12 mixers on 26 tasks and the best validation loss in a 123M-parameter nanoGPT setting. On a retrieval task gated by an interval statistic it reaches 34.3% accuracy, against 20.4% for its endpoint-only configuration, and it holds the best average downstream rank at both 340M and 1.3B parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sequence Modeling, Linear Attention, Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gRJPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work introduces a new linear-time sequence mixer, FasterWeight-MLP Attention. They evaluate this alternative attention layer on settings such as the NanoGPT benchmark and various commonsense reasoning tasks for 340M & 1.3B models pretrained on fineweb. This work also introduces a view of attention’s query-key interaction which is used to derive the proposed method.

Critical strengths
Critical weaknesses

This work appears to be almost entirely written by LLMs and is very difficult to read. The prose states a lot of buzz words, but does not actually convey clear information. For example, here is a topic sentence of an introductory paragraph: “Cost linear in sequence length constrains how the context may enter a fast MLP: only through statistics that carry forward in a running form.” This sentence structure is confusing and unnatural.

The math is not clear. Variables are not defined before they are used. I am unsure when an object is a function, a trainable parameter, etc., nor its dimensions. The surrounding text does not motivate or explain the point of many equations.

Questions for authors

Can you write the state update rules in a clear way (with dimensions clearly defined) all as a function of the tokens x1,x2,…,xt?x_1, x_2, …, x_t? Please denote which terms are trainable parameters vs. activations / online, fast weights.

What does st,pt,rts_t, p_t, r_t represent and how does this relate to softmax attention? The text of lines 140-143 connects this to Eq. 1, yet the choice of summing over tokens seemingly has no relationship to any of the above equations for softmax attention.

Reviewer_GtA1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, what is proposed is a linear-time attention architecture, dubbed FasterWeight-MLP (FW-MLP), which is motivated by viewing attention as a neural net whose weights are instantiated by the context. They parameterize their attention calculation by separately considering query-side prefix, key-side prefix, and the interval between the two positions. In fact, the authors provide some theoretical results characterizing the expressivity of the attention score.

In particular, the authors indeed compare their architecture to several existing attention mechanisms, such as softmax attention, Gated DeltaNet, GLA, and Mamba-3. They did experiments across 26 benchmark tasks, an interval-dependent retrieval task, and training nanoGPT and LMs at 340M and 1.3B parameters.

Empirically, they find that FW-MLP has the best average rank across the benchmark suite and the downstream LM evaluations.

Critical strengths
  • The central architecture contribution is interesting and mostly well-motivated, but as expressed in the weak points, writing quality could be improved. In any case, the idea is a natural extension of existing linear attention mechanisms in my opinion.
  • While I am not able to follow all of the proofs very closely, the theory does seem strong; as mentioned, they establish linear-time recurrence and discuss results on expressiveness of their score class, and relate it to other linear-attention architectures.
  • The ablations are extensive. There are numerous architectures and model scales, and ablations of the individual architectural components. There was also a notable ablation on the interval-gated retrieval, which is of course intertwined to the central claim of the paper.
Critical weaknesses
  • The writing and diagrams are not well-presented. The introduction in particular is written in a very confusing and circular way, with sentences winding around, and some grammatical rules such as use of colons are not followed well.
  • It is not clear to me that the theoretical results themselves are particularly general. E.g., the universal approximation results depend on sufficiently expressive branches and potentially increasing the number of heads, channels, and tracks. I do not know how much these results tell us something about the finite architecture actually implemented (but this may be a comment more generally about theoretical proofs for such things).
  • While the empirical results are promising, I don't think they have a striking advantage over the baseline. Looking at the nanoGPT validation loss, it only improves by 0.0048 compared to the Gated DeltaNet. Further, each config is trained only once. And, in the main benchmark suite, the improvement over FW-MLP-No$\Delta$ is not statistically significant (Table 10).
  • The architecture seems sensitivity to parameter budget. Appendix R.4 shows that when we reduce param-budget from 8d^2 to 4d^2, FW-MLP goes from 1st to 6th place. While this behavior is discussed by the authors, it raises questions about the practicality of their architecture. Also, as mentioned previously, the small budget results are only reported for a single seed.
  • The paper matches attention-side parameter counts, but I would have liked to see a more extensive comparison under matched training compute. In particular, given the additional complexity of the architecture, it is not entirely clear whether the performance improvements justify the additional computation.
Questions for authors
  • Re: weak point 2, to what extent do the universal approximation results apply to the finite architecture in the paper's experiments? What can we say about the expressivity when fixing number of heads and intervals?
  • What is the justification for assuming the cancellation through prefix subtraction which leads to the interval representation to be an additive potential?
  • Seeding. Were there experiments done at different seeds? How robust are the results to different seeding? Is nanoGPT always slightly improved over Gated DeltaNet?
  • If you match training FLOPs, how does FW-MLP perform relative to baselines?
Reviewer_NUpuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper views a causal attention layer as a network whose weights are instantiated by the context and proposes FasterWeight-MLP Attention (FW-MLP). For a query position t and a key position i, the score is the product of three factors: two endpoint "fast MLPs" that read the prefixes X_{1:t} and X_{1:i} (a first-stage linear-attention prefix statistic followed by a SwiGLU activation, used as the second-stage query and key), and an interval factor that reads X_{i+1:t} as a learned, x_t-weighted mixture of R exponential tracks over nonnegative additive potentials. Arguing from Cauchy's functional equation that this additive form is the one available for prefix subtraction, the authors derive an exact linear-time recurrence (Theorem G.1), show that linear attention, retention and gated decays are special cases, and give compact-domain density results. Experiments compare FW-MLP with 11 other mixers under a matched attention-side budget on a 26-task suite (VariantBench), an interval-gated retrieval diagnostic, a 123M nanoGPT setting, and 340M/1.3B language models, where FW-MLP obtains the best aggregate rank or loss; an endpoint-only ablation, FW-MLP-NoΔ, is included in the first three settings.

Critical strengths

Clear computational structure with an exact algebraic basis. The layer is built from explicit components (endpoint fast MLPs and an interval factor), and the core linear-time recurrence is derived exactly from the materialized tri-region score (Theorem G.1), with a worked scalar example and a numerical equivalence check (Table 26). This makes the method easy to analyze and to place relative to linear attention and gated decays.

Broad and transparent cross-mixer evaluation. The experiments cover 26 tasks across several domains, a nanoGPT setting, and language models at two scales, and the cross-mixer comparisons use a shared scaffold and a matched attention-side budget with verified parameter counts. The authors report bootstrap intervals, paired tests, and alternative aggregation rules, and release code and per-seed results, which made it possible to examine the results in detail.

Critical weaknesses

Training failures on MQAR-long. Adding the interval branch lowers MQAR-long accuracy from 93.5% (FW-MLP-NoΔ) to 32.4% at budget 8 and from 60.5% to 0.03% at budget 6 (Tables 8 and 17). The released per-seed results show that at budget 8 FW-MLP solves the task in one seed (0.973) and stays near chance in the other two (0.00025 and 0.000125), while FW-MLP-NoΔ is stable across seeds (0.93–0.94). The paper reports only the mean and does not discuss this. Since FW-MLP contains FW-MLP-NoΔ as a special case, this points to an optimization or numerical problem introduced by the branch rather than a capacity limit. Long-sequence recall is a central concern for linear-attention models, so this needs an explanation.

The ablations do not isolate the interval branch. Apart from the single-stage variant, all component ablations (Table 20) are run on FW-MLP-NoΔ rather than on the full model, and the components of the branch itself (input-dependent increments, multiple tracks, x_t-dependent readout) are never ablated. Moreover, in the released code FW-MLP-NoΔ is not endpoint-only: its second stage applies an input-dependent per-channel decay, and its first stage builds queries and keys from prefix averages rather than as in FW-MLP. The paper instead describes it as setting F^Δ ≡ 1 with the interval-generator parameters removed (Sections 1 and 3.1, Appendix S.1). The FW-MLP vs. FW-MLP-NoΔ comparison therefore changes several things at once, and its outcome is weak in any case: on VariantBench, FW-MLP ranks above FW-MLP-NoΔ on 11 tasks and below it on 8 (sign test p = 0.648, Table 10), and on the interval-gated diagnostic FW-MLP's 34.3% is about the 1/3 that key matching alone would give, since the three same-key candidates carry distinct values.

Scalability. Together, these results raise doubts about whether the interval branch remains beneficial as the recall load grows or as the parameter budget shrinks (at budget 4, FW-MLP places 6th of 12; Table 13). The 340M and 1.3B experiments do not resolve this, because they include no FW-MLP-NoΔ and no long-context evaluation, so the contribution of the branch at scale is not isolated.

Questions for authors

Why does FW-MLP stay near chance on MQAR-long in two of three seeds at budget 8 (and in the budget-6 run) while FW-MLP-NoΔ trains stably, and do the training curves show the loss diverging or never decreasing?

The released FW-MLP-NoΔ applies an input-dependent decay in its second stage and builds its first stage differently from FW-MLP. Was this the version used for the reported results, and if so, what does the FW-MLP vs. FW-MLP-NoΔ comparison isolate?

Which readout form, number of tracks R, and scale λ were used in each experiment, given that the released code uses query- and key-side track weights (1 + Wx)/√R without tanh and a default of R = 8, whereas Eq. 9 uses (1 + tanh(x_t W_ω))/R?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.