- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ALM, an attention-free architecture that replaces self-attention with a local bilinear mixer. Hidden states are first projected into a low-dimensional space, local token pairs are encoded using a symmetric bilinear map that replaces the Plücker embedding of Chong (2025), and the resulting features are gated back into the hidden states. The method is evaluated on WikiText-2 against a size-matched Transformer baseline, with reported validation perplexity improvements of approximately 23–28% and linear asymptotic complexity in sequence length.
Critical strengths
Bilinear, attention-free token mixing is a reasonable research direction, and the paper is candid that its predecessor, Chong (2025), did not improve perplexity. The authors provide runnable code in the supplementary material, which makes the implementation verifiable. I credit the authors for this.
Critical weaknesses
-
The proposed mixer is not causal: position (t) is mixed with future representations (z_{t+\Delta}), including (z_{t+1}), so the representation used to predict (x_{t+1}) can directly depend on the target token. The released implementation appears to realize the same future-looking computation, meaning the Table 1 perplexities cannot be interpreted as valid autoregressive language-modeling results. Fixing this requires rerunning the experiments with a strictly causal mixer, and the current formulation also leaves autoregressive generation at the final position unclear.
-
The claimed Grassmannian interpretation is also unsupported because the symmetric encoding (p_{ij}=u_i v_j+u_j v_i) is not invariant to a change of basis of the same subspace, so it does not define a representation on (Gr(2,r)). The ALM-vs-Plücker comparison is further confounded if the released baseline uses past-token interactions while ALM uses future-token interactions, so the experiment does not isolate the proposed encoding. Finally, the computational advantage is only argued asymptotically, with no runtime or memory measurements, while Table 1 reports only relative perplexity changes and several textual percentages are not transparently recoverable from the table.
-
The bibliography also raises reliability concerns: Kocon et al. (2019), cited as a reference for ChatGPT, is actually an unrelated CoNLL paper on sentiment analysis of Polish consumer reviews, and the bibliographic metadata reported for Gu et al. (2025) is incorrect.
Questions for authors
-
Were the Table 1 results produced using the released future-looking ALM implementation? If so, please rerun all experiments using a strictly causal ((z_t,z_{t-\Delta})) formulation and report absolute test perplexities over multiple seeds.
-
Does the proposed symmetric encoding satisfy any invariance property with respect to changes of basis of the underlying subspace? If not, what formally justifies interpreting it as a representation of (Gr(2,r))?
-
How is autoregressive generation performed under the stated formulation, given that the final position has no future token from which to construct its mixing feature?