acceptodds
Under review as a conference paper at ICLR 2027

An Attention Free Alternative to Language Modeling

Abstract

Most of the recent Transformer models have attention mechanism as a key component. Attention mechanism is the de-facto standard of transformer model as it enhances performance significantly. Nevertheless, it is well-established that attention mechanisms entail significant computational expenses. This places a restriction on model and input size when computing is limited. Additional limitations of attention mechanisms include the attention sink phenomenon. The primary goal of language model is to have efficient sentence representation. In attention mechanism this goal is achieved through similarity computation of tokens based on a starting representation, which is subsequently refined across different levels. We employ an algebraic approach. Tokens are mapped to a lower dimension first. Token pairs are subsequently generated based on the specified attention window size. The two-dimensional space spanned by the token pairs are encoded through a similarity based encoding approach. Our proposed method, the Algebraic language model (ALM), was evaluated using Wikitext2 data. Our experimental results using a laptop demonstrate improvement of the perplexity score by more than 25% in ALM relative to conventional Transformer models. Beside, ALM has substantially less computational complexity compared to Transformer models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Representation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rupXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ALM, an attention-free architecture that replaces self-attention with a local bilinear mixer. Hidden states are first projected into a low-dimensional space, local token pairs are encoded using a symmetric bilinear map that replaces the Plücker embedding of Chong (2025), and the resulting features are gated back into the hidden states. The method is evaluated on WikiText-2 against a size-matched Transformer baseline, with reported validation perplexity improvements of approximately 23–28% and linear asymptotic complexity in sequence length.

Critical strengths

Bilinear, attention-free token mixing is a reasonable research direction, and the paper is candid that its predecessor, Chong (2025), did not improve perplexity. The authors provide runnable code in the supplementary material, which makes the implementation verifiable. I credit the authors for this.

Critical weaknesses
  1. The proposed mixer is not causal: position (t) is mixed with future representations (z_{t+\Delta}), including (z_{t+1}), so the representation used to predict (x_{t+1}) can directly depend on the target token. The released implementation appears to realize the same future-looking computation, meaning the Table 1 perplexities cannot be interpreted as valid autoregressive language-modeling results. Fixing this requires rerunning the experiments with a strictly causal mixer, and the current formulation also leaves autoregressive generation at the final position unclear.

  2. The claimed Grassmannian interpretation is also unsupported because the symmetric encoding (p_{ij}=u_i v_j+u_j v_i) is not invariant to a change of basis of the same subspace, so it does not define a representation on (Gr(2,r)). The ALM-vs-Plücker comparison is further confounded if the released baseline uses past-token interactions while ALM uses future-token interactions, so the experiment does not isolate the proposed encoding. Finally, the computational advantage is only argued asymptotically, with no runtime or memory measurements, while Table 1 reports only relative perplexity changes and several textual percentages are not transparently recoverable from the table.

  3. The bibliography also raises reliability concerns: Kocon et al. (2019), cited as a reference for ChatGPT, is actually an unrelated CoNLL paper on sentiment analysis of Polish consumer reviews, and the bibliographic metadata reported for Gu et al. (2025) is incorrect.

Questions for authors
  1. Were the Table 1 results produced using the released future-looking ALM implementation? If so, please rerun all experiments using a strictly causal ((z_t,z_{t-\Delta})) formulation and report absolute test perplexities over multiple seeds.

  2. Does the proposed symmetric encoding satisfy any invariance property with respect to changes of basis of the underlying subspace? If not, what formally justifies interpreting it as a representation of (Gr(2,r))?

  3. How is autoregressive generation performed under the stated formulation, given that the final position has no future token from which to construct its mixing feature?

Reviewer_HeJ5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an Algebraic Language Model (ALM) as an alternative to self-attention. The model projects token representations into a lower-dimensional space. It then constructs local token pairs and encodes them with symmetric bilinear features. These features are projected back to the model dimension and combined with the original representations through a gating mechanism. The paper reports a 23–28% reduction in validation perplexity over a Transformer baseline on WikiText-2. It also presents a complexity analysis that shows linear scaling with sequence length for fixed model and window settings.

Critical strengths

The proposed mixing block has a simple structure. It uses a fixed set of token offsets and avoids constructing a full attention matrix. Its computational cost scales linearly with sequence length under fixed model and window settings. This property provides a plausible motivation for further study in resource-constrained language modeling.

Critical weaknesses

1.The method has unresolved correctness issues. Section 3.2 uses future tokens (z_{t+\Delta}) to update the representation at position (t), but describes this operation as causal. Under standard next-token prediction, this introduces information leakage and invalidates the perplexity comparison. The paper does not explain whether the implementation follows a different indexing convention. The symmetric encoding also changes under a change of basis within the same subspace. It therefore does not define a Grassmannian embedding as claimed. These issues undermine the validity of the method and its geometric interpretation.

2.The experiments do not establish the claimed performance and efficiency advantages. Table 1 reports only relative changes in validation perplexity. Absolute perplexities, test results, and variation across runs are missing. The evaluation covers only WikiText-2 and two small ALM configurations. It includes no comparisons with efficient attention methods, attention-free models, or local attention with a comparable context window. It also provides no runtime or memory measurements. The reported results are therefore insufficient to demonstrate a reliable advantage over existing methods.

Questions for authors

1.Does the implementation pair each position with future tokens, as stated in Section 3.2? Could the authors specify the input–target alignment and explain how the model prevents future-token access during both training and evaluation?

2.Is the symmetric encoding intended to represent a subspace or a particular pair of vectors? How does the claimed Grassmannian interpretation account for changes of basis within the same subspace?

3.What are the absolute validation and test perplexities for all models in Table 1? What evidence supports the reported gains over efficient attention alternatives under comparable training settings and the claim of practical computational savings?

Reviewer_jykdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an attention-free language model. Each attention block is replaced with a causal algebraic mixing layer. Hidden states are first projected to a low dimension, and each position is paired with positions 1, 2, 4, 8, 12 and 16 steps later. For each pair, the layer builds a symmetric pairwise product feature, described as a similarity-based alternative to Plücker coordinates. The feature is normalized, projected back to the model dimension, averaged over offsets with weights inversely proportional to distance, gated with the original hidden state, and followed by an FFN. On WikiText-2 with ~7M-parameter models trained on a laptop, they report 23–28% lower validation perplexity than a Transformer baseline and a Grassmann-flow baseline.

Critical strengths

The question of whether a cheap local geometric mixer can stand in for attention is reasonable, and the layer is simple.

Critical weaknesses
  1. The method may leak future tokens. Sec. 3.2 pairs position t with positions after t. For next-token prediction this means the model can see its target. If the code does this, the perplexity numbers are meaningless.

  2. The experiments are too thin to support the claims: one small dataset, one ~7M scale, one seed, only relative perplexity changes, no test set, and no efficiency measurements. Obvious baselines are missing (sliding-window attention, causal convolution, Mamba/RWKV).

Questions for authors
  1. Exactly which positions does the representation at a given position depend on? Please show that changing later tokens leaves the logits at that position unchanged.

  2. Please give absolute validation and test perplexities for all models, over several seeds, with the Transformer's hyperparameters.

  3. Please compare against sliding-window attention and a causal convolution with the same receptive field and parameter count.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.