- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LEMA binarizes query and key vectors and retrieves the value at the latest exactly matching key, allowing dictionary-based lookup. On the theory side, LEMA uses a more restrictive construction than prior recency-based hard attention and shows simulations in both directions: word-RAMs can simulate LEMA transformers and LEMA transformers can simulate word-RAMs. On the empirical side, experiments with models of up to 834M parameters show positive results on the reported benchmarks and in inference performance. However, LEMA’s broader advantage is less convincing. More details are provided below.
Critical strengths
- Extends the prior word-RAM connection for recency-based hard attention to the more restrictive LEMA rule and establishes simulations in both directions.
- Shows competitive long-range recall against GDN on selected tasks with competitive compute.
- Demonstrates strong length extrapolation on the synthetic associative-recall task.
Critical weaknesses
- The word-RAM results do not explain how the choice of backward surrogate affects learning. The paper studies the effect of capping the backward sharpening parameter, but a small ablation across different surrogate derivatives, like sigmoid, would help establish whether the training result depends on the particular choice of backward pass function.
- The comparisons omit several relevant attention baselines, including stick-breaking attention beyond the limited ablation, path attention (https://arxiv.org/pdf/2505.16381), which also reports targeted recall experiments, and other related improved mechanisms like GDN-2 (https://arxiv.org/pdf/2605.22791).
- The empirical case relies heavily on selected recall tasks. The harder multi-key retrieval results in the appendix are less favorable to LEMA and deserve more discussion; standard downstream language-modeling benchmarks, like TinyBenchmark, would also help assess the quality cost of exact-match attention.
Questions for authors
No specific question apart from the concerns mentioned in the weaknesses.