acceptodds
Under review as a conference paper at ICLR 2027

Learning to Remember with Möbius Updates

Abstract

Recurrent sequence models with matrix-valued hidden states provide fixed-size associative memory, but their update rules affect what information is kept. We study a family of linear-fractional memory updates that keep the state bounded in spectral norm, including affine updates and nonlinear updates induced by matrix Riccati dynamics. We give sufficient conditions for boundedness, analyze memory writing and retention, and derive a chunkwise form exactly equivalent to the sequential recurrence. Experiments on synthetic associative recall and state-tracking tasks examine accumulation, overwriting, and deletion, as well as generalization to longer sequences and increased memory load. Results show improvements over recurrent baselines in selected settings, with the relative performance of affine and nonlinear variants depending on the task and parameterization.

Primary area:
learning theory
Keywords:
sequence modeling, recurrent neural networks, associative memory, linear attention, Riccati equations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Xd1ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper study matrix memories for recurrent neural network and propose a non-linear update based on Mobius transformation. They show the proposed method has boundedness guarantees and distinguish from common affine-based updates. Empirical result show that the performance Mobius-based write outperform existing baselines.

Critical strengths
  1. Explored non-linear write to memory based on Mobius transformation, which is significantly different from common affine updates. Intuitively, this allow richer interactions between stored associations, and the paper provides a theoretical explanation of its expressivity relative to affine updates.

  2. Show theoretical boundedness guarantee and exact chunk-wise algorithm for efficient parallel update.

Critical weaknesses
  1. The paper is mathematically dense and poorly written. For example, in Eq. (1), it requires strong prior knowledge on Gated DeltaNet-2 architecture and Eq. (3) - Eq. (6) have a lot of notation not explained clearly. The intuition behind memory update, role of each variable and the headline motivation, the Mobius transformation, is not covered at all.

  2. The main result in Table 1 show that the gain is low, and the standard deviation are overlapping, the experiment scope is narrow and limited to Gated DeltaNet-2 only for the main experiment. Table 3 also shows that the runtime for Curved\mathrm{Curved} is significantly higher than Native\mathrm{Native}.

I believe by improving the clarity and explaining the background, this paper can be potentially very good.

Questions for authors
  1. Is it possible to repeat the main experiment on another model?

  2. Is it possible to have a section to define the task, problem and notation clearly? It is not clear to me the background, for example: what is the structure of Gated DeltaNet-2, and how it link to Section 3 motivation? The paper still have more than 1 page to the page limit (2 for post-rebuttal).

  3. The current Section 7 briefly list a huge array of prior works. Given the page limit is 9 pages and there is 1.5 page left, it is better that the author include in the main paper instead of leaving it in the appendix.

Reviewer_PjaQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a nonlinear memory update based on Mobius transformations for recurrent sequence models like Gated-DeltaNet. The authors also derive an exact chunkwise computation. The experiments mainly study chess state tracking and synthetic memory tasks.

Critical strengths

The proposed Mobius write supports exact chunkwise computation. This allows the nonlinear memory to be evaluated in parallel during training.

Critical weaknesses
  • The paper lacks clarity. Essential concepts and notation are used before they are explained, and the experiments are presented before the proposed update is defined. A preliminary section explaining the memory formulation, Mobius transformations, and essential notation would make the paper easier to follow.

  • The choice of Mobius writes is insufficiently motivated. The introduction jumps from studying bounded nonlinear updates to selecting Mobius transformations without explaining why this particular family stands out.

  • The reported gains are mainly demonstrated on chess state tracking and synthetic memory tasks. Although Appendix E includes language modeling results, they do not follow standard pretraining settings as those in Gated-DeltaNet.

  • The current implementation has substantial computational cost. Table 3 reports ~3x forward and backward time and ~5x the additional peak memory of native Gated DeltaNet-2. The benchmark also uses only one batch and one GPU. It remains unclear whether the accuracy gains justify this cost.

Questions for authors
  • Could the authors provide the necessary background setting, formulation, and notation in a preliminary section?

  • Why was this particular family of writes selected? Could the authors explain the construction of Eq (2) and provide a concrete memory example that motivates its design?

  • Could the authors provide broader and standard language-modeling evaluations, e.g. 300M, 700M language models with 6B-20B training tokens?

Reviewer_2FjEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies matrix memories whose write is a Möbius (linear-fractional) automorphism of a spectral ball, while the read stays linear. On Chess World Model (about 7.7M parameters, 128M training moves, three paired runs), the Möbius write (Curved) is compared with native Gated DeltaNet-2 and its matched affine Tangent. The results show some improvement.

Critical strengths
  • The theoretical aspects are interesting, but as they are heavily influenced by frontier AI, I would say it is nice to have rather than major focal point of interest.
  • The algorithm is meaningful, and well-characterised.
Critical weaknesses
  • The central question is posed but not answered: does the nonlinear write provide computation that an equal-cost affine memory lacks?
  • Limited and inconsistent empirical benefits -- the evidence does not establish a broadly useful advantage over affine memory updates.
  • There are gaps between theories and experiments.
Questions for authors
  • The 3.61× time and 5.14× memory cost is measured against Native's FLA kernel. Tangent through the same generic LU costs 51.44 ms against 51.95 ms for Curved, and 43.46 ms with the triangular solver (Table 3). Is it correct that the curved write's intrinsic overhead over the tangent is about 1.2×, and that most of the 3.6× is the prototype dense solver? This changes how expensive the method really is.
  • How do you reconcile the consistent Board64-over-Tangent gain at 128M moves with the mixed results of the independent 32M study (Fig. 6)? Is the effect expected to grow with training, and is there evidence for that?
  • During training, how close does the normalized state come to the ball boundary? Is the pivot 1 + g uᵀXk bounded away from zero in practice? Appendix C.1's example shows cond(H) can grow exponentially without an interior margin.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.