acceptodds
Under review as a conference paper at ICLR 2027

Implicit Bias of Optimizers Guides Length Generalization in Forgetting Attention

Abstract

Autoregressive language models such as transformers process variable-length inputs, yet whether a model trained on bounded-length inputs stays accurate on much longer ones (or length generalize), is known to depend heavily on design choices. We approach it from the optimization perspective, through a task in which a sequence of key-value pairs may reassign a key and a query asks for its key's latest value. Repeated keys force the model to discount older records by their distance from the query, yet a recency bias alone cannot tell a stale record from a correct value that merely sits far back behind newer records of other keys. We analyze this tension in a two-layer attention model with fixed attention masks and value projections, which learns key-value binding, query-key matching, a forget gate, and a linear decoder from binary answer labels. Our main result separates gradient descent (GD) from Adam on this model under suitable per-parameter-group learning rates. From a shared orthogonal initialization of the query and key embeddings with zero initial content logits, GD with a single schedule succeeds whenever the target sits at most two records further back than any training target, and at each greater distance it fails on a fixed sequence. Adam, with its stabilizer set to zero on the scalar parameters and positive on the embedding matrices, instead guarantees a retrieval distance proportional to its cumulative learning rate, which diverges. Both guarantees hold for an arbitrary prefix of earlier records. Adam's guarantee extends to independent Gaussian initialization. SGD inherits GD's cutoff with increasing batch sizes and exact averages over all key pairs. Controlled experiments compare finite retrieval ranges and isolate the role of the scalar stabilizer .

Primary area:
optimization
Keywords:
length generalization, implicit bias, Adam, attention, training dynamics, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HGWROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper under review studies the length generalization problem in transformers. The authors study this problem from the lens of the optimizer essentially seeking to understand whether the optimizer itself brings an implicit bias to how the transformer performs on length generalization. On a particular transformer model the authors show that both GD and Adam bring a different bias for length generalization.

Critical strengths

The authors study an interesting problem for transformers from a very nice lens, namely optimizer bias. I do feel this is a very novel lens to try and understand how transformers perform on length generalization and how the optimizer affects that generalization performance.

Critical weaknesses

I did find the paper quite difficult to read especially to understand the point the authors were giving with their paper. Given that Adam and GD bring a different bias to this problem, which I think people working in the field would have guessed, what does that say about how we should be optimizing such models for problems like length generalization? No real effort was made in bringing deeper insight. While I admire the honesty of the authors in their AI statement in saying that Codex and Claude helped with various aspect of the work, I think it is therefore imperative that the authors put significant time into expositing the work well. I feel this was not the case.

Questions for authors
  1. Does your work tell us anything about how we should design optimizers for such models for good performance for problems like length generalization? What is the key insight this tells us about how we should optimizer transformers for such problems.

  2. I understand the approach you took was to understand the situation for a 2-layer transformer. Is it possible to carry out some experiments for deeper transformers and at least show empirically what is happening in that case?

Reviewer_yYTKOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the choice of optimizer affects the ability of a small attention model to learn the latest value of the retrieved key, given a fixed architecture, data, and objective function. The results show that gradient descent is only suitable for fixed context distances after training, while the Adam optimizer, with its stabilizer parameters set to zero, can handle context distances that increase over training time. Small simulations demonstrate that the Adam optimizer can achieve larger context distances than gradient descent, but its performance reverts to the limits of gradient descent when its stabilizer parameters are set to standard values. Under unrestricted evaluation conditions, there is no significant difference between the trained Transformer model and the Adam optimizer.

Critical strengths

The theory is concrete and tested by training: the predicted gradient descent constants are close to the measured values, so the analysis quantitatively describes the trained model. It also avoids the usual simplifying assumptions (limited attention, pre-learned representations) and provides guarantees for arbitrarily long contexts, which makes this claim about length generalization and why I rate this paper as informative.

Critical weaknesses

The core prediction is that gradient descent will fail beyond a fixed distance, a phenomenon that has never been observed. According to the paper's own formula, it should only occur during astronomically long training times, far exceeding the stated run time, and the authors acknowledge that it has not yet materialized. A test on the learned model shows no Adam-SGD separation under unrestricted evaluation. Experiments support the gains of Adam, but not complete separation. The guarantee of Adam depends on the stabilizer being set to zero. Under standard values, the effect is lost after a single run of testing, while the training loss remains largely unchanged. The results are narrower than the title suggests, and the paper itself does not determine whether this effect is sustainable.

Questions for authors
  1. Is there some setting (allowed constants or population of data) where the prediction failure of gradient descent occurs within the scope of the simulation?

  2. By adjusting the learning rate of each optimizer, does the comparison of learned Transformers still show no separation?

Reviewer_DC8KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors analyze a structured two-layer attention model that learns key–value binding, query–key representations, forgetting, and a binary decoder, while keeping attention masks and value projections fixed. Under specified initialization and learning-rate conditions, they prove an asymptotic separation between GD and a specially configured Adam optimizer. GD achieves uniform retrieval through lag R+2, where R is the maximum training recall lag, but admits a fixed failure example at every larger lag. Adam guarantees a retrieval range proportional to its cumulative learning rate. Extensions cover Gaussian initialization for Adam and a restricted growing-batch SGD scheme.

Experiments illustrate retrieval behavior and sensitivity to Adam’s scalar stabilizer. A more fully learned Transformer experiment does not exhibit a consistent Adam–SGD separation when target predecessors are unrestricted.

Critical strengths

The paper addresses an interesting question: how optimizer choice affects generalization to sequences longer than those encountered during training. The theoretical separation between GD’s fixed retrieval boundary and Adam’s growing retrieval range, under the stated assumptions, is insightful, and the analysis connects these outcomes to the evolution of matching strength and recency penalties. The presentation is clear, with illustrative examples and figures that help explain the task, model, and main results.

Critical weaknesses

The analysis is restricted to a specific latest-value retrieval task, where the ground truth is the value in the most recent assignment to the queried key, and values are binary -1 or +1. This task is closely aligned with the model’s recency bias, leaving it unclear how strongly the optimizer separation depends on this alignment and the binary-value setting. Although the authors acknowledge the restricted scope, the current results provide limited evidence that the identified mechanism extends to other retrieval rules, richer value spaces, or broader Transformer applications.

Questions for authors

How dependent is the optimizer separation on the latest-value retrieval rule and the restriction to binary values -1,+1?

In particular, if the ground truth were the earliest value assigned to the queried key rather than the latest, would the conclusions change?

Comments on the paper

1 comment

I thank the authors for taking the time to respond to my reviewer and appreciate the clarity. After reading the paper several more times and their rebuttal I am still convinced this paper is not at the level of ICLR. While the authors have put effort in showing how optimizer bias affects length generalization I don't believe their insights are at the level of a top tier publication. I do thank the authors for doing deeper experiments and that does add to their paper, I would encourage them to include such results in future iterations. I am thus keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.