acceptodds
Under review as a conference paper at ICLR 2027

PoFormer: Recurrent Pointer Addressing for Length Generalization

12 views

Abstract

Length generalization requires a model trained on short sequences to apply the same algorithm to longer inputs, enabling compact, reusable computational subcircuits rather than length-specific behaviors. Looped Transformers provide recurrent computation but do not explicitly retain previously attended locations, making precise traversal difficult when repeated symbols render content matching ambiguous or when the next address depends on the current state or retrieved content. We introduce PoFormer, a recurrent Transformer whose recurrent state is factorized into a controller state that carries intermediate computation and a pointer state that tracks the active memory location. Selected attention heads act as pointer heads that maintain and update active addresses over the input, enabling local address shifts and global content-conditioned jumps while preserving content-based retrieval. We formalize this mechanism with P_RASP, a recurrent pointer extension of RASP, and constructively show that PoFormer can simulate P_RASP programs over finite execution horizons. Across controlled algorithmic tasks, PoFormer substantially improves OOD length generalization over standard and looped Transformer baselines. Its interpretable pointer trajectories reveal stable traversal strategies and expose address-level failures such as pointer drift and positional aliasing. Together, these results establish explicit recurrent address state as an effective inductive bias for length generalization in tasks requiring precise, stateful addressing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Length generalization, recurrent Transformers, looped Transformers, pointer attention, RASP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QukgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper describes a way to improve length generalization in (looped) transformers. The approach is based on modifying RoPE positional embeddings to allow for elaborate (learned) pointer operations. The paper also introduces a formalism derived from RASP, called Pointer-RASP, that makes it possible to characterize the computations that can be expressed with the model. Experimental results show that the pointer operations improve length generalization on several tasks, and that a model can retain performance on standard language modeling tasks in their presence.

Critical strengths

The formalism based on pointers is interesting and seems novel and worth exploring. The inability to learn length-generalizing mechanisms in algorithmic tasks is widely considered a potentially critical Achilles’ heel for transformer-based models. Solutions like the one proposed here are important and could have a large impact. The paper is very well written and easy to follow.

Critical weaknesses

Length generalization performance (the main point of the paper) is not very good as compared to known results in the literature. For example, on Parity, the performance degrades sharply even below twice the training sequence length. Furthermore, the model does not learn the harder tasks (bi-directional copy and binary addition) at all, unless it is trained to copy the execution trace of a P-RASP program that solves these tasks. Length generalization performance of the model improves as compared to the baseline models, but the baseline performance on many tasks seems very bad even in-distribution. I would have expected these models to work well in-distribution, and suspect that something is wrong with the implementation or experimental setup. See also, the questions under “Questions for Authors” below.

Questions for authors

Does the presence of pointer operations change (or does it not change) training parallelism?

Is the model able to learn multiple tasks at once (for example, by using the prompt to distinguish between them)? My understanding is that the experiments study the model on one task at a time.

Related, is the model able to learn the task while also being trained on natural language (like in Section 5.4)?

Reviewer_y6m6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

in this paper, the authors propose a new recurrent architecture called PoFormer, which introduces an explicit recurrent pointer. They suggest that learning with explicit pointer information provides a better inductive bias for algorithmic length generalisation tasks. To demonstrate the expressivity of PoFormer, the authors define P-RASP, a pointer-based program, showing that PoFormers are sufficient to implement pointer-based algorithms such as Duplicate-Token Copy, Prefix Parity, and Shuffled Linked-List Traversal. In the end, they empirically demonstrate that PoFormers outperform both standard and looped transformers on these tasks.

Critical strengths
  1. Introducing the explicit pointer information are quite interesting and well motivated for tackling the important length generalisation tasks.
  2. Providing an expressivity analysis of PoFormer to show it is sufficient for implementing pointer-style programs with p_shift and p_jump operators. furthermore, the set of the tasks including Duplicate-Token Copy, Prefix Parity, and Shuffled Linked-List Traversal can be simulated by the PoFormer.
  3. PoFormer appears to outperform baseline models on algorithmic length generalisation tasks.
Critical weaknesses

The clarity of the writing should be significantly improved!:

  1. The overall process of the model and the descriptions of the notations are currently unclear. for example, how is the static memory (Eq. 3) obtained? Does it require an additional encoder? in the recurrent computation stage, are q, k, and v encoded from the static memory m, the input token embedding, or the controller state c (Eq. 5)? It is also unclear where the controller state is utilised and how to get it? several notations, such as t, ℓ, in Eq. 5, are introduced without any explanation.
  2. The paper lacks intuition as to why PoFormer (or P-RASP) is better suited for length generalisation tasks. For example, what specific advantages does PoFormer's pointer attention offer over typical or looped transformers? Why should we "explicitly" learn the pointer information.

What information does the theoretical framework convey?

  1. why are p_shift and p_jump necessary, and what benefits does P-RASP provide compared to RASP-L style programs? Reference [1] has already demonstrated that RASP-L (the loop version) can implement Parity, Copy, and Addition. Probably, the RASP-L might be able to implicitly simulated the pointer primitives.

  2. While the theory demonstrates the strong expressiveness of PoFormers, looped transformers are already known to be highly expressive [2].

  3. theoretical expressiveness does not guarantee better length generalisation when using gradient descent-based learning algorithms.

The comparison in the empirical result are questionable

  1. While the baselines use the same model size, it is also important to consider computational cost. A comparison against baselines with equivalent compute is needed.

  2. It is unclear whether the advantages of the PoFormer stem from a superior inductive bias or simply from the limited expressiveness of other models. Further investigation is needed to determine if this performance gap remains significant when both the baselines and the PoFormer are scaled up.


[1] Ying Fan et al. Looped Transformers for Length Generalization (ICLR 2025)

[2] Angeliki Giannou at al. Looped Transformers as Programmable Computers (ICML 2023)

Questions for authors

Most of the questions are in the weaknesses.

  1. Could you explain why the baselines perform so poorly even in the in-distribution regime? Although Appendix E.3 shows improved results, they remain well below 100%. For instance, using NoPE positional embeddings, both the looped and non-looped models in [1] achieve nearly 100% accuracy in the in-distribution regime across tasks like Unique Set, Binary Addition, Copy, and Parity.

  2. How fast do the trainings reach the low loss? could you provide the training dynamics of the baseline and PoFormers?

  3. Could you provide some intuition as to why PoFormer performs better on length generalisation tasks? What kinds of tasks might it be less suitable for?

Reviewer_BJo6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PoFormer, a recurrent Transformer architecture that incorporates an explicit inductive bias for addressing over input tokens to enhance length generalization. It formalizes scans, jumps, and finite-state updates as recurrent pointer programs with Pointer-RASP (P-RASP). On algorithmic tasks, PoFormer demonstrates strong length generalization, and address-error analyses provide insight into its failure modes.

Critical strengths
  1. The explicit addressing mechanism is intuitive and well motivated as an inductive bias for improving length generalization.
  2. The P-RASP formalization provides a useful framework for describing the computations PoFormer is designed to perform.
Critical weaknesses

The evaluation focuses primarily on algorithmic tasks. Although PoFormer appears to generalize well to longer sequences in these settings, the paper does not evaluate whether this benefit extends to practical data, such as general language modeling, which limits the evidence for its broader length-generalization claims.

Questions for authors
  1. The baselines perform poorly on both the basic and advanced algorithmic tasks. Could the authors compare against baselines that achieve reasonable interpolation performance, even if they fail to generalize to longer sequences? The evaluation would also benefit from comparisons with contemporaneous methods specifically designed for length generalization.
  2. Could the authors evaluate length generalization on general language modeling tasks to test whether the proposed pointer inductive bias extends beyond algorithmic settings? A looped Transformer baseline could help clarify whether the gains com from explicit pointer addressing or recurrence more generally.
  3. Could the authors provide an analysis of the computational and memory costs associated with the pointer mechanism, particularly in general language modeling settings? This would help clarify the practical efficiency implications of the proposed approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.