- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper describes a way to improve length generalization in (looped) transformers. The approach is based on modifying RoPE positional embeddings to allow for elaborate (learned) pointer operations. The paper also introduces a formalism derived from RASP, called Pointer-RASP, that makes it possible to characterize the computations that can be expressed with the model. Experimental results show that the pointer operations improve length generalization on several tasks, and that a model can retain performance on standard language modeling tasks in their presence.
Critical strengths
The formalism based on pointers is interesting and seems novel and worth exploring. The inability to learn length-generalizing mechanisms in algorithmic tasks is widely considered a potentially critical Achilles’ heel for transformer-based models. Solutions like the one proposed here are important and could have a large impact. The paper is very well written and easy to follow.
Critical weaknesses
Length generalization performance (the main point of the paper) is not very good as compared to known results in the literature. For example, on Parity, the performance degrades sharply even below twice the training sequence length. Furthermore, the model does not learn the harder tasks (bi-directional copy and binary addition) at all, unless it is trained to copy the execution trace of a P-RASP program that solves these tasks. Length generalization performance of the model improves as compared to the baseline models, but the baseline performance on many tasks seems very bad even in-distribution. I would have expected these models to work well in-distribution, and suspect that something is wrong with the implementation or experimental setup. See also, the questions under “Questions for Authors” below.
Questions for authors
Does the presence of pointer operations change (or does it not change) training parallelism?
Is the model able to learn multiple tasks at once (for example, by using the prompt to distinguish between them)? My understanding is that the experiments study the model on one task at a time.
Related, is the model able to learn the task while also being trained on natural language (like in Section 5.4)?