- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies several different NLP architectures on the task of outputting when the input is . They fix all parameters but , which allows to study how robust the different models are to adding noise after the value of interest. By studying this task and a few variations of it, they conclude that no model is robust to longer dependencies in general, and all fail in some cases.
Critical strengths
The paper studies the capability of different NLP models in information retrieval tasks, and how adding noisy context modifies this capability. This is a relevant setting, of interest to many in the community.
Critical weaknesses
The paper has two main weaknesses.
First, it lacks clarity, a lot of notions are not formalized, and the motivation is not clearly exposed. Mostly, a lot of things are barely defined. Those include:
- the specific models being studied,
- how they are trained,
- how and are chosen. This is especially problematic as those are central to the main message of the paper. As a result, it is quite hard to read and I believe it requires substantial rewriting before it can contribute to the community.
Second, the relevance with regards to prior work is barely discussed. As such, it is not clear to me what the contributions of this paper are. The main message seems to be that the architectures under study (which are not clearly defined) don't generalize on the specific tasks under study (not clearly defined either). However, the generalization limits of transformers are already a clearly established result [1,2,3]. Moreover, information retrieval tasks and copying as well as their limits have been widely studied [4,5,6].
[1] A Formal Framework for Understanding Length Generalization in Transformers, Huang et al, ICLR 2025
[2] Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count, Cho et al, ICLR 2025
[3] Quantitative Bounds for Length Generalization in Transformers, Izzo et al, ICLR 2026
[4] RULER: What’s the Real Context Size of Your Long-Context Language Models?, Hsieh et al, COLM 2024
[5] Transformers need glasses! Information over-squashing in language tasks, Barbero et al, NeurIPS 2024
[6] Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?, Roberts et al, ICLR 2025
Questions for authors
What are the specific architectures you are considering (saying simply RoPE or NoPE is not nearly enough to identify a precise architecture)?
And why did you choose those specifically?
How do your specific tasks and results compare to all the existing literature? (eg Why can your results not be derived from existing length generalization results?)