- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a new metric/notion to predict when length generalization happens in single layer Transformers without residual connections and without layer normalization. This new metric called NESV is a function of input length and measures the fraction of parameters (where the parameter values are bounded) that achieve an exact solution on every input of length . They derive asymptotic bounds for various formal languages - FIRST, MAJORITY, INDEX and PARITY, and find that empirically the exact volume metric as a function of is predictive of how well Transformers length generalize. Further, on one of the tasks where Transformers do struggle, i.e. INDEX (return symbol at a position specified by one of the input tokens), they find two things which cause a worse NESV, and consider a slightly altered Transformer that is designed to eliminate one of the sources causing the error, and this said Transformer (by design improves NESV) empirically length generalizes better.
Critical strengths
-
NESV is an interesting way to think about length generalization: This new notion introduced by the authors, while in some indirect ways can be inferred from past papers, especially Kövér et al 2026[1] and Hahn & Rofin 2024 [2]. is still quite valid, and it is worth working this out formally. The fact that, this line of work that can side step the complexity of gradient descent and still be able to say something useful about models, is quite nice.
-
NESV helping in designing dedicated improvements in the architecture as seen in the INDEX task is quite promising as well.
-
The proofs look correct to me.
-
The experiments across 20 seeds on each of these tasks and multiple positional encodings, shows that there is some correlation between length generalization and this metric.
[1] Kövér et al ICML 2026: Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions
[2] Hahn & Rofin ACL 2024: Why are sensitive functions hard for Transformers?
Critical weaknesses
Some bigger problems --
-
This notion of exact volume solution and length generalization are correlating for now, but there are no formal links that are established between these completely different notions. This correlation exists, but without any formal link, this correlation may not always hold. In fact, the authors themselves find a counterexample in the case of MAJORITY, where the NESV is not as predictive across the model types considered in the paper.
-
This piggybacks on the last point, the authors also have currently ignored what afaik is the most successful formal framework for studying length generalization: which are all RASP based [1, 2]. The wealth of tasks on which these RASP based variants are predictive seems much higher than this NESV. I understand that this NESV could potentially could help in getting more fine grained perspectives, instead of just the binary answers that one usually gets from [1,2], however for now, a stronger formal link, or more experiments on a wider variety of tasks would be required to convincingly establish the utility of this method. Without that, this paper feels like a good first step, but one that is not enough.
-
The results are also only for very toy models -- single layer, no residuals, no layer norm, very specific kind of positional embeddings which are not naturally used. While these assumptions are understandable, as this paper is the first one to introduce this notion. The no residuals and no layer norm could easily be missed by someone skimming the paper, this detail only appears when the authors mathematically write out the equations. Such critical assumptions should be more clearly highlighted in the abstract and introduction. Also, somewhere providing some discussion / remarks on these assumptions also feel missing to me.
Some minor problems --
-
The 'Kover et al' 2026 citation should be spelled more carefully (Kövér). There are non-English characters in the name.
-
The current paper is ignoring a lot of work that has been done in understanding length generalization in Transformers, the RASP line of work was an example, which itself has some critical papers. The related works section should more adequately address papers in this area.
[UPDATE] During the rebuttal period, the authors productively engaged, made important clarifications and incorporated what I think is some useful hedging in many places, making the technical claims more well scoped and accurate. Despite the hedging, I think that the idea deserves merit, and would be valuable to the community. I have therefore increased my score to reflect this.
[1] Zhou et al ICLR 2024: What Algorithms can Transformers Learn? A Study in Length Generalization
[2] Huang et al ICLR 2025: A Formal Framework for Understanding Length Generalization in Transformers
Questions for authors
-
The current Figure 2 is also very unintuitive to me, could you explain what you wanted to show with that Figure here ?
-
How would things like Chain of Thought, looping, padding etc. fit in with NESV ?
-
The results across seeds was not clear to me. What does seed bootstrap 95% mean in Table 2? And for such results, the seed variance maybe should be reported more clearly in the main paper.
5 comments
Thank you for your response. Could you actually make the changes you are promising here and update the PDF? It would be much more helpful for me to see those changes actually reflected in the PDF. ICLR allows an extra page for the rebuttal, so hopefully that's plenty space to add most things.
Also, it seems that the authors missed responding in full to weakness 3 that I listed.
Such critical assumptions should be more clearly highlighted in the abstract and introduction. Also, somewhere providing some discussion / remarks on these assumptions also feel missing to me.
Essentially I have no issues with the assumptions, I would just think that the title of the paper, abstract and introduction was not very clear in managing expectations about what's to come. The fact that the results for extremely toy models is alright, but I definitely think that the authors need to be upfront about it in the paper.
For instance, the following detail needs to be mentioned way earlier -- We do not use residual connections or layer normalization.
We also agree that at present, the connection between NESV and learnability is intuitive and only empirically supported. We acknowledge that developing a formal, mathematical account of when and how solution volume relates to learnability is an important direction for future work.
Great, no problem with that. Again please be more upfront about it in the introduction/ abstract. Essentially I am happy with your response, the major thing I would like to see is that the 'advertisement' of the results of the paper should be correct and not misleading. When I was personally going through the paper, I felt a bit deceived when finding out the actual results, vs what the introduction setup. This is not an issue as this is easily fixable and therefore I would really want to see this reflected in the PDF, so that we hopefully can have another back and forth during the rebuttal period itself.
If updating all the things is too difficult in this short period, then kindly at least give the abstract + intro another pass, I would be happy to have another round of back and forth after (at least) some updates in the PDF. Also kindly mark the new things in blue / any color so that it's easy to quickly check what changed.
Thank you again.
Thank you for the clarification. We have now updated the PDF to make the scope and assumptions of our study clearer from the outset. In particular, we have made the architectural assumptions more explicit in the abstract and introduction, and clarified the current empirical nature of the connection between NESV and learnability. We have also corrected the typos. The changes are marked in blue.
We have not yet expanded the related work discussion on RASP and the broader length-generalization literature, but plan to include this in subsequent revisions. We hope these changes address your main concerns, and would be happy to hear your thoughts on the updated PDF.
Thank you very much for your thoughtful review! We apologize for misspelling Blanka Kövér's name. We will correct it in the final version.
You asked for a more extensive discussion of previous work on length generalization, especially related to RASP. Our discussion and related-work section were shortened because of the page limit at submission. We would be happy to extend this section to discuss the broader literature on length generalization.
We agree with you that NESV provides fine-grained answers instead of the binary answers one gets by hypothesizing that expressivity in a RASP variant is equivalent to length generalization in a transformer. The two papers cited [1,2] argue for equivalence with two different RASP variants, and one might also add the work of [Li and Cotterell 2025], which argues that expressivity in LTL[P] is equivalent to length generalization in a fixed-precision transformer. One way to reconcile these results is to say that weaker logics/RASPs define languages that are "more" length-generalizable, but there isn't yet a unifying framework. While we would welcome such a framework, we propose that NESV gives a simple way to talk about how some languages are more learnable than others.
We also agree that at present, the connection between NESV and learnability is intuitive and only empirically supported. We acknowledge that developing a formal, mathematical account of when and how solution volume relates to learnability is an important direction for future work.
Regarding the omission of LayerNorm and residual connections: If we follow the placement of layer normalization in the original transformers paper (that is, post-norm), adding LayerNorm alone would not affect our bounds. Adding LayerNorm together with a residual connection would introduce an additional n−1 factor for both the unnormalized and normalized models, changing the bounds to Θ(n−4) and Θ(n−2), respectively. This is because the residual path carries the query index k directly into the downstream representation, creating an additional length-dependent constraint that must hold uniformly for all k≤n. We would be happy to add this discussion to the final version of the paper if desired.
Answers to the questions:
The intention of Figure 2 is to show the difference in length-generalization performance between the normalized and unnormalized models on INDEX. We would be happy to clarify the figure and its caption further if any aspect remains confusing.
The notion of NESV should extend to these settings as long as there is a defined parameter space and a criterion for final-answer correctness. We don't make any claims, however, about how difficult it would be to derive NESV bounds in these settings. In particular, a prerequisite for studying looping would be NESV bounds for multi-layer transformers, which is our priority for future work.
For Table 2, we resample the 20 seed-level accuracies with replacement, compute the mean accuracy for each sample, and report the 2.5th and 97.5th percentiles as the pointwise 95% confidence interval for the mean accuracy across seeds. If the reviewer would prefer an additional measure of between-seed variability, we would be happy to add it to the appendix.