- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies counting in a single-layer, single-head attention model. It analyzes how Softmax, LayerNorm placement, and read-out choice affect the parameter norms needed to count up to length L. The main finding is that learned post-LayerNorm requires slower norm growth than no normalization, while bare pre-LayerNorm imposes a finite counting limit. Classification avoids growing norms but requires more storage and has shrinking prediction margins.
Critical strengths
- The paper goes beyond whether attention can count and provides tight bounds on the parameter norms required. This offers a useful comparison of different normalization choices.
- The advantage of learned post-LayerNorm is surprising. The geometric explanation helps clarify why normalization can improve, rather than simply harm, counting.
Critical weaknesses
- Limited scope and connection to learning. The results focus on a minimal architecture and may not directly extend to standard Transformers. They mainly establish representational capacity, rather than whether training can find these solutions or generalize to longer sequences.
- Norm bounds do not capture all practical costs. Bounded norms do not necessarily imply robust or efficient counting. The classifier needs a growing output head and has shrinking margins, so further analysis of numerical precision and robustness would strengthen the comparison.
- The related-work section is difficult to follow. It moves between sequential counting, transformer expressivity, histogram tasks, and arithmetic without a clear organizing structure. Detailed descriptions of prior constructions also make the main comparisons harder to identify.
Questions for authors
Refer to weakness.
2 comments
Thank you for the clarifications. While I appreciate these points and have raised my score from 1 to 2 accordingly, my main concern regarding the limited scope remains unresolved. The gap between this minimal architecture and standard Transformers significantly limits the practical contribution of the paper.
We thank the reviewer for their time and feedback.
One main issue is that we only study this problem in a simplified setting of a single attention head, and that it is unclear how this connects to real-world transformer models. We understand this concern. We picked this setting precisely because it is simple enough for us to thoroughly analyze and prove matching upper and lower bounds. These bounds allow us to be precise about when normalization is an obstacle to counting and when it helps. Earlier papers seem to point in different directions on this (Ouellette et al., 2023; Chiang & Cholak, 2022). The setting we use in the paper isn't unprecedented: earlier work on counting also analyzed one-layer models, for example Behrens et al. (2025). The rates may well be different in deeper models with more heads, which is why the abstract and limitations paragraph scope our claims to this setting.
On the point that our theorems are about what the model can express and don't claim anything about how training might reach such a solution: this is true of the theorems, although a lower bound on the norm holds for any parameters, so it also holds for whatever training ends up with. We did run training experiments from random initialization in the empirical results section of the paper. When we cap the norms during training, the largest length the model can count to grows with the cap at a fitted exponent of 1.00 without normalization and 0.68 with bare post-LN, and the predicted values are 1 and 2/3 (Fig. 2d). For the learned-gain variant the fitted exponent did not match, which we report in Section 6. In Appendix J.1 we also train the model without normalization using Adam. On generalizing to longer sequences, Section 7 discusses what our bounds imply: once training stops the norms are fixed, and a norm that is enough to count to L is too small for much longer inputs. We only present this as a hypothesis for larger models.
On numerical precision and robustness for the classification head: some of this analysis is already in the appendix, though we realize it is easy to miss. In Appendix G we give the margin rates for our classifiers and a condition under which the prediction is stable to perturbations (Prop. 9). The storage comparison with regression is Prop. 10 in the same appendix. We will also keep your comments on the related work section in mind when we reorganize it.
We couldn't tell if there is a specific problem with one of the proofs or experiments. If there is, we would appreciate it if it could be pointed out to us so we can check.