acceptodds
Under review as a conference paper at ICLR 2027

Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

Abstract

Counting is a simple task on which transformers can nevertheless fail at long sequence lengths. We study the cost of counting in a single attention layer, up to a maximum length , focusing on how softmax attention and layer normalization shape the representation from which the count is recovered. We mainly focus on regression read-out but also consider classification read-out. For regression, we show unnormalized linear attention counts exactly at parameter norms independent of . For softmax attention, regression cannot count exactly with finite parameters; it recovers the count after rounding, but only when the vector parameter norms grow with . We measure this cost by , the smallest bound that can be placed on every trainable vector to ensure exact counting after rounding, and derive tight rates. At fixed model dimension, grows as without normalization, as under bare post-normalization, and as with learned post-normalization. The last rate is unexpected: post-normalization discards the magnitude that carries the count and might be expected to make counting harder, yet with a learnable gain it counts at a smaller norm than softmax without normalization. Bare pre-normalization is different: it bounds the attention scores, so beyond a finite length no parameter norm suffices. A classification read-out keeps every norm bounded under every placement, but a dense head stores one output vector per count and costs more in storage. Experiments on the post-normalized model reproduce this split between the two read-outs. Together, our theoretical and empirical findings characterize how self-attention, layer-normalization placement, and read-out choice shape the representation geometry and parameter cost of counting in the minimal single-head architecture studied here.

Primary area:
learning theory
Keywords:
layer normalization, counting, softmax attention, transformers, expressivity, length generalization, parameter norm, representation geometry, pre-LN, post-LN, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_abQSOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies counting in a single-layer, single-head attention model. It analyzes how Softmax, LayerNorm placement, and read-out choice affect the parameter norms needed to count up to length L. The main finding is that learned post-LayerNorm requires slower norm growth than no normalization, while bare pre-LayerNorm imposes a finite counting limit. Classification avoids growing norms but requires more storage and has shrinking prediction margins.

Critical strengths
  1. The paper goes beyond whether attention can count and provides tight bounds on the parameter norms required. This offers a useful comparison of different normalization choices.
  2. The advantage of learned post-LayerNorm is surprising. The geometric explanation helps clarify why normalization can improve, rather than simply harm, counting.
Critical weaknesses
  1. Limited scope and connection to learning. The results focus on a minimal architecture and may not directly extend to standard Transformers. They mainly establish representational capacity, rather than whether training can find these solutions or generalize to longer sequences.
  2. Norm bounds do not capture all practical costs. Bounded norms do not necessarily imply robust or efficient counting. The classifier needs a growing output head and has shrinking margins, so further analysis of numerical precision and robustness would strengthen the comparison.
  3. The related-work section is difficult to follow. It moves between sequential counting, transformer expressivity, histogram tasks, and arithmetic without a clear organizing structure. Detailed descriptions of prior constructions also make the main comparisons harder to identify.
Questions for authors

Refer to weakness.

2 comments

We thank the reviewer for their time and feedback.

One main issue is that we only study this problem in a simplified setting of a single attention head, and that it is unclear how this connects to real-world transformer models. We understand this concern. We picked this setting precisely because it is simple enough for us to thoroughly analyze and prove matching upper and lower bounds. These bounds allow us to be precise about when normalization is an obstacle to counting and when it helps. Earlier papers seem to point in different directions on this (Ouellette et al., 2023; Chiang & Cholak, 2022). The setting we use in the paper isn't unprecedented: earlier work on counting also analyzed one-layer models, for example Behrens et al. (2025). The rates may well be different in deeper models with more heads, which is why the abstract and limitations paragraph scope our claims to this setting.

On the point that our theorems are about what the model can express and don't claim anything about how training might reach such a solution: this is true of the theorems, although a lower bound on the norm holds for any parameters, so it also holds for whatever training ends up with. We did run training experiments from random initialization in the empirical results section of the paper. When we cap the norms during training, the largest length the model can count to grows with the cap at a fitted exponent of 1.00 without normalization and 0.68 with bare post-LN, and the predicted values are 1 and 2/3 (Fig. 2d). For the learned-gain variant the fitted exponent did not match, which we report in Section 6. In Appendix J.1 we also train the model without normalization using Adam. On generalizing to longer sequences, Section 7 discusses what our bounds imply: once training stops the norms are fixed, and a norm that is enough to count to LL is too small for much longer inputs. We only present this as a hypothesis for larger models.

On numerical precision and robustness for the classification head: some of this analysis is already in the appendix, though we realize it is easy to miss. In Appendix G we give the margin rates for our classifiers and a condition under which the prediction is stable to perturbations (Prop. 9). The storage comparison with regression is Prop. 10 in the same appendix. We will also keep your comments on the related work section in mind when we reorganize it.

We couldn't tell if there is a specific problem with one of the proofs or experiments. If there is, we would appreciate it if it could be pointed out to us so we can check.

Thank you for the clarifications. While I appreciate these points and have raised my score from 1 to 2 accordingly, my main concern regarding the limited scope remains unresolved. The gap between this minimal architecture and standard Transformers significantly limits the practical contribution of the paper.

Reviewer_tNahOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how layer normalization affects counting in a single attention layer. It derives bounds on the parameter norms needed for counting under different normalization settings and compares regression and classification readouts. The paper also provides experiments to illustrate some of the theoretical results.

Critical strengths

The paper provides a theoretical analysis of a simple but relevant problem. The comparison between different normalization placements is interesting, and the results help explain how normalization may affect counting in this setting.

Critical weaknesses

1.The main limitation is the simplified setting. The analysis considers a single attention head with identity projections and assumes zero epsilon for layer normalization. The authors also note that the results concern what the model can represent, rather than whether training can find such representations. Therefore, it remains unclear how directly these findings apply to practical transformer models.

2.The experiments provide some support for the analysis, but further evidence would help clarify the relevance of the theoretical results to more realistic settings.

Questions for authors

See Weakness

2 comments

We thank the reviewer for their time and feedback.

One main issue is that we only study this problem in a simplified setting of a single attention head, and that it is unclear how this connects to real-world transformer models. We understand this concern. We picked this setting precisely because it is simple enough for us to thoroughly analyze and prove matching upper and lower bounds. These bounds allow us to be precise about when normalization is an obstacle to counting and when it helps. Earlier papers seem to point in different directions on this (Ouellette et al., 2023; Chiang & Cholak, 2022). The setting we use in the paper isn't unprecedented: earlier work on counting also analyzed one-layer models, for example Behrens et al. (2025). The rates may well be different in deeper models with more heads, which is why the abstract and limitations paragraph scope our claims to this setting.

On the identity projections: these are fixed in all our main results, as we state in Section 3. One of our results does depend on this assumption. In Section 4.3 and Appendix E.3 we show that learned query and key projections remove the finite counting limit of bare pre-LN.

On the point that our theorems are about what the model can express and don't claim anything about how training might reach such a solution: this is true of the theorems, although a lower bound on the norm holds for any parameters, so it also holds for whatever training ends up with. We did run training experiments from random initialization in the empirical results section of the paper. When we cap the norms during training, the largest length the model can count to grows with the cap at a fitted exponent of 1.00 without normalization and 0.68 with bare post-LN, and the predicted values are 1 and 2/3 (Fig. 2d). For the learned-gain variant the fitted exponent did not match, which we report in Section 6. In Appendix J.1 we also train the model without normalization using Adam.

For experiments in a more realistic setting, we attempt this in Appendix J.3, where we look at how Qwen3-4B represents counts and we see a similar crowding of large counts in its later layers, though we only present this as a qualitative comparison.

We couldn't tell if there is a specific problem with one of the proofs or experiments. If there is, we would appreciate it if it could be pointed out to us so we can check.

Thanks for the response which has address some of my concerns. I would like to keep my initial rate.

Reviewer_fpCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the theoretical limitations and parameter costs of counting with single-layer attention architectures. It analyzes how softmax attention, different layer normalization placements, and regression or classification read-outs influence counting accuracy and the required parameter norms. Theoretical results establish bounds on the parameter norms needed for exact counting after rounding, while experiments provide empirical support for some of these findings.

Critical strengths

The paper provides a theoretical analysis of counting with attention mechanisms, offering insights into how normalization and read-out strategies influence counting capabilities.

Critical weaknesses
  1. The theoretical analysis is mainly restricted to a minimal single-layer, single-head attention architecture. It is unclear whether the derived conclusions and bounds can be generalized to more complex Transformer architectures with multiple layers, attention heads, and nonlinear components.

  2. The paper investigates specific layer normalization configurations, but the applicability of the theoretical findings to other commonly used normalization methods remains unclear.

  3. The practical implications of the theoretical bounds are insufficiently discussed. In particular, it is unclear how the derived parameter norm requirements relate to training difficulty, convergence behavior, and generalization performance in real-world applications.

Questions for authors
  1. How can the theoretical derivations and parameter norm bounds be extended to more complex Transformer architectures, particularly those with multiple attention layers and multiple heads?

  2. Do the conclusions remain valid when additional nonlinear components, such as feed-forward networks, are introduced?

  3. How would other normalization methods, such as RMSNorm, affect the theoretical counting capability and parameter norm requirements?

  4. What is the relationship between the derived parameter norm bounds and practical optimization difficulty or convergence behavior?

2 comments

We thank the reviewer for their time and feedback.

One main issue is that we only study this problem in a simplified setting of a single attention head, and that it is unclear how this connects to real-world transformer models. We understand this concern. We picked this setting precisely because it is simple enough for us to thoroughly analyze and prove matching upper and lower bounds. These bounds allow us to be precise about when normalization is an obstacle to counting and when it helps. Earlier papers seem to point in different directions on this (Ouellette et al., 2023; Chiang & Cholak, 2022). The setting we use in the paper isn't unprecedented: earlier work on counting also analyzed one-layer models, for example Behrens et al. (2025). The rates may well be different in deeper models with more heads, which is why the abstract and limitations paragraph scope our claims to this setting.

On the relation to optimization: our bounds are necessary conditions, so any trained model that counts to LL must have norms at least that large, but we do not have a convergence result. In Appendix J.1 we train the model without normalization using Adam. The norms reach the lower bound early, but most runs are still not exact on every input: after 20M steps at L=250L = 250, only one of six runs has worst-case error below 1/21/2. What lags is the saturation scale γ\gamma, so the norm is not what makes training slow here.

The questions about how our conclusions would change with more layers and heads, feed-forward layers and RMSNorm are good ones and we do not have complete answers to them yet. We plan to look at these in future work.

We couldn't tell if there is a specific problem with one of the proofs or experiments. If there is, we would appreciate it if it could be pointed out to us so we can check.

Thanks for the response. However, I agree with Reviewer abQS that the scope is a major issue and is not resolved. So, I would like to keep my initial rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.