- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper systematically characterizes LLM counting failures in two settings: counting items in lists and counting target occurrences in natural text. It evaluates error frequency, magnitude, and direction, showing that exact-match accuracy alone obscures important error patterns. The study further finds that homogeneous lists are harder to count than heterogeneous ones, and that partial contextual corruption produces more errors than either intact or fully corrupted contexts.
Critical strengths
This work examines counting as a fundamental capability of LLMs and presents a unified evaluation framework that characterizes counting errors in terms of their frequency, magnitude, and direction.
Critical weaknesses
- My major concern is that the findings remain largely descriptive and empirical, offering limited guidance on how to improve LLM counting. For example, a controlled intervention demonstrating that these findings lead to better counting performance would strengthen the claimed implications for model design and deployment.
- The evaluation focuses on closed-source models from different families, making it difficult to disentangle the effects of model size, architecture, and training. Comparing models within the same family across sizes and training checkpoints would better characterize how counting capabilities develop with scale and training.
- There are several formatting issues. Table captions should appear above their respective tables, and Figure 1 should be positioned at the top of the page. In line 250, ''(Figure 3 )'' should be corrected to ''(Figure 3)''.
Questions for authors
Refer to weakness.
3 comments
We thank the reviewer for their feedback and valuable suggestions for future work: devising better solutions for enhancing counting in LLMs, and studying how model size, architecture and training shape counting ability. We agree that both of these are important directions that future work should aim to address. As LLMs are increasingly deployed in cognitively demanding tasks, they are required to perform elementary tasks like counting in indirect ways. Tracing how counting develops across model sizes, architectures and training checkpoints in open models is an important question in its own right, requiring a different experimental design and a focused study.
Thank you for the clarifications. I appreciate the additional analysis and have raised my Soundness score from 1 to 2 accordingly. However, my primary concerns remain only partially addressed. The central research question highlighted in the rebuttal (i.e., evaluating the reliability of frontier models) should be clearly shown or motivated in the manuscript. I will maintain my overall score at 2.
We thank the reviewer for their review and suggestions. Their main concerns relate to the scope of the paper's contribution, namely how far the findings guide improvement and how far they generalize across model families. We address these below.
W1. The findings are descriptive and offer limited guidance on improving counting. We agree that the paper's primary contribution is behavioral characterization, and that it is not explicitly prescriptive in how its findings can improve counting performance. That said, some of the experimental interventions we present can be viewed as evidence of how models can, in principle, count with higher precision when a favorable perturbation is applied to the input as part of preprocessing. In particular, letter case variations in just 20% of otherwise identical items can reduce error rates and sharply reduce error magnitudes for GPT-5.2 and Gemini 2.5 Pro (Claude Opus 4.5, having much smaller errors to begin with, shows a slight increase in magnitude, however). When the target is embedded within a natural context, masking words in which no surface form of the target appears also shows a similar effect (see table below for details).
Miscount rates are from Table 8. $M_{RE}$: median relative error over miscounted instances (Eq. 2), computed from the same experiments and added to Appendix C in the revision. * Significant change in miscount rate ($p_{Holm} < 0.05$, exact McNemar test). † 64 pairs, as Claude refuses most masked passages.
Finally, our scratchpad analysis (response to Reviewer agW8, W4) shows that contextual miscounts arise from occurrences that are never enumerated rather than from misjudged ones, indicating that interventions should target enumeration coverage rather than matching rules.
In deployed systems, however, counting is typically embedded in more complex queries that also require abilities such as entity resolution and tracking, which we believe are outside the scope of this work. The mechanisms by which counting failures propagate in such settings and exploring ways to resolve them is complex enough to warrant a dedicated investigation. Our findings provide the behavioral foundation and motivation for that work, and we have revised the introduction to state its implications in these terms.
W2. Models from different families make it hard to separate the effects of size, architecture and training. We agree. Comparisons across families cannot attribute differences to size or training. Our aim, however, was to explore the limits of counting in state-of-the-art systems and characterize failures that persist at the current capability frontier. Frontier models from different providers also test whether these failures generalize across independently trained systems, and several do: undercounting dominates contextual counting in all three models, and list diversity reduces miscounts in all three. The cross-family design helps distinguish systematic failures of current LLMs from model-specific ones. A common error pattern across independently trained systems is unlikely to be an artifact of a single training pipeline or tokenizer, and the divergences we observe (e.g. Claude Opus 4.5 refusing corrupted contexts while GPT-5.2 makes larger errors under partial than full corruption) are findings that a single-family study would miss. Conclusions from any within-family study would also need cross-family replication to generalize.
We also acknowledge that disentangling scale and training is an important research question and calls for a different experimental design. Such a study would involve a comprehensive investigation exploring how model sizes, architectures, and intermediate checkpoints shape counting abilities. Ours, however, is designed to answer a different question: How do we characterize counting errors and how reliably do widely-deployed frontier models perform on simple counting tasks?. We believe these are related but different research questions. The revised manuscript will list within-family studies across model sizes and training checkpoints as a future direction.
W3. Formatting. We thank the reviewer for drawing our attention to these issues. We will place table captions above their tables, move Figure 1 to the top of the page, and fix the spacing in "(Figure 3 )".