acceptodds
Under review as a conference paper at ICLR 2027

The Context Counts: Characterizing LLM Counting Failures Across Input Regimes

Abstract

Counting is an essential skill underpinning complex forms of quantitative reasoning, yet large language models (LLMs) remain unreliable even on seemingly simple counting tasks. Prior work has shown that contextual cues and training frequency biases affect counting performance. Building on this literature, we characterize not only how often LLMs miscount, but also the magnitude and direction of their errors across two regimes: non-contextual counting (tallying items in line-separated lists) and contextual counting (a filter-then-count operation over unstructured text passages). Our experiments reveal three findings: (1) miscount rates rise with target count, although most errors are relatively small and undercounting predominates; (2) greater list diversity improves performance, even when the added variation comes from case perturbations on a fraction of instances, interspersed sporadically in an otherwise homogeneous list; and (3) similar miscount rates across degrees of context corruption conceal distinct error patterns: GPT 5.2 makes larger relative errors on partially corrupted than fully corrupted contexts, while Claude Opus 4.5 begins to refuse to perform the counting task on contexts with corruption rates as low as 25%. Together, these results show how changes to input structure affect counting errors in ways that miscount rates alone do not capture. We hope this behavioral account provides a basis for future mechanistic studies of LLM counting.

Primary area:
interpretability and explainable AI
Keywords:
model behavior, evaluation, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_giTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper systematically characterizes LLM counting failures in two settings: counting items in lists and counting target occurrences in natural text. It evaluates error frequency, magnitude, and direction, showing that exact-match accuracy alone obscures important error patterns. The study further finds that homogeneous lists are harder to count than heterogeneous ones, and that partial contextual corruption produces more errors than either intact or fully corrupted contexts.

Critical strengths

This work examines counting as a fundamental capability of LLMs and presents a unified evaluation framework that characterizes counting errors in terms of their frequency, magnitude, and direction.

Critical weaknesses
  1. My major concern is that the findings remain largely descriptive and empirical, offering limited guidance on how to improve LLM counting. For example, a controlled intervention demonstrating that these findings lead to better counting performance would strengthen the claimed implications for model design and deployment.
  2. The evaluation focuses on closed-source models from different families, making it difficult to disentangle the effects of model size, architecture, and training. Comparing models within the same family across sizes and training checkpoints would better characterize how counting capabilities develop with scale and training.
  3. There are several formatting issues. Table captions should appear above their respective tables, and Figure 1 should be positioned at the top of the page. In line 250, ''(Figure 3 )'' should be corrected to ''(Figure 3)''.
Questions for authors

Refer to weakness.

3 comments

We thank the reviewer for their review and suggestions. Their main concerns relate to the scope of the paper's contribution, namely how far the findings guide improvement and how far they generalize across model families. We address these below.

W1. The findings are descriptive and offer limited guidance on improving counting. We agree that the paper's primary contribution is behavioral characterization, and that it is not explicitly prescriptive in how its findings can improve counting performance. That said, some of the experimental interventions we present can be viewed as evidence of how models can, in principle, count with higher precision when a favorable perturbation is applied to the input as part of preprocessing. In particular, letter case variations in just 20% of otherwise identical items can reduce error rates and sharply reduce error magnitudes for GPT-5.2 and Gemini 2.5 Pro (Claude Opus 4.5, having much smaller errors to begin with, shows a slight increase in magnitude, however). When the target is embedded within a natural context, masking words in which no surface form of the target appears also shows a similar effect (see table below for details).

Input RegimeIntervention (correct count unchanged)ModelMiscount rate$M_{RE}$
Non-contextualCase mixing (20% of items)GPT-5.21.00 → 0.76*30.7% → 0.7%
Gemini 2.5 Pro0.97 → 0.92*17.0% → 0.9%
Claude Opus 4.50.80 → 0.74*2.6% → 3.9%
ContextualContext maskingGPT-5.20.59 → 0.47*11.4% → 10.5%
Gemini 2.5 Pro0.42 → 0.395.7% → 7.1%
Claude Opus 4.5†0.17 → 0.205.9% → 10.0%

Miscount rates are from Table 8. $M_{RE}$: median relative error over miscounted instances (Eq. 2), computed from the same experiments and added to Appendix C in the revision. * Significant change in miscount rate ($p_{Holm} < 0.05$, exact McNemar test). † 64 pairs, as Claude refuses most masked passages.

Finally, our scratchpad analysis (response to Reviewer agW8, W4) shows that contextual miscounts arise from occurrences that are never enumerated rather than from misjudged ones, indicating that interventions should target enumeration coverage rather than matching rules.

In deployed systems, however, counting is typically embedded in more complex queries that also require abilities such as entity resolution and tracking, which we believe are outside the scope of this work. The mechanisms by which counting failures propagate in such settings and exploring ways to resolve them is complex enough to warrant a dedicated investigation. Our findings provide the behavioral foundation and motivation for that work, and we have revised the introduction to state its implications in these terms.

W2. Models from different families make it hard to separate the effects of size, architecture and training. We agree. Comparisons across families cannot attribute differences to size or training. Our aim, however, was to explore the limits of counting in state-of-the-art systems and characterize failures that persist at the current capability frontier. Frontier models from different providers also test whether these failures generalize across independently trained systems, and several do: undercounting dominates contextual counting in all three models, and list diversity reduces miscounts in all three. The cross-family design helps distinguish systematic failures of current LLMs from model-specific ones. A common error pattern across independently trained systems is unlikely to be an artifact of a single training pipeline or tokenizer, and the divergences we observe (e.g. Claude Opus 4.5 refusing corrupted contexts while GPT-5.2 makes larger errors under partial than full corruption) are findings that a single-family study would miss. Conclusions from any within-family study would also need cross-family replication to generalize.

We also acknowledge that disentangling scale and training is an important research question and calls for a different experimental design. Such a study would involve a comprehensive investigation exploring how model sizes, architectures, and intermediate checkpoints shape counting abilities. Ours, however, is designed to answer a different question: How do we characterize counting errors and how reliably do widely-deployed frontier models perform on simple counting tasks?. We believe these are related but different research questions. The revised manuscript will list within-family studies across model sizes and training checkpoints as a future direction.

W3. Formatting. We thank the reviewer for drawing our attention to these issues. We will place table captions above their tables, move Figure 1 to the top of the page, and fix the spacing in "(Figure 3 )".

We thank the reviewer for their feedback and valuable suggestions for future work: devising better solutions for enhancing counting in LLMs, and studying how model size, architecture and training shape counting ability. We agree that both of these are important directions that future work should aim to address. As LLMs are increasingly deployed in cognitively demanding tasks, they are required to perform elementary tasks like counting in indirect ways. Tracing how counting develops across model sizes, architectures and training checkpoints in open models is an important question in its own right, requiring a different experimental design and a focused study.

Thank you for the clarifications. I appreciate the additional analysis and have raised my Soundness score from 1 to 2 accordingly. However, my primary concerns remain only partially addressed. The central research question highlighted in the rebuttal (i.e., evaluating the reliability of frontier models) should be clearly shown or motivated in the manuscript. I will maintain my overall score at 2.

Reviewer_5Ji4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper evaluates LLM model's counting failures by their frequency and direction across distinct inputs. It tests two input regimes, counting items in a list, and selected counting nouns in wikipedia snippets. The paper finds that LLMs tend to undercount.

Critical strengths

The paper in principle investigates an interesting problem: counting. But at the same time it is also one of its key limitations: how important is it to study very simple counting problems with LLMs, especially now in the era of tool-based LLMs and agentic AI? There is prior work on counting objects in generated images, thus language vision problems. There I would find the problem much more interesting to tackle. In the text domain as scope here the counting is trivial and can be solved with near zero compute (see my question section for solutions).

Critical weaknesses

The paper is an analysis paper over two input formats (lists and paragraph), and thus very limited in scope. The setups are also mostly synthetic. No more complex counting benchmark (like for math problems) is considered.

The paper analysis is quiet limited, and its key contribution is to go beyond exact accuracy, but also measure how far the answer is off from the target value (plus/minus) and relates with frequency and context.

The paper does not go beyond the analysis, no attempt to try to devise a solution.

The setup of counting in a list and counting words in a paragraph is very limiting, and is quiet unrealistic to try to solve it with an LLM. It is the opposite of what should be done: searching a problem for a solution (LLM), rather than finding a solution to a problem.

Overall, I do find this study severely underscoped both in scope and analysis for an ICLR paper.

Questions for authors

What was the motivation to use an LLM for these two rather trivial problems? Because counting items in a list can be done with a one call to a simple Python function, for example, and counting tokens in a text can be done with a single Linux command line. Why throw an LLM at such problems, isn't that an overkill?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5Ji4 (1/3)

We thank the reviewer for engaging with the paper and recognizing counting as an interesting problem. The reviewer's concerns relate to the motivation for studying LLMs on a task that “can be done with a single call to a simple Python function”, particularly in the era of tool-based LLMs and agentic AI. We answer the reviewer's questions and address their concerns here.

"What was the motivation to use an LLM for these two rather trivial problems? […] Why throw an LLM at such problems, isn't that an overkill?"

  • We agree that LLMs should not be used to solve tasks like counting string occurrences where a simple grep and wc could do the job. However, the simplicity of this task is what makes it a useful testbed for evaluating the robustness of LLM counting. The evaluation strategies and controlled input perturbations presented in our work characterize the kinds of errors LLMs are prone to and identify how generalizable they are under various input conditions. In short, we use counting as a diagnostic probe for LLMs, not as a benchmark to improve performance on.

"How important is it to study very simple counting problems with LLMs, especially now in the era of tool-based LLMs and agentic AI?"

  • Elementary failures matter more as capabilities grow. AI systems now propose solutions to open problems in mathematics (OpenAI, 2026) making it all the more important to study their limitations. Success on hard problems assumes mastery over elementary ones. However, that assumption is not always guaranteed (West et al., ICLR 2024).
  • Real-world counting is rarely greppable, it is embedded within semantic contexts and cannot be offloaded to a tool call. Counting appears inside larger tasks (aggregating over long documents, tracking steps in an agentic loop; Hsieh et al., 2024; Song et al., 2025), with filters that are rarely greppable. In such settings, a tool call without semantic understanding is insufficient to perform the task with precision. Even under our most favorable, exact-match definition, frontier models miscount 44–59% of real Wikipedia passages (Fig. 5a).
  • Tools do not remove the need to understand the model. A model must recognize when to defer to a tool. Because its counting errors are mostly small undercounts (often within 20% of the target), its answers look plausible, which is exactly when it is least likely to defer and a user is least likely to notice.

"There is prior work on counting objects in generated images, thus language vision problems. There I would find the problem much more interesting to tackle."

  • We appreciate the pointer to object counting in images. We firmly believe that the fundamental aspect of counting, namely enumeration, is captured more directly in text, despite being less complex. Counting failures in vision mix perceptual errors (detection, occlusion, segmentation) with numerical ones (for instance, Rahmanzadehgervi et al., ACCV 2024 shows that VLMs struggle to count when object occlusion is at play). In text, the items can be identified and cleanly segmented to isolate the enumeration and accumulation aspects of counting. Moreover, any account of VLM counting presupposes an understanding of how the language backbone counts; Hasani et al. (2026), which we cite, study counting mechanisms in both LLMs and VLMs for this very reason.
AuthorsOpenReview ↗

Response to Reviewer 5Ji4 (2/3)

(continued)

"... study [is] severely underscoped both in scope and analysis for an ICLR paper."

These studies have deepened our understanding of LLMs by identifying how LLMs can fail on simple tasks when minimal input variations are applied. Our work follows a similar approach to characterize counting failures to show that frontier models are still lacking in generalizability.

"The paper is an analysis paper over two input formats (lists and paragraph), and thus very limited in scope. The setups are also mostly synthetic."

We agree and strongly believe that the key findings of our work hinges on the broader set of interventions we target within these two input formats:

  • List composition:
    • single-entity vs. multi-entity
    • single entity vs. single entity with case mixing at 20%
  • Target substitution:
    • original noun vs. nonce word vs. cipher vs. random hash.
  • Context degradation:
    • incremental context corruption from 0% to 100%
    • target substitution with nonce / cipher / random hash
  • Context masking,
    • context counting (unmodified) compared with masking out words in context that are irrelevant to the task.

The synthetic nature of the study enables the systematic and controlled treatment of the failure mode diagnosis. However, we also believe that devising a solution in a synthetic setting does not necessarily translate to real-world settings, especially because counting (in particular, enumeration) is embedded within more complex tasks and can be obscured by closely related linguistic aspects like entity recognition and disambiguation. Our work focuses on identifying failure modes that are inherent and exclusive to the enumeration aspect of counting. If anything, it should be seen as motivation for studying the more complex use cases involving counting and we will explicitly state that in the revised version of the paper.

"The paper does not go beyond the analysis, no attempt to try to devise a solution."

We believe a precise characterization is crucial to devise targeted remedies and such solutions prove valuable only when it can be reliably applied in real-world applications involving counting. However, such scenarios often require resolution of other closely related aspects of counting and thus requires a more holistic treatment, which is outside the scope of our work. That said, some of the interventions presented in the paper can be viewed as preliminary evidence of more precise counting abilities in LLMs (response to Reviewer giTx, W1). For example, letter case variations in as little as 20% of list items significantly reduces miscounts in all three models and sharply reduces error magnitude for GPT-5.2 and Gemini 2.5 Pro. Masking the context to only include words with substrings matching the target's surface form reduces miscount rates, suggesting that pre-filtering long inputs to candidate spans can help.

AuthorsOpenReview ↗

Response to Reviewer 5Ji4 (3/3)

(continued)

References in the response:

(Already cited in the paper and used above: Hasani et al. 2026; Hsieh et al. 2024; Song et al. 2025.)

Reviewer_agW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper characterizes text-counting errors in line-separated lists and exact target-word occurrences in Wikipedia passages. Three models are evaluated using error frequency, relative magnitude, and directional asymmetry. Experiments vary list diversity, letter case, target identity, and context corruption. Reported findings include increasing errors at larger counts, improvements with list diversity, model-dependent contextual performance, and a non-monotonic corruption pattern for GPT 5.2 (Secs. 3–7; Figs. 1–9).

Critical strengths
  1. Reporting frequency, size, and direction provides a richer description than exact accuracy alone. Figure 1 illustrates why near misses and large errors should be distinguished, while Figures 2–5 expose differences between directional frequency and typical magnitude (Sec. 3.3, pp. 3–5).

  2. The contextual interventions separately manipulate target identity and surrounding text while attempting to preserve target occurrence counts and positions. This organized design makes the non-monotonic corruption pattern a concrete, testable observation (Sec. 5.2; Tables 1–2; Figs. 7–9).

  3. The appendix supplies prompts, lexical inventories, and detailed outcomes. Tables 8 and 11 distinguish corrected errors from newly introduced errors, providing information beyond aggregate changes (Appendices A and C).

Critical weaknesses
  1. The magnitude metric changes its conditioning population. Equation 2 defines median relative error over incorrect predictions only, and Figure 2 repeats that restriction. Table 18 includes exact predictions, yet Sec. 5.2.2 calls its 13.33 value the same M_RE. An unconditional median can change because the proportion of exact answers changes, even if the severity of actual errors does not. The interpretation of the corruption curve therefore conflates failure frequency with conditional severity. These definitions need to be reconciled before the curve supports the proposed diagnostic account (pp. 4, 8, and 27).

  2. The directional indices do not provide the claimed protection against severe tail errors. Section 6 says that tracking both indices ensures such errors remain visible. Consider targets of 100 and signed errors (−1,−1,−1,1,1,L), with integer L>1. Each direction occurs three times, and both conditional median relative errors equal 1%, so both indices are zero while the largest error is unbounded. This counterexample directly contradicts the guarantee. The indices remain descriptions of directional balance in typical errors; a tail-sensitive summary or a narrower claim is needed (Eqs. 3–4; Sec. 6, p. 9).

  3. Several interventions change more than the input property they are used to explain. As documented in Table 3, ordinary single-entity counting uses a word-occurrence instruction, while mixed-case counting uses a total-item instruction. The contrast therefore cannot isolate letter case without establishing that the actual experiments used identical instructions, or reporting the crossed comparison (Sec. 5.1; Fig. 6). The diversity and corruption experiments have a related identification problem: multiword entities, ciphers, and hexadecimal strings can change token segmentation and sequence length alongside semantic coherence. Matching target counts or character lengths does not hold those properties fixed. This limits the explanatory increment over the counting and elicitation work discussed in Sec. 2, including Xu and Ma (2025), even though the observed responses to the complete interventions remain informative (Tables 1 and 7; Sec. 6).

  4. The final count does not locate the source of contextual failure. The task excludes inflections and embedded substrings, while Sec. 6 associates inflectional forms with poorer performance. A wrong count can result from selecting the wrong occurrences, incorrectly accumulating selected occurrences, or both. The scratchpad requests identification but the evaluation only scores the final number. No direct evidence found in the manuscript. Occurrence-level selection accuracy is not reported, leaving the distinction between target recognition and enumeration unresolved. That distinction matters for interpreting why contextual manipulations help or hurt counting (Sec. 3.1; Table 5; Fig. 15).

  5. The evaluated population and treatment of invalid responses are insufficiently traceable. Section 3.1 reports 156 passages, Table 18 begins with 158 and varies denominators across corruption levels, and Tables 8 and 14 use 151 and 144 pairs. Valid filtering could explain these differences, but identifiers and exclusion rules are missing. The count grid also differs between Sec. 5.1's 30 counts below 2000 and Table 6's two groups of 30. Meanwhile, Sec. 5.2 attributes Claude refusals to guardrails without a response classification rule or refusal-rate breakdown, although Eqs. 1–4 require numerical predictions. Endpoint versions and effective generation limits are also unspecified (Sec. 3.2). These details are needed to determine whether comparisons use the same examples and how refusals affect the numerical summaries.

  6. The appendix prose contradicts some of its own results. Appendix C says Gemini's aggregate relative error decreases under context ablation, but Table 9 reports the mean increasing from 3.102 to 5.436. The discussion also says GPT target-ablation effects do not survive the reported tests, whereas Table 12 includes corrected results supporting reduced error magnitude. Readers cannot identify which analysis version supports the prose and the model comparisons. This is a substantive presentation problem because the direction of the reported effect changes, and it should be reconciled throughout the results discussion (Appendix C, pp. 23–25; Sec. 5.2).

Questions for authors
  1. Which exact metric, example set, and invalid-response rule produced Figure 9 and the 13.33 value? Please reconcile the conditional definition in Equation 2 with Table 18, including the changing denominators and handling of refusals, so the corruption trend can be interpreted on a consistent population (Sec. 3.1; Sec. 5.2.2).

  2. Were the mixed-case and ordinary-list runs actually given the different instructions shown in Table 3, or is that a documentation error? If the instructions differed, is there an existing comparison that separates the instruction effect from the case effect (Sec. 5.1; Figs. 6, 12–13)?

  3. Are Tables 9 and 12 or the surrounding Appendix C statements authoritative? After reconciling them, which conclusions about context and target ablation still hold for Gemini and GPT (Sec. 5.2; Appendix C)?

3 comments
AuthorsOpenReview ↗

Response to agW8 (1/2)

We thank the reviewer for their detailed review and the careful reading of the appendix content. We address the reviewer’s concerns and incorporate their suggested changes in the paper. None of the changes alter the paper’s original findings.

W1 / Q1. Magnitude metric. We thank the reviewer for catching this. Previously, the 13.33 value reported as $M_{RE}$ was computed over all instances rather than over miscounted instances only, as Eq. 2 requires. Recomputing the metric over the appropriate conditioning population does not change the overall trend. After correction, GPT-5.2's median relative error is 11.39% (95% CI [10.00, 15.56]) with no context perturbation, 20.00% ([16.67, 24.09]) at 50% corruption and 11.27% ([8.99, 14.63]) with full corruption. The interval at 50% does not overlap either endpoint, confirming that the non-monotonic effect of incremental context degradation persists under the corrected metric. We have revised Section 5.2.2 and Figure 8, in light of this.

W2. Directional indices do not provide the claimed protection against severe tail errors. We will revise this discussion point in Section 6 to be less ambiguous in the manuscript.

To clarify, Section 6 originally stated:

"[…] while the model undercounts most of the time, its rare overcounting errors are catastrophically large hallucinations. Tracking both indices ensures severe tail-end errors are not obscured by sheer error frequency."

This discussion point was meant to highlight how the asymmetry indices proposed in the paper helps with identifying directional miscounting biases in frequency and magnitude that aggregate metrics like miscount rates and median relative error are likely to miss. The asymmetry we refer to here is between directions, not that of outliers within a direction (which is what the reviewer’s counterexample points to).
In this context, the usage "tail-end errors" was referring to errors in the less frequent direction whose typical (median) magnitude lies in the upper tail end of the error distribution across the dataset, not to individual outliers for a given target count. Consequently, in the case of the reviewer’s example with errors $

−1,−1,−1,1,1,L≫1-1,-1,-1,1,1,L\gg1

$, both indices correctly report no directional asymmetry in frequency or median magnitude, since medians are insensitive to isolated outliers.

However, we acknowledge that the original phrasing is open to the reading that the asymmetry metrics guarantee the visibility of extreme individual errors and will update the discussion point to state it with more clarity.

W3 / Q2. Interventions change more than input property. The discrepancies in the instructions for the single entity and its mixed case variation reported in the appendix was a formatting error that is now fixed. Variations in tokenization and input length are inherent to any text-level manipulation, and we acknowledge this limitation in Section 6. However, we also believe that the answer-preserving nature of these perturbations allows us to test the robustness of LLM counting to superficial variations in the inputs.

AuthorsOpenReview ↗

Response to agW8 (2/2)

(continued)

W4. Analyzing the source of counting failures. We thank the reviewer for this suggestion and have conducted the proposed analysis on the existing model outputs. Across all three models, we find that overcounts are rare even in passages containing inflectional forms of the target, indicating that models rarely count invalid forms. On analyzing the scratchpads of miscounted instances (in Claude and Gemini; GPT-5.2's API exposes only reasoning summaries), we find that these errors arise at the selection stage rather than at accumulation: models mostly fail to list out all the matching occurrences from the passage rather than explicitly rejecting them. For instance, for the target begging (true count 83), Claude Opus 4.5's scratchpad lists 63 occurrences, correctly marks them all as valid, and reports Total count: 63; the remaining 20 occurrences are never listed. This analysis is included in the appendix of the revision.

W5. Traceability. The dataset contains 158 unperturbed Wikipedia passages (the "156" value in Section 3.1 was a typo). Denominators vary because of the exclusion criteria (namely, refusals, truncations, and non-compliant responses to instructions) and paired tests are performed only on instances with a valid count under both conditions. The 144 pairs in the GPT-5.2 was from an incomplete run and is now corrected to reflect the full 158 pairs (no changes in effects reported). The single- and mixed-entity list counting instances use the same 30 target counts (from 1-2000 sampled uniformly at random) and the case-variant version of single-entity counting uses the same system prompt. Endpoints and output limits (Sep–Nov 2026): GPT-5.2 and Claude Opus 4.5 via OpenRouter (10k tokens), Gemini 2.5 Pro via an organization-specific API router (32k tokens).

In the revised version, we explicitly state these exclusion criteria in Section 3.3, remove the refusal attribution to "security guardrails", add refusal rate breakdown by model, and list endpoints and generation settings in Section 3.2. Item identifiers and validity flags will be released.

W6 / Q3. Appendix C. The tables are correct; the Appendix C text and one sentence in Section 5.2 had not been updated after the final analysis. Both now state that context ablation significantly increases Gemini 2.5 Pro's relative error (3.10 → 5.44, $p_{Holm} = 0.018$; submitted Table 9, now Table 14) and that target ablation significantly reduces GPT-5.2's (10.00 → 6.48, $p_{Holm} = 0.002$; submitted Table 12, now Table 17). Neither ablation significantly changes miscount frequency after Holm correction.

We have revised the paper to make the metrics, population specifications, and claims precise as suggested by the reviewer. We confirmed that none of these modifications affect any of the findings from the original submission. In particular, errors are predominantly undercounts, list composition affects downstream counting performance, and partial corruption produces larger errors for GPT-5.2 under the reviewer's stricter metric. We appreciate the careful reading by the reviewer and hope to have adequately addressed their comments and concerns in our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and AC

We thank the reviewers for their feedback. Their concerns fall into three groups, summarized below with the corresponding revisions. None of the revisions change the paper's original findings.

1. Corrections in the Appendix (agW8).
We have applied the population conditioning described in the evaluation metrics and updated the appendices to address any documentation discrepancies (e.g. the prompts in Table 3, the passage count in Section 3.1, the count grid in Section 5.1).

2. Traceability (agW8). Our revision now states the exclusion criteria (refusals identified by the provider's finish reason, responses without a count in the requested tag, chiefly truncations) and lists endpoints and generation limits. We will release the generated responses with Item identifiers and validity flags.

3. Scope and contribution (5Ji4, giTx). The paper uses counting as a diagnostic probe of frontier models, following an established line of model-analysis research at ICLR. We have clarified the motivation and the stated implications in the introduction, and point to the paper’s experimental interventions be viewed as avenues for improving task performance: namely, introducing letter case variations in single-entity lists and context masking when counting occurrences within a natural context.

New analysis. At agW8's suggestion, we scored the models' scratchpads against ground-truth occurrences. Contextual miscounts are more likely when inflectional forms of the target are present, however, the miscounts are a result of undercounting. In GPT-5.2 and Claude Opus 4.5, we analyzed the model’s reasoning outputs and traced the source of this undercounting to limited coverage (with entire spans containing the target being not enumerated at all) rather than a false rejection of valid occurrence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.