acceptodds
Under review as a conference paper at ICLR 2027

A minimal probe of task modeling in language models

Abstract

Language models use long contexts and score well on knowledge and reasoning benchmarks, but neither says whether they can carry out work whose next step depends on everything that has happened so far, as a research program or a long-running agent does. We call that ability task modeling. Counting identical items is its minimal case, a total to which every item contributes and that no item records; the question is whether a model's limit on counting is a limit of that ability. Here we introduce counting capacity (CC), the length up to which a model's exact count is its majority behavior, and measure it for 205 models in 296 configurations. Counting fails early: every resolved measurement loses count, at a median of 50 items and at best 627, and across release cohorts the median rose 1.6-fold while the median context window grew about 80-fold. As far as the measurements can tell, the limit is not a simple limit of counting: the failures are abrupt, structured collapses rather than careless slips; the capacity shifts severalfold with the encoding while the ordering of models holds; native reasoning gives no median gain across 77 verified contrasts, with large model-specific gains and losses; and a concurrently executed rule task consumes it in both models tested. Mechanistic analyses of six open-weight models locate the count in an internal state that stays readable in magnitude past the boundary, and an intervention that carries the count forward carries, unchanged, the state of an object history. Counting capacity tracks the METR task horizon beyond general capability on the 24 models shared, scales with a ladder of grid-execution tasks, and co-varies across vendor families with the length at which a state-tracking task fails when reasoning is off, a limit the same models escape when reasoning is on and the state can be written down. These findings are consistent with counting measuring, at a few hundred characters per probe, a distinct, silent dimension of capability that a chain of thought masks whenever the state is nameable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
counting capacity, state tracking, long-horizon execution, language model evaluation, mechanistic interpretability, chain-of-thought reasoning, scaling, task horizon, linear probes, activation patching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uHbrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates "task modeling" (the capacity to maintain a global, unwritten aggregate across context) via a minimal probe (exact counting of repeated tokens (a,a,a,…a, a, a, \dots)). The authors introduce Counting Capacity (CC), estimated via a BIC-gated, two-regime Bernoulli change-point estimator (p1>0.5>p2p_1 > 0.5 > p_2). Evaluating 205 language models across 296 configurations, they report that models fail abruptly at a median of 50 items (maximum 627). Mechanistic interventions across six open-weight models (linear probes, decode-point clamps, mid-depth activation shifts, and boundary-window transplants) indicate that count representations saturate and are not read directly from early linear directions. The paper further connects CC to agentic benchmarks (METR task horizon, grid execution, and state tracking), concluding that CC reflects a distinct, domain-general latent capacity.

Critical strengths
  1. The BIC-gated change-point Bernoulli likelihood (p1>0.5>p2p_1 > 0.5 > p_2) is a major methodological improvement over naive first-error metrics, properly handling isolated early slips (observed in 284/296 runs).
  2. Comprehensive evaluation spanning 205 models (296 configurations) with rigorous tracking of bounds (256 mistake-resolved, 39 budget-bounded), full prompt disclosure, and low replication cost ($\approx$$200).
  3. Commendable candor in reporting replicate run-to-run variance (Table 1), instruction-following failures (3,887/45,287 deviations in App. D), and reasoning-trace failure modes (App. G).
  4. The probing suite extends beyond passive linear probes to include causal tests: decode-point clamps with random controls, chimera decompositions, mid-depth item shifts, and boundary transplants with sham conditions.
Critical weaknesses
  1. Construct Validity: Surface-Form Sensitivity vs. Latent "Task Modeling" (W1). The central claim that homogeneous counting probes a domain-general capacity is directly challenged by the paper's own data:
  • Delimiter sensitivity: Changing the separator from a comma to a section sign drops CC fourfold (89 to 19).
  • Heterogeneous collapse: On mixed sequences, accuracy drops to 10–48% at counts well below the homogeneous CC threshold.
  • Known failure mode: Repeated-token regimes trigger well-documented attention sink instabilities and representational collapse. Rather than isolating abstract task modeling, CC appears to measure model stability under long runs of identical tokens.
  1. Structured errors cluster heavily around offsets of ±100\pm 100 and −50-50, alongside round-number attractors. This strongly suggests digit-chunking failures in tokenization and multi-digit emission rather than representational loss. The authors do not evaluate standard formatting controls (e.g., words, zero-padded strings, or modular arithmetic like count mod 10).
  2. The claim of a shared capacity across counting, grid execution, and state tracking rests on single, unreplicated runs across only two models with divergent behaviors: grid load degrades Sonnet's grid accuracy without shifting its CC, whereas pointer chains degrade Sonnet's CC by 29% but leave Gemini unaffected.
  3. The METR correlation was identified after screening 66 benchmarks without multiple-comparison adjustments. In a partial correlation controlling for general capability (ECI, n=24n=24), the incremental out-of-sample R2R^2 gain is 0.060.06 with a 95% confidence interval containing zero ([−0.24,0.30][-0.24, 0.30]), heavily leveraged by a few high-end models.
  4. Nominal 95% profile intervals derived from single scans severely understate run-to-run uncertainty. In replicate testing (Table 1), 100% of Claude Sonnet 4.6 replicate boundaries (370–424) fall outside its catalogue interval ([315,320][315, 320]), indicating that catalogue intervals do not support fine-grained model comparisons.
  5. The paper omits citation of Dai & Fan (arXiv:2605.02028, May 2026), which predates this submission by 4.5 months. Dai & Fan uses the identical prompt string, stimulus structure, and "Counting Capacity (CC)" terminology across 126 models with mechanistic probing. The authors must explicitly articulate their incremental contribution and reconcile large metric discrepancies (e.g., Claude Opus 4 at 2,176 vs. 580/627 here).
Questions for authors
  1. How does your methodology and causal probing suite differentiate from this preprint? Why are the reported CC values so divergent on shared models? Dai & Fan (arXiv:2605.02028, May 2026)
  2. In a two-way mixed-effects model across diverse encodings, what percentage of CC variance is attributable to the model versus the surface format?
  3. Does the boundary or error clustering shift when prompting for outputs in written words, zero-padded integers, or under modular arithmetic (count(mod10)\text{count} \pmod{10})?
  4. Can the dual-task experiments in Table 14 be replicated across ≥3\ge 3 runs and extended to open-weight models with a non-executive control task?
  5. How do you reconcile the claim that the internal readout "keeps advancing" with the flat post-boundary probe slopes observed in four of the six models?
2 comments

Thank you for the detailed review. We ran the three experiments you asked for and revised the paper. Some of the results cut against claims in the submitted version, and we have changed those claims.

W6 / Q1. arXiv:2605.02028 and the different CC values

We have written to the AC about this preprint under the ICLR policy on arXiv versions. On the technical question, the two numbers measure different things. The preprint calls a length stable when the mean absolute error over 16 sampled targets is below 5% of the length. Our CC requires the exact count to be the model's majority answer.

For Claude Opus 4 with reasoning off, exact counting stops being the majority at 580. From 581 to 988 (the end of the scan) the reply is exact in 45% of probes, but 84% of the errors are exactly one short, the mean relative error is 0.08%, and no reply is more than 3.4% off. A 5% tolerance passes that whole range; an exact-majority rule does not. The 627 in our catalogue is the same model with reasoning on.

Relative to that preprint, the submission covers 205 models (against 126), replaces the tolerance rule with the change-point estimator you describe in Strength 1, and adds the METR, execution-ladder, state-tracking, concurrent-load and transplant analyses.

W1 / Q2. How much of CC is the model and how much is the surface form

Across 16 API models from seven vendors and all eleven encodings (176 scans; new Table 2, Appendix A.6), the model accounts for 77% of the variance in log CC and the encoding for 6%. Among six open-weight models with similar capacities, the two shares are about equal.

PanelModelEncodingInteraction + scan noiseKendall's W
16 API models76.6% (95% CI 50.3 to 88.4)5.6% (2.5 to 16.9)17.8% (7.9 to 34.9)0.85
6 open-weight models, local38.5%29.8%31.7%0.71
all 2271.9%8.8%19.3%0.83

The API panel's median capacities run from 12 to 295; the open-weight models lie within a factor of three of each other. One encoding can also move one model a long way: the section sign cuts CC to a third or less in five of the 16 API models and leaves it within 6% in three. The revised text says that CC is a property of a model under an encoding.

Accuracy on mixed sequences is lower than on homogeneous ones at the same count. That argues against instability under long runs of identical tokens as the cause of the limit, since removing the runs makes counting harder. The first ladder task (counting a target digit among distractors) and object tracking over natural sentences have no such runs either, and both scale with CC, the second between vendor families. We agree that none of this proves a domain-general capacity. The abstract, introduction and discussion now state task modeling as a hypothesis and say that we do not show counting to be necessary or sufficient for agentic work.

W2 / Q3. Words, zero-padded digits, and count mod 10

The boundary does not shift consistently with the output format. The hundred-item errors do not shift either in the one model that makes them, but that test is weaker than we would like, for the reason given below.

We ran all three formats on nine configurations (Appendix A.7, Table 3). Two Claude configurations did not follow the formats and are excluded: one wrote its counting out in prose until the output cap, the other stated the integer first and then converted it. In the seven that remain, CC with words is 0.80 to 1.28 of its bare-integer value (median 1.00). With the last digit alone it is 0.81 to 1.03 in four configurations and higher in the two Gemini models, and one small open-weight model fails the format itself.

Only Gemini 3.7 Flash makes hundred-item errors in this set. Its reply is exactly 100 short in 32 of 47 errors past the boundary with integers, 31 of 62 with words ("one hundred ninety-seven" for 297), and 42 of 58 with one digit per token. Asked for the last digit alone, it is right in 233 of 240 probes up to 479. The hundred is lost and the units are kept, whatever the format.

This model reasons before it answers (about 1,800 reasoning tokens per probe), so the format only changes how an already-formed count is written. The result rules out the writing of the final answer as the source of the error and leaves open a digit-level slip inside the reasoning text. The configurations that answer immediately were scanned to at most 300 items and make no hundred-item errors. The nearest evidence from them is an error of ten in Gemma 4 26B, which is as common with words (14 of 42 errors) as with integers (11 of 49). The paper states this scope, and the test on an immediate-answer model with a capacity of several hundred has not been run.

W3 / Q4. Replicating the dual-task experiment

The replication changes our claim. The row-rule drops we reported (36% and 28%) do not replicate, and a second task lowers CC beyond its controls for one load in each model.

We scanned six conditions five times each in both models (60 scans; Appendix H, Table 17), adding a second control the model has to read: a passage as long as the grid load with a four-option question. Ratios are to the mean of five solo scans.

LoadGemini 3.7 FlashClaude Sonnet 5
solo CC, mean ± SD357 ± 4295.0 ± 4.6
control, matched length0.910.93
control, reading1.030.90
pointer chain0.930.67 (62 to 64 in all five scans)
row rule, 60%1.080.92
grid rule, 60%0.81 (p = 0.03 vs solo, 0.24 vs control)0.89, grid answers 65% correct

Your reading of the single scans was right. The two row-rule scans fell where an occasional scan of each model collapses early. The loads that hold are the pointer chain in Sonnet, in every scan, and the grid in Gemini, less securely. We removed the dual-task claim from the abstract, rewrote the paragraph in Section 3 and Fig. 3C around the replicates, and now say in Sections 5 and 6 that the evidence for a shared capacity is limited to one load per model and does not separate a shared resource from other dual-task interference.

We also ran the conditions on three open-weight models. That run does not test the question and we report it as such: answering immediately, the models solve the rule and grid loads in 0 to 7% of probes, and in two of the three the controls alone lower CC by 22 to 48%.

W4. METR and multiple comparisons

The METR partial correlation survives adjustment for the whole screen, and we agree that the gain in prediction is not established. We computed the partial correlation given ECI for every benchmark in the screen that shares enough models. METR's is the largest positive one (0.74; the next are 0.51 on ten models and 0.46 on fifty). Its unadjusted p is 4.9e-5, which stays below 0.004 after Bonferroni adjustment for every benchmark screened. The bootstrap puts the out-of-sample gain at 0.06 with an interval of -0.24 to 0.30. Section 5 now reports both numbers and says the relation rests on 24 models and should be tested on more.

W5. Profile intervals and run-to-run variation

Agreed. The new scans make the point more strongly than Table 1 did, because the Table 1 rescans start at 0.8 of CC and cannot see an early collapse. Five solo scans from short lengths give a standard deviation of 12% of CC for Gemini 3.7 Flash (290 to 394) and 5% for Claude Sonnet 5, and a single scan can fall 20 to 30% from the others. Section 3 and Appendix A.2 now say so and state that we do not interpret differences of that size between two models; the catalogue spans two orders of magnitude.

Q5. "Keeps advancing" against the flat slopes

The wording was too strong. Readout slope between 1.0 and 1.2 times CC (the stable-regime slope is 1 by construction):

Modelfirst probed layersecondthirdlast
gemma-4-26b0.330.290.24-0.59
gpt-oss-120b0.32-0.09-0.27-0.78
gpt-oss-20b0.060.290.13-0.33
llama-3.3-70b0.36-0.67-0.70-1.00
muse-glimmer-30b0.330.19-0.52-1.69
qwen3.6-35b0.510.370.15-0.60

At the first probed layer the readout still rises, at a third to a half of its earlier slope in five models and 0.06 in the sixth. At the last layer it falls in all six. Section 4, the Fig. 4D caption and Appendix I.1 now give these numbers in place of "keeps advancing".

Reviewer_RkXAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose to use counting tasks with sequences composed of identical items, to probe the ability of language models carrying out the whole context. They introduce Counting Capacity (CC), the length up to which a model's exact count is its majority behavior, to measure this ability. The authors evaluate 205 models and 296 settings, finding that the increase of CC lags far behind the expansion of context length, and that enabling reasoning mode does not yield an overall improvement in CC. Through further analysis, the paper suggests that counting may reflect a form of task modeling capability that is not yet adequately captured by conventional benchmarks.

Critical strengths
  • The research question is clear, and the probing task is straightforward and easy to interpret. The paper transforms a simple failure phenomenon into a systematic measurement problem, which is highly insightful.

  • The evaluation is comprehensive, and the measurement methodology is more robust than relying on a single instance of failure. The experiments cover multiple model families and various inference configurations. Furthermore, the study examines factors such as prompt and encoding, which helps disentangle intrinsic model behavior from the effects of the measurement protocol itself.

  • The paper not only analyzes the structure of errors but also explores underlying mechanisms. These experiments provide a deeper understanding of counting failures.

Critical weaknesses
  • My main concern is the jump from counting to task modeling. I can see why counting is a useful test. But it is less clear why doing well on this particular task should relate much to running an agent or carrying out research. The paper shows some connections, but does not establish that good counting is either enough for these tasks or required to solve them. A useful proxy should satisfy at least one condition.

  • CC changes quite a lot when the symbol changes, even though the underlying counting problem stays the same. This makes it hard to tell how much of the score reflects a general ability and how much reflects familiarity with a particular input format.

Questions for authors

See weaknesses.

2 comments

Thank you for the review. Both weaknesses are fair.

W1. From counting to task modeling

We agree that the paper does not establish that good counting is required for, or enough for, running an agent. The revision says this in the abstract, the introduction and the discussion, and states "counting is a minimal case of task modeling" as a hypothesis. The introduction now defines a task model by two properties (every step so far contributes to it, and no token in the context records it), which is what counting shares with the long tasks we have in mind.

Where the evidence stands:

  • CC relates to long-task measures more than to knowledge benchmarks. Its correlation with the METR horizon survives conditioning on general capability (partial ρ = 0.74 on 24 models, p below 0.004 after correction for the whole benchmark screen), while its correlations with HLE, SWE-bench and GPQA are weak or explained by general capability. Whether CC improves prediction of the horizon is not established (gain 0.06, interval -0.24 to 0.30).
  • Counting is not required when the state can be written down. With reasoning on, object-tracking capacity rises by a median factor of 4.6 and CC by 1.07. If CC matters for agents, it should matter for state that is never written, and we say this is untested.
  • The direct test of a shared capacity became weaker in revision. We replicated the dual-task experiment five times per cell, and a second task lowers CC beyond matched controls for only one load in each of two models. The paper now reports that.

W2. CC changes with the symbol

When models differ widely, most of the score is the model. We extended the encoding sweep from 3 models to 16 models from seven vendors (176 scans; Appendix A.6, Table 2). In a two-way decomposition of log CC, the model accounts for 77% of the variance (95% interval 50 to 88%), the encoding for 6%, and their interaction with scan noise for 18%. Models keep their rank across the eleven encodings (Kendall's W = 0.85).

The share depends on the panel: among six open-weight models with similar capacities, the model accounts for 39% and the encoding for 30%. A single encoding can also move one model a long way: the section sign cuts CC to a third or less in five of the 16 models and barely changes it in three. We now describe CC as a property of a model under an encoding and do not compare models across encodings.

Follow-up to Reviewer cV1z

We have uploaded a revised version. The changes made for readability are:

  • The abstract is rewritten in short sentences and is a quarter shorter.
  • The introduction defines task modeling by two properties and gives the formal definition of CC (the cut between a mostly-exact and a mostly-inexact regime in the record of right and wrong replies). It then says what each section tests.
  • Paragraph headings in Sections 3 and 4 now state the finding of the paragraph, and each new experiment is described before its numbers.
  • The discussion separates what the measurements show from the hypothesis and lists the limits of the study.

If particular passages are still unclear, line numbers would help us fix them.

Thanks for your detailed response. I will maintain my original score of 3.

Reviewer_cV1zOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is impossible for me to write a summary of this paper, because it is extremely unclear. The motivation seems about probing the capacity of LLMs to solve a task when the task requires a complex understanding of the input, that cannot be represented sequentially and with a summary. Counting capacity is introduced, based on the task of counting how many characters are in a given string with repeats. There seems to be a lot of experiments that want to show that this measure is fundamentally different from others and useful.

Critical strengths

Cannot really find critical strengths, sorry. There is a need to study the reasoning capability of LLMs, indeed. This paper might be a try in that direction. Many experiments.

Critical weaknesses

Paper is not well written. It is for me impossible to understand, despite a thorough try. There is sloppiness in the language, for example "it" or "this" that is unclear to match with what it should relate to. Capability task modelling and CC are badly defined. Experiments are not well described, motivated and often remain unclear, I cannot say what the contribution of this paper is.

Questions for authors

You have no red line in your paper, the research results are not logically described. What is your paper about, really? What have you discovered?

Comments on the paper

1 comment

We thank the reviewers. The revised PDF is ten pages of main text. The changes are:

  1. Framing. Task modeling is stated as a hypothesis throughout, with an explicit statement that we do not show counting to be necessary or sufficient for long-horizon work.
  2. Encodings (uHbr Q2, RkXA W2). The sweep now covers 16 API models and 6 open-weight models. The model accounts for 77% of the variance in log CC in the API panel, less in a panel of similar models.
  3. Output formats (uHbr Q3). The boundary does not shift consistently when the answer is requested in words, spaced digits or as the last digit. The hundred-item error survives these formats in the one model that shows it, which reasons before answering, so the control excludes the writing of the final answer only.
  4. Dual-task replication (uHbr Q4). Five scans per cell. The row-rule effect in the submitted version did not replicate, and the claim is reduced to one load per model.
  5. Statistics (uHbr W4, W5). Multiple-comparison adjustment for the METR relation; run-to-run spread from scans that start at short lengths.
  6. Wording (uHbr Q5, cV1z). Post-boundary readout slopes reported as numbers; abstract, introduction and discussion rewritten.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.