acceptodds
Under review as a conference paper at ICLR 2027

Counting on Thinking: Tracing Evidence Integration in Language Models

Abstract

Computational resources are finite, so no intelligent system can afford extended computation for every decision. Humans and animals handle frequent operations with automatic system 1 processes and reserve costly system 2 computation for problems that warrant it. Large language models (LLMs) can likewise spend extra inference-time computation on hard problems, yet their direct answers struggle even with counting, an elementary integration of input that humans and animals perform automatically. We ask why such an operation requires thinking in LLMs. Evidence integration is so basic that psychology and neuroscience have long used it to probe decision-making; our LLM version presents one letter per conversational turn and asks the model which of two target letters appeared more often. A running difference between the two counts solves the task optimally, weighting every letter equally, and fits this format exactly: the tokens at each turn can represent the difference accumulated over all earlier letters and update it with the new one. Direct responses nonetheless weighted evidence unevenly, with a strong recency effect, and assigned less probability to the correct answer as the task grew harder. Thinking improved performance and made the integration weights nearly uniform, yet final-query attention stayed concentrated on the ends of the sequence in both modes. Instead, reasoning trajectories showed the model revisiting the input, recounting the letters, and checking intermediate counts that the answer then read, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. This recounting cost reasoning tokens that grew with the number of letters far more than with coherence. Nor did outcome feedback bring the computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and the recency effect strengthened, yet the models grew more confident. Humans and animals amortize such elementary computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available is a central question for how future models allocate computation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
interpretability, reasoning, large language models, resource rationality, evidence integration, in-context reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hqB8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The authors study the performance of open-weight LLMs, e.g., Qwen, Gemma and Nemotron models, on a simple counting task. Each LLM is given an input sequence containing multiple letters and is asked to determine which of two letters appears more often in the sequence. The paper questions why LLMs have difficulty solving this counting task when directly predicting the answer, compared to when using reasoning. They suggest that an LLM should keep a running count the way a human or animal might and should therefore amortize its reasoning cost over the sequence of inputs. However, the reason why the LLM does not keep a running count of the letters is because the authors explicitly do not allow it to do so! They do not allow the LLM to output any intermediate words while receiving the letters, except for a placeholder “WAIT” word, so there is no mechanism by which an LLM could keep a running count of the letters while receiving the input. The authors show, using a regression analysis, that LLMs tend to overweight recent letters more than earlier letters when producing the answer via direct prediction. However, when reasoning, the LLMs give a roughly equal weight to all letters across the sequence. The authors establish that this is because the LLMs preferably attend to a different part of the input sequence at each reasoning step. In addition, the authors show that in-context reinforcement learning does not help the LLM produce more correct answers. They did not try any other methods of training. I would expect that simply asking the LLMs to maintain a running count in its context, or fine-tuning the models to do so, would work perfectly on this task.

The authors also draw many analogies to human and animal reasoning. They claim that a human or animal would solve this task by keeping a running count of the letters while receiving inputs. While both humans and animals can accumulate evidence over time in different circumstances, there is no good reason to expect that they would solve this particular counting task by keeping a running count of the letters instead of reasoning after receiving all the inputs. In fact, the latter strategy might be preferred in many cases, e.g., if the letters arrive very quickly and are recorded in a scratchpad.

Critical strengths

The task and experiment setup were simple and described well. The Appendix seems reasonably thorough.

The authors studied various model classes and sizes, including Gemma, Qwen and Nemotron models.

It is interesting that reasoning can mitigate the recency bias of LLMs.

The authors used an easily interpretable regression analysis to study how much the LLMs weighted old vs. recent evidence in their final output.

It is interesting that attention during reasoning attends to a few input tokens at any time and moves across the sequence with time.

Critical weaknesses

The result that LLMs attend to recent tokens more than earlier or middle tokens has already been established.

The main result is that LLMs do not continuously integrate evidence over time in the counting task. However, this is because the authors explicitly did not allow the models to emit any tokens while the input is being fed. I would expect that, if the authors allowed this, it would be fairly easy to fine-tune or prompt the LLMs to maintain a running count of the letters.

The relevance of this particular problem to scientific or practical questions about LLMs is unclear. The authors give some loose analogies to System 1 and System 2 thinking in the brain, but it is not clear how those are related to the mechanics of inference using LLMs.

Overall, the analogies to human and animal thinking are confusing and unclear. The authors cite several well-known results related to how humans or animals continuously integrate evidence over time, but they do not spend much time discussing what this kind of evidence integration would look like in an LLM. Since LLMs are mechanistically different from brains, it should look quite different.

Questions for authors

Have the authors tried using supervised fine-tuning or prompting to elicit the desired behavior from an LLM, where it keeps a running count of each letter on this task?

Can the authors show that recency bias is mitigated by reasoning in some more realistic tasks? I think this is the most interesting result in the paper.

Is there a framing or motivation for this paper that does not require making loose analogies to human or animal thinking?

Reviewer_w2kiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how thinking affects evidence integration in language models. The authors introduce a sequential letter-counting task in which models receive one letter per conversational turn and must determine which of two target letters appears more frequently. They estimate behavioral integration kernels across seven models and examine final-query attention and selected reasoning trajectories for four of them.

The results show that direct responses exhibit strong recency bias and assign little weight to evidence from the middle of the sequence. Thinking reduces the negative log-likelihood of the correct answer and produces nearly uniform integration weights, even though final-query attention to the original letters remains concentrated near the beginning and end of the sequence. Selected reasoning trajectories show models revisiting the input and writing intermediate counts, suggesting a possible explanation for the improvement.

The authors also examine whether outcome feedback can improve direct counting over ten consecutive games. Under the tested protocol, performance deteriorates rather than improves. The main contribution is a controlled analysis of how thinking changes temporal evidence weighting, particularly the finding that behavioral evidence integration and final-step attention can differ substantially.

Critical strengths

1. A well-designed task for studying evidence integration.

The sequential counting task provides a clear setting in which the optimal evidence-weighting rule is precisely defined. By estimating temporal integration kernels across seven models, the authors go beyond conventional performance measures and reveal a strong recency bias in direct responses. Comparing thinking and non-thinking modes using identical input histories provides a useful basis for examining how additional reasoning changes evidence use.

2. An interesting dissociation between evidence integration and attention.

One of the most interesting findings is that thinking produces nearly uniform behavioral evidence weights, while final-query attention to the original letters remains concentrated near the beginning and end of the sequence. This suggests that attention at the final decision step does not necessarily reflect how evidence was used throughout the reasoning process. The analysis of selected reasoning trajectories further illustrates how models revisit the input and generate explicit intermediate counts. This provides a useful perspective on what changes when models are allowed to think before answering.

Critical weaknesses

1. The mechanistic interpretation remains underdetermined.

The paper aims to distinguish whether thinking constructs an accumulated count or improves access to one already represented internally. However, the integration kernels characterize behavioral evidence weighting, while the attention analysis and selected reasoning trajectories provide descriptive rather than causal evidence about the underlying mechanism. The observed recounting is consistent with construction, but does not rule out readout. Although the authors acknowledge this limitation, the distinction is central to the paper's motivation. Without evidence that separates these explanations, the paper offers a stronger account of how behavior changes during thinking than of why thinking improves performance.

2. The negative ICRL result is difficult to interpret.

The ICRL experiment examines adaptation over ten games per conversation, during which task experience, feedback history, and context length increase together. Figure 5f also shows that less attention is allocated to the current game's letters as the conversation progresses. While the observed performance degradation is clear, it is difficult to determine whether this reflects a failure to improve evidence integration through feedback or interference from the growing context and the particular feedback protocol. The authors acknowledge these limitations, but the lack of controls makes it difficult to draw broader conclusions about whether repeated experience can improve direct counting.

Questions for authors

1. Construction versus readout

Section 4.2 argues that the evidence favors the construction hypothesis, while Section 4.4 acknowledges that readout remains possible. Which of the reported findings actually distinguishes these two explanations? Could the recounting observed during thinking also occur if the model had already formed an internal representation of the accumulated evidence?

2. Interpretation of the ICRL results

In the ICRL experiment, feedback history and context length increase together across the ten games. Can the existing results distinguish between a failure to improve evidence integration through feedback and interference from the growing context? If these factors cannot be separated, should the conclusion be explicitly limited to the specific ten-game feedback protocol?

Reviewer_LmsCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts a psychophysics-style evidence-integration task to language models. Letters arrive one per conversational turn, and the model identifies which of two target letters occurred more frequently. In the single-game experiment, nonfinal WAIT responses are teacher-forced, and thinking and non-thinking evaluations branch from the same stimulus history. Seven models are evaluated across sequence lengths, coherences, and distractor conditions. A separate experiment tests ten consecutive non-thinking games with correctness feedback and permits early commitment.

The analyses combine correct-answer negative log-likelihood, regression-fitted temporal integration kernels, final-query attention, and selected reasoning trajectories. Direct responses show uneven temporal weighting, often with pronounced recency. Thinking lowers NLL and produces flatter kernels, while final-query attention to the original letters remains concentrated at the sequence ends. Selected trajectories show models revisiting the letters and writing intermediate counts. Across repeated games, mean NLL increases, action entropy decreases, and the final-position kernel weight increases. The authors interpret these observations as suggesting that thinking constructs an accessible count rather than merely improving access to one already formed.

Critical strengths
  1. A controlled behavioral comparison with an explicit reference computation. The task has a transparent solution, and the matched single-game branches control the stimulus sequence and preceding conversation history. This makes the contrast between direct and thinking responses informative about what changes between receiving the final letter and producing the answer, subject to the disclosed mode-specific sampling settings. The temporal kernels provide a useful description of positional dependence beyond aggregate performance, particularly the recency of direct responses. The individual-model and condition-specific results help establish the scope and heterogeneity of this pattern.

  2. A useful combination of behavioral and attention measurements, with clearly documented limitations. The contrast between flatter behavioral kernels and persistently end-concentrated final-query attention is informative: the answer’s final attention profile does not directly summarize the temporal dependence of the full computation. The held-out selection of stimulus-selective heads and their reuse across compared modes or game stages strengthen this descriptive comparison. The repeated-game experiment also provides a useful negative result for the tested feedback protocol. The appendix documents eligibility rules, score coverage, and several block-bootstrap comparisons, while the limitations section explicitly identifies important unresolved alternatives.

Critical weaknesses
  1. The evidence does not adequately distinguish construction from improved access to an existing count. The paper introduces construction versus readout as a central question and states that the evidence favors construction. However, the principal mechanistic evidence consists of four illustrative trajectories and correlational attention measurements. Revisiting letters, writing counts, and attending to those counts are consistent with construction, but they are also compatible with verification or reconstruction despite an existing internal representation.
    The manuscript acknowledges this limitation in 4.4, but the abstract’s suggestion that thinking constructs a count that direct responses lack is stronger than the experiments establish. Moreover, the selected trajectories do not quantify how frequently the proposed process occurs across trials or models.

  2. The additional explanatory value of the flatter thinking kernels needs clearer calibration against the fixed-count task design. Within each condition, the generator fixes the absolute signed count difference at (g). If a responder assigns a constant correct-direction score margin (m), its regression target satisfies y=m,sign!(∑ixi)=mg∑ixiy=m,\mathrm{sign}!\left(\sum_i x_i\right) =\frac{m}{g}\sum_i x_i. Thus, even without specifying an internal accumulation mechanism, this responder has a uniform linear representation; the fitted ridge coefficients will additionally reflect regularization and finite sampling.
    This does not mean that accuracy determines kernel shape: position-dependent confidence can produce a nonuniform kernel even when every answer is correct. It does mean that a flat kernel is an expected behavioral consequence of approximately constant-margin correct responding under this stimulus design. Consequently, its mechanistic interpretation requires care.
    The paper appropriately calls the kernel a functional description, but the thinking results would be more informative with an explicit reference responder passed through the same stimulus generation and estimation procedure, a quantitative measure of departure from uniformity with uncertainty, and accuracy alongside NLL.

Questions for authors

Q1. What evidence would distinguish construction from readout, and how representative are the displayed trajectories?
Can you provide additional evidence about whether count information is available before the reasoning continuation, with suitable controls and held-out evaluation? Such evidence should be distinguished from evidence that the model actually uses that information.

Q2. How much information do the thinking kernels add beyond correct responding and score separation under the fixed-count design? Could you compare the fitted kernels with a fixed-margin correct responder using the same sequences, sample sizes, and ridge estimator, with the margin explicitly specified or matched?

Q3. How robust is the repeated-game deterioration to scoring and commitment differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.