Counting on Thinking: Tracing Evidence Integration in Language Models

Computational resources are finite, so no intelligent system can afford extended computation for every decision. Humans and animals handle frequent operations with automatic system 1 processes and reserve costly system 2 computation for problems that warrant it. Large language models (LLMs) can likewise spend extra inference-time computation on hard problems, yet their direct answers struggle even with counting, an elementary integration of input that humans and animals perform automatically. We ask why such an operation requires thinking in LLMs. Evidence integration is so basic that psychology and neuroscience have long used it to probe decision-making; our LLM version presents one letter per conversational turn and asks the model which of two target letters appeared more often. A running difference between the two counts solves the task optimally, weighting every letter equally, and fits this format exactly: the tokens at each turn can represent the difference accumulated over all earlier letters and update it with the new one. Direct responses nonetheless weighted evidence unevenly, with a strong recency effect, and assigned less probability to the correct answer as the task grew harder. Thinking improved performance and made the integration weights nearly uniform, yet final-query attention stayed concentrated on the ends of the sequence in both modes. Instead, reasoning trajectories showed the model revisiting the input, recounting the letters, and checking intermediate counts that the answer then read, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. This recounting cost reasoning tokens that grew with the number of letters far more than with coherence. Nor did outcome feedback bring the computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and the recency effect strengthened, yet the models grew more confident. Humans and animals amortize such elementary computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available is a central question for how future models allocate computation.