- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses a needle-in-a-haystack counting task to investigate why chain-of-thought improves long-context reasoning. City-score records are inserted into Paul Graham essays and the model must report how many there are. Across twelve model groups, thinking mode is far more accurate than non-thinking mode, and the gap widens as the number of records grows. Mechanistic analyses of Qwen3-8B and Gemma-4-E4B identify two contrasting strategies. Without thinking, a set of heads at the answer token attends to all records simultaneously and the count information they gather is noisy. With thinking, the model enumerates the records one at a time, a head attends to the next record at each step, and the hidden states along the enumeration separate cleanly by count. Head ablation and activation patching support both accounts, most notably in an experiment where transplanting the state of a later item to an earlier position causes the model to skip ahead and continue enumerating correctly. A small transformer trained from scratch reproduces the two retrieval patterns.
Critical strengths
- The empirical work is thorough. The behavioral comparison covers twelve model groups with thirty paired seeds per condition, retrieval heads are selected on one set of seeds and evaluated on another, ablations are compared with layer-matched random heads, and patching is performed in both directions. The state-transplant experiment in Figure 6C is particularly persuasive. After a single intervention the model goes on to list the correct subsequent records, which shows that the state is used by the model and not merely decodable from it. The appendix on instructed enumeration with thinking disabled is also a useful contribution for researchers working with closed models.
- The counting task offers a controlled setting that brings together long-context retrieval, the geometry of numerical representations and state tracking in chain-of-thought. The account in which per-step retrieval errors compound over many steps is simple and agrees well with the degradation of thinking mode when many records are combined with long contexts. The release of code and an interactive visualization further strengthens the paper.
Critical weaknesses
- The main comparison of representational compactness is confounded, and the claim of an internal counter goes somewhat beyond the evidence. Figure 1C contrasts prompt states in non-thinking mode at one layer with trace states in thinking mode at another, and the thinking traces carry explicit numbering of the form 2. City, 3. City. The trace format and the data split were chosen after inspecting the data, and the appendix itself describes the comparison as exploratory. When the index is written in the context, it is unsurprising that states at the kth item separate by k, so the contrast between 98 and 46 percent says little about whether chain-of-thought makes representations more compact. The authors acknowledge that numbering and position may contribute, yet the abstract still presents the result as a central finding. The patching experiment raises a similar issue. The transplanted state encodes both the count and the identity of the record just retrieved, so continuing with the next record is equally consistent with a pointer to the last retrieved position, in the spirit of the iteration heads described by Cabannes et al., without any counter being involved. The non-thinking analysis already shows that an accurate linear readout can coexist with failed steering, and the same caution should apply to the thinking analysis.
- Novelty and scope are limited. That chain-of-thought enumeration retrieves records sequentially by attending to the next one is fairly intuitive and closely related to prior work on retrieval heads and iteration heads. The records follow a rigid template and stand out lexically from the surrounding essays, so it is not clear that the conclusions extend to realistic long-context reasoning, where the relevant information is not marked by fixed phrasing. The mechanistic analysis is restricted to two models at around ten thousand tokens, and the causal tests rely on ten seeds or a few dozen prompt pairs. The synthetic experiment uses a single training seed, and the thinking model is trained on traces that list the targets explicitly, so the large accuracy gap there is to be expected.
Questions for authors
- Could the comparison in Figure 1C be repeated with unnumbered traces, such as the natural Qwen traces analyzed in Section 4.2, at matched layers, and reported in the main text? If the separation persists, my first concern would largely be resolved.
- Is it possible to dissociate count from content in the patching experiment, for instance by transplanting the kth-item state from a prompt with different cities and checking whether the model continues with record k+1 of the target prompt? Such a test would distinguish a counter from a positional pointer.
- At the final answer, is the count read from an internal counter state or simply from the last number written in the trace?
2 comments
Weaknesses
W1: Confounded compactness comparison and an overly strong counter interpretation.
- The evidence extends beyond numbered traces and comparisons across different layers. Our response to Q1 provides unnumbered representation results and a same-layer comparison; Q2 tests the effect of transferred counting progress while matching current record identity and retrieval position; Q3 extends answer-state patching to unnumbered traces. Together, these support functionally used trace states that track progress. We will distinguish these findings from the stronger hypothesis of a content-independent counter and make trace selection, token positions, and layer selection explicit in the main text.
W2: Limited novelty, task scope, and mechanistic coverage.
-
Beyond retrieval heads and iteration heads. Retrieval heads copy information from individual needles, and iteration heads (Cabannes et al.) were identified in small transformers trained on synthetic iterative tasks. We study naturally generated CoT in pretrained LLMs, contrast broad and targeted retrieval, connect them to distinct counting-error laws, and show that trace states carry accumulated count beyond a current-record pointer (our response to Q2).
-
Template and scope. The fixed record template provides the exact retrieval targets and progress labels needed for causal tests. Our new TableBench experiment extends the analysis to counting over real tables, where records are table rows rather than template sentences: Thinking achieves 90% accuracy compared with 20% for Non-thinking, and patching shifts subsequent counts as predicted in 8/10 trials, versus 0/10 for self and random patches.
-
Synthetic experiment. Listing the targets is what defines enumeration CoT; the experiment aims to show that targeted and broad retrieval mechanisms emerge under standard autoregressive training.
-
Please see the Novelty claim and Agentic tasks sections of our General Response for details.
We hope our responses address your concerns and encourage a more favorable assessment of our paper.
Response to Reviewer dNxd
We thank the reviewer for recognizing the multi-step intervention evidence. We address the representation controls, causal interpretation, and contribution below.
Questions
Q1: Unnumbered traces at matched layers.
Count separation remains clear when we do match layers and control trace grammar. At the same layer, L19 (the layer selected by NCC), unnumbered comma-separated records (
CITY with SCORE, …) and quoted records on separate lines ("In the 2024 city score audit, CITY received a score of SCORE.") achieve NCC balanced accuracy of 85% and 97%, respectively, compared with 54% for Non-thinking. The interactive supplement (https://anonymous.4open.science/r/CoT-NIAH-Counting-rebuttal/index.html) shows both formats and the pooled unnumbered traces without grammar filtering across all layers. We will include these comparisons in the main text of the revised manuscript. Click “JS off” to enable the interactive 3D figures.Note that different computational mechanisms might not be implemented at the same layer. Non-thinking aggregates source-record information for direct answering, whereas Thinking retrieves records successively and tracks progress in its trace.
Q2: Separating a counter from a record carrier or a positional pointer.
Thanks for pointing out this confound. We tested the proposed cross-city transfer. Transplanting states between different cities introduces a record-content mismatch: in 22/30 transfers, the model rechecks or departs from enumeration, which complicates the interpretation of unsuccessful transfers. Prior work also has only identified item-by-item counter updates, and isolating a content-independent counter from retrieval position and record identity in LLMs remains challenging.
We therefore use multi-content counting to functionally distinguish accumulated count from current record identity and retrieval position. Prompts contain city and flower records, and the model counts only cities. Source and target traces share the same current record and remaining records but differ in accumulated city count. An illustrative example:
Successful transfer: the target still retrieves Tokyo and Lima, but counts them as 5 and 6, rather than 3 and 4.
Across 30 cross-prompt pairs, endpoint-aligned item-span patching at L22, using the main-text patching procedure, changes the first subsequent target count in 19/30 cases, versus 0/30 for self and norm-matched random patches. This shows that transferred trace-state information affects counting beyond a current-record pointer alone.
Q3: Reading an internal counter versus copying the last numeral.