- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors study the performance of open-weight LLMs, e.g., Qwen, Gemma and Nemotron models, on a simple counting task. Each LLM is given an input sequence containing multiple letters and is asked to determine which of two letters appears more often in the sequence. The paper questions why LLMs have difficulty solving this counting task when directly predicting the answer, compared to when using reasoning. They suggest that an LLM should keep a running count the way a human or animal might and should therefore amortize its reasoning cost over the sequence of inputs. However, the reason why the LLM does not keep a running count of the letters is because the authors explicitly do not allow it to do so! They do not allow the LLM to output any intermediate words while receiving the letters, except for a placeholder “WAIT” word, so there is no mechanism by which an LLM could keep a running count of the letters while receiving the input. The authors show, using a regression analysis, that LLMs tend to overweight recent letters more than earlier letters when producing the answer via direct prediction. However, when reasoning, the LLMs give a roughly equal weight to all letters across the sequence. The authors establish that this is because the LLMs preferably attend to a different part of the input sequence at each reasoning step. In addition, the authors show that in-context reinforcement learning does not help the LLM produce more correct answers. They did not try any other methods of training. I would expect that simply asking the LLMs to maintain a running count in its context, or fine-tuning the models to do so, would work perfectly on this task.
The authors also draw many analogies to human and animal reasoning. They claim that a human or animal would solve this task by keeping a running count of the letters while receiving inputs. While both humans and animals can accumulate evidence over time in different circumstances, there is no good reason to expect that they would solve this particular counting task by keeping a running count of the letters instead of reasoning after receiving all the inputs. In fact, the latter strategy might be preferred in many cases, e.g., if the letters arrive very quickly and are recorded in a scratchpad.
Critical strengths
The task and experiment setup were simple and described well. The Appendix seems reasonably thorough.
The authors studied various model classes and sizes, including Gemma, Qwen and Nemotron models.
It is interesting that reasoning can mitigate the recency bias of LLMs.
The authors used an easily interpretable regression analysis to study how much the LLMs weighted old vs. recent evidence in their final output.
It is interesting that attention during reasoning attends to a few input tokens at any time and moves across the sequence with time.
Critical weaknesses
The result that LLMs attend to recent tokens more than earlier or middle tokens has already been established.
The main result is that LLMs do not continuously integrate evidence over time in the counting task. However, this is because the authors explicitly did not allow the models to emit any tokens while the input is being fed. I would expect that, if the authors allowed this, it would be fairly easy to fine-tune or prompt the LLMs to maintain a running count of the letters.
The relevance of this particular problem to scientific or practical questions about LLMs is unclear. The authors give some loose analogies to System 1 and System 2 thinking in the brain, but it is not clear how those are related to the mechanics of inference using LLMs.
Overall, the analogies to human and animal thinking are confusing and unclear. The authors cite several well-known results related to how humans or animals continuously integrate evidence over time, but they do not spend much time discussing what this kind of evidence integration would look like in an LLM. Since LLMs are mechanistically different from brains, it should look quite different.
Questions for authors
Have the authors tried using supervised fine-tuning or prompting to elicit the desired behavior from an LLM, where it keeps a running count of each letter on this task?
Can the authors show that recency bias is mitigated by reasoning in some more realistic tasks? I think this is the most interesting result in the paper.
Is there a framing or motivation for this paper that does not require making loose analogies to human or animal thinking?