- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the capacity limitations of internal and external memory in language model reasoning. The authors formulate a finite-state constraint reconstruction problem and derive a theoretical bound on exact reconstruction success under a shared memory budget. They further extend the analysis to non-uniform information distributions and answer-time external access. To examine these theoretical predictions, the paper introduces a two-call retention protocol and evaluates several language models under different memory representations, information entropy levels, and external access conditions.
Critical strengths
-
Interesting theoretical perspective. The paper provides a clear formulation of memory capacity constraints and highlights the distinction between nominal context length and information that remains recoverable during answer generation.
-
Useful empirical observations. The experiments investigate multiple language models and analyze how scratchpad representations, information redundancy, and external access affect reconstruction performance. These observations provide useful insights into memory retention behavior.
Critical weaknesses
-
Limited theoretical novelty. The central Joint Memory Budget Theorem is essentially derived from a standard finite-state counting argument. The exponential decay of exact reconstruction probability follows directly from the number of distinguishable memory states. Although the formulation is clear, the theoretical contribution appears relatively incremental compared with established information-theoretic capacity bounds.
-
Insufficient connection between theoretical assumptions and empirical validation. The theoretical analysis considers the joint capacity of internal and external memory, whereas the empirical two-call protocol primarily measures information recoverable from explicitly written representations. This protocol does not directly characterize the internal memory capacity of an LLM during continuous reasoning. Moreover, the experiments focus on synthetic constraint reconstruction rather than realistic reasoning tasks, limiting the evidence for the broader claims about LLM reasoning and external scratchpads.
Questions for authors
-
What is the fundamental theoretical novelty of the Joint Memory Budget Theorem compared with classical finite-state counting and information-theoretic capacity bounds? Could the authors clarify which conclusions are not already implied by these established results?
-
How can the theoretical joint memory budget be meaningfully related to the internal states of real LLMs? Since the two-call protocol resets the model's internal state and retains only the written representation, how does the experiment validate the claimed joint internal-external memory bottleneck? Would additional experiments on realistic multi-constraint reasoning tasks provide stronger evidence?