- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the generalization difference between standard chain-of-thought (CoT) reasoning and recursive language models (RMs), focusing on whether restricting context access can improve out-of-domain (OOD) generalization. The authors develop an MDL-based theoretical framework suggesting that, while CoT can efficiently represent recursive rules and achieve comparable IID generalization, access to the full reasoning trace may enable simpler shortcut solutions that fail under distribution shift. In contrast, RM isolates subproblems and therefore removes some of these shortcut opportunities. Controlled symbolic experiments are provided to support the theoretical arguments.
Critical strengths
- The paper studies an interesting and relatively underexplored question regarding the relationship between context accessibility, simplicity bias, and OOD generalization.
- The comparison between IID and OOD generalization provides an interesting perspective on why greater expressiveness or access to more information may not always improve generalization.
- The paper attempts to connect recursive reasoning, MDL, shortcut learning, and invariance within a unified theoretical framework.
Critical weaknesses
-
The theoretical motivation and presentation could be substantially improved. The transition from simplicity bias to the MDL formulation is rather abrupt, and the paper introduces many definitions and intermediate constructs for concepts that are often relatively intuitive. This makes the genuinely non-trivial theoretical contributions difficult to identify. Given the amount of formalism, a high-level roadmap showing the dependencies among the definitions, propositions, and theorems would greatly improve readability.
-
The scope of the theoretical OOD claim appears narrower than the paper’s broader framing. The invariance analysis primarily concerns changes to the surrounding context while preserving a visible subproblem encountered during training, rather than generalization to genuinely unseen subproblems or semantic domains. The distinction between this form of context shift and more general OOD generalization should be made clearer.
-
The empirical evidence is limited relative to the breadth of the conclusions. The experiments rely primarily on a controlled symbolic expression-evaluation testbed, with several OOD shifts and shortcuts explicitly constructed to reflect the theoretical setting. While useful for mechanism verification, this alone provides limited evidence that the conclusions extend to realistic reasoning tasks, natural distribution shifts, or pretrained LLMs.
-
The generality beyond the specific CoT–RM comparison remains unclear. Although the paper frames context restriction and specificity as general principles, the analysis focuses almost exclusively on RM. Comparisons with alternative context-control mechanisms, such as masking, pruning, or restricted-context CoT, would help determine whether the findings reflect a general principle or properties specific to RM.
-
The current comparison does not cleanly isolate context isolation as the causal factor. As the paper itself acknowledges, visibility, sequence segmentation, and positional exposure change simultaneously between CoT and RM. More controlled ablations are needed before attributing the OOD improvements specifically to context isolation.
-
The connection between the MDL explanation and the empirical results remains somewhat indirect. The theory attributes failure to a preference for shorter shortcut rules, whereas the experiments mainly demonstrate that CoT can exploit planted shortcuts. Stronger empirical evidence connecting the learned shortcuts to the proposed simplicity/description-length mechanism would make the theoretical explanation more convincing.
-
The manuscript would benefit from more careful proofreading and formatting consistency. There are several inconsistencies in notation and capitalization throughout the paper, such as “Fig.” vs. “fig.” and “IID” vs. “iid”. The authors should carefully check and standardize these expressions throughout the manuscript.
Questions for authors
No