- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the compositional generalization ability of recursive language models (RLMs), with a focus on hierarchical reasoning tasks. Specifically, the authors first introduce a controlled synthetic benchmark based on Countdown tasks, where the evaluation splits explicitly test novel combinations of components observed during training. Experiments with SmolLM2-135M, Gemma3-270M, and SmolLM2-1.7B consistently show that recursive execution substantially outperforms the linearized baseline in compositional generalization. The authors further investigate the effect of inference-time computation and find that increasing recursive search width (i.e., the number of subcalls) leads to greater performance improvements than increasing the output length of linearized baselines. Finally, they propose an adaptive curriculum method to further improve compositional generalization, assuming access to held-out validation compositions.
Critical strengths
-
The paper investigates an interesting question: whether recursive language models (RLMs) can improve hierarchical compositional generalization. As interest grows in using external scaffolding and recursive inference to solve problems that models struggle to solve on their own, this work provides a useful perspective on when and why RLM-style approaches may be beneficial.
-
The paper introduces a controlled Countdown benchmark that could be useful for future research on hierarchical compositional reasoning. The empirical results are promising, showing substantial improvements in compositional generalization over the linearized baseline. The additional investigations into search budget allocation and curriculum learning also provide interesting insights.
Critical weaknesses
- The study is conducted exclusively on the controlled Countdown task, and it remains unclear whether the observed improvements in hierarchical compositional generalization would transfer to more realistic reasoning tasks. It is also unclear how the proposed composition-aware curriculum could be applied to realistic LLM training scenarios, where the compositional relationships between tasks are typically not explicitly available. This limits the broader applicability of the findings.
- The experiments are conducted on relatively small models, with SmolLM2-1.7B being the largest. It would be important to verify whether the conclusions hold for more capable contemporary models, such as Qwen3 models at similar scales (e.g., 1.7B or 4B), or larger models (e.g., Qwen3-8B), as evaluated in the original RLM work [1].
[1] Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive Language Models. arXiv preprint arXiv:2512.24601, 2026.
Questions for authors
- The proposed adaptive curriculum assumes access to the compositional structures and per-pattern errors of held-out validation tasks. Do the authors have insights into how this method could apply to realistic LLM training scenarios where such information is typically unavailable? Or does the method have to rely on knowing the target validation distribution?