- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes an LLM based pipeline for discovering cognitive models using program refinement. Their pipeline has 4 modules, 3 of which are LLM based, and one (the Inference module) uses standard methods for probabilistic inference. Using human data collected on a single task, a colored light switching puzzle, they analyze the algorithms discovered and the refinements proposed by their pipeline. They show that for many of the participants, a revised strategy proposed by their pipeline provides the overall best-fit.
Critical strengths
- The scientific setup of evaluating human hypotheses using LLMs for discovering better programs is useful.
Critical weaknesses
- The comparison is between the revised programs and the 3 original models that were refined. Some more baselines, for eg., a comparison of the programs discovered by their pipeline from scratch (instead of starting from the 3 defined models) with the 3 baseline models would be informative.
- Using LLMs for program refinement/discovery is not something new (e.g. [1]), and hence I'm unclear about the novelty of the proposed approach.
[1] Castro, Pablo Samuel, et al. "Discovering symbolic cognitive models from human and animal behavior." bioRxiv (2025): 2025-02.
Questions for authors
Please look at the Weaknesses section above.
We thank the reviewer for recognizing the usefulness of combining human hypotheses with LLM-based program discovery.
De novo baseline. We added the suggested generation-from-scratch ablation in Appendix G. We removed the base cognitive program while retaining the same pipeline and six generation attempts per trajectory. Across 49 trajectories, only 69 of 294 outputs (23.5%) produced a finite maximized log likelihood, 7 (2.4%) supported the complete inference grid, and 6 (2.0%) passed all black-box validity checks. Thus, without an executable base model, the pipeline usually fails to produce a program that can be reliably compared with the three cognitive models. This result supports our central design choice: existing cognitive programs provide both theoretical guidance and the executable structure required for reliable refinement. We do not interpret this ablation as establishing the contribution of every individual pipeline module.
Relationship to Castro et al. We agree that LLM-based cognitive-program discovery is not itself novel, and we have revised Section 3 to cite and distinguish Castro et al. (2025). Their method uses evolutionary LLM-based search to discover symbolic cognitive models directly from behavioral data, whereas our framework begins with human-specified probabilistic programs and uses trajectory-level diagnosis and constrained refinement to produce interpretable extensions of existing cognitive theories. Our contribution is therefore this hypothesis-guided refinement workflow, rather than the first use of LLMs for cognitive-model discovery.
Concluding remarks We believe the new de novo ablation strengthens the motivation for refinement rather than unconstrained generation, while the comparison with Castro et al. clarifies the complementary methodological contribution of our work. We hope these additions and clarifications support acceptance.