- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies child behavior in a Box Task, in which they need to figure out which keys open which boxes, which is governed by a latent rule. It uses a particle filtering approach in which hypotheses either take the form of candidate rules or of LLM-synthesized symbolic programs, considering different sets of restrictions within both of these classes. They find that candidate rules represent child behavior well only when combined with subjective reliability parameters and online generation of new rules, what they call the full model. They find that LLM-suggested programs consistently suggest simpler rules and solve the task faster than children, but that some of the LLM-processes are also best modeled by the full candidate rules models.
Critical strengths
- Comprehensive modeling approach: It's a strength of the paper that it combines using LLM-guided program synthesis with more traditionally generated rules (SoC). This allows the LLM-guided program synthesis to be compared with the results of the traditional models, and indeed for the traditional models to be used to analyze the LLMs. Using Bayesian particle filters also is a good, interpretable method of guiding synthesis that has a long tradition in cognitive modeling. To my mind, bringing all of this together in a cognitive decision-making task and making comparisons to real data represents the primary contribution of the paper.
Critical weaknesses
-
Insufficient analysis of behavior: The stated aim of the paper is to understand the computational principles underlying children exploratory behavior and rule inference better. The filter based on candidate rules does this to some extent, for instance making it clear that children weigh evidence subjectively. I'm less sure what the learnings from the LLM program synthesis are. It reads a bit more like a validation of the program synthesis as a method, and in particular, it seems like it does not predict the behavior as well as the other models, for instance comparing Figure 2 and 4. Overall, I would have appreciated a broader, more qualitative analysis of the predictions made by the program synthesis, for instance by including them in the behavioral descriptive analysis shown in Figure 3.
-
Technicalities: The models are only evaluated with goodness of fit and AIC (in appendix) rather than validation on any held-out data. Clarity in the figures could be improved, for instance in Figure 2 the top row is plotted on a shared y-axis when the it is changing between the different panels (indeed I would appreciate using the same axis across panels to facilitate comparisons). I was unsure how the POMDP formulation related to the modeling approaches used; for instance, the SoC-models do not allow for any principled action selection. (The formulation is also technically incomplete, without describing for instance the reward function.)
Questions for authors
Clarification: What would you say are the main learnings from the LLM modeling, and why?
Clarification: How is action selection implemented in the behavioral comparisons for SoC and LLM models other than LLM-PS-P, for instance to compute the NLL for behavior of the children? The figures show, "N trials to solve" for the different models, what criterion is used for solving here, is it when the correct hypothesis becomes the most likely one?
Future Work: What does Figure 3 look like for the different classes of SoC and LLM models? I think the plot only shows SoC full.
4 comments
Q1. What are the main learnings from LLM modeling?
The principal contribution of LLM-PS is to demonstrate how hypothesis generation can be implemented over an expressive program space by using LLM as a proposal distribution. We found that while our method can produce simple and compositional rules, the generated programs predominantly expressed straigh-forward salient relationships, and did not generate unrelated key–box assignments the SoC generator supports. This difference can inform directions of future work. To recap, our conclusion is not that LLMs reproduce children's inference, but to examine LLM-based program synthesis as a computational means of open-ended hypothesis generation under a variety of conditions.
Q2. Action selection and criterion for solving the task
For SoC and the LLM-PS variants, ATTEMPT actions are selected using expected information gain over the current particle-weighted hypothesis distribution, as defined in Section 2.3. The SoC models operate over candidate key–box mappings; LLM-PS uses executable programs as hypotheses. The principal difference in action space is that LLM-PS-P additionally supports OBSERVE actions, which reveal previously unavailable information about the boxes. SoC models implement ATTEMPT actions only.
The NLL values reported for children's behavior in Figure 2 and Appendix D are calculated for the SoC models only, as only SoC are treated as a model of children's behaviour. They measure the agreement between simulated and observed cumulative numbers of boxes opened across trials.
The "N trials to solve" refers to the number of ATTEMPT actions required to open all five boxes. It does not require the correct hypothesis to become the most probable one, as models can complete the task without discovering the underlying general rule.
For LLM-PS, successful rule discovery is evaluated separately by examining whether the highest-weight hypothesis at task completion implements the correct number-matching rule.
Q3. Behavioral predictions across model variants (Figure 3)
Figure 3 currently presents SoC-Full alongside random-search baselines, to illustrate how the fitted cognitive model reproduces several descriptive features of children's exploration that are not captured by random sampling.
As noted in W1, we are examining additional comparisons extending the analysis in Figure 3 to LLM-PS simulations and will add them to the revised Appendix, however we note that LLM-PS simulations are not directly matched to children's observational conditions, particularly because the different LLM variants manipulate evidence reliability and observability. As discussed above, their purpose is to characterize generated hypothesis representations rather than provide fitted predictions of children's behavior. We will clarify this distinction in the revised manuscript.
We will upload a revised version of that paper with this analysis included by the end of rebuttals
Thank you for your replies. I will maintain my score for now, waiting to see the revised version, and the discussions with the other reviewers and AC.
Thanks -- we have uploaded the revised manuscript, with all changes highlighted in blue. The full list of revisions is given in a top-level comment.
W1. Behavioral analysis and interpretation of LLM-PS
We thank the reviewer for this observation. We would like to clarify that SoC and LLM-PS address different questions, and Section 2.3.2 explicitly distinguishes them.
SoC serves as our cognitive model of children's behavior to investigate the contributions of subjective evidence reliability and online hypothesis generation through parameter fitting and lesion analysis.
LLM-PS is not intended as a cognitive model of children's behavior. It investigates what representations generated hypotheses may take if we do not assume a predefined hypothesis space and use an LLM as a proposal distribution. Notably, LLM-PS exhibits important differences from children: its proposals favor relatively simple, salient rules, and its exploration policy differs from children's observed behavior. This is interesting in light of previous work (Piriyakulkij & Ellis, 2024), which uses a similar approach to model children's inference, although unlike our work it does not fit the model to individuals. Please also see Figure 5, which elaborates on this analysis by using SoC models to examine LLM outputs.
Per reviewer's request, we are extending the behavioral comparisons in Figure 3 to LLM-PS to provide additional insight into these differences, while noting that these comparisons are descriptive, since the LLM-PS conditions differ from the children's experimental environment in observability and evidence reliability.
We will expand the Discussion to clarify the distinct purposes of SoC and LLM-PS, and include extended the behavioral comparisons in Figure 3 in the revised Appendix, to be uploaded by the end of rebuttal period
W2. Behavioral validation and POMDP formulation
Behavioral validation. We agree that additional validation beyond fitting task-completion trajectories would strengthen the behavioral conclusions. We therefore conducted an additional analysis using the previously fitted parameters, without refitting, to evaluate the probability assigned to children's actual key–box choices conditional on their observed evidence histories.
For each child and each SoC variant, we initialized repeated SMC-S simulations using the parameters previously fitted to that child's cumulative task-completion trajectory. At each trial, we recorded the action proposed by the model before supplying the child's actual key–box choice and observed outcome, which were then used to update the particle population; each simulation receives the child's actual evidence history despite stochastic differences in its internal hypotheses. We estimate the probability of each observed action from the proportion of Monte-Carlo simulations proposing that action, and calculated the mean negative log probability per action, using a common lapse probability of 0.001 to avoid zero likelihoods.
The results show that SoC-Full achieves the lowest mean NLL per action (4.405), followed by SoC-Gen (4.466), SoC-Rel (4.994), and SoC-Lesioned (5.001). At the individual level, SoC-Full provided the best predictions for 45 children, SoC-Gen for 44, SoC-Rel for 8, and SoC-Lesioned for 3.
These results provide additional behavioral validation of the fitted models, showing that the advantage of online hypothesis generation extends beyond predicting cumulative task progress to predicting children's individual exploratory choices.
This approach is more robust than splitting each child's trajectory into training and validation segments because the available sequences contain only 5–72 ATTEMPT actions, and because the data is sequential, making within-child splits problematic for the shortest trajectories. This validation is appropriate, as the identities of individual actions were not directly used in parameter estimation.
We are adding the results and corresponding figures to the revised Appendix.
Figure 2A uses identical x- and y-axis scales across all four panels, allowing direct comparison between model variants. The shared y-axis labels are displayed on the leftmost panel only.
POMDP formulation. The POMDP formalizes the task environment, including latent states, observations, actions, transitions, and the reward function, which assigns a reward of 1 when all five boxes are opened (Appendix B). We cannot solve this POMDP optimally due to the intractability of exact inference and planning over the open-ended hypothesis space. We therefore approximate belief inference using particles and select actions using expected information gain over the current hypothesis distribution. This action-selection procedure is explicitly defined in Section 2.3. In SoC, the policy selects among ATTEMPT actions.