acceptodds
Under review as a conference paper at ICLR 2027

Learning Mental Models From Unreliable Evidence

Abstract

Children expertly construct mental models under uncertainty despite limited world knowledge. Which computational principles make this possible? We study this question in an interactive physical Box Task in which participants infer a latent cause through interaction with a stochastic and partially observed environment. The Box Task was originally designed to study children's puzzle-solving qualitatively. We contribute a computational analysis of behavior that formalizes the cognitive process by which children generate new hypotheses and respond to environmental uncertainty as inference unfolds. Our modeling framework is based on Bayesian particle-based inference, which maintains a small set of working hypotheses and continually generates new ones. We give two complementary implementations of hypothesis representations within this framework: (1) as constraints on evidence, and (2) as executable programs evaluated against evidence. Using the constraint-based formulation, we show that the model needs to account both for subjective reliability of evidence and online hypothesis generation to explain key signatures of children’s behavior. Using the programmatic formulation, we show that hypotheses may take the form of partial compositional rules to guide information gathering when evidence interpretation is hard. The programmatic hypotheses are implemented using LLM-based program synthesis, evaluated across three prompt variants and several LLM backends, demonstrating robustness of our approach. Together, these results extend current models of children's inference to account for subjective reliability of evidence and open-ended hypothesis spaces, and provide a computational analysis of the form generated hypotheses may take in practice.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Bayesian inference, program synthesis, LLM agents, causal reasoning, Cognitive modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tNiaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies child behavior in a Box Task, in which they need to figure out which keys open which boxes, which is governed by a latent rule. It uses a particle filtering approach in which hypotheses either take the form of candidate rules or of LLM-synthesized symbolic programs, considering different sets of restrictions within both of these classes. They find that candidate rules represent child behavior well only when combined with subjective reliability parameters and online generation of new rules, what they call the full model. They find that LLM-suggested programs consistently suggest simpler rules and solve the task faster than children, but that some of the LLM-processes are also best modeled by the full candidate rules models.

Critical strengths
  • Comprehensive modeling approach: It's a strength of the paper that it combines using LLM-guided program synthesis with more traditionally generated rules (SoC). This allows the LLM-guided program synthesis to be compared with the results of the traditional models, and indeed for the traditional models to be used to analyze the LLMs. Using Bayesian particle filters also is a good, interpretable method of guiding synthesis that has a long tradition in cognitive modeling. To my mind, bringing all of this together in a cognitive decision-making task and making comparisons to real data represents the primary contribution of the paper.
Critical weaknesses
  • Insufficient analysis of behavior: The stated aim of the paper is to understand the computational principles underlying children exploratory behavior and rule inference better. The filter based on candidate rules does this to some extent, for instance making it clear that children weigh evidence subjectively. I'm less sure what the learnings from the LLM program synthesis are. It reads a bit more like a validation of the program synthesis as a method, and in particular, it seems like it does not predict the behavior as well as the other models, for instance comparing Figure 2 and 4. Overall, I would have appreciated a broader, more qualitative analysis of the predictions made by the program synthesis, for instance by including them in the behavioral descriptive analysis shown in Figure 3.

  • Technicalities: The models are only evaluated with goodness of fit and AIC (in appendix) rather than validation on any held-out data. Clarity in the figures could be improved, for instance in Figure 2 the top row is plotted on a shared y-axis when the it is changing between the different panels (indeed I would appreciate using the same axis across panels to facilitate comparisons). I was unsure how the POMDP formulation related to the modeling approaches used; for instance, the SoC-models do not allow for any principled action selection. (The formulation is also technically incomplete, without describing for instance the reward function.)

Questions for authors

Clarification: What would you say are the main learnings from the LLM modeling, and why?

Clarification: How is action selection implemented in the behavioral comparisons for SoC and LLM models other than LLM-PS-P, for instance to compute the NLL for behavior of the children? The figures show, "N trials to solve" for the different models, what criterion is used for solving here, is it when the correct hypothesis becomes the most likely one?

Future Work: What does Figure 3 look like for the different classes of SoC and LLM models? I think the plot only shows SoC full.

4 comments

W1. Behavioral analysis and interpretation of LLM-PS

We thank the reviewer for this observation. We would like to clarify that SoC and LLM-PS address different questions, and Section 2.3.2 explicitly distinguishes them.

SoC serves as our cognitive model of children's behavior to investigate the contributions of subjective evidence reliability and online hypothesis generation through parameter fitting and lesion analysis.

LLM-PS is not intended as a cognitive model of children's behavior. It investigates what representations generated hypotheses may take if we do not assume a predefined hypothesis space and use an LLM as a proposal distribution. Notably, LLM-PS exhibits important differences from children: its proposals favor relatively simple, salient rules, and its exploration policy differs from children's observed behavior. This is interesting in light of previous work (Piriyakulkij & Ellis, 2024), which uses a similar approach to model children's inference, although unlike our work it does not fit the model to individuals. Please also see Figure 5, which elaborates on this analysis by using SoC models to examine LLM outputs.

Per reviewer's request, we are extending the behavioral comparisons in Figure 3 to LLM-PS to provide additional insight into these differences, while noting that these comparisons are descriptive, since the LLM-PS conditions differ from the children's experimental environment in observability and evidence reliability.

We will expand the Discussion to clarify the distinct purposes of SoC and LLM-PS, and include extended the behavioral comparisons in Figure 3 in the revised Appendix, to be uploaded by the end of rebuttal period

W2. Behavioral validation and POMDP formulation

Behavioral validation. We agree that additional validation beyond fitting task-completion trajectories would strengthen the behavioral conclusions. We therefore conducted an additional analysis using the previously fitted parameters, without refitting, to evaluate the probability assigned to children's actual key–box choices conditional on their observed evidence histories.

For each child and each SoC variant, we initialized repeated SMC-S simulations using the parameters previously fitted to that child's cumulative task-completion trajectory. At each trial, we recorded the action proposed by the model before supplying the child's actual key–box choice and observed outcome, which were then used to update the particle population; each simulation receives the child's actual evidence history despite stochastic differences in its internal hypotheses. We estimate the probability of each observed action from the proportion of Monte-Carlo simulations proposing that action, and calculated the mean negative log probability per action, using a common lapse probability of 0.001 to avoid zero likelihoods.

The results show that SoC-Full achieves the lowest mean NLL per action (4.405), followed by SoC-Gen (4.466), SoC-Rel (4.994), and SoC-Lesioned (5.001). At the individual level, SoC-Full provided the best predictions for 45 children, SoC-Gen for 44, SoC-Rel for 8, and SoC-Lesioned for 3.

These results provide additional behavioral validation of the fitted models, showing that the advantage of online hypothesis generation extends beyond predicting cumulative task progress to predicting children's individual exploratory choices.

This approach is more robust than splitting each child's trajectory into training and validation segments because the available sequences contain only 5–72 ATTEMPT actions, and because the data is sequential, making within-child splits problematic for the shortest trajectories. This validation is appropriate, as the identities of individual actions were not directly used in parameter estimation.

We are adding the results and corresponding figures to the revised Appendix.

Figure 2A uses identical x- and y-axis scales across all four panels, allowing direct comparison between model variants. The shared y-axis labels are displayed on the leftmost panel only.

POMDP formulation. The POMDP formalizes the task environment, including latent states, observations, actions, transitions, and the reward function, which assigns a reward of 1 when all five boxes are opened (Appendix B). We cannot solve this POMDP optimally due to the intractability of exact inference and planning over the open-ended hypothesis space. We therefore approximate belief inference using particles and select actions using expected information gain over the current hypothesis distribution. This action-selection procedure is explicitly defined in Section 2.3. In SoC, the policy selects among ATTEMPT actions.

Q1. What are the main learnings from LLM modeling?

The principal contribution of LLM-PS is to demonstrate how hypothesis generation can be implemented over an expressive program space by using LLM as a proposal distribution. We found that while our method can produce simple and compositional rules, the generated programs predominantly expressed straigh-forward salient relationships, and did not generate unrelated key–box assignments the SoC generator supports. This difference can inform directions of future work. To recap, our conclusion is not that LLMs reproduce children's inference, but to examine LLM-based program synthesis as a computational means of open-ended hypothesis generation under a variety of conditions.

Q2. Action selection and criterion for solving the task

For SoC and the LLM-PS variants, ATTEMPT actions are selected using expected information gain over the current particle-weighted hypothesis distribution, as defined in Section 2.3. The SoC models operate over candidate key–box mappings; LLM-PS uses executable programs as hypotheses. The principal difference in action space is that LLM-PS-P additionally supports OBSERVE actions, which reveal previously unavailable information about the boxes. SoC models implement ATTEMPT actions only.

The NLL values reported for children's behavior in Figure 2 and Appendix D are calculated for the SoC models only, as only SoC are treated as a model of children's behaviour. They measure the agreement between simulated and observed cumulative numbers of boxes opened across trials.

The "N trials to solve" refers to the number of ATTEMPT actions required to open all five boxes. It does not require the correct hypothesis to become the most probable one, as models can complete the task without discovering the underlying general rule.

For LLM-PS, successful rule discovery is evaluated separately by examining whether the highest-weight hypothesis at task completion implements the correct number-matching rule.

Q3. Behavioral predictions across model variants (Figure 3)

Figure 3 currently presents SoC-Full alongside random-search baselines, to illustrate how the fitted cognitive model reproduces several descriptive features of children's exploration that are not captured by random sampling.

As noted in W1, we are examining additional comparisons extending the analysis in Figure 3 to LLM-PS simulations and will add them to the revised Appendix, however we note that LLM-PS simulations are not directly matched to children's observational conditions, particularly because the different LLM variants manipulate evidence reliability and observability. As discussed above, their purpose is to characterize generated hypothesis representations rather than provide fitted predictions of children's behavior. We will clarify this distinction in the revised manuscript.

We will upload a revised version of that paper with this analysis included by the end of rebuttals

Thank you for your replies. I will maintain my score for now, waiting to see the revised version, and the discussions with the other reviewers and AC.

Thanks -- we have uploaded the revised manuscript, with all changes highlighted in blue. The full list of revisions is given in a top-level comment.

Reviewer_Ng1wOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines how children construct and revise hypotheses when interacting with an environment where evidence may be unreliable, and the space of possible explanations is unknown in advance. The authors use the Box Task to study this and develop a computational framework based on Bayesian particle-based inference. They consider two ways to represent hypotheses: as sets of constraints and as executable programs generated by LLMs. The model captures allow agents to arrive at successful key-box assignments without learning a general rule. The authors also show that LLMs can generate and revise executable hypotheses, including rules that can generalise to unseen boxes, although this ability varies across LLMs and experimental conditions.

Critical strengths
  1. The results are largely supporting the claims. The comparison between the SoC variants shows that accounting for both subjective evidence reliability and online hypothesis generation significantly improves the fit to children's observed behaviour.
  2. The paper provides a great insight into thinking about the interactions between rules, evidence, uncertainty and hypotheses.
  3. The paper considers two ways of representing hypotheses, as sets of constraints and as executable programs. The LLM-based experiments demonstrate how structured hypotheses can be generated and revised using accumulated evidence, including under unreliable and partially observable conditions.
Critical weaknesses

Overall, there is not critical flaws that I could find. However, I had this remark:

  1. The results show that GPT-5.2 can complete the task using a single particle, whereas the other LLMs require multiple particles. Does this suggest that the need for maintaining multiple hypotheses depends on the capability of the underlying LLM? It would be useful to understand how the authors interpret this result in relation to the role of particle-based inference in their framework.
Questions for authors
  1. Does the result regarding GPT 5.2 suggest that the need for maintaining multiple hypotheses depends on the capability of the underlying LLM?
  2. Have the authors attempted to see what happens with GPT 6?
1 comment

We thank the reviewer for this observation. We agree that the number of particles needed for successful inference appears to depend on the capability of the underlying LLM.

In our framework, the LLM serves as a proposal mechanism for generating and revising hypotheses, while SMC-S provides the inference scaffolding for maintaining and updating multiple hypotheses. A more capable LLM may generate better hypotheses and revise them more effectively in response to evidence, reducing the need to maintain multiple particles.

During pilot experiments, we tested different particle counts. GPT-5.2 successfully completed the Box Task with a single particle, effectively reducing inference to sequential hypothesis generation and revision, and therefore we did not evaluate it with more particles. We did not evaluate stronger models because we found GPT-5.2 to be already sufficiently strong to solve this task with one particle. Based on the GPT-5.2 results, we hypothesize that GPT-6 may also succeed with a single particle in this particular task domain. Because GPT-5.1 and DeepSeek V3.2 did not reliably complete the task with a single particle, during pilots we found that approximately 10 particles appeared necessary for reliable completion in our pilots. While particle count was not systematically investigated as a free parameter, these observations suggest that stronger proposal mechanisms may require less SMC-S scaffolding.

From an engineering perspective, SMC-S may therefore be particularly useful for enabling less capable, less expensive LLMs to perform inference by maintaining and evaluating multiple candidate hypotheses. This suggests a potential trade-off between the capability of the proposal mechanism and the computational resources / budget allocated to inference.

Separately, SoC provides a tool for interpreting LLM behavior (Figure 5), and this application does not depend on whether the underlying LLM requires multiple particles to complete the task.

We will elaborate on these clarifications in the revised Discussion

Reviewer_NxQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study describes a model of the way in which children behave in the Box Task. The children are given a false rule regarding colour matching and are then asked to open five boxes using 13 keys; the correct rule is that the number on the key must match the number of shapes on the box. The authors regard the task as a POMDP and approximate it by means of particle based inference and EIG based action selection. They incorporate two mechanisms: a subjective evidence reliability parameter ρ and an online hypothesis generator governed by p_gen. The hypotheses are represented either as sets of key/box constraints, which are fitted to the children's data, or as Python programs suggested by an LLM, which are employed to explore the various possible forms that hypotheses might take. The evidence comprises (i) the per child fits of the cumulative boxes opened trajectories, where SoC Full performs best in terms of NLL and AIC and is the best fit for 91 out of 100 children, (ii) comparisons with random-search baselines, and (iii) LLM PS simulations using three different backends that demonstrate a dissociation between task completion and rule generalization.

Critical strengths

The task and the data include a misleading instruction, genuinely random results, partial observability, and a counter intuitive rule; this represents a real improvement on the simple tasks usually employed to test the Sampling Hypothesis, and it is a worthwhile objective to model inference when dealing with unreliable evidence and an open hypothesis space.

Transparency is achieved since the lesion design, along with the appendices, makes the claims verifiable; it is only because of this transparency that several of the problems listed below could be identified.

Critical weaknesses

The fitting does not support the mechanistic conclusion since it only considers the total number of boxes opened by the trial in question and the simulations are not based on the child's actual history. Consequently, the fits mainly show how fast a child finishes the task rather than which key–box pairs they chose.

The findings from the generalization study and the LLM-PS analysis are hard to interpret. In most cases, the SoC-Full procedure ends up with per box “generator” constraints, which fail to make a prediction about a new box. The fact that there is a match between the 20% and the 15% figures therefore does not show how children actually generalize. In the LLM-PS prompts both the Key and the Box have an attribute known as number, and the shapes are described as being numbered from 1 to N. ReAct also manages to solve the task in fewer than 15 attempts. As a result, the number rule stands out clearly and so the conclusions regarding the LLM's hypothesis priors cannot be separated from the prompt design or from contamination.

Questions for authors

How are the NLLs in Table 4 calculated (specifically with regard to the definition of T_j, the base of the logarithm, ε, and any summation over simulations)? Do the model rankings stay valid after any corrections have been made?

Is ρ a constant scalar or a Beta prior that is updated online? How many runs of the LLM were carried out for each condition? Appendix G states that the total number is less than 300, while Fig. 10 shows N = 100.

2 comments
AuthorsOpenReview ↗

Response to reviewer NxQr

W1

We thank the reviewer for raising the concern about whether the models explain children's individual action choices. We conducted an additional behavioral validation analysis evaluating whether the models also predict the identities of children's key–box choices. We conditioned each model on the child's observed action and outcome history, and used repeated SMC-S simulations to estimate the predictive distribution over the next action. We then evaluated the probability assigned to the child's actual choice. This procedure accounts for the stochastic nature of SMC-S inference where different simulations can maintain different hypothesis populations and select different actions under identical evidence.

This approach is more robust than splitting each child's trajectory into training and validation because behavioral sequences range from 5 to 72 ATTEMPT actions per child, often leaving insufficient observations for within-child training/validation split. Moreover, the sequential nature of inference means that early and late actions reflect different states of knowledge, making temporal splitting problematic. Our analysis therefore provides an appropriate validation of parameters against action choices.

Results. SoC-Full achieved the lowest mean NLL per action (4.405, 95% CI [4.248, 4.562]), followed by SoC-Gen (4.466, [4.322, 4.611]), SoC-Rel (4.994, [4.861, 5.126]), and SoC-Lesioned (5.001, [4.869, 5.134]).

Relative to SoC-Full, the mean NLL per action was higher by 0.596 for SoC-Lesioned, 0.589 for SoC-Rel, and 0.061 for SoC-Gen.

At the individual level, SoC-Full provided the best predictions for 45 children, SoC-Gen for 44 children, SoC-Rel for 8 children, and SoC-Lesioned for 3 children. SoC-Full performs best on average, although its improvement over SoC-Gen is smaller in this additional analysis of best prediction for individual children.

We are adding the corresponding plots to the revised supplement.

W2

The reviewer is correct that hypotheses generated as independent key–box constraints do not generalize to unseen boxes. This is an intended feature of the model, as our central assumption is that successful task completion may result either from inferring a generalizable rule or learning a collection of unrelated key–box mappings.

In SoC-Full, the online generator can construct such mappings by sampling new key–box assignments conditioned on accumulated evidence without an underlying rule. This assumption is consistent with cognitive modeling approaches that combine model-based inference with stochastic exploration. The approximately 20% of SoC-Full simulations that discover the true rule illustrates the relatively low frequency of rule discovery, rather than provides a prediction of children's responses to the generalization phase.

We will clarify this interpretation in the revised manuscript.

W3

As the reviewer notes, the task specification provided to the LLM in fully observed prompt conditions makes the number-matching rule salient, unlike children's task. We clarify that we do not intend LLM-PS to serve as a cognitive model of children's behavior or to establish that LLMs share children's hypothesis priors. As stated in Section 2.3.2, the purpose of LLM-PS is to examine how expressive, interpretable, and executable hypotheses can be generated and revised within the same particle-based inference framework. The LLM experiments therefore explore what forms generated hypotheses can take when the hypothesis space is not restricted to a predefined set of rules. We find that LLM-generated hypotheses can express both unified rules and partial, compositional rules that account for individual observations. They do not, however, produce the unrelated key–box assignments that the SoC generator can produce.

These experiments illustrate the representational possibilities of open-ended hypothesis generation using an LLM as a proposal distribution. The ReAct baseline operates under fully observable, reliable conditions, demonstrating that under these conditions the task can be completed efficiently without explicit SMC-S inference.

We are expanding the Discussion in the revised paper to clarify the distinction between the purposes of SoC and LLM-PS, and to acknowledge the limitations imposed by prompt design

Q1. Calculation of NLL (Table 4)

We thank the reviewer for catching the inconsistency. The actual probability floor in our code is 10^-10, not 0.01. Rerunning with 0.01 gives:

\textbf{Model} & \textbf{Mean NLL} & \textbf{SD} & \textbf{95\% CI} & \textbf{$t$} & \textbf{$p$} & \textbf{Cohen's $d$} & \\
\hline
SoC-Lesioned & 101.12 & 63.04 & [ 88.76, 113.47] & 13.046 & $<$.001 & 1.305 & *** \\
SoC-Rel      &  84.99 & 50.62 & [ 75.07,  94.92] & 13.279 & $<$.001 & 1.328 & *** \\
SoC-Gen      &  64.97 & 51.61 & [ 54.86,  75.09] &  6.078 & $<$.001 & 0.608 & *** \\
SoC-Full     &  45.33 & 24.00 & [ 40.63,  50.04] & ---    & ---     & ---   &     \\
\hline

Therefore, upon rerunning the analysis with \epsilon=0.01 the model rankings remain unchanged, and SoC-Full continues to outperform all three lesioned variants (all p<.001, paired t-tests). We also ensured that best-fitting model per child with \epsilon=0.01 remain unchanged.

The likelihood is calculated using natural logarithms. T_j denotes the number of evaluated ATTEMPT trials for child j. For each parameter setting, we estimate the distribution of cumulative boxes opened at each trial from 100 independent simulations, then sum the log probabilities assigned to the child's observed cumulative box counts across trials. There is no additional summation over simulations.

** We will fix the typo that reported \epsilon incorrectly in the revised paper. **

Q2. Reliability parameter (\rho)

The model-fitting procedure searches over Beta prior parameters ((\alpha,\beta)), as described in Appendix D, with the corresponding expected reliability

[ \mathbb E[\rho]=\frac{\alpha}{\alpha+\beta}. ]

The value of (\rho) was not updated during simulation. The implementation uses the expected reliability (\rho=\alpha/(\alpha+\beta)) throughout inference. Thus, the model performs online inference over hypotheses, while treating subjective evidence reliability as fixed.

We will clarify this distinction in the revised Methods.

Q3. Number of LLM simulations

To clarify the ambiguity in Appendix G, the statement that fewer than 300 runs were conducted refers to each LLM-PS variant, including development and prompt-debugging runs, rather than to the total number of simulations across all experimental conditions in the paper.

The (N=100) in Figure 10 refers specifically to the 100 LLM-PS-P simulations using GPT-5.2 (low reasoning) represented in that figure.

We will revise Appendix G to clarify this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions.

Dear Reviewers,

We have uploaded the revised manuscript, with all changes highlighted in blue.

Summary of revisions:

  1. Model fitting robustness: Corrected the reported probability floor to ε=10−10\varepsilon = 10^{-10} and added supplementary model-fitting results using ε=0.01\varepsilon = 0.01 to demonstrate the robustness of our findings.

  2. Action-level predictive validation: Added a new appendix section, Action-Level Validation of Fitted Model Parameters, including new figures evaluating the models' predictions of children's individual key–box choices.

  3. Behavioral comparison with LLM-based agents: Added a new appendix section, Behavioral Comparison with LLM-Based Agents, including a figure extending the behavioral analyses in Figure 3 to LLM-based agents.

  4. Clarifications and expanded discussion: Added clarifications throughout the manuscript and expanded the Discussion to address the reviewers' comments, including the interpretation and limitations of the LLM-based analyses.

All revisions are highlighted in blue in the updated manuscript.

Thank you for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.