- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents AgentBayes, a two-agent framework for automated Bayesian model discovery. An Interactor performs data exploration and posterior critique, while a Modeler generates and fits Stan programs. The method is evaluated against expert Bayesian models, BoxLM, and symbolic regression baselines.
Critical strengths
The integration of executable data exploration with automated Bayesian modeling has practical utility.
Critical weaknesses
he presentation of this paper is disastrous. After reading the entire introduction, I could not grasp what the core contributions of the paper actually are. Upon finishing the full text, it appears that the paper merely repurposes the existing "LLM candidate generation – execution evaluation – feedback refinement" loop as its primary contribution, without clearly proposing any novel mechanisms that go beyond general agent search. Furthermore, the Odorant program (Appendix A.5, lines 1191–1207) conditions posterior predictive generation on observed responses, which forces predictions to zero whenever the observation is non-positive. This invalidates the posterior predictive procedure presented in the paper and raises serious concerns regarding the reliability of the critique mechanism. The experimental evaluation is also severely inadequate. The paper fails to justify the necessity of the two-agent architecture. Additionally, the ablation study removes multiple capabilities simultaneously, and the scalability comparison omits BoxLM's existing visual-only variant. More importantly, improved predictive performance does not establish the recovery of the underlying data-generating process. The baseline comparisons are also insufficient; notably, there is a complete lack of comparison with existing studies that leverage LLMs to accelerate Bayesian Optimization (BO). In conclusion, this paper suffers from severe flaws and requires substantial improvements before it can be considered for publication.
Questions for authors
No
2 comments
Response to Reviewer fbbN (2/2)
- “The scalability comparison omits BoxLM’s existing visual-only variant”
BoxLM provides neither the code to generate visual representation of datasets nor the structure to propose plots in a data-agnostic way. There is no way to generate the visual-representation of PosteriorDb that was used in BoxLM and to attempt to generate visual representations on new datasets would require a domain expert to decide on how to generate the plots. In AgentBayes, the Interactor can propose plots to represent the data without a human expert pre-defining the plots.
- Improved predictive performance and the recovery of the underlying data-generating process
We agree that predictive metrics do not certify that a discovered structure is correct, and we will narrow our wording accordingly. This limitation applies to Bayesian modeling generally, not only to AgentBayes: under misspecification the posterior concentrates on the parameter values closest in KL divergence to the data generating process (Berk, 1966; Kleijn & van der Vaart, 2006), and expected log predictive density ranks models by the KL divergence of their predictive distributions from that process (Gelman et al., 2014; Vehtari et al., 2017). AgentBayes searches for the best-supported model among those it proposes and checks, and its parameter and variance-component estimates are interpreted conditional on that model. We will replace “recover the data-generating process” with “propose, test, and select probabilistic models”.
We target the problem of real world scientific domains, which attempt to model imperfectly observed phenomena where the ground truth structure is unknown. We do devote the case studies to dive into these datasets and show how the models are useful for real scientific tasks and how the modeling choices shed light on underlying structures within the data that we verify.
- “There is a complete lack of comparison with existing studies that leverage LLMs to accelerate Bayesian Optimization (BO).”
Despite the shared term “Bayesian”, the two settings address different problems. BO, including LLM-assisted variants, use a Bayesian surrogate model to choose where to evaluate an expensive black-box objective, searching a fixed, typically low-dimensional parameter space for a best configuration. AgentBayes, instead automates the Bayesian modeling workflow itself (Box’s loop of model building, inference, and criticism). Its search space is the open-ended set of probabilistic programs, which the system writes, fits, and criticizes against the data. The output is not an optimal point, but a fitted scientific model. The closest comparisons for our setting are prior LLM-driven model-discovery systems (like BoxLM, which we compare against in Table 2), the more distantly related symbolic regression systems (which we compare against), and general-purpose coding agents tasked with model discovery (which we now report in Table A).
Response to Reviewer fbbN (1/2)
We thank the reviewer for their review and are glad they acknowledge that our proposal has practical utility.
We thank the reviewer for identifying this. The A.5 program, written by the LLM, generates y_rep conditioned on the observed zero; the posterior predictive procedure itself is standard, but this program implemented it incorrectly for those observations. To assess how much this affects our results and the critique mechanism, we analyzed all AgentBayes selected models. Only 1 of 26 has this fault. We agree this is a limitation of LLM generated models. Predictive checks are one of several inputs to the critic, alongside ELPD-LOO, Pareto-k, convergence diagnostics, and the code itself. To make identification of such issues more reliable, we have added mechanical invariance checks to the pipeline, which test whether y_rep[n] changes when only y[n] changes.
We further clarify that this model was not used for the generation of Table 3, but was from the case study section, of how a scientist might apply AgentBayes (and use the entire dataset, as is common practice in Bayesian modeling). However, we will update Figure 3 to accurately represent the correct posterior. Visually, it remains almost identical. Table C shows the exact changes in the figure.
Table C. Figure 3 before and after correcting the A.5
y_repNecessity of the two-agent architecture and Ablations.
We have conducted new experiments to compare AgentBayes with a general-purpose coding agent, Codex with access to the same machinery (stan, plotting, analysis environment) and underlying LLM (GPT 5.2). We show test-set predictions on the held-out Radon, Odorant, and Wingbeat datasets. To better quantify the predictive distribution, we include the Continuous Ranked Probability Score (CRPS), a generalization of the mean absolute error. We find that Codex approximately matches Radon, but AgentBayes outperforms Codex on both Odorant and the unpublished dataset Wingbeat.
Table A. AgentBayes vs Codex, mean test set CRPS, NMSE
To help separate the contributions of the different roles of the Interactor, exploration and critique, we ran additional held-out ablations on Radon and Odorant. Removing the Interactor (proposer alone) degrades prediction on average on both datasets. Much of the benefit of the Interactor comes from exploring the data before modeling, as exploration alone can help match the full system performance. A system with the explorer ablated, but critic intact can reach full system performance on Radon but not on Odorant. Radon has a well-established hierarchical structure, and all interactive configurations converge to similar accuracy. Odorant is more complex and has no standard model, and interaction matters more there. We believe the benefit of interaction is problem-dependent and is largest when structure must be discovered from data.
Table B. Stage ablation (held-out test set), CRPS and NMSE