- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a falsification-driven paradigm for scientific belief learning and implements it in a system called SEER. The system maintains a set of falsifiable theories simultaneously, selects an experiment at each step to test them, and updates the confidence in each theory based on the results. The scientific theory mentioned in the article actually boils down to single-variable intervals or interval combination rules for two variables. The revised theory is mainly reflected in adjusting the scope of application for these rules. Therefore, this implementation is much more specific and narrower than the scientific discovery described in the abstract.
Critical strengths
The paper targets a good problem. Moving AI-for-science systems from optimizing a fixed objective toward forming explicit, falsifiable hypotheses and revising them as evidence comes in is an important direction.
Critical weaknesses
-
The readability of this article feels very poor. For example, Figure 1 looks very fancy, but in reality, I find it incomprehensible; it cannot explain itself. The figure contains many elements that are not explained in the legend, caption, or main text. The peak chart in the middle of Figure 1 doesn't have defined axes, and it's unclear what the positions of the peaks represent. There is also no explanation of what the points and connecting lines actually are. The curve in the bottom left of Figure 1 also has a flaw. The caption says these are illustrative posteriors, but the probability curve goes over 1.0 and ends up around 1.2. A probability can't be greater than 1.
-
There’s a disconnect between the problem the authors claim to solve and the work they actually do, and they are missing necessary baseline comparisons. In the introduction, the authors state that scientific discovery needs to answer why a system behaves a certain way and when that explanation fails. The challenge, they argue, is being able to autonomously build, evaluate, and revise those beliefs. Then, in section 3.1, they state that the learning objective is the explanatory quality of the final B_T, but this explanatory quality is never defined anywhere in the paper. When it comes to the actual experiments, however, the task shifts entirely:
- SEER are given a queryable black-box function f(x) with a query budget of 300 to 500 steps.
- It outputs a batch of interval rules or box rules in the form of "when x is in the closed interval [a, b], y is higher."
- They compare these against pre-defined ground-truth rules to see how much recall they ultimately achieve.
So while the problem they claim to tackle is grand, the actual problem they solve is very specific, and they fail to bridge the gap between the two.
Furthermore, the three groups of existing methods they contrast SEER with (optimization and active learning, causal discovery, and LLM agents) were not designed to recover box rules from black-box functions. Criticizing them for not producing explanations is essentially grading other people’s work against a task of the authors’ own definition. The problem categories that are actually closest to this real task (such as active level set estimation, subgroup discovery, or PRIM) are completely unmentioned.
These are also the baselines the paper is missing. The main comparison (Table 2) gives Random/GA/BO a post-hoc extractor that keeps at most one interval rule per dimension (Appendix E.3). More appropriate baselines would include:
- Active level set estimation, which spends a query budget on finding where f exceeds a threshold, e.g. LSE (Gotovos et al., IJCAI 2013; https://www.ijcai.org/Abstract/13/202).
- Box-rule learners applied to the queried data, e.g. PRIM (Friedman & Fisher, Statistics and Computing 1999; https://doi.org/10.1023/A:1008894516817), or REDS, which finds box-shaped scenarios of an expensive simulator from few runs (Arzamasov & Böhm, SIGMOD 2021; https://doi.org/10.1145/3448016.3457301). Even the authors’ own depth-4 tree extractor already exceeds SEER’s aRR on three of the five domains (Table 7).
- Methods that keep a posterior over competing rules and choose experiments to discriminate between them, e.g. Piriyakulkij et al. (NeurIPS 2024; https://proceedings.neurips.cc/paper_files/paper/2024/hash/5f1b350fc0c2affd56f465faa36be343-Abstract.html). Without this comparison, it is unclear whether the falsification term adds anything beyond standard discriminative experiment design.
Questions for authors
I have no questions for the authors. My main concerns are described in the Weaknesses section.