- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UNSATZ, a benchmark designed to test whether LLMs can discover scientific equations without knowing their structure beforehand. It includes 80 simulated problems across 20 scientific domains. The benchmark consists of 80 simulated scientific systems across 20 domains, where agents must design experiments, infer hidden variables, and propose governing equations. The results show that stronger LLMs perform better and make fewer basic errors than weaker models. However, even the strongest models often struggle to discover the correct relationships between variables and tend to favor equations that fit the observed data rather than generalize well to new experiments.
Critical strengths
The paper addresses an interesting and important problem: whether LLMs can discover scientific models without being given the underlying equations. One strength is the diversity of the benchmark, which includes 80 problems across 20 scientific domains. Finally, the analysis of model failures is useful, especially the finding that LLMs can often identify relevant variables but struggle to discover the correct relationships between them. An other strength strength of the paper is that the authors provide a detailed procedure for generating new scientific problems, rather than relying only on a fixed set of benchmark examples. This makes the benchmark potentially scalable and useful for future evaluations.
Critical weaknesses
The authors do not provide a formal guarantee that the generated scientific problems have never appeared before. That is the major problem of this works and questions the soundness of the approach.
The authors evaluate 80 problems across 20 scientific domains, meaning only four problems per domain. This seems limited for drawing broad conclusions about scientific-discovery capabilities. For example, solving three out of four ecology problems does not necessarily mean a model is generally good at discovering ecological mechanisms.
In Section 2.5, the authors explicitly score models based on their predictions, not whether they recover the true equations.
Several existing benchmarks, particularly DiscoverPhysics, PhysGym, and NewtonBench, already evaluate scientific discovery through interactive experimentation and hypothesis generation. DiscoverPhysics also considers hidden mechanisms and evaluates predictions on unseen trajectories. Given this overlap, I am not fully convinced of the novelty of UNSATZ is sufficiently established.
Questions for authors
Could the authors clarify what fundamentally new capabilities their benchmark measures beyond those already covered by existing work?How is UNSATZ fundamentally different from existing benchmarks such as DiscoverPhysics and PhysGym? How can we be sure that the models are actually discovering scientific mechanisms rather than parroting familiar equations or using prior knowledge? Since the benchmark evaluates predictive accuracy rather than recovery of the true underlying equations, how do the authors justify the claim that it measures scientific mechanism discovery rather than simply predictive modeling? If UNSATZ were released tomorrow, would researchers learn something important about scientific-discovery agents that they could not already investigate with DiscoverPhysics or PhysGym?
5 comments
It compares every candidate world with a list of 300 named laws. LLMs have likely encountered substantially more than 300 scientific laws and mathematical formulations during pretraining. ! This is why soundness is my main concern. Also, the paper does not clearly demonstrate the practical usefulness of the benchmark, for example, whether it could be used to train LLMs to improve their scientific-discovery capabilities in specific domains. instead it just benchmark existing LLMs. Finally I can not see code in the submission which also raises reproducibility concerns. Therefore I will maintain my score.
Contamination. Your comment reads the comparison with 300 named laws as our sole defense against contamination. But our primary defense is that every evaluated world is new. The generator drew each world at random from a private seed, and neither its program nor its hidden tests has been published (Section 3, Appendix P.1). A model that had memorized every law ever written would still have to find, by experiment, a program it has never seen, so the number of laws a model has read is beside the point. The concern does apply to benchmarks whose answers are known laws, such as NewtonBench, whose 108 problems are modifications of 12 canonical laws (Table 1), but UNSATZ was explicitly designed to avoid it.
Nonetheless, we also tested recognition directly to more directly address your soundness concerns:
- We gave a frontier LLM (Claude Opus 5.5) the 92 evaluated true programs, shuffled blind with 19 named laws written in the same language, all with neutral variable names, and asked it to identify every named model. It named 18 of the 19 laws and none of the 92 worlds.
- Among 76,249 recorded draws of twelve of the generators that the comparison covers, no two draws of one generator have the same equations up to their coefficients (Appendix P.1).
- Only 6 of the 230 passing programs have the terms of the true program; the other 224 are different programs that predict the same behavior (Table 29). The passing agents built their programs from the data.
On a formal guarantee: known laws are finitely many, while the number of programs the modeling language can express grows without bound with their size. Among programs of size at most n, the fraction that are known laws therefore tends to zero as n grows, so in the limit known laws are a null set. UNSATZ bounds program size so that worlds stay solvable from affordable experiments, and for this bounded generator the guarantee holds by construction: the generator discards every candidate that matches one of the 300 named laws, whatever its coefficients and variable names, so the probability that an evaluated world there is one of them is exactly zero (Section 2.6). We also wish to contextualize the relative size of our list: 300 is longer than the English Wikipedia list of named differential equations, which has only 216 entries.
Practical usefulness. The review asks whether the benchmark is useful beyond benchmarking existing LLMs. It is useful for what it shows about how LLM agents do science, which earlier benchmarks cannot measure because they fix the variables or the form of the answer (Table 1). We restate our findings here:
- The structure is the bottleneck. In 268 of the 490 failed episodes, the true program with unknown coefficients, fitted to the agent's own experiments, passes the hidden tests (Section 4.1).
- Finding the latent states is not enough. Final models with the true number of latent states pass no more often (8 of 69) than the others (11 of 93), and with every latent state recorded, 55 of 72 episodes still fail (Section 4.4).
- The training error misleads revisions. In 235 of the 510 episodes with more than one submission, the final model predicts worse than an earlier one (Section 4.4).
- Agents reuse a discovered mechanism. Agents holding a correct program of one world solve a related world in 15 of 42 episodes, against 5 of 64 without one (Section 4.5).
- Classical methods solve 14 of the 64 deterministic worlds over all their datasets, against 39 for GPT (Section 4.1).
Although we did not claim this in our manuscript, we agree that UNSATZ can be used to train LLMs in specific domains, for example as a reinforcement-learning environment. It draws any number of new worlds in each of its 20 domains, so a training set drawn from other seeds shares no world with the evaluation set, and its hidden tests give a verifiable reward without a judge model (Section 2.5).
However, none of the six interactive benchmarks in Table 1, including the three the review names, trains a model, and Gravity-Bench, PhysGym and DiscoverPhysics leave training to future work. Since the review cites this as a reason to keep its score, could the reviewer say what demonstration of practical usefulness would be needed to support our benchmark?
Code. The code is now in the supplementary material: the benchmark and its tests, one script per experiment with a README that maps each script to its section, and the analysis scripts behind the reported results.
We hope your concerns are addressed sufficiently to motivate raising the score; otherwise, we would appreciate the opportunity to clarify or improve our work further upon your feedback.
We thank sgu2 for the review, and address the questions below:
W1. The generator excludes the known laws. This has been made clear in the revision. In twelve domains and in the transfer problems, it compares every candidate world with a list of 300 named laws, (e.g. Lotka–Volterra, van der Pol, Michaelis–Menten, etc.), and discards any match, whatever the coefficients and the naming of the variables; this check was part of the generator that drew the evaluated worlds and is now described in Section 2.6. Applied to the true programs of the evaluated worlds whose class it covers, it finds no named law; the 24 delay, spatial, lattice and agent-rule worlds match none of 70 named models of their classes. Named laws are rare even without the check: a candidate of the kinetics generator is the Lotka–Volterra system with probability one in 22 million (Appendix P.1). Every world is a new draw from a private seed, fixed before any agent ran.
W2. The paper makes no claim about an agent's skill in a single domain. Every claim about agents rests on all 80 worlds, on which the agents are compared in pairs; resampling whole domains with their worlds, GPT is first in 99.7% of 200,000 samples (Appendix O.1).
W3, Q4. A curve fit and a mechanism agree on the data they were fitted to and differ beyond them, which is where the hidden tests are: outside the ordinary ranges, under interventions and over long runs (Section 2.5). The score separates the two: 163 of the 321 executable final programs that fail in deterministic worlds fit their own experiments closely, and the true dependency graph, which variables enter each equation, is found in 77% of passing final programs against 43% of failing ones (Appendix N.2). A score on the equations would do worse: only 6 of the 230 passing programs have exactly the true terms, and it would reject the other 224, of which 171 have the true dependency graph (Table 29).
W4, Q2. UNSATZ differs from all three (Table 1). UNSATZ can be regenerated arbitrarily and can generate arbitrarily many worlds, thus overcoming contamination without requiring the benchmark be unpublished: its worlds are sampled programmatically by a generator from seeds in 20 domains, whereas a person wrote or selected the 22 DiscoverPhysics worlds, the 97 PhysGym problems and the 108 NewtonBench laws. Moreover, the agent writes a complete program and decides how many states exist, which are latent and how the sensors read them, whereas the others ask for a force law or a closed-form law in given variables. The answer is scored on predictions alone, with no judge model, and experiments are priced under a budget rather than counted.
Q1, Q5. The capability measured is proposing the ansatz of a system when neither its variables nor the form of the answer is given. What researchers can learn here and not with the other benchmarks rests on two features: a true program that can be fitted to any agent's data, and a generator that can change one property of a world or place one mechanism in two worlds. With them, the revision shows that agents fail at the structure rather than the experiments (general response, A), that recording the latent states leaves most failures in place (55 of 72 episodes still fail; Section 4.4), and that a correct program of one world helps on a related world (15 of 42 episodes, against 5 of 64 without one; Section 4.5).
Q3. The worlds are not solved by recall: 211 of the 230 passing final programs match no named law, and familiar final programs, those that match or contain a named law or are entirely linear, pass less often (21% against 36%; Appendix P.2). Agents do propose named laws, in 449 of the 717 episodes with a program, but in 360 episodes they replace them, in 267 of these with a program that fits the data better.
We hope our response answers the reviewer’s concerns and, if so, would be grateful if the reviewer would consider raising their score.