acceptodds
Under review as a conference paper at ICLR 2027

Unsatz: Evaluating Scientific Discovery Without a Given Ansatz

Abstract

Before a scientist fits any parameter, they must decide what the model contains: which quantities exist, which are hidden, how they interact, and how they are measured. Physicists call such a trial form an ansatz. Benchmarks for automated equation discovery mostly supply it, by fixing the variables, a library of candidate terms, or a law to be modified. We introduce Unsatz, a benchmark in which an LLM agent must construct the ansatz itself. Each task is a simulated system, generated from a private seed as a program in a small modeling language: ordinary, delay, or partial differential equations, discrete maps, static and causal models, or stochastic processes, possibly with hidden states, distorted sensors, and uninformative measurement channels. The agent buys experiments within a budget, writes candidate programs in the same language, obtains their coefficients from the benchmark's fitter, and is scored only on how its final program predicts experiments it never saw: conditions outside the range it could explore, interventions, and longer runs. Across 80 systems in 20 domains, nine language models solve between 21% and 60%. On 44 of these systems, classical equation-discovery methods almost never succeed, although the same data usually suffice once the true equations are given with unknown coefficients; the difficulty lies in finding the structure, not in estimating it. Because mechanisms recur across systems, we also hand agents the program they discovered for a related system; it helped only when the new system shared the mechanism and the first investigation had succeeded. Stronger agents make fewer unscientific errors; beyond these, agents find which variables a system has but not how they act, and prefer revisions that lower the training error to revisions that predict better.

Primary area:
datasets and benchmarks
Keywords:
benchmark, llm

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_sgu2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UNSATZ, a benchmark designed to test whether LLMs can discover scientific equations without knowing their structure beforehand. It includes 80 simulated problems across 20 scientific domains. The benchmark consists of 80 simulated scientific systems across 20 domains, where agents must design experiments, infer hidden variables, and propose governing equations. The results show that stronger LLMs perform better and make fewer basic errors than weaker models. However, even the strongest models often struggle to discover the correct relationships between variables and tend to favor equations that fit the observed data rather than generalize well to new experiments.

Critical strengths

The paper addresses an interesting and important problem: whether LLMs can discover scientific models without being given the underlying equations. One strength is the diversity of the benchmark, which includes 80 problems across 20 scientific domains. Finally, the analysis of model failures is useful, especially the finding that LLMs can often identify relevant variables but struggle to discover the correct relationships between them. An other strength strength of the paper is that the authors provide a detailed procedure for generating new scientific problems, rather than relying only on a fixed set of benchmark examples. This makes the benchmark potentially scalable and useful for future evaluations.

Critical weaknesses

The authors do not provide a formal guarantee that the generated scientific problems have never appeared before. That is the major problem of this works and questions the soundness of the approach.

The authors evaluate 80 problems across 20 scientific domains, meaning only four problems per domain. This seems limited for drawing broad conclusions about scientific-discovery capabilities. For example, solving three out of four ecology problems does not necessarily mean a model is generally good at discovering ecological mechanisms.

In Section 2.5, the authors explicitly score models based on their predictions, not whether they recover the true equations.

Several existing benchmarks, particularly DiscoverPhysics, PhysGym, and NewtonBench, already evaluate scientific discovery through interactive experimentation and hypothesis generation. DiscoverPhysics also considers hidden mechanisms and evaluates predictions on unseen trajectories. Given this overlap, I am not fully convinced of the novelty of UNSATZ is sufficiently established.

Questions for authors

Could the authors clarify what fundamentally new capabilities their benchmark measures beyond those already covered by existing work?How is UNSATZ fundamentally different from existing benchmarks such as DiscoverPhysics and PhysGym? How can we be sure that the models are actually discovering scientific mechanisms rather than parroting familiar equations or using prior knowledge? Since the benchmark evaluates predictive accuracy rather than recovery of the true underlying equations, how do the authors justify the claim that it measures scientific mechanism discovery rather than simply predictive modeling? If UNSATZ were released tomorrow, would researchers learn something important about scientific-discovery agents that they could not already investigate with DiscoverPhysics or PhysGym?

5 comments

We thank sgu2 for the review, and address the questions below:

W1. The generator excludes the known laws. This has been made clear in the revision. In twelve domains and in the transfer problems, it compares every candidate world with a list of 300 named laws, (e.g. Lotka–Volterra, van der Pol, Michaelis–Menten, etc.), and discards any match, whatever the coefficients and the naming of the variables; this check was part of the generator that drew the evaluated worlds and is now described in Section 2.6. Applied to the true programs of the evaluated worlds whose class it covers, it finds no named law; the 24 delay, spatial, lattice and agent-rule worlds match none of 70 named models of their classes. Named laws are rare even without the check: a candidate of the kinetics generator is the Lotka–Volterra system with probability one in 22 million (Appendix P.1). Every world is a new draw from a private seed, fixed before any agent ran.

W2. The paper makes no claim about an agent's skill in a single domain. Every claim about agents rests on all 80 worlds, on which the agents are compared in pairs; resampling whole domains with their worlds, GPT is first in 99.7% of 200,000 samples (Appendix O.1).

W3, Q4. A curve fit and a mechanism agree on the data they were fitted to and differ beyond them, which is where the hidden tests are: outside the ordinary ranges, under interventions and over long runs (Section 2.5). The score separates the two: 163 of the 321 executable final programs that fail in deterministic worlds fit their own experiments closely, and the true dependency graph, which variables enter each equation, is found in 77% of passing final programs against 43% of failing ones (Appendix N.2). A score on the equations would do worse: only 6 of the 230 passing programs have exactly the true terms, and it would reject the other 224, of which 171 have the true dependency graph (Table 29).

W4, Q2. UNSATZ differs from all three (Table 1). UNSATZ can be regenerated arbitrarily and can generate arbitrarily many worlds, thus overcoming contamination without requiring the benchmark be unpublished: its worlds are sampled programmatically by a generator from seeds in 20 domains, whereas a person wrote or selected the 22 DiscoverPhysics worlds, the 97 PhysGym problems and the 108 NewtonBench laws. Moreover, the agent writes a complete program and decides how many states exist, which are latent and how the sensors read them, whereas the others ask for a force law or a closed-form law in given variables. The answer is scored on predictions alone, with no judge model, and experiments are priced under a budget rather than counted.

Q1, Q5. The capability measured is proposing the ansatz of a system when neither its variables nor the form of the answer is given. What researchers can learn here and not with the other benchmarks rests on two features: a true program that can be fitted to any agent's data, and a generator that can change one property of a world or place one mechanism in two worlds. With them, the revision shows that agents fail at the structure rather than the experiments (general response, A), that recording the latent states leaves most failures in place (55 of 72 episodes still fail; Section 4.4), and that a correct program of one world helps on a related world (15 of 42 episodes, against 5 of 64 without one; Section 4.5).

Q3. The worlds are not solved by recall: 211 of the 230 passing final programs match no named law, and familiar final programs, those that match or contain a named law or are entirely linear, pass less often (21% against 36%; Appendix P.2). Agents do propose named laws, in 449 of the 717 episodes with a program, but in 360 episodes they replace them, in 267 of these with a program that fits the data better.


We hope our response answers the reviewer’s concerns and, if so, would be grateful if the reviewer would consider raising their score.

It compares every candidate world with a list of 300 named laws. LLMs have likely encountered substantially more than 300 scientific laws and mathematical formulations during pretraining. ! This is why soundness is my main concern. Also, the paper does not clearly demonstrate the practical usefulness of the benchmark, for example, whether it could be used to train LLMs to improve their scientific-discovery capabilities in specific domains. instead it just benchmark existing LLMs. Finally I can not see code in the submission which also raises reproducibility concerns. Therefore I will maintain my score.

Contamination. Your comment reads the comparison with 300 named laws as our sole defense against contamination. But our primary defense is that every evaluated world is new. The generator drew each world at random from a private seed, and neither its program nor its hidden tests has been published (Section 3, Appendix P.1). A model that had memorized every law ever written would still have to find, by experiment, a program it has never seen, so the number of laws a model has read is beside the point. The concern does apply to benchmarks whose answers are known laws, such as NewtonBench, whose 108 problems are modifications of 12 canonical laws (Table 1), but UNSATZ was explicitly designed to avoid it.

Nonetheless, we also tested recognition directly to more directly address your soundness concerns:

  • We gave a frontier LLM (Claude Opus 5.5) the 92 evaluated true programs, shuffled blind with 19 named laws written in the same language, all with neutral variable names, and asked it to identify every named model. It named 18 of the 19 laws and none of the 92 worlds.
  • Among 76,249 recorded draws of twelve of the generators that the comparison covers, no two draws of one generator have the same equations up to their coefficients (Appendix P.1).
  • Only 6 of the 230 passing programs have the terms of the true program; the other 224 are different programs that predict the same behavior (Table 29). The passing agents built their programs from the data.

On a formal guarantee: known laws are finitely many, while the number of programs the modeling language can express grows without bound with their size. Among programs of size at most n, the fraction that are known laws therefore tends to zero as n grows, so in the limit known laws are a null set. UNSATZ bounds program size so that worlds stay solvable from affordable experiments, and for this bounded generator the guarantee holds by construction: the generator discards every candidate that matches one of the 300 named laws, whatever its coefficients and variable names, so the probability that an evaluated world there is one of them is exactly zero (Section 2.6). We also wish to contextualize the relative size of our list: 300 is longer than the English Wikipedia list of named differential equations, which has only 216 entries.

Practical usefulness. The review asks whether the benchmark is useful beyond benchmarking existing LLMs. It is useful for what it shows about how LLM agents do science, which earlier benchmarks cannot measure because they fix the variables or the form of the answer (Table 1). We restate our findings here:

  1. The structure is the bottleneck. In 268 of the 490 failed episodes, the true program with unknown coefficients, fitted to the agent's own experiments, passes the hidden tests (Section 4.1).
  2. Finding the latent states is not enough. Final models with the true number of latent states pass no more often (8 of 69) than the others (11 of 93), and with every latent state recorded, 55 of 72 episodes still fail (Section 4.4).
  3. The training error misleads revisions. In 235 of the 510 episodes with more than one submission, the final model predicts worse than an earlier one (Section 4.4).
  4. Agents reuse a discovered mechanism. Agents holding a correct program of one world solve a related world in 15 of 42 episodes, against 5 of 64 without one (Section 4.5).
  5. Classical methods solve 14 of the 64 deterministic worlds over all their datasets, against 39 for GPT (Section 4.1).

Although we did not claim this in our manuscript, we agree that UNSATZ can be used to train LLMs in specific domains, for example as a reinforcement-learning environment. It draws any number of new worlds in each of its 20 domains, so a training set drawn from other seeds shares no world with the evaluation set, and its hidden tests give a verifiable reward without a judge model (Section 2.5).

However, none of the six interactive benchmarks in Table 1, including the three the review names, trains a model, and Gravity-Bench, PhysGym and DiscoverPhysics leave training to future work. Since the review cites this as a reason to keep its score, could the reviewer say what demonstration of practical usefulness would be needed to support our benchmark?

Code. The code is now in the supplementary material: the benchmark and its tests, one script per experiment with a README that maps each script to its section, and the analysis scripts behind the reported results.


We hope your concerns are addressed sufficiently to motivate raising the score; otherwise, we would appreciate the opportunity to clarify or improve our work further upon your feedback.

Reviewer_VNRDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that most automated science benchmarks effectively provide the ansatz: variables are known, functional families are constrained, or candidate terms are predefined. In contrast, real scientific discovery requires deciding what latent variables exist, how they interact, and how observations relate to hidden states.

To address this gap, the authors introduce UNSATZ, a benchmark where: Hidden systems are generated from private seeds. Systems span 20 domains, including ODEs, PDEs, stochastic systems, causal models, delay systems, lattice models, and networks. Agents purchase experiments under a budget. Agents propose executable programs in a shared DSL. Coefficients are fitted automatically by a common fitter. Scoring uses only held-out extrapolation, intervention, and long-horizon tests.

The benchmark evaluates nine language models on 80 worlds and transfer tasks. GPT‑6 Astra solves 48/80 worlds and Claude Fable 5.1 solves 38/80, while classical equation-discovery baselines perform substantially worse. The authors conclude that: Model structure discovery is substantially harder than parameter estimation. Strong agents make fewer procedural mistakes. Agents often identify latent variables or interaction graphs but fail to infer governing mechanisms. Agents are strongly biased toward improving training error rather than predictive validity.

Figure 1 provides an illustrative ecology example involving hidden resource states. Figure 2 clearly shows the benchmark evaluation loop.

Critical strengths

1 Strongly Motivated Problem The paper identifies a real and important weakness in current scientific-discovery benchmarks: most evaluate parameter estimation or constrained structure search rather than open-ended model formulation. The distinction between "discovering the ansatz" and fitting its parameters is compelling and clearly articulated.

2 Novel Benchmark Design UNSATZ meaningfully differs from prior benchmarks summarized in Table 1. The benchmark introduces: Partial observability. Latent states. Observation distortions. Auxiliary channels. Experiment design decisions. Budget constraints. Held-out intervention testing. This yields a richer scientific reasoning challenge than symbolic regression alone.

3 Emphasis on Predictive Validity One of the strongest ideas is evaluating on: Out-of-distribution conditions, Interventions, Long-horizon prediction, rather than symbolic equation matching

4 Careful Attempt to Isolate Structure Discovery The benchmark explicitly separates: Structure search, Coefficient estimation. A shared fitter estimates coefficients for all agents and baselines. This design strengthens the paper's central claim that failures are largely attributable to structure discovery rather than optimisation.

5 Broad Domain Coverage The benchmark spans 20 domains: ODEs, PDEs, delay systems, causal models, metrology, spectroscopy, stochastic systems, networked dynamics, agent-based models, lattice systems. This breadth is impressive and helps reduce overfitting to a single scientific area.

6 Interesting Scientific Findings Several findings are genuinely insightful: Agents identify hidden-state existence better than hidden-state dynamics. Agents often recover causal structure but not governing mechanisms. Training error frequently misleads model revision. Previously discovered mechanisms transfer only when correctly discovered. These observations feel substantive rather than merely leaderboard-oriented.

7 Reproducibility The authors commit to releasing: generators, seeds, prompts, episode records, evaluation code. This is exemplary benchmark practice.

Critical weaknesses

1 Limited External Validity The benchmark simulates scientific discovery entirely within a DSL. While the ansatz problem is important, agents ultimately search within: a predefined language, bounded operator vocabulary, fixed parameter limits, shared structural priors. Consequently, agents are not discovering arbitrary scientific models but rather models expressible within carefully engineered program generators. The paper occasionally risks overstating the connection between benchmark success and authentic scientific discovery.

2 Potential Benchmark–Generator Coupling Worlds are generated using the same modelling language that agents employ. This creates an "in-language" discovery setting: Truth is always representable. Correct abstractions already exist within the DSL. Discovery becomes a search over the benchmark's own generative grammar. This is useful for controlled evaluation but weakens claims about broader scientific reasoning. A stronger discussion of this limitation is warranted.

3 Statistical Power Concerns Each domain contributes only four worlds. Although there are 80 total worlds, many domain-level conclusions rest on extremely small samples. Examples include: latent-state difficulty, transfer effectiveness, domain-specific comparisons. Confidence intervals are provided only for aggregate results. Additional uncertainty analyses would strengthen the paper.

4 Evaluation Depends Heavily on Generated Worlds The benchmark's conclusions depend critically on: world generators, calibration procedures, budget allocation, held-out test construction. Because these design decisions are hand-crafted, benchmark difficulty could be substantially affected by generator idiosyncrasies. The paper would benefit from ablations showing robustness to alternative generator choices.

5 Transfer Evaluation Is Underpowered Transfer claims rely on: four transfer problems, 36 studies overall. The observed transfer effect is intriguing but statistically fragile. At present, the transfer component feels more exploratory than conclusive.

6 Lack of Human Baseline A significant omission is the absence of human performance. Given that the paper is motivated by scientific reasoning, it would be valuable to know: whether skilled humans solve these worlds, which domains are genuinely difficult, whether benchmark difficulty aligns with expert intuition. Without such baselines it is difficult to calibrate the benchmark.

Questions for authors
  1. How much do the conclusions depend on the benchmark generator and DSL? The worlds are generated using the same modelling language that agents use to express hypotheses. Can the authors provide sensitivity analyses showing whether rankings and success rates remain stable under: alternative operator vocabularies, different latent-state distributions, different observation-model complexity, or generators not sharing the same DSL as the submitted models?

  2. How much of the performance gap comes from experiment design versus hypothesis generation? The benchmark combines two difficult skills: selecting informative experiments under a budget; constructing the correct model structure. Can the authors quantify the contribution of each component via ablations such as: fixed optimal experimental datasets with free structure search, agent-designed experiments with oracle structural hints, or replaying one agent's experiments through another agent's modelling pipeline?

  3. How representative is UNSATZ difficulty of real scientific discovery? Have the authors compared benchmark performance with human experts (e.g., controls engineers, physicists, systems biologists) on a subset of worlds, or otherwise validated that the generated tasks capture difficulties encountered in real scientific model construction?

2 comments

We thank VNRD for the detailed review and for recognizing that the design separates structure discovery from coefficient fitting. We address the questions below in 2 parts:

(1/2)

W1. The benchmark isolates three steps of an investigation, choosing experiments under a budget, proposing a model and revising it against data, and holds the language and the fitter fixed so that success and failure at these steps can be attributed; the revised introduction states this scope. Within it, the language is broad and the model is determined by the agent:

  • 362 of 368 named models from the literature can be written within an agent's limits; the exceptions include the Hodgkin–Huxley model, a five-variable circadian oscillator and four PDEs with spatial derivatives above order two (Appendix D.5).
  • The limit of 16 coefficients rarely binds: the true programs use 4 to 15, and 104 of the 35,917 programs the agents fitted or submitted (0.3%) exceeded it (Appendix D.5).
  • Only 6 of the 230 passing final programs contain exactly the terms of their true program (Appendix N.2).

W2. This is deliberate: because the true program can be fitted to any dataset an agent bought, a failure of the structure can be told apart from a failure of the experiments (Q2); the revision discusses it where the language is introduced (Section 2.3). It does not turn discovery into a search over the grammar. A search over complete programs with 4,096 proposals solves 1, 0 and 1 of the 48 ODE, map, static and causal worlds it applies to on its three datasets, where GPT solves 23 of them (Table 17). And 105 of the 230 passing final programs lack a transcendental function of their true program, such as a hyperbolic tangent, and any exact substitute for it, yet solve 30 of the 54 solved worlds (Appendix N.2).

W3. Latent states now rest on an experiment (Section 4.4, Appendix N.3). We reran the 18 ODE and map worlds that have latent states with every latent state recorded on an additional channel. GPT, Claude, DeepSeek and Gemini solve 17 of the 72 episodes, against 8 with the states hidden, and 45 of 60 on the worlds without a latent state: hiding the states adds to the difficulty, but most of it remains. Transfer now rests on an experiment with intervals and an exact test (W5). Domains and agents (Appendix O.1): the per-domain table is descriptive, and every comparison between agents is paired on all 80 worlds. In 200,000 bootstrap samples of the 20 domains, each keeping its four worlds, GPT is first in 99.7% and Claude ahead of the other seven agents in 99.9%; GPT leads Claude by 10 worlds (95% interval 3 to 18) and Claude leads the third agent by 14 (7 to 22). Table 31 gives each agent's interval and range of rank.

W4. Each of these choices varies across the 80 worlds or can be varied after the fact, and under every variation GPT is first or tied for first (Appendix O.1). Generators: split in two on each of 13 recorded attributes, such as the number of latent states and of auxiliary channels, the signal-to-noise ratio, the budget and the size of the true program, over all worlds and within domains, GPT is first or tied for first in all 50 halves, and no attribute reorders the agents more than a random split does (smallest p-value 0.15). Budgets: in the 40 worlds with the smaller budget factors GPT solves 23 and Claude 18, in the other 40 they solve 25 and 20, and no other agent solves more than 14 in either half. Hidden tests and threshold: with any one test group, any pair of groups or their mean, and at every threshold on the error from 0.02 to 1.0, GPT is first and Claude second (Tables 33 and 34).

W5. The transfer effect is now tested by a controlled experiment, and it is significant (p = 0.002; Section 4.5, Appendix E.10). In all 36 studies, a fresh agent is also given the true program of the first world, which separates the mechanism from the agent's own success. Given the true program, agents solve the related world in 10 of 36 studies, against 6 of 35 with their own program and 4 of 35 starting fresh; where the agent failed the first world, its own program does no better than a fresh start (1 of 29 against 2 of 29), while the true program solves 7 of 30 (Table 11). A correct program helps whether or not the agent found it. Pooled, agents that start from a correct program, the true one or their own after solving the first world, solve the related world in 15 of 42 episodes and the others in 5 of 64 (exact permutation test within studies, p = 0.002); on the control world, which lacks the mechanism, the counts are 2 of 42 and 1 of 64 (p = 0.44). Nine first-run episodes ended early through the provider; with those runs in place of their reruns, p = 0.006.

(2/2)

W6, Q3. No human study could be run in the discussion period, because it requires ethics review, or a certified exemption, before any participant is contacted; the generator, which will be released with the benchmark, and the agent's interface (Appendix C) allow one on fresh worlds. The revision instead answers the review's three questions with measurements.

  • Can the worlds be solved? Every world is solvable within its budget from its true structure with the fit started near the true coefficients (Section 2.6), and at least one agent solves 54 of the 80 (Section 4.2).
  • Which domains are difficult? Of the 36 episodes of each domain, kinetics, reservoir and lattice are solved in one each and reaction–diffusion in 34 (Table 25).
  • Is the difficulty that of real model construction? It lies in choosing the structure (general response, A), only partly in unmeasured states (W3), and in predicting beyond the data: 163 of the 321 executable final programs that fail in deterministic worlds fit their own experiments with the square root of their training error below 0.1, yet fail the hidden tests (Appendix N.2).

Q1. The ranking holds under the first three scenarios; the fourth is not in our data, but two results address it nonetheless (Appendix O.1). On the 36 worlds whose true program uses only signed powers and hyperbolic tangents, GPT solves 26 and Claude 19, and no other agent more than 14; on the 35 with a latent state, 16 and 13 against at most 10; on the 15 with a sensor that mixes or saturates states, 6 and 3 against at most 1 (Table 32, Appendix O.1). With any one of the seven response functions excluded, GPT and Claude stay ahead of the other seven agents, and with the latent states recorded (W3) the order of the four agents is unchanged. For generators outside the language: the hidden tests compare behavior and never read the program, so the scoring applies to a system in any form, and solving a world does not require the functions of its true program (W2).

Q2. Hypothesis generation separates the agents far more than experiment design does. We answer the three ablations with the baselines and with refits of the recorded episodes (Section 4.1, Appendices I.1 and N.1):

  1. Fixed dataset, free structure search. On each world's calibration design, bought by a procedure that knows the true program (Section 2.6), the classical and neural baselines solve 6 of the 64 deterministic worlds under any of three noise seeds, against 39 for GPT in its own episodes (Appendix I.1).
  2. Agent-designed experiments, true structure as the hint. Fitted to the experiments of the 490 failed episodes, the true structure passes in 448, in 268 of them from random starts; for every agent the rate from random starts is 50% to 60% (Table 27), so the agents' experiments are similarly sufficient and their structures are not.
  3. One agent's experiments through another's structure. In the worlds GPT solved, GPT's structure fitted to a failing agent's experiments passes in 180 of 197 episodes, whereas the failing agent's structure fitted to GPT's experiments passes in 11 (Table 28).

Refitting the agents' 248 executable but failing final programs on 40 worlds with 24 starts of 200 evaluations turns none into a pass (Section 4.1).


We would appreciate if the reviewer considers raising their score if our responses are satisfactory.

Reviewer_fMG8OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an LLM benchmark for ansatz construction, essentially performing this essential step of the scientific method. Tasks are procedurally genetated for variety, and the task is sufficiently diffucult that current models do not saturate.

Critical strengths

This is a very well motivated paper. Scientific automation is a critical topic, and there is indeed a lacuna in the literature that this benchmark fills. The paper is clearly written and executed, and I expect this benchmark to be impactful

Critical weaknesses

The classical baselines are weak, and I am not convined they represent the best of classical methods. This does not undermine the central purpose of the paper, but it should be addressed. Many experiments were run only one time, and thus the presented results are potentially noisy (line 335).

Questions for authors
  1. Can you ensure each run is done at a minimum of 3 times?
  2. Can you present stronger classical baselines, or at least a justification for why these are sufficient?
1 comment

We thank fMG8 for recognizing the utility of our benchmark and address questions below:

Q1. We ran three episodes under different noise seeds for four agents: DeepSeek and Gemini on all 80 worlds, and GPT and Claude, the two leading agents, on one world of each domain (400 new episodes; Appendix O.2). GPT solves 11, 13 and 13 of these 20 worlds and Claude solves 9, 8 and 12, so GPT is ahead in every episode; DeepSeek solves 18, 18 and 16 of the 80 and Gemini solves 17, 16 and 15. A repeat shares the outcome of the first episode on 88% of worlds for DeepSeek and Gemini and on 78% for GPT and Claude.

The differences between agents are larger than this variation. The exact McNemar tests of Section 4.2 (GPT against Claude, p = 0.013; Claude against the third agent, p = 0.001) pair the agents on the same worlds with one episode each, so run-to-run variation is already in their noise, and the gaps between the first three agents are 2.7 and 3.8 times the standard error of a difference, which includes that variation (Appendix O.1). For the same number of episodes, one episode on each of more worlds measures an agent at least as precisely as repeats on fewer worlds, so the other five agents keep one episode per world.

Q2. We rebuilt the baselines with nine methods (Appendix I.1): SINDy with a library of the language's operators, in integral form with bootstrap term selection (Schaeffer & McCalla, 2017; Fasel et al., 2022), with and without a latent coordinate, polynomial SINDy and PDE-FIND; PySR; an MLP and two neural ODEs; and a grammar search with 4,096 proposals, eight times the submission's. Each method runs on every deterministic world it applies to, with three datasets bought within the budget: the calibration design, chosen by a procedure that knows the true program; GPT's experiments; and a random design. The baselines collectively solve 14 of the 64 deterministic worlds; GPT solves 39 and Claude solves 31 with one episode per world. The baselines keep up only where the task reduces to regression, with 10 of the 16 static and causal worlds against 9 for GPT, and solve 4 of the 48 worlds with dynamics, against 30 for GPT (Table 18).

Comments on the paper

2 comments

We thank the reviewers. The revision adds 544 episodes in three new experiments and seven analyses; section, appendix and table numbers refer to the revised paper. Three results answer several reviews.

A. Failures are failures of structure rather than data. In 448 of the 490 failed episodes, the true structure fitted to the failing agent's own experiments passes the hidden tests, in 268 of them from the fitter's random starts (Section 4.1).

B. Classical baselines rarely find a model that predicts. Nine methods, each given three datasets per world, together solve 14 of the 64 deterministic worlds; GPT solves 39 and Claude 31 with one episode per world (Appendix I.1).

C. The ranking is stable. GPT is first and Claude second under every removal of up to five domains, at every error threshold from 0.02 to 1.0 and under every combination of test groups, and GPT solves more worlds than Claude in each of three episodes on one world of each domain (Section 4.2, Appendix O).

AuthorsOpenReview ↗

Summary of rebuttal phase

As the discussion period comes to an end, we would like to thank the reviewers and the area chair for their time and expertise in reviewing and helping to improve the work; all three reviewers recommend acceptance, and to our knowledge, no concern remains open. We summarize the paper and the revision.

Motivation. Frontier labs and public programs now invest in LLM agents for science, for example OpenAI for Science, Google DeepMind's co-scientist and the U.S. Department of Energy's Genesis Mission. Benchmarks for automated equation discovery mostly supply the ansatz: they fix the variables, a library of candidate terms, or a law to be modified. Whether agents can construct the ansatz themselves, deciding which quantities exist, which are hidden, how they interact and how they are measured, has not been measured. In UNSATZ the agent does this, buys experiments under a budget, and is scored only on how its final program predicts experiments it never saw. fMG8 writes that "there is indeed a lacuna in the literature that this benchmark fills", and VNRD that "the distinction between 'discovering the ansatz' and fitting its parameters is compelling and clearly articulated".

Contamination and reproducibility. LLM benchmarks usually have to choose between the two. A published test set can be reproduced and audited, but its items can enter training data, and items built from known laws may be familiar to a model before the benchmark is released. A private test set resists contamination, but no one else can check the evaluation. UNSATZ does not have to choose, because its worlds are generated. The generator, the fitter and the scorer are public and can be audited, while the evaluated worlds are new and have never been published. Once we release them with the episode records, anyone can rescore every episode without a judge model or a model provider, and later evaluations simply draw new worlds from the same generator. Among the benchmarks in Table 1, only UNSATZ can draw new worlds without limit. sgu2 finds that the approach "reduces this risk and it is quite valuable", and VNRD calls the planned release "exemplary benchmark practice".

Findings. Nine models solve between 21% and 60% of 80 worlds in 20 domains, and their failures follow consistent patterns:

  • The structure is the bottleneck. In 268 of the 490 failed episodes, the true program fitted to the agent's own experiments passes the hidden tests (Section 4.1).
  • Agents find the variables but miss how they act. With every latent state recorded, 55 of 72 episodes still fail (Section 4.4).
  • The training error misleads. In 235 of the 510 episodes with more than one submission, the final model predicts worse than an earlier one (Section 4.4).
  • A correct program helps on a related world: 15 of 42 episodes with one, against 5 of 64 without (p = 0.002; Section 4.5).
  • Classical and neural methods solve 14 of the 64 deterministic worlds, against 39 for GPT 6 Astra (Section 4.1).

In VNRD's words, these observations "feel substantive rather than merely leaderboard-oriented".

Revision. The revision adds 544 episodes in three new experiments and seven analyses, each answering a reviewer's request:

  • Three episodes per world for four agents, 400 in all (fMG8; Appendix O.2).
  • Classical baselines rebuilt with nine methods (fMG8; Appendix I.1).
  • The ranking under splits by generator attribute, budget, test group and threshold, and under resampling of whole domains (VNRD, sgu2; Section 4.2, Appendix O.1).
  • Fits of the true program to each agent's experiments, which separate experiment design from model construction (VNRD; Section 4.1).
  • An experiment that records every latent state (VNRD; Section 4.4).
  • A controlled transfer experiment (VNRD; Section 4.5).
  • The comparison with 300 named laws, now described, and evidence that agents do not solve worlds from memory (sgu2; Section 2.6, Appendix P).
  • The code, with one script per experiment, in the supplementary material (sgu2).

Audience. In our view, the paper is of interest to three groups. Those building LLM agents for science learn where agents fail: at the structure, with the data in hand. Those building benchmarks gain a way to resist contamination without keeping a private test set. And, as sgu2 notes, those training agents gain an environment with new worlds and a reward that needs no judge model. fMG8 expects the benchmark "to be impactful", and we believe its results are of interest to much of the ICLR community, beyond those who work on scientific discovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.