acceptodds
Under review as a conference paper at ICLR 2027

Learning to Outgrow a Theory: Experimental Discovery Beyond the Initial Hypothesis Space

Abstract

Scientific discovery systems typically optimize experiments within a fixed hypothesis space. This creates a failure mode when all available candidates omit the same missing mechanism: candidate disagreement can collapse even while the model class is systematically wrong. We formulate experimental model-class revision, in which a discovery policy jointly proposes a structural edit and a diagnostic experiment that tests whether that edit is necessary. The method couples a class-level distinguishability objective, in which one shared parameterization must explain all selected experiments, with anytime-valid sequential evidence that triggers structural revision only after the current class is rejected. On 400 held-out controlled dynamical environments, the joint policy reaches 89.5% exact recovery with a budget of 32 real experiments, improving the strongest matched baseline by 10.0 percentage points while requiring fewer executed experiments and candidate fits. The learned revision–experiment pairing transfers across unseen mechanism combinations, held-out but expressible primitives, parameter extrapolation, and shifted experiment costs; when the true mechanism is outside the edit grammar, it detects library insufficiency in 88% of cases with a 5.5% false-support rate. Revision gains also transfer to ODEBench and ODEBase model-library tasks, as well as DiscoverPhysics worlds. These results support a view of scientific discovery in which deciding what mechanisms a theory should make expressible and where to collect evidence are treated as a single sequential decision problem.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
automated scientific discovery, experimental design, model-class revision, hypothesis space expansion, active learning, sequential hypothesis testing, dynamical systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ssejOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies discovery when the current model class omits a mechanism shared by all its candidates. A learned policy jointly proposes an edit from a predefined grammar and an experiment intended to distinguish the revised class from the current one. Class-level testing requires a shared parameterization across experiments, and ideal sequential evidence governs revision. On 400 held-out dynamical environments, exact recovery is 89.5%, compared with 79.5% for falsification with independent edit and experiment proposals. Additional evaluations cover distribution shifts, out-of-grammar detection, public ODE collections, and simulated physics environments. The experiments demonstrate model revision in executable environments rather than newly conducted physical experiments.

Critical strengths
  • The shared-parameter class-level formulation addresses a concrete discovery failure. Requiring one parameterization to explain multiple experiments prevents separate refits from absorbing evidence of structural misspecification. Coupling this criterion to an edit and a diagnostic experiment gives a substantive framework for reasoning about when revision is necessary.

  • The empirical evaluation extends beyond one controlled task family. Recovery, predictive error, discovery costs, distribution shifts, and external dynamical collections are all examined. These results support the tested pipeline’s utility and provide more information than a single recovery endpoint.

Critical weaknesses
  • The novelty should be positioned more precisely within a predefined edit grammar. The paper studies revision beyond the initial class, while the candidate revisions remain expressible in a supplied grammar. The shared-parameter criterion and the joint-versus-independent comparison provide a meaningful contribution in this setting. A sharper comparison with active model selection, structural search, and sequential class testing would clarify which elements supply the new knowledge and prevent a broader interpretation as unrestricted mechanism invention.

  • Component-level attribution requires clearer training-resource accounting. The joint policy uses teacher-search supervision followed by return optimization. The reported experiment counts, candidate fits, and discovery-time compute do not establish equivalence of teacher supervision, training resources, and fitting configurations for the relevant learned controls. This limits attribution of the recovery gain specifically to joint decision learning, although the recovery and transfer results remain evidence for the complete tested pipeline.

Questions for authors
  • Which aspect of the proposed objective or policy goes beyond active selection over the same expanded model library with the same sequential class test? Can you isolate that aspect while holding the edit grammar and available experiments fixed?
  • For the joint and independent policies, please compare teacher-search access, supervision volume, training objectives and compute, architecture capacity, and fitting settings. Does the advantage persist when these are matched?
  • I encourage the authors to relate the proposed class-revision loop to interactive biological model-reconstruction environments such as SciGym[1]. Explaining what is added by an explicit revision grammar, shared-parameter class test, and coupled edit–experiment policy would make the contribution and its intended experimental use clearer :)
  • For one external benchmark task, what practical experimental operations, measurements, and costs do the available queries represent? Which restrictions of that application are encoded in the benchmark, and which remain simulator assumptions?

[1] Duan, H., Lu, S., Harrigan, C. F., Desai, N., Lu, J., Koziarski, M., ... & Maddison, C. (2026). Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab. Advances in Neural Information Processing Systems, 38.

Reviewer_Hxe3OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes joint edit–experiment discovery for the task of fitting ODE models to ODE-produced datasets. The key innovations are (1) a class-level distinguishability score; (2) a learned joint edit–experiment policy; (3) anytime-valid sequential evidence gathering process that triggers structural revision based on class-level evidence;

It is evaluated on synthetic-data benchmarks, ODEBase, ODEBench and DiscoverPhysics.

Critical strengths

Model explicitly reports that the edit library is insufficient, instead of brute force fitting a wrong-class models. This is a very valuable signal, as it indicates that the system might be driven by previously unaccounted physical processes.

The model shows improved exact recovery on ODEBench and ODEBase benchmarks.

The paper addresses a plausible structural limitation of baseline models which use model disagreement to choose experiments.

Critical weaknesses

Library insufficiency is the core contribution of the method. A few key metrics are missing. What is the false-insufficiency rate on the 400 in-grammar environments? How is “library insufficiency” outcome separated from a plainly difficult problem for which the budget was exhausted?

Questions for authors

In the benchmark presented in Table 1, how much performance difference is due to the proposed mechanism of shared structural misspecification? Did the baselines have access to the same candidate pool as the proposed method?

What are “target-relevant edits” (L190)?

How is “post-revision support” defined in Section 4.3?

How was the template library for DiscoverPhysics constructed? In the original paper LLMs wrote free-form equations.

1 comment

Thank you for your careful reading of our response and for raising your score. We sincerely appreciate your time and constructive feedback, which has helped us improve the paper.

Reviewer_DQudOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper targets a specific failure in automated scientific discovery: every candidate model can share the same missing mechanism. When that happens, the candidates agree with each other even though the whole model class is wrong. The authors frame the problem as experimental model-class revision. At each round, a learned policy πϕ(m,e∣St)\pi_\phi(m, e \mid S_t) jointly proposes a typed structural edit mm and an experiment ee to test it.

The method has three parts:

Distinguishability score. Candidate edit–experiment pairs are ranked by a class-level score J^t\hat J_t (Eq. 5). One shared parameter vector of the current class must explain every experiment in the design, so a high score marks a mismatch that refitting experiment by experiment cannot hide. Revision gate. Revision is allowed only after the current class is rejected by an anytime-valid likelihood-ratio statistic Ej,nE_{j,n} (Eq. 6), with error budget αj=α/[j(j+1)]\alpha_j = \alpha/[j(j+1)] per stage. Proposition 2 gives the resulting false-rejection control. Training. The policy is trained by teacher search on synthetic tasks whose mechanisms are known, then imitation learning, then return optimization.

Results:

Controlled benchmark. On 400 held-out dynamical environments with budget B=32B = 32, exact recovery is 89.5% versus 79.5% for the strongest baseline, which uses the same class test but picks edits and experiments independently. It also uses fewer experiments (18.7 vs 21.6) and fewer candidate fits (46 vs 68). Robustness. Gains hold across budgets, five distribution shifts, all six mechanism families and three seeds. Out-of-library mechanisms. When the true mechanism is outside the edit grammar, library insufficiency is detected in 88% of cases with 5.5% false support. External benchmarks. Exact revision improves on ODEBench (57.1 → 69.8%) and ODEBase (60.0 → 73.3%). DiscoverPhysics passes rise from 34/55 to 42/55.

Critical strengths
  • A clear, well-motivated problem, backed by a principled revision gate. Proposition 1 shows that agreement within the class can hide a shared omission. This identifies a real blind spot in disagreement- and information-gain-based discovery. Gating structural revision on an anytime-valid class test is a principled way to decide when to change the hypothesis space. The test appears to be well calibrated in practice: 2.1% false rejection against a 2.5% budget (Fig. 10). This framing is the paper’s most lasting contribution.
  • Consistent paired gains, with useful ablations. The improvement holds across every mechanism family, budget, shift and seed. On the external ODE sets it holds per system (9:1 discordant pairs). The per-experiment-refit ablation (89.5 -> 70.5%) directly supports the shared-parameter idea behind J^\hat J.
Critical weaknesses
  • The learned policy, which drives the gains, is essentially unspecified. Table 2 shows the coupling gives no gain without training: “No policy training” reaches 78.8%, versus 79.5% for the independent baseline. The headline improvement therefore comes from the learned proposer. Yet the paper gives no account of: the architecture beyond Fig. 2 (size, encoders); the teacher-search procedure and its budget; the imitation and RL algorithms and their hyperparameters; how the predictive densities qm,tq_{m,t} are built; how the design distribution wt,ew_{t,e} is chosen; how J^\hat J is combined with complexity, calibration and cost terms; what “Score only” means. Training compute is excluded from all cost accounting (Table 9). There is no code or reproducibility statement. The comparison may also be uneven. If Falsification + independent, LLM-ACES and AutoSciLab were not trained on the same 2,000 synthetic environments, then “joint vs independent” is partly “trained on this grammar vs not”. Without these details the main result can’t be reproduced or properly attributed.

  • Evaluation is mostly synthetic, and several reported quantities don’t add up. The controlled benchmark, the training distribution and the edit grammar come from the same generator. Real data appear only through public ODE libraries and DiscoverPhysics, and the paper doesn’t say how the typed grammar was instantiated for those. Out-of-library numbers. In Table 11 the joint policy rejects the class in 92.5% of cases, detects insufficiency in 88.0%, and returns false support in 5.5%. Under either reading of “false support” this is inconsistent: If false support only happens after rejection, then 88.0+5.5>92.588.0 + 5.5 > 92.5. If unrejected cases also count as false support, false support must be at least 7.5%. The text describes the gap as 4.5 points. Which 200 environments? The open-set set is called “200 mechanisms excluded from the grammar”. Yet the 200-environment diagnostic split in Table 3 is broken down by in-grammar families, and it is also used for sufficient-class and identifiability diagnostics. The J(w)J(w) example. The Gaussian example in Prop. 1 has Y∣e2∼N(0,σ2)Y\mid e_2 \sim N(0,\sigma^2) with no free parameter. Under that model, J(w)=(1−w)Δ2/2σ2J(w) = (1-w)\Delta^2/2\sigma^2, which is largest when e2e_2 is used alone. The paper’s J(w)=2w(1−w)J(w) = 2w(1-w), with “either single design has zero separation”, instead requires a class whose single parameter must explain both experiments. So the analysis in Fig. 4(a) doesn’t follow from the example it is presented with.

Questions for authors
  • Were the independent and LLM baselines trained or tuned on the same 2,000 training environments with comparable compute? If not, can you report Falsification + independent with an independently trained edit proposer and experiment proposer? That would separate the effect of coupling from the effect of training.
  • Please give the full specification of the policy and its training: architecture, parameter count, teacher-search budget, imitation and RL algorithms, and total training compute. Will code be released?
  • Please reconcile Table 11: how is “false support” defined, and what happens in the remaining 100−92.5=7.5%100 - 92.5 = 7.5\% of cases? Is the open-set set the same as the 200-environment diagnostic split in Table 3?
  • Is the headline “exact recovery” exact syntax or equivalence-class coverage? Fig. 11 suggests the two differ a lot on non-identifiable tasks.
  • Which model class gives J(w)=2w(1−w)J(w) = 2w(1-w), and how does it relate to the setup of Prop. 1?
  • How was the edit grammar instantiated for ODEBench, ODEBase and DiscoverPhysics? Was the policy retrained or fine-tuned for them? - Which LLM backbones did LLM-ACES and AutoSciLab use?
  • “Uncalibrated predictive density” raises unnecessary revision to 5.9%, above the overall α=0.05\alpha = 0.05. Proposition 2 only requires qq to be normalized and fixed in advance. Does this variant break either condition?
3 comments
AuthorsOpenReview ↗

Follow-up on our response

Thank you for your comments. We wanted to politely check whether our rebuttal has addressed your concerns. Please let us know if anything remains unclear.

Reviewer_DQudOpenReview ↗

Response to rebuttal

Thank you for the thorough response and revision. Most of my concerns are resolved.

The Table 11 decomposition (Table 16) is now consistent, and adding the Inconclusive outcome makes the open-set results interpretable. The J(w) clarification is correct. Separating Proposition 1’s information-gain blind spot from the shared-parameter example of design mixtures fixes the issue. Renaming the “uncalibrated” variant as a non-predictable density, and showing that all normalized, predictable variants stay below α₁, addresses my calibration question. The architecture, training and compute details (Tables 25–27), the strict-syntax recovery numbers, and the LLM baseline details resolve my reproducibility concerns. I also accept the correction on attribution: my 78.8 vs 79.5 comparison crossed training conditions. Table 20 is the right control set.

I have raised my scores. A few remaining questions:

  • Source of the joint-head gain. Both trained policies rank pairs with the same edit-conditioned score over a 16×32 pool, so the +10 pp must arise from (a) which designs enter the pool or (b) the learned term ℓ̃_φ in s_t. Could you report the number of distinct designs evaluated per round for each policy? If the joint head proposes designs per edit and so evaluates more of them, could you add a control with matched design counts (e.g., untrained global-pair selection with an enlarged design pool)? Relatedly, the trained independent head gains only +0.75 pp over untrained global-pair selection. Is there evidence that this head is not undertrained? External evaluation. ODEBench and ODEBase remain simulated from known equations, so the paper has no real experimental data. This should be stated as a limitation. For how many of the 63/60 systems is the removed term expressible in the Table 4 grammar? For DiscoverPhysics, the two added central-force primitives were chosen with access to the public worlds. Results on the private worlds, even via the maintainers, would strengthen the external claim considerably. Table 15. The paired cells (35 / 9 / 1) are identical across ODEBench and ODEBase. This may be coincidence, but please include per-system outcomes in the appendix.
  • Open-set robustness. Open-set detection uses a single checkpoint, and the bootstrap CI only resamples environments. Could you report all three seeds? Please also confirm that none of the four excluded mechanism families appear in the development data used to fix the support thresholds.

Overall, the paper’s framing and revision gate are a lasting contribution, and the matched controls now support the main claim on the synthetic benchmark. My remaining reservations concern how far the results generalize beyond simulation and exactly where the joint-head gain comes from.

Thank you for your careful re-evaluation and for raising your scores. We appreciate your detailed follow-up questions and will address them in the final version, including the design-count analysis, the limitation on simulated data, per-system outcomes, and open-set results across all three seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.