acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Discover Scientific Laws in Real and Parallel Settings?

Abstract

Scientific law discovery has long been central to scientific progress, proceeding through iterative cycles of generating hypotheses, testing them against empirical evidence, and refining them under scientific constraints. As large language models (LLMs) become increasingly involved in scientific research, whether they can discover scientific laws and how to evaluate this ability remain open questions. A central evaluation challenge is to move beyond familiar published equations while keeping discovery tasks grounded in scientific data and constraints. We introduce SCILAWS-BENCH, a curated collection of scientific task packages grounded in the source literature, each linking a scientific problem, supporting data, published reference equations, and scientific-validity rubrics. Through agent-assisted curation and human verification, we assemble 118 problems spanning six disciplines, drawing on 381 papers, 291 candidate laws, and roughly 8M data points. Each problem supports two complementary evaluation settings. SCILAWS-REAL uses fixed scientific data to evaluate proposed laws for held-out predictive fit and scientific validity. SCILAWS-PARALLEL evaluates recovery of a newly synthesized structural variant of a published equation through active queries to a simulator calibrated to the source data. Our evaluation reveals three limitations: good predictive fit need not imply scientific validity, recovering a published formula does not establish recovery of its new structural terms, and candidate selection remains a bottleneck in scientific law discovery.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Scientific Discovery, Equation Discovery, AI for Science, Benchmark Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZDw7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SciLaws-Bench, a benchmark for evaluating whether LLMs can discover scientific laws from data. It consists of 118 problems across six scientific disciplines, collected from published research. The benchmark has two settings: SciLaws-Real, where models propose equations that fit real scientific data while satisfying scientific constraints, and SciLaws-Parallel, where models conduct experiments in simulated environments to recover hidden equations derived from published laws.

The authors evaluate many different and frontier LLMs using a common agent framework. They find that good predictive performance does not necessarily imply scientific validity, and that models often recover familiar equations but struggle to discover new mathematical structure. They also find that models frequently generate better candidate equations than the ones they ultimately select, suggesting that evaluating and selecting hypotheses remains an important bottleneck in scientific discovery.

Critical strengths
  1. The benchmark and its data seem valuable even to the frontier research.
  2. The evaluations span many different models, even including Opus 5.5. Moreover, the provided insights are interesting.
  3. The evaluations are rigorous and well documented.
Critical weaknesses
  1. While frontier models are very important to the community, I believe evaluating at least one specialized system (such as LLM-SR) is important for such benchmark, but it is currently missing.
  2. Given closely related benchmarks such as LLM-SRBench and NewtonBench, I think the novelty of this paper is somewhat limited. Although the authors discuss how their benchmark differs from previous works, I believe several of the main ideas have already been explored. Nevertheless, I think the proposed benchmark is still a useful contribution.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Author's Response: Specialized Baseline & Novelty of SciLaws-Bench

Novelty and Contribution of SciLaws-Bench

Thank you for the positive assessment of the benchmark, the breadth of the evaluation, and its documentation. Beyond previous work, SciLaws-Bench contributes open research problems with real data, two complementary settings, a scientific validity evaluation, and findings on fit versus validity, memorization, and the verification bottleneck that are new. We give the full discussion in the General Response: Contributions and New Findings of SciLaws-Bench.

Results of the Specialized Baseline (LLM-SR)

Your suggestion is very constructive. We added LLM-SR [1] as a specialized baseline and ran it with three base models, GPT-5.4-mini, GPT-5-mini, and GPT-6, on all 118 REAL tasks with its iterative proposal loop, multi-island program buffer, and numerical coefficient fitting. Multi-group tasks follow the same protocol as every other method, and all laws are scored with the same numerical evaluator and validity judge as the main table.

Base modelMethodS_NS_V
GPT-5.4-miniReAct37.1772.45
GPT-5.4-miniLLM-SR40.5972.41
GPT-5-miniReAct40.6977.03
GPT-5-miniLLM-SR38.5571.90
GPT-6ReAct51.3382.91
GPT-6LLM-SR40.9883.83

The results suggest that the specialized framework, to some extent, limits what a stronger base model can do. With LLM-SR, GPT-6 and GPT-5.4-mini reach almost the same S_N (40.98 and 40.59), while with ReAct the same two models are 14 points apart (51.33 and 37.17). In LLM-SR the model only proposes equation skeletons inside a fixed search loop, whereas in ReAct it can inspect the data and run its own analyses, so the capability of a stronger model is not fully used by the fixed loop.

We will add this table and the adaptation details to the paper.

[1] Shojaee et al. LLM-SR: Scientific equation discovery via programming with large language models. ICLR, 2025.

I thank the authors for their response. I will keep the same positive score.

We are happy to see that your concerns are fully addressed. Thank you for your efforts devoted to the review and discussion!

Reviewer_x836OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on the question of how to evaluate the ability of LLMs to discover scientific laws. This leads to the creation of a benchmark for scientific equation discovery (SED), named SCILAWS-BENCH. There are 2 variants of the benchmark: SCILAWS-REAL, which includes datasets and ground-truth equations from real-world sources, and SCILAWS-PARALLEL, which is derived from synthesized variants of published equations.

Findings from running evaluations on SCILAWS-BENCH are: (1) predictive fit ≠ scientific validity; (2) LLMs struggle to discover novel equation structures; and (3) candidate selection is a bottleneck in the process.

Critical strengths
  • The presentation is clear and easy to follow.
  • The benchmarks are created from a moderate number of research papers across 6 domains. The process of collecting and synthesizing the benchmark is sound.
  • The finding that good equations are presented in intermediate candidate laws within trajectories but often not selected by small models is interesting.
Critical weaknesses
  • There are several prior SED benchmarks that have similar characteristics, such as perturbing real-world physical equations to avoid the memorization issue [1, 2] or using synthetic equations with grounded structure [1]. There is a lack of detailed discussion of SCILAWS-BENCH's advantages or differences compared to prior benchmarks.
  • Several findings are not novel. For example, [1] already shows that LLMs can accelerate the discovery process by reciting memorized equations, and numerical metrics can miss the interpretability of discovered equations.
Questions for authors

Please address the above weaknesses and the following question:

  1. How do the different reasoning efforts affect the discovery capability or search styles of thinking models?
7 comments
AuthorsOpenReview ↗

Clarification request for references

Dear Reviewer x836,

Thank you for your thoughtful review. The reference list for [1] and [2] does not appear in the review text. Could you please share their titles or links, so that we can discuss the differences from these works precisely?

Best regards, The Authors

Dear Authors,

Apologies for the omission in the original review. Here are the intended references:

[1] Shojaee, P., Nguyen, N.-H., Meidani, K., Farimani, A.B., Doan, K.D. and Reddy, C.K., 2025. LLM-SRBench: A new benchmark for scientific equation discovery with large language models. arXiv preprint arXiv:2504.10415.

[2] Zheng, T., Tam, K.K., Nguyen, N.H.K., Xu, B., Wang, Z., Cheng, J., Tsang, H.T., Wang, W., Bai, J., Fang, T., Song, Y., Wong, G.Y. and See, S. (2025) NewtonBench: Benchmarking generalizable scientific law discovery in LLM agents. arXiv:2510.07172.

Best regards,

AuthorsOpenReview ↗

Author's Response: Novelty of the Task Setting and Findings

Thank you for the careful review and clarification. We are glad that you find the benchmark construction sound and the candidate-selection finding interesting.

We would like to emphasize what is new in the task setting of SciLaws-Bench:

  • open research problems anchored in real data;
  • paired REAL and PARALLEL settings, in which simulators calibrated to real data allow active experimentation;
  • scientific validity evaluation with expert-verified rubrics, and a cold-recall audit for memorization.

This new evaluation is what makes our findings possible, since earlier work could not observe or quantify them: the gap between fit and scientific validity, the effect of memorization on recovering new structure, and the verification bottleneck in candidate selection.

Please see the General Response: Contributions and New Findings of SciLaws-Bench for details.

Reviewer_ZtQaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SciLaws-Bench contains 118 scientific problems across six domains, linking source papers, data, executable reference laws, and scientific-validity rubrics. The REAL setting evaluates prediction on held-out scientific data separately from scientific validity. The PARALLEL setting asks agents to recover a synthesized hidden equation through active queries to a simulator calibrated to source data. Experiments with fourteen models examine predictive fit, scientific validity, formula recall, and candidate selection. The paper finds that these capabilities are not interchangeable and that agents often fail to select their strongest available candidates.

Critical strengths
  1. The benchmark addresses an important distinction between fitting observations and discovering a scientifically valid law. The inclusion of source-grounded constraints and multi-group problems makes it more informative than evaluating recovery of isolated textbook equations.

  2. The evaluation is detailed and reasonably transparent. It covers fourteen models and provides judge prompts, blinded expert validation, cold-recall controls, and task-bootstrap intervals. The worked examples also make the task and scoring protocols easier to understand.

  3. The analysis provides useful insights into scientific agents. In particular, separating predictive fit, validity, and structural recovery reveals differences that a single aggregate score would miss, while the candidate analysis identifies potentially useful room for improving selection.

Critical weaknesses
  1. It is unclear whether the added structural terms are identifiable under the experimental budget. Noise is calibrated against the complete hidden law, so a high overall signal-to-noise ratio need not make a small added term recoverable. A comparison against the best refitted base law would help calibrate task difficulty.

  2. The simulator's debiasing procedure needs validation. Subtracting a local residual mean at source points and then sampling from different neighborhoods does not guarantee zero conditional noise mean at query points (Appendix A.3). Measuring the remaining bias would clarify whether observations are centered on the intended hidden law.

  3. The selection gap does not directly measure an improvement available to the agent. The oracle selects using held-out test performance, while the agent sees training data only. A training-only validation or complexity-based selector is needed to distinguish a weak selection strategy from an unavoidable information gap (Section 3.4).

  4. Most task packages cannot yet be independently inspected in full. The anonymous release includes complete data and simulators for only four of 118 tasks. The hosting limitation is understandable, but an alternative anonymous archive would make the benchmark's construction and reported results substantially easier to assess.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Author's Response 1/4: Complete Dataset Anonymous Archive

We sincerely thank the reviewer for the careful reading and the positive assessment of our work. We are glad that the reviewer recognizes our work on all fronts: the problem we study, the design of our evaluation, and the insights from our analysis.

We now provide the complete data and simulators of all 118 tasks in an anonymous Hugging Face repository: https://huggingface.co/datasets/scilawbench-anon/SciLaws-Bench. We will make the full release public with the camera-ready version.

Below we discuss the three weaknesses in turn. We will incorporate the tables and details below into the paper.

AuthorsOpenReview ↗

Author's Response 2/4: Identifiability of the added structure

We thank the reviewer for this insightful point. A high overall signal-to-noise ratio does not establish that a small added term is recoverable, so we ran the comparison that the reviewer suggests, against the best refitted base law. The result is clear: in all 118 tasks, the added structure is identifiable within the experimental budget.

Comparison against the best refitted base law

For a query design (points, groups, repeats) we compute

λ = min_θ Σ_i [φ(F(x_i)) − φ(f_base(x_i; θ))]² / σ²(x_i),

where f_base is the published base family with all its constants refitted and σ(x) is the noise SD. λ is the evidence that the data carry for the hidden law over the best refitted base law, and √λ is the separation between the two in units of the noise. A likelihood-ratio test at α = 0.05 reaches 90% power when λ = (1.96 + 1.28)² ≈ 10.5. We use a stricter criterion and call a task identifiable when λ ≥ 25 at its row budget, which corresponds to a 5σ separation. The design is the set of inputs that the 14 agents actually queried, pooled and scaled to the budget, so the criterion does not assume an optimal experimenter. All 118 tasks in our dataset are identifiable (smallest λ = 30.8, median 2,778): within the budget, the data separate the hidden law from the best refitted base law by more than 5σ in every task, and by about 50σ in a typical task.

Task difficulty calibration

The comparison establishes that every task is identifiable, but the evidence available at the budget does not calibrate how hard a task is. Across tasks, λ at the budget is only weakly related to recovery (rank correlation 0.16 with the number of models that recover the full law): once a task is identifiable, its difficulty is not set by the amount of evidence that the simulator can provide.

What matters is the evidence in the data that an agent collects itself. We applied the same comparison to the agent runs, computing λ on the queries that each run actually made instead of on the full budget. The table below groups the runs by this value and reports, for each group, the share of runs that recover the full law (S_S = 1) and the share that reach S_S ≥ 0.75. Recovery rises with the evidence in the agent's own data, from 0.8% to 28.2%. Identifiability is therefore not what limits the agents: the median run uses 2.8% of its row budget, and the runs in the first row a median of 0.9%, although these tasks are identifiable at full budget. Deciding how much to sample and where to spend the budget is itself an ability that the PARALLEL setting is designed to test.

Evidence in the agent's own data (λ)Full recoveryS_S ≥ 0.75
< 10.50.8%3.5%
10.5–257.1%16.7%
25–1007.3%16.7%
100–100018.5%31.5%
> 100028.2%46.3%
AuthorsOpenReview ↗

Author's Response 3/4: Conditional bias at query points

We thank the reviewer for this careful observation. Our statement of zero-mean noise in Appendix A.3 is not precise, and we will tighten it in the revision. The procedure centres the noise overall rather than locally: averaged over all queries the noise has zero mean, but at an individual query point x a bias can remain, equal to the mean of the centred residuals in the neighbourhood from which the noise is sampled. Subtracting this neighbourhood mean at query time makes the noise zero-mean at every query point without changing its variance. We will include this correction in the next version of the code.

The remaining local bias is small and does not affect our conclusions. As the reviewer suggests, we measured it, computing the bias exactly at every query the agents made (518,056 observations).

Bias relative to single-observation noise SDPer observationPer task (median)
Median |b|/σ0.130.14
90th percentile0.540.40
Signed mean b/σ0.020.00
  • It is small relative to the noise. The bias is typically 0.13 of the noise SD of a single observation, and 2.6% of the SD of the hidden signal.
  • It has no preferred direction. The signed mean is 0.02 noise SD, so the overall bias is also small and the observations are not shifted away from the hidden law.
  • It does not change which law the data support. When both the hidden family and the base family are refitted to the biased mean, all 118 tasks remain identifiable at budget (W1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Contributions and New Findings of SciLaws-Bench

We thank all reviewers for their constructive comments and for recognizing the value of this work. Scientific law discovery with LLMs is not a new problem, and benchmarks such as LLM-SRBench [1] and NewtonBench [2] have made important progress on it. It remains unclear how models perform on real scientific data from problems that scientists are still studying, and whether the laws they propose are scientifically valid. SciLaws-Bench is designed to answer these questions, and we summarize its contributions below.

1. Real and broad scientific tasks

  • Open research problems. We select problems that scientists are still studying, not textbook laws, and the published law is a baseline that a proposed law can outperform. Our memorization audit confirms that these laws are not over-exposed in training data: no model can recall the published law for 39% of the problems, and only 9% are recalled by all models (Appendix E).
  • Real data as the anchor. We collected real data records from published research, about 8M points from 381 papers, where earlier benchmarks generate data from the target equation. Proposed laws are evaluated on these records, and the noise in our simulators is estimated from them.
  • Broad coverage. The collection has 118 distinct problems in six disciplines. In 52 of them one law must hold across groups such as materials, species, or countries.

2. Novel and practical task design

  • Two complementary settings for each problem. REAL fixes the existing data and leaves the law open, so models can go beyond what is published. PARALLEL fixes a hidden law and lets models query a simulator, so recovery can be checked against a known target.
  • Realism together with active experimentation. As [2] notes, constructed laws can "lack correspondence to real-world phenomena". Our PARALLEL setting addresses this as far as possible: each hidden law extends a published law with additional structure, and its simulator is calibrated to the real data. Models can therefore design and run their own experiments in an environment close to the real scientific problem, and we score recovery of the published form and of the added structure separately.
  • Scientific validity evaluation. Earlier benchmarks check only how well an equation fits the data or matches the target. For every task we summarize validity rubrics from its source literature, drafted by agents and verified by human experts. A code agent then tests each proposed law against these rubrics, and its verdicts agree with the majority labels of five domain experts (Cohen's κ = 0.82, Appendix D).

3. New findings and analysis

  • Fit versus scientific validity. Earlier work also observes that numerical metrics can miss part of an equation's quality, which supports our finding, but without a validity evaluation this could not be analyzed quantitatively or at scale. Our validity scores make this analysis possible across all tasks and models. Laws that fit the data well are often not scientifically valid: within the same task, ranking models by fit agrees with ranking them by validity in only 53.6% of model pairs, close to chance (Fig. 4).
  • A quantitative analysis of memorization. We design a recall audit that measures, for every task and model, whether the model can reproduce the published law before seeing any data, and we relate this recall to what is recovered. Recall helps models reproduce the published form but does not help them find the new structure, and on problems they cannot recall, models more often propose laws that exceed the published reference (Fig. 5).
  • Search styles and the verification bottleneck. We analyze the trajectories and find distinct search styles: some models screen many candidates in a few turns, and others refine a few candidates over many turns. Candidate pools often contain laws that score better on held-out data than the one a model submits, which Reviewer x836 lists among the strengths (Section 3.4). The selectors added in our response to Reviewer ZtQa (W3) show that this gap is partly achievable without test data: on the same GPT-5.4-mini candidates, GPT-6 as the selector scores 4.9 points higher in fit and 7.1 points higher in validity than GPT-5.4-mini's own selection. The bottleneck is therefore not only proposing laws but also verifying them.

We believe our dataset and these findings can help the design of future scientific agent systems.

[1] Shojaee et al. LLM-SRBench: A new benchmark for scientific equation discovery with large language models. arXiv:2504.10415, 2025.

[2] Zheng et al. NewtonBench: Benchmarking generalizable scientific law discovery in LLM agents. arXiv:2510.07172, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.