- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SciLaws-Bench, a benchmark for evaluating whether LLMs can discover scientific laws from data. It consists of 118 problems across six scientific disciplines, collected from published research. The benchmark has two settings: SciLaws-Real, where models propose equations that fit real scientific data while satisfying scientific constraints, and SciLaws-Parallel, where models conduct experiments in simulated environments to recover hidden equations derived from published laws.
The authors evaluate many different and frontier LLMs using a common agent framework. They find that good predictive performance does not necessarily imply scientific validity, and that models often recover familiar equations but struggle to discover new mathematical structure. They also find that models frequently generate better candidate equations than the ones they ultimately select, suggesting that evaluating and selecting hypotheses remains an important bottleneck in scientific discovery.
Critical strengths
- The benchmark and its data seem valuable even to the frontier research.
- The evaluations span many different models, even including Opus 5.5. Moreover, the provided insights are interesting.
- The evaluations are rigorous and well documented.
Critical weaknesses
- While frontier models are very important to the community, I believe evaluating at least one specialized system (such as LLM-SR) is important for such benchmark, but it is currently missing.
- Given closely related benchmarks such as LLM-SRBench and NewtonBench, I think the novelty of this paper is somewhat limited. Although the authors discuss how their benchmark differs from previous works, I believe several of the main ideas have already been explored. Nevertheless, I think the proposed benchmark is still a useful contribution.
Questions for authors
None
3 comments
I thank the authors for their response. I will keep the same positive score.
We are happy to see that your concerns are fully addressed. Thank you for your efforts devoted to the review and discussion!
Author's Response: Specialized Baseline & Novelty of SciLaws-Bench
Novelty and Contribution of SciLaws-Bench
Thank you for the positive assessment of the benchmark, the breadth of the evaluation, and its documentation. Beyond previous work, SciLaws-Bench contributes open research problems with real data, two complementary settings, a scientific validity evaluation, and findings on fit versus validity, memorization, and the verification bottleneck that are new. We give the full discussion in the General Response: Contributions and New Findings of SciLaws-Bench.
Results of the Specialized Baseline (LLM-SR)
Your suggestion is very constructive. We added LLM-SR [1] as a specialized baseline and ran it with three base models, GPT-5.4-mini, GPT-5-mini, and GPT-6, on all 118 REAL tasks with its iterative proposal loop, multi-island program buffer, and numerical coefficient fitting. Multi-group tasks follow the same protocol as every other method, and all laws are scored with the same numerical evaluator and validity judge as the main table.
The results suggest that the specialized framework, to some extent, limits what a stronger base model can do. With LLM-SR, GPT-6 and GPT-5.4-mini reach almost the same S_N (40.98 and 40.59), while with ReAct the same two models are 14 points apart (51.33 and 37.17). In LLM-SR the model only proposes equation skeletons inside a fixed search loop, whereas in ReAct it can inspect the data and run its own analyses, so the capability of a stronger model is not fully used by the fixed loop.
We will add this table and the adaptation details to the paper.
[1] Shojaee et al. LLM-SR: Scientific equation discovery via programming with large language models. ICLR, 2025.