- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether language models can derive scientific mechanisms from experiments rather than simply pattern-fit from observations. The authors argue that empirical fitting and general deduction can produce similar answers, and to test this, they use three LLMs on simulated experiments on 10 areas of physics. For each topic, they create three different versions that have similar experiments but are operating under different mechanisms. In order to separate numerical accuracy from real deduction authors define six dimensions to evaluate the models’ conclusions. The authors state that models can produce findings beyond fitting. Through the two case studies, stochastic motion and lattice invariants, the authors find that models can identify useful relationships and derive mathematical conclusions, but often fail to explain the complete mechanisms behind the observations.
Critical strengths
-
Comparing environments with the same experimental interfaces but different mechanisms provides a useful way to examine whether models distinguish competing explanations rather than merely reproduce superficial mathematical relationships. This addresses an important limitation of scientific-agent evaluations based primarily on prediction accuracy or task completion.
-
The stochastic-motion andlattice case studies distinguish observations from interventions and understanding
Critical weaknesses
-
The paper has insufficient validation of the evaluation framework. The scoring framework is a central contribution but its reliability was not established. The authors state that the six-dimensional claim scores are assigned retrospectively with AI assistance, without blinded expert assessment or inter-rater validation (Sec. 2.2). Several dimensions involve subjective judgment that evaluators may interpret differently. Without independent validation it is unclear whether the scores measure the intended aspects of reasoning.
-
The paper includes only one primary investigation per model and setting. In Appendix A some trajectories even contain incomplete rounds, making comparisons less controlled. Repeated trials are necessary to assess the stability of the findings.
-
There is limited evidence for generalizable discovery capabilities. The experiments lack controlled baselines such as nonadaptive and reasoning-only baselines. Many conclusions also involve established physical or mathematical relationships, making it difficult to distinguish actual scientific reasoning from prior knowledge. For example, identifying properties of stochastic motion may reflect existing knowledge recall rather than learning new relationships.
-
The paper provides limited evidence for generalizable scientific-discovery capabilities. Although the paper covers 30 simulated settings, the two detailed cases are retrospectively selected. They provide qualitative insights buㄴt the main analysis emphasizes two selected examples. Without repeated experiments or validated scores it is difficult to assess how consistently models achieve meaningful findings.
Questions for authors
-
Can the authors provide results with expert annotations or inter-rater agreement statistics for the six scoring dimensions? This would substantially strengthen the reliability of the central evaluation framework. I feel that this is essential to prove the reliability of the framework.
-
How consistent are the results across repeated testing? One trajectory per model and setting makes it difficult to assess whether the findings are robust and representative.
-
Can the authors compare adaptive investigations against nonadaptive or reasoning-only baselines? Such a comparison would clarify how much of the observed success arises from experimentation rather than existing scientific knowledge in the LLMs