acceptodds
Under review as a conference paper at ICLR 2027

What Can Large Language Models Discover from Experiments? Scientific Claims and Discovery Value across Multiple Worlds

Abstract

Large language models are increasingly used as scientific agents, yet successful prediction alone cannot establish whether they infer mechanisms or merely fit observations. We investigate this distinction through a controlled multiworld framework for mechanism identification, active experimental discrimination, and conditional deduction beyond observed data. The environment spans ten physics topics and thirty synthetic worlds: worlds within each topic share experimental interfaces but differ in hidden generative rules. Models select experiments, commit to predictions before receiving feedback, and state the assumptions and limits of their conclusions. We analyze 90 final answers from GPT-6, Claude, and Gemini together with their experimental trajectories. Our evaluation separates empirical, mechanistic, and deductive evidence along six claim-level dimensions and relates synthetic rules to established physics, distinguishing mechanistic correspondences, effective approximations, and counterfactual constructions. The results reveal meaningful but bounded achievements. In stochastic motion, calibrated temporal statistics and force interventions constrain hidden dynamics without uniquely identifying them. In lattice systems, closed-path invariants support a complete four-class classification under explicit assumptions, although the apparatus prepares only two classes. Across models, valid local findings coexist with unsupported or falsified stronger explanations. These results support within-task reconstruction and conditional mathematical discovery, but do not establish reliable autonomous discovery or previously unknown physical laws. Evaluating machine scientific discovery therefore requires jointly examining experimental discrimination, generalizable implications, and validity boundaries rather than accuracy or plausibility alone.

Primary area:
datasets and benchmarks
Keywords:
large language models, scientific discovery, active experimentation, counterfactual evaluation, mechanism identification, scientific reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_e29wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether language models can derive scientific mechanisms from experiments rather than simply pattern-fit from observations. The authors argue that empirical fitting and general deduction can produce similar answers, and to test this, they use three LLMs on simulated experiments on 10 areas of physics. For each topic, they create three different versions that have similar experiments but are operating under different mechanisms. In order to separate numerical accuracy from real deduction authors define six dimensions to evaluate the models’ conclusions. The authors state that models can produce findings beyond fitting. Through the two case studies, stochastic motion and lattice invariants, the authors find that models can identify useful relationships and derive mathematical conclusions, but often fail to explain the complete mechanisms behind the observations.

Critical strengths
  1. Comparing environments with the same experimental interfaces but different mechanisms provides a useful way to examine whether models distinguish competing explanations rather than merely reproduce superficial mathematical relationships. This addresses an important limitation of scientific-agent evaluations based primarily on prediction accuracy or task completion.

  2. The stochastic-motion andlattice case studies distinguish observations from interventions and understanding

Critical weaknesses
  1. The paper has insufficient validation of the evaluation framework. The scoring framework is a central contribution but its reliability was not established. The authors state that the six-dimensional claim scores are assigned retrospectively with AI assistance, without blinded expert assessment or inter-rater validation (Sec. 2.2). Several dimensions involve subjective judgment that evaluators may interpret differently. Without independent validation it is unclear whether the scores measure the intended aspects of reasoning.

  2. The paper includes only one primary investigation per model and setting. In Appendix A some trajectories even contain incomplete rounds, making comparisons less controlled. Repeated trials are necessary to assess the stability of the findings.

  3. There is limited evidence for generalizable discovery capabilities. The experiments lack controlled baselines such as nonadaptive and reasoning-only baselines. Many conclusions also involve established physical or mathematical relationships, making it difficult to distinguish actual scientific reasoning from prior knowledge. For example, identifying properties of stochastic motion may reflect existing knowledge recall rather than learning new relationships.

  4. The paper provides limited evidence for generalizable scientific-discovery capabilities. Although the paper covers 30 simulated settings, the two detailed cases are retrospectively selected. They provide qualitative insights buㄴt the main analysis emphasizes two selected examples. Without repeated experiments or validated scores it is difficult to assess how consistently models achieve meaningful findings.

Questions for authors
  1. Can the authors provide results with expert annotations or inter-rater agreement statistics for the six scoring dimensions? This would substantially strengthen the reliability of the central evaluation framework. I feel that this is essential to prove the reliability of the framework.

  2. How consistent are the results across repeated testing? One trajectory per model and setting makes it difficult to assess whether the findings are robust and representative.

  3. Can the authors compare adaptive investigations against nonadaptive or reasoning-only baselines? Such a comparison would clarify how much of the observed success arises from experimentation rather than existing scientific knowledge in the LLMs

Reviewer_2cTxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates what scientific conclusions LLM agents can support through interaction with controlled experimental environments. It introduces thirty synthetic worlds across ten physics topics, with three worlds per topic sharing an interface but differing in their hidden generating rules. Three models independently investigate each world over four rounds, with at most two instrument requests per round. Final claims and predictions are committed before the last round of feedback.

The study analyzes ninety final answers using six claim-level dimensions: insight, concept, explanation, experimental discrimination, implication, and boundary testing. Two detailed cases examine stochastic motion with hidden memory and the classification of binary lattice configurations through closed-path invariants. The paper distinguishes supported empirical findings, conditional mathematical deductions, and stronger explanations contradicted by observations. Its principal conclusion is that the recorded agents can reconstruct useful within-task relations and conditional results, while these achievements do not establish complete mechanism identification, reliable autonomous discovery, or previously unknown physical laws.

Critical strengths
  1. The experimental design makes the distinction between observational fit and mechanistic support concrete. Worlds within a topic share an interface while differing in their hidden rules, and final-round predictions are frozen before feedback. The stochastic-motion case usefully separates temporal anticorrelation, force response, and claims about equilibrium; the lattice case distinguishes sampled observations from a conditional completeness proof. These examples provide substantive guidance on what evidence should support different types of scientific claims.

  2. The paper carefully preserves both partial achievements and failed stronger explanations. It does not interpret a successful local prediction as complete mechanism recovery, and it separates the four mathematically permitted lattice classes from the two classes actually prepared by the apparatus. The appendices document failed predictions, invalid operations, continuation provenance, and evaluator-only checks. This explicit accounting makes the case analyses more informative than a binary success judgment and helps prevent overstating the scientific significance of the results.

Critical weaknesses
  1. The contribution of experimental interaction is not sufficiently isolated. The study provides trajectories in which observations support or refute claims, but includes no matched no-feedback or fixed-experiment controls. This matters especially for the strongest deductive achievements: the P09 classification follows from explicit binary-lattice and relabeling assumptions, so a model could potentially derive much of it from the task description without interacting with the apparatus. The paper appropriately acknowledges that behavior cannot establish whether training knowledge was recalled; my concern is narrower. The current design does not determine which parts of the reported achievement depend on acquiring experimental evidence, or whether adaptive experiment selection adds value over simpler access to observations. This limits what the study establishes specifically about discovery from experiments.

  2. The central claim-level assessment is insufficiently validated. Section 2.2 and Appendix D explicitly describe retrospective, AI-assisted ratings without blinded domain-expert assessment or inter-rater validation. Although scoring anchors are provided, selecting the claim to score and separating a supported weaker statement from a failed stronger explanation require substantial judgment. Figure 6 and Table 7 also evaluate claims of different scope, making independent verification of those selections particularly important. The two main cases were selected retrospectively, and Appendix F.2 presents a proof that follows the mathematical substance of the model answer rather than reproducing it verbatim. The manuscript would benefit from an independently checked mapping between original claims, original arguments, experimental evidence, and assigned scores. This concern applies even to the paper's deliberately limited case-study conclusions; it does not depend on requiring a model leaderboard or an aggregate success rate.

Questions for authors
  1. Under the identical initial task descriptions, what can the models establish without receiving instrument feedback? A matched control for P06C and P09C would be particularly informative. Which observables, assumptions, or proof steps first appear after discriminating observations, and which can already be derived from the supplied interface? A fixed-experiment comparison would additionally clarify the contribution of adaptive selection.

  2. Could you provide the exact model-produced passages supporting the P09C completeness proofs, aligned with Proposition 1 and its proof in Appendix F.2? Please distinguish translation or notation cleanup from any mathematical steps supplied during evaluation. The same mapping for representative high-scoring and falsified claims would clarify how the scored claim was selected from each answer.

  3. Have independent domain experts checked a subset of the claim selections, proof-completeness judgments, and six-dimensional scores? If so, please report the procedure and disagreements. If not, an independent assessment of the two main cases and a sample of other worlds could materially change my confidence in the evaluation.

  4. Please specify the exact Claude and Gemini model identifiers, collection dates, and inference settings, together with the corresponding settings for the GPT-6 continuations. How do the qualitative cross-model profiles change when restricted to complete, non-continued trajectories? This is relevant because five GPT-6 trajectories were continued, while Gemini's missing rounds were not rerun, including two missing intermediate rounds in P09C.

Reviewer_jw9pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs can obtain scientifically meaningful knowledge from experiments or rather tend to reproduce familiar explanation instead. The authors introduce a framework consisting of 30 synthetic physical environments across 10 topics, in which GPT-6, Claude, and Gemini conduct experiments and formulate scientific claims. Within each topic, the three environments share the same experimental interface but differ in their hidden mechanisms. Two detailed case studies out of these 30 concern stochastic motion and the classification of lattice configurations. The authors find that LLMs can often produce useful measurements and experiments but often propose a bigger explanation that fails when tested.

Critical strengths

The problem of distinguishing genuine inference from pattern-matching in LLM-driven science is interesting and a worthwhile research direction.

Using multiple environments with common experimental interfaces but different hidden mechanisms is a promising approach to study how the LLM comes to its conclusions.

Critical weaknesses

The writing is very hard to follow. Key terms like “world,” “interface” are used before they are explained. The language generally is very verbose and unnecessarily complicated. For example, the introduction, which should motivate the work, includes complicated, technical sentences such as "Temporal memory, for example, can arise from active forcing or from an unobserved elastic degree of freedom in equilibrium dynamics. Local lattice records can omit global differences." or "Useful observables, cross-condition constraints, and conditional classification proofs organize testable knowledge; they do not establish human-unknown laws of nature."

The six-dimensional scoring is AI-assisted. This is concerning because judging the validity of scientific reasoning is itself the challenge addressed by the paper.

The method cannot tell whether a model’s correct results were derived from the experiments or recalled from training. Some included problems seem to be widely known material.

The two detailed cases seem to have been picked because they were interesting. The paper does not discuss how typical they are.

The paper does not specify which versions of Claude and Gemini were used.

Questions for authors

Which AI models were involved in assessing scientific claims and assigning the six-dimensional scores? What measures were taken to reduce potential biases?

Reviewer_dfW3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work studies LLMs experiment design in 30 synthetic physics environments. The work proposes 6 dimensions to evaluate scientific discoveries on, and present more detailed qualitative studies across 3 cases.

Critical strengths

Using the same experimental setup to study worlds with different hidden rules is an interesting idea.

The paper makes an effort to distinguish valid findings from unsupported claims and reports some cases where models make incorrect predictions.

Critical weaknesses

Scientific clarity. It's hard to clearly understand from the paper its intended novelty, or its contrast with prior work in this space.

Evaluation. The scoring rubric is subjective and evaluated using AI models. It's hard to trust the scores or draw meaningful conclusions from them.

Limited experiments. There is only one run per model per environment. The two main case studies are selected in retrospect.

Scientific novelty in experiments. Much of the paper describes models recovering known physics relationships or standard mathematical results. It is unclear what is actually being discovered vs memorized from the data.

Missing baselines. There is no comparison against models answering without experiments. The paper does not establish that the proposed experimental setup adds value here.

Questions for authors

How reliable are the six evaluation scores? Why was there no independent human evaluation or inter agreement analysis?

How much of the reported discoveries comes from the experiments rather than the models' existing knowledge? Could the models reach similar conclusions without interacting with the environments?

Why were the two main case studies selected retrospectively? How stable are the findings across runs?

What is the main contribution compared to existing benchmarks for scientific discovery?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.