- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes SCOPE, a multi-agent framework for scientific discovery, in which multiple AI scientists collaboratively discover scientific laws through an iterative process of proposing scientific hypotheses, searching for mathematical equations, actively designing experiments, analyzing evidence, and refining hypotheses.
Critical strengths
-
The paper is well-written and clearly organized, making it easy to follow.
-
The paper presents a systematic framework that integrates hypothesis generation, equation search, experimental design, and evidence-based refinement into a unified scientific discovery pipeline.
-
The experimental evaluation is comprehensive, covering a diverse set of benchmarks and baseline methods, with clearly presented results.
Critical weaknesses
1. Concerns about Active Experimental Design
A key challenge in equation discovery is identifying mathematical expressions from limited and noisy data, where many candidate equations may fit equally well. SCOPE can actively acquire new experimental data, which substantially reduces this difficulty. However, I consider the use of active experimental design in equation discovery to be somewhat of a "shortcut," as it does not truly address the fundamental challenge of discovering equations from limited and noisy data. It remains unclear whether SCOPE’s improvements come from better equation search or more informative data from active experiments. Although the authors control the amount of data, equal data quantity does not imply equal information content. SCOPE actively selects experimental conditions, while baselines such as PySR and LLM-SR rely on randomly generated data. I strongly suggest evaluating these baselines using the same actively selected data as SCOPE to isolate the contribution of the search method.
2. Fairness of Search Budgets Across Multiple Agents
SCOPE employs multiple agents that independently search for equations over several rounds, whereas most baselines use a single search process. Even with the same limit on LLM calls, the actual computational and search budgets may differ. Therefore, the performance gains could partly result from greater search effort. A more direct comparison would be to evaluate all methods under the same total token budget.
3. Reliability of the ClosureBench Dataset
According to the paper, ClosureBench's target equations are generated and modified by LLMs, followed by manual inspection and numerical validation. However, these checks establish physical and numerical plausibility rather than ruling out data contamination. If the target equations resemble expressions encountered during pretraining, models may simply recall existing knowledge instead of discovering new laws from data. Since the complete training data of the models used (e.g., GPT-5.4-mini, Kimi K2.6, and Qwen3.6-35B) are not publicly available, how can the authors rule out potential leakage? I strongly recommend conducting additional experiments with fully open-source models (not merely open-weight models), such as the OLMo series, whose training data are publicly disclosed. More importantly, even if a benchmark was free from contamination when initially released, this cannot be guaranteed for future models. This is a broader concern affecting many recent LLM-based symbolic regression studies, rather than an issue specific to this paper.
Questions for authors
See the Weaknesses.
2 comments
Author Response to Reviewer KX2J (2/2)
📝 W2 Fairness of Search Budgets in Multi-Agent Systems
Thank you for your feedback. We understand your concern about the fairness of the search budgets. We use a common limit of 1,000 LLM calls for LLM-based methods and a limit of 30 simulation experiments for interactive methods. The 1,000-call limit was adopted to remain consistent with the evaluation protocols of existing symbolic regression methods such as LLM-SR and SR-Scientist, allowing comparisons under the same call-budget rules rather than introducing a new limit specifically for SCOPE.
To assess computational-budget fairness more directly and comprehensively, we additionally report each method’s actual LLM calls and token consumption. As shown in Table R3.3, SCOPE achieves 41.67% ID and 25.00% OOD Acc@1% using 4.70M tokens per task on average. It outperforms Reflexion-Exp, LLM-SR, and SR-Scientist with fewer tokens, and substantially surpasses ReAct in accuracy despite higher token usage, demonstrating a favorable trade-off between computational cost and discovery performance.
Table R3.3: Accuracy, LLM calls, and total token consumption by method.
| Method | ID Acc@1% (%) ↑ | OOD Acc@1% (%) ↑ | LLM Calls / Task | Total Tokens / Task (M) |
|---|---|---|---|---|
| SCOPE | 41.67 | 25.00 | 109.5 | 4.70 |
| ReAct | 26.67 | 19.44 | 95.4 | 3.09 |
| Reflexion-Exp | 30.56 | 17.78 | 133.1 | 5.58 |
| LLM-SR | 21.11 | 15.56 | 1000 | 8.17 |
| SR-Scientist | 26.11 | 18.89 | 682.7 | 17.06 |
📝 W3 Clarification on Data Contamination and Benchmark Reliability
Thank you for your feedback. We understand your concern about data leakage, which we also considered carefully when constructing ClosureBench.
(1) Dataset construction. To ensure that ClosureBench is free from LLM memorization effects, we drew on and cited the construction approach used for the LSR-Synth dataset in LLM-SRBench[1] when designing the closure relations. This is currently the most widely used dataset for LLM-based symbolic regression methods, and its construction approach is widely regarded in the field as effective in resisting LLM memorization.
(2) Additional comparison. To further demonstrate the absence of LLM memorization effects in ClosureBench, we conducted an additional comparison. Table R3.4 presents the convergence behavior and final performance of LLM-SR, an established LLM-based method, on SRBench-Feynman and ClosureBench using fixed datasets. Following the reviewer’s suggestion, we evaluated LLM-SR with the open-source OLMo-3.1-32B model and with Qwen3.6-35B, the backbone used in the main experiments.
On Feynman, both backbones exhibits convergence patterns characteristic of memorization, with accuracy rising sharply early in the search. After only 32 calls, Acc@1% reaches 62.64% with Qwen3.6-35B and 56.32% with OLMo-3.1-32B, suggesting data leakage in the Feynman for both backbones. In contrast, performance on ClosureBench improves gradually under both backbones, with substantial room for further improvement.
The main experiments in our text likewise show substantial room for improvement for all advanced methods evaluated on ClosureBench, including SCOPE. None exhibits the same rapid convergence to high performance observed on Feynman. This comparison indicates that ClosureBench does not exhibit the characteristic signs of data leakage.
(3) Evaluation with an open-source model. To further address the reviewer’s concern, we additionally evaluated SCOPE using open-source model OLMo-3.1-32B-Instruct (Table R3.5), achieving 35.56%/26.11% ID/OOD Acc@1%, with MI, EF, and EE of 0.357, 0.628, and 18.7, respectively. Its overall performance is comparable to that obtained with the three backbones reported in the submitted manuscript.
The reviewer also raises an important broader issue: even if a benchmark is free of contamination when first released, this cannot be guaranteed for future models. We consider this a worthwhile direction for future research.
Table R3.4: LLM-SR convergence on Feynman and ClosureBench (ID Acc@1%).
| Backbone | Dataset | 32 calls | 100 calls | 300 calls | 1000 calls | Final SA |
|---|---|---|---|---|---|---|
| Qwen3.6-35B | Feynman | 62.64 | 76.72 | 88.22 | 91.38 | 83.62 |
| Qwen3.6-35B | ClosureBench | 8.89 | 15.00 | 18.33 | 21.11 | 4.44 |
| OLMo-3.1-32B | Feynman | 56.32 | 68.39 | 75.29 | 83.91 | 73.28 |
| OLMo-3.1-32B | ClosureBench | 6.67 | 13.89 | 15.56 | 17.78 | 3.89 |
Table R3.5: SCOPE performance on ClosureBench across backbones.
| Backbone | ID Acc@1% (%) ↑ | OOD Acc@1% (%) ↑ | MI ↑ | EF ↑ | EE ↑ |
|---|---|---|---|---|---|
| Qwen3.6-35B | 41.67 | 25.00 | 0.378 | 0.573 | 17.4 |
| GPT-5.4-mini | 33.89 | 32.78 | 0.371 | 0.676 | 19.6 |
| Kimi K2.6 | 39.44 | 38.33 | 0.474 | 0.664 | 18.9 |
| OLMo-3.1-32B-Instruct | 35.56 | 26.11 | 0.357 | 0.628 | 18.7 |
[1] LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. ICML 2025 Oral.
Author Response to Reviewer KX2J (1/2)
We thank the reviewer for recognizing the clarity of our presentation, the systematic design of SCOPE, and the breadth of our evaluation. We appreciate the constructive suggestions on experimental controls, budget fairness, and potential data contamination, which we address below.
📝 W1 Clarifications on Active Experimentation
(1) Evaluating method performance with original and new experimental data. Thank you for this suggestion. We understand the concern that introducing active experimentation makes it difficult to compare the methods themselves fairly. We therefore added a comparison using the same data (Table R3.1): we reran equation search with PySR, LLM-SR, and SR-Scientist on the original data and data actively collected by SCOPE and compared their final results. With the data aligned, SCOPE retains a clear performance advantage, indicating that its gains come not only from active experimentation but also from its framework design.
(2) The role and effectiveness of active experimentation. As shown in Table R3.1, compared with their results on the original data, PySR, LLM-SR, and SR-Scientist improve their ID Acc@1% by 2.78, 1.67, and 2.22 percentage points, respectively, and their OOD Acc@1% by 1.67, 1.66, and 1.11 percentage points. These improvements indicate that active experimentation can improve performance and is a useful component of automated scientific discovery. In recent years, equation discovery has increasingly developed toward a more intelligent scientific discovery process, in which active experimentation plays an important role because scientists do not rely solely on a fixed dataset. Accordingly, equation discovery through active experimentation has been explored in recent work at leading conferences [1, 2]. Our comparison with conventional equation-discovery methods was intended to examine the benefit of active experimentation. In our main experiments, we also compared SCOPE with two agents that support experimental design and active experimentation: ReAct and Reflexion-Exp.
(3) Comparisons with Latest Active Experimentation Methods. We additionally compare SCOPE with LLM-AutoSciLab [2] (NeurIPS 2026), the most recent accepted work on equation discovery through active experimentation. This comparison further assesses SCOPE’s competitiveness among methods that support active experimentation. Because LLM-AutoSciLab’s original experimental interface differs from ClosureBench’s, we adapted it to the OpenFOAM environment to obtain observations through admissible experimental conditions while preserving its core algorithm. Since LLM-AutoSciLab focuses solely on equation discovery and does not support scientific report generation, we compare only ID/OOD equation accuracy, excluding report metrics such as EF and MI.
As shown in Table R3.2, SCOPE outperforms LLM-AutoSciLab on all four accuracy metrics under both backbones. Meanwhile, LLM-AutoSciLab shows mixed results relative to Reflexion-Exp. In contrast, SCOPE’s structured conjectures, question-driven experiments, and evidence exchange among multiple scientist agents make it better suited to continuously revising candidate mechanisms in this constrained experimental setting. These comparisons indicate that SCOPE also offers a competitive advantage over methods that support active experimentation.
[1] NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents. ICLR 2026.
[2] LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs. NeurIPS 2026.
Table R3.1: Performance comparison using SCOPE’s actively collected data.
·Here, SCOPE Data denotes the union of the original data and the data collected through SCOPE's active experimentation.
Table R3.2: Comparison of scientific discovery methods with active experimentation under two backbones.