acceptodds
Under review as a conference paper at ICLR 2027

Beyond Equation Search: Conjecture-Guided Discovery with SCOPE and ClosureBench

Abstract

Most symbolic regression methods optimize candidate equations, but improving an equation's fit does not establish whether its underlying scientific conjecture should be retained, revised, or rejected. A conjecture may admit multiple mathematical realizations, and similarly accurate equations may imply different explanations. This motivates a shift from refining individual expressions to developing and testing the conjectures that guide them. We introduce SCOPE, a framework that sustains multiple conjecture-driven research trajectories. Multiple scientist agents independently pursue conjectures through equation modeling, question formulation, and active experimentation, then exchange evidence to revise their conjectures and research directions. Their findings are consolidated into a scientific report linking models, conclusions, and evidence. Existing equation-discovery benchmarks primarily assess recovered expressions, leaving the evidence and scientific conclusions established through the investigation underexamined. We therefore introduce ClosureBench, an interactive benchmark of 60 closure-discovery tasks across five scientific domains. ClosureBench focuses on closure-discovery tasks to give equation discovery a concrete scientific purpose. Each task embeds an unknown algebraic relation in a known PDE system, where agents control experimental conditions in OpenFOAM to acquire new data for equation discovery. Beyond equation accuracy, ClosureBench proposes three metrics to evaluates the discovery process: evidence faithfulness, mechanism insight, and experimental effectiveness. Compared with diverse baselines, SCOPE achieves higher equation accuracy on both ClosureBench and real-world scientific discovery case, while producing more accurate, reliable, and effective scientific discoveries.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
symbolic regression, automated scientific discovery, bench for scientific discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KX2JOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SCOPE, a multi-agent framework for scientific discovery, in which multiple AI scientists collaboratively discover scientific laws through an iterative process of proposing scientific hypotheses, searching for mathematical equations, actively designing experiments, analyzing evidence, and refining hypotheses.

Critical strengths
  1. The paper is well-written and clearly organized, making it easy to follow.

  2. The paper presents a systematic framework that integrates hypothesis generation, equation search, experimental design, and evidence-based refinement into a unified scientific discovery pipeline.

  3. The experimental evaluation is comprehensive, covering a diverse set of benchmarks and baseline methods, with clearly presented results.

Critical weaknesses

1. Concerns about Active Experimental Design

A key challenge in equation discovery is identifying mathematical expressions from limited and noisy data, where many candidate equations may fit equally well. SCOPE can actively acquire new experimental data, which substantially reduces this difficulty. However, I consider the use of active experimental design in equation discovery to be somewhat of a "shortcut," as it does not truly address the fundamental challenge of discovering equations from limited and noisy data. It remains unclear whether SCOPE’s improvements come from better equation search or more informative data from active experiments. Although the authors control the amount of data, equal data quantity does not imply equal information content. SCOPE actively selects experimental conditions, while baselines such as PySR and LLM-SR rely on randomly generated data. I strongly suggest evaluating these baselines using the same actively selected data as SCOPE to isolate the contribution of the search method.

2. Fairness of Search Budgets Across Multiple Agents

SCOPE employs multiple agents that independently search for equations over several rounds, whereas most baselines use a single search process. Even with the same limit on LLM calls, the actual computational and search budgets may differ. Therefore, the performance gains could partly result from greater search effort. A more direct comparison would be to evaluate all methods under the same total token budget.

3. Reliability of the ClosureBench Dataset

According to the paper, ClosureBench's target equations are generated and modified by LLMs, followed by manual inspection and numerical validation. However, these checks establish physical and numerical plausibility rather than ruling out data contamination. If the target equations resemble expressions encountered during pretraining, models may simply recall existing knowledge instead of discovering new laws from data. Since the complete training data of the models used (e.g., GPT-5.4-mini, Kimi K2.6, and Qwen3.6-35B) are not publicly available, how can the authors rule out potential leakage? I strongly recommend conducting additional experiments with fully open-source models (not merely open-weight models), such as the OLMo series, whose training data are publicly disclosed. More importantly, even if a benchmark was free from contamination when initially released, this cannot be guaranteed for future models. This is a broader concern affecting many recent LLM-based symbolic regression studies, rather than an issue specific to this paper.

Questions for authors

See the Weaknesses.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer KX2J (1/2)

We thank the reviewer for recognizing the clarity of our presentation, the systematic design of SCOPE, and the breadth of our evaluation. We appreciate the constructive suggestions on experimental controls, budget fairness, and potential data contamination, which we address below.


📝 W1 Clarifications on Active Experimentation

(1) Evaluating method performance with original and new experimental data. Thank you for this suggestion. We understand the concern that introducing active experimentation makes it difficult to compare the methods themselves fairly. We therefore added a comparison using the same data (Table R3.1): we reran equation search with PySR, LLM-SR, and SR-Scientist on the original data and data actively collected by SCOPE and compared their final results. With the data aligned, SCOPE retains a clear performance advantage, indicating that its gains come not only from active experimentation but also from its framework design.

(2) The role and effectiveness of active experimentation. As shown in Table R3.1, compared with their results on the original data, PySR, LLM-SR, and SR-Scientist improve their ID Acc@1% by 2.78, 1.67, and 2.22 percentage points, respectively, and their OOD Acc@1% by 1.67, 1.66, and 1.11 percentage points. These improvements indicate that active experimentation can improve performance and is a useful component of automated scientific discovery. In recent years, equation discovery has increasingly developed toward a more intelligent scientific discovery process, in which active experimentation plays an important role because scientists do not rely solely on a fixed dataset. Accordingly, equation discovery through active experimentation has been explored in recent work at leading conferences [1, 2]. Our comparison with conventional equation-discovery methods was intended to examine the benefit of active experimentation. In our main experiments, we also compared SCOPE with two agents that support experimental design and active experimentation: ReAct and Reflexion-Exp.

(3) Comparisons with Latest Active Experimentation Methods. We additionally compare SCOPE with LLM-AutoSciLab [2] (NeurIPS 2026), the most recent accepted work on equation discovery through active experimentation. This comparison further assesses SCOPE’s competitiveness among methods that support active experimentation. Because LLM-AutoSciLab’s original experimental interface differs from ClosureBench’s, we adapted it to the OpenFOAM environment to obtain observations through admissible experimental conditions while preserving its core algorithm. Since LLM-AutoSciLab focuses solely on equation discovery and does not support scientific report generation, we compare only ID/OOD equation accuracy, excluding report metrics such as EF and MI.

As shown in Table R3.2, SCOPE outperforms LLM-AutoSciLab on all four accuracy metrics under both backbones. Meanwhile, LLM-AutoSciLab shows mixed results relative to Reflexion-Exp. In contrast, SCOPE’s structured conjectures, question-driven experiments, and evidence exchange among multiple scientist agents make it better suited to continuously revising candidate mechanisms in this constrained experimental setting. These comparisons indicate that SCOPE also offers a competitive advantage over methods that support active experimentation.

[1] NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents. ICLR 2026.
[2] LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs. NeurIPS 2026.


Table R3.1: Performance comparison using SCOPE’s actively collected data.

MethodDataID Acc@1% (%) ↑OOD Acc@1% (%) ↑
PySROriginal14.4412.22
PySRSCOPE Data17.2213.89
LLM-SR (Qwen3.6-35B)Original21.1115.56
LLM-SR (Qwen3.6-35B)SCOPE Data22.7817.22
SR-Scientist (Qwen3.6-35B)Original26.1118.89
SR-Scientist (Qwen3.6-35B)SCOPE Data28.3320.00
SCOPE (Qwen3.6-35B)SCOPE Data41.6725.00

·Here, SCOPE Data denotes the union of the original data and the data collected through SCOPE's active experimentation.

Table R3.2: Comparison of scientific discovery methods with active experimentation under two backbones.

BackboneMethodID Acc@1% (%) ↑ID [email protected]% (%) ↑OOD Acc@1% (%) ↑OOD [email protected]% (%) ↑
Qwen3.6-35B-A3BLLM-AutoSciLab (NeurIPS 2026)28.3319.4420.5613.33
Qwen3.6-35B-A3BSCOPE41.6723.8925.0017.22
Kimi K2.6LLM-AutoSciLab (NeurIPS 2026)31.6721.1130.5622.78
Kimi K2.6SCOPE39.4429.4438.3327.22
AuthorsOpenReview ↗

Author Response to Reviewer KX2J (2/2)

📝 W2 Fairness of Search Budgets in Multi-Agent Systems

Thank you for your feedback. We understand your concern about the fairness of the search budgets. We use a common limit of 1,000 LLM calls for LLM-based methods and a limit of 30 simulation experiments for interactive methods. The 1,000-call limit was adopted to remain consistent with the evaluation protocols of existing symbolic regression methods such as LLM-SR and SR-Scientist, allowing comparisons under the same call-budget rules rather than introducing a new limit specifically for SCOPE.

To assess computational-budget fairness more directly and comprehensively, we additionally report each method’s actual LLM calls and token consumption. As shown in Table R3.3, SCOPE achieves 41.67% ID and 25.00% OOD Acc@1% using 4.70M tokens per task on average. It outperforms Reflexion-Exp, LLM-SR, and SR-Scientist with fewer tokens, and substantially surpasses ReAct in accuracy despite higher token usage, demonstrating a favorable trade-off between computational cost and discovery performance.

Table R3.3: Accuracy, LLM calls, and total token consumption by method.

MethodID Acc@1% (%) ↑OOD Acc@1% (%) ↑LLM Calls / TaskTotal Tokens / Task (M)
SCOPE41.6725.00109.54.70
ReAct26.6719.4495.43.09
Reflexion-Exp30.5617.78133.15.58
LLM-SR21.1115.5610008.17
SR-Scientist26.1118.89682.717.06

📝 W3 Clarification on Data Contamination and Benchmark Reliability

Thank you for your feedback. We understand your concern about data leakage, which we also considered carefully when constructing ClosureBench.

(1) Dataset construction. To ensure that ClosureBench is free from LLM memorization effects, we drew on and cited the construction approach used for the LSR-Synth dataset in LLM-SRBench[1] when designing the closure relations. This is currently the most widely used dataset for LLM-based symbolic regression methods, and its construction approach is widely regarded in the field as effective in resisting LLM memorization.

(2) Additional comparison. To further demonstrate the absence of LLM memorization effects in ClosureBench, we conducted an additional comparison. Table R3.4 presents the convergence behavior and final performance of LLM-SR, an established LLM-based method, on SRBench-Feynman and ClosureBench using fixed datasets. Following the reviewer’s suggestion, we evaluated LLM-SR with the open-source OLMo-3.1-32B model and with Qwen3.6-35B, the backbone used in the main experiments.

On Feynman, both backbones exhibits convergence patterns characteristic of memorization, with accuracy rising sharply early in the search. After only 32 calls, Acc@1% reaches 62.64% with Qwen3.6-35B and 56.32% with OLMo-3.1-32B, suggesting data leakage in the Feynman for both backbones. In contrast, performance on ClosureBench improves gradually under both backbones, with substantial room for further improvement.

The main experiments in our text likewise show substantial room for improvement for all advanced methods evaluated on ClosureBench, including SCOPE. None exhibits the same rapid convergence to high performance observed on Feynman. This comparison indicates that ClosureBench does not exhibit the characteristic signs of data leakage.

(3) Evaluation with an open-source model. To further address the reviewer’s concern, we additionally evaluated SCOPE using open-source model OLMo-3.1-32B-Instruct (Table R3.5), achieving 35.56%/26.11% ID/OOD Acc@1%, with MI, EF, and EE of 0.357, 0.628, and 18.7, respectively. Its overall performance is comparable to that obtained with the three backbones reported in the submitted manuscript.

The reviewer also raises an important broader issue: even if a benchmark is free of contamination when first released, this cannot be guaranteed for future models. We consider this a worthwhile direction for future research.

Table R3.4: LLM-SR convergence on Feynman and ClosureBench (ID Acc@1%).

BackboneDataset32 calls100 calls300 calls1000 callsFinal SA
Qwen3.6-35BFeynman62.6476.7288.2291.3883.62
Qwen3.6-35BClosureBench8.8915.0018.3321.114.44
OLMo-3.1-32BFeynman56.3268.3975.2983.9173.28
OLMo-3.1-32BClosureBench6.6713.8915.5617.783.89

Table R3.5: SCOPE performance on ClosureBench across backbones.

BackboneID Acc@1% (%) ↑OOD Acc@1% (%) ↑MI ↑EF ↑EE ↑
Qwen3.6-35B41.6725.000.3780.57317.4
GPT-5.4-mini33.8932.780.3710.67619.6
Kimi K2.639.4438.330.4740.66418.9
OLMo-3.1-32B-Instruct35.5626.110.3570.62818.7

[1] LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. ICML 2025 Oral.

Reviewer_watvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To advance symbolic regression from automated equation discovery toward automated scientific discovery, the authors propose SCOPE and ClosureBench. SCOPE implements a scientific conjecture-oriented multi-agent parallel exploration framework, in which multiple agents propose structured hypotheses, refine them through symbolic regression, and exchange insights via round-table discussions, finally producing scientific reports that extend beyond individual mathematical formulas. Meanwhile, ClosureBench introduces a new benchmark for scientific discovery that constructs novel problems by augmenting known PDEs with additional closure terms, in which algorithms are required not only to derive the final formula but also to identify the underlying mechanisms and provide corresponding evidence.

Critical strengths

The proposed SCOPE framework combines independent research with periodic exchanges of evidence; rather than merely searching for a single formula, it generates scientific reports based on experimental findings, thereby advancing symbolic regression from the simple discovery of formulas toward the discovery of scientific knowledge. This framework performs well on the proposed ClosureBench, demonstrating not only superior evidence faithfulness, mechanism insight, and experimental effectiveness, but also an improvement in the numerical performance of the discovered formulas themselves.

Critical weaknesses

My concerns primarily center on ClosureBench.Specifically, its evaluation of two key metrics (Evidence Faithfulness and Mechanism Insight) relies on LLMs, which inevitably introduces arbitrariness and randomness. Although the authors demonstrate that LLM-based evaluations largely align with human judgments through cross-comparisons, the evaluation criteria remain subjective and ambiguous. This makes it difficult to use this benchmark for cross-model comparisons or to scale it effectively.

Moreover, even when the same mathematical formula is used, it can often yield multiple distinct yet equally plausible mechanistic explanations. The data available to a model are insufficient to distinguish between these different mechanistic interpretations, and the problem formulation itself fails to provide enough evidence for an evaluator to either endorse or rule out equivalent mechanistic explanations. This problem of indistinguishability further exacerbates the pressure on evaluation standards.

Questions for authors

Q1. Experience suggests that employing multiple distinct base models in parallel to tackle the same task often enhances diversity and leads to improvements in overall performance compared to relying on a single model. In SCOPE, could utilizing a variety of foundational models (e.g., Qwen3.6-35B + GPT-5.4-mini + Kimi K2.6) yield superior results compared to using just one?

Q2. Although the ClosureBench is designed to evaluate the ability to autonomously discover scientific knowledge, I am not sure whether it is well-aligned with this objective, since the benchmark differs significantly from the real world: In ClosureBench, the formulas to be discovered are themselves fundamental laws (i.e., axioms). This largely gives rise to the "indistinguishability" issue mentioned above (an algorithm might arrive at the same formula but offer different mechanistic explanations, with no clear way to evaluate which is superior). In contrast, in the real world, formulas discovered via symbolic regression are often projections of fundamental laws under specific conditions, rather than the underlying laws themselves. This allows for the evaluation of competing results, like through counterfactual experiments or theoretical derivation -- which is, however, not feasible within the framework of ClosureBench.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer watv (1/3)

We thank the reviewer for the positive recommendation and for recognizing SCOPE’s contribution to evidence-based scientific discovery. We also appreciate the constructive feedback on ClosureBench’s evaluation and scope, and the suggestion to explore diverse model backbones, which we address below.


📝 W1 Evaluation Metric Subjectivity and Stability: Clarifications and Additional Experiments

Thank you for your feedback. We understand your concerns about the evaluation.

To enable efficient, automated evaluation at scale, recent studies have widely adopted LLM evaluators and assessed their reliability against human judgments, including DeepResearch Bench [1], PaperBench [2], and HealthBench [3]. We follow the same validation approach while making the evaluation criteria explicit: Evidence Faithfulness (EF) checks whether report conclusions are fully supported by the investigation record; Mechanism Insight (MI) assesses correspondence with predefined, human-reviewed reference properties using full-match, partial-match, and no-match categories. The final scores are computed programmatically.

To further examine evaluation stability across models, we conducted two additional experiments.

(1) Consistency Between Cross-LLM Evaluators and Human Validation. We used different LLM evaluators to score the same set of reports, compared their EF and MI scores, and assessed agreement with human annotations. Table R2.1 reports this agreement: Mistral Medium 3.5 achieves LLM–Human agreement of 90.5% for EF and 89.3% for MI. These results indicate that both LLM evaluators show high agreement with the human labels.

(2) Consistency in the Direction of Differences Across LLM Evaluators. Table R2.2 compares the EF and MI score differences between methods under the two evaluators. SCOPE’s score differences relative to ReAct and Reflexion-Exp remain positive for both metrics under both evaluators. Thus, the direction of the relative performance differences is consistent across the two evaluators examined.

In addition, the main-experiment results reported in the submitted manuscript are averaged over three independent runs, providing some robustness to evaluation variability.

Table R2.1: LLM–Human agreement across evaluators.

Evaluation taskHuman-labeled itemsHuman–Human (Submitted manuscript)GPT-OSS-120B LLM–Human (Submitted manuscript)Mistral Medium 3.5 LLM–Human (New)
EF: Evidence support20094.5%92.0%90.5%
MI: Mechanism matching15092.7%88.0%89.3%

Table R2.2: Performance differences across evaluators.

EvaluatorEF: SCOPE − ReActEF: SCOPE − Reflexion-ExpMI: SCOPE − ReActMI: SCOPE − Reflexion-Exp
GPT-OSS-120B+0.093+0.072+0.147+0.129
Mistral Medium 3.5+0.071+0.087+0.115+0.093

[1] DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. ICLR 2026.
[2] PaperBench: Evaluating AI’s Ability to Replicate AI Research. ICML 2025.
[3] HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775, 2025.


AuthorsOpenReview ↗

Author Response to Reviewer watv (2/3)

📝 W2 How SCOPE Handles Indistinguishable Mechanistic Explanations

Thank you for this valuable feedback. We fully agree that the same mathematical equation can be compatible with multiple distinct mechanistic explanations. A major source of this ambiguity is that previous discovery approaches start from equations and interpret the best equation post hoc, leaving multiple possible mechanistic interpretations. This motivates our approach to scientific discovery guided by structured conjectures: conjectures and their proposed mechanisms guide equation discovery, so the resulting equations can be traced back to the corresponding conjectures and mechanisms.

SCOPE addresses this concern in two ways.

(1) Starting from conjectures rather than equations. Unlike methods that begin their search directly from candidate equations, SCOPE starts from a structured conjecture H=(m,v,R,P,F,e)H=(m,\mathbf v,\mathcal R,\mathcal P,\mathcal F,e), as defined in line 180 of the submitted manuscript. This conjecture includes a mechanistic explanation mm, testable predictions P\mathcal P, and candidate equation families F\mathcal F. Equation search is therefore guided by the conjecture and its proposed mechanism, and SCOPE checks whether the resulting equations are consistent with the predictions P\mathcal P. Active experiments investigate potential shortcomings of the current equation and inform conjecture refinement. The final equation developed under a conjecture thus has a traceable mechanistic interpretation.

(2) Exploring conjectures in parallel. When multiple conjectures remain plausible, each supported by a complete chain of evidence, roundtable discussions can generate new questions aimed at resolving the ambiguities. These questions are added to the question record Q\mathcal Q. Subsequent investigation selects questions from this record and designs new active experiments to distinguish between the competing conjectures, helping identify a better-supported scientific explanation. If equations developed under two conjectures are equivalent on all observed data, SCOPE considers not only predictive accuracy but also the completeness of the evidence linking each equation to its corresponding conjecture when selecting the most convincing research trajectory.


📝 Q1 Additional Experiments with Mixed LLMs

Thank you for this insightful suggestion. We conducted an additional experiment to evaluate SCOPE with multiple base models, as reported in Table R2.3. The heterogeneous combination achieves an ID Acc@1% of 38.33%, an OOD Acc@1% of 35.56%, an EF of 0.61, and an MI of 0.42.

Compared with using Qwen3.6-35B-A3B alone, the heterogeneous combination improves OOD Acc@1% from 25.00% to 35.56%, but reduces ID Acc@1% from 41.67% to 38.33%. It also does not outperform Kimi K2.6 across all metrics. These results indicate that using scientist agents with different backbones can improve some metrics, but does not necessarily yield better performance than the best single-backbone configuration.

Table R2.3: SCOPE performance with single and heterogeneous backbones.

BackboneID Acc@1% (%) ↑OOD Acc@1% (%) ↑EF ↑MI ↑
Qwen3.6-35B-A3B41.6725.000.570.38
GPT-5.4-mini33.8932.780.680.37
Kimi K2.639.4438.330.660.47
Qwen3.6-35B-A3B + GPT-5.4-mini + Kimi K2.6 (heterogeneous)38.3335.560.610.42

AuthorsOpenReview ↗

Author Response to Reviewer watv (3/3)

📝 Q2 Clarification on the Indistinguishability Issue

Thank you for this valuable question. We agree that the same mathematical relationship can admit multiple mechanistic explanations. If these explanations produce the same observations under all permitted experimental conditions, it is difficult to determine which microscopic mechanism is better supported. However, the three new metrics introduced in ClosureBench do not directly assess the quality or correctness of the final mechanistic explanation itself. Instead, they evaluate the reliability and effectiveness of the scientific discovery process as a whole.

(1) Evidence Faithfulness (EF) checks whether the report’s conclusions are fully supported by the investigation record, including their direction, scope, and degree of certainty. Even when different explanations remain indistinguishable, it is still possible to assess whether the report faithfully represents the evidence and whether it presents unverified hypotheses as established conclusions. If two explanations have equal evidential support, EF does not determine which mechanism is superior.

(2) Mechanism Insight (MI) assesses whether the report correctly identifies properties of the hidden closure, including variable dependencies, directions of influence, limiting behavior, invariances, and coupling relationships. It evaluates these properties rather than the model’s final mechanistic explanation itself. Different explanations that express exactly the same properties receive the same assessment.

(3) Experimental Effectiveness (EE) measures the improvement in predictive accuracy from the initial to the final equation, reflecting the numerical progress achieved jointly through experimentation and subsequent modeling. This assessment does not depend on whether a unique mechanism has been identified.

The metrics proposed in ClosureBench are intended to provide quantitative evaluations from multiple perspectives, rather than directly determining the best scientific explanation report. The reviewer highlights a valuable direction for future research: how to directly evaluate the correctness and distinguishability of different mechanistic explanations associated with the same equation.

Reviewer_Hem5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores how AI can move beyond fitting equations to testing scientific explanations. Its framework, SCOPE, lets several AI agents pursue different ideas, run experiments, and share what they learn. The authors also introduce ClosureBench to assess both equation accuracy and how well the evidence supports the agents’ conclusions. The results are promising, though the claims about scientific understanding need stronger support.

Critical strengths
  1. The paper addresses an important gap between fitting equations and testing scientific explanations. SCOPE connects competing conjectures with targeted experiments and evidence-based revision. I find this valuable because predictive accuracy alone does not establish whether an explanation is supported, and the framework makes that distinction central to the discovery process.

  2. ClosureBench provides a useful setting for evaluating active scientific discovery. Agents choose experimental conditions rather than directly selecting the local states they observe, which introduces a meaningful experimental constraint. The benchmark also evaluates whether reported conclusions are supported by evidence. Together with the gains across several model backbones, this makes the contribution worth considering for acceptance.

Critical weaknesses
  1. The crystallization example raises doubts about the claimed mechanistic understanding. The reference closure in Appendix C.5 decreases monotonically with crystallinity and allows a nonzero rate at zero crystallinity. Yet SCOPE’s report describes early enhancement followed by depletion and claims that an existing crystal seed is necessary. If these refer to different task variants, the paper needs to make that clear. Otherwise, this inconsistency directly undermines a central claim: that the system produces scientifically correct, evidence-supported explanations.
  2. The experiments do not clearly establish what drives SCOPE’s gains. The Single-Conjecture ablation suppresses independent exploration, so its weaker performance does not isolate the benefit of structured conjectures. A budget-matched parallel equation-search baseline would help distinguish that benefit from broader search diversity and additional computation. Without this control, I find the numerical improvements encouraging, but the evidence for the proposed framework’s central mechanism remains incomplete.
Questions for authors
  1. Clarification on the crystallization example: Does the report on pages 36–37 concern the same closure shown in Appendix C.5? The displayed closure decreases monotonically with crystallinity and generally permits a nonzero rate at zero crystallinity, whereas the report describes early enhancement and seed-dependent onset. If these are the same task, how do you reconcile these statements, and how were they scored by the mechanism evaluator?
  2. Clarification on the experimental controls: Do you have a comparison with independent parallel equation searches using the same tools and matched search and token budgets, but without structured conjectures? The Single-Conjecture ablation also restricts exploration diversity, so I am unsure how much of the gain can be attributed specifically to conjecture-guided reasoning.
  3. Clarification on evaluator validation: Were complete held-out reports evaluated by humans, including claim extraction, deduplication, and matching to reference facts, or was validation limited to preselected claims and claim–fact pairs? Also, does MI distinguish properties supplied in the public task description from those learned during the investigation? These details would affect how confidently I interpret the reported gains in scientific understanding.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hem5 (1/2)

We thank the reviewer for the thoughtful review and for recognizing the value of SCOPE’s conjecture-guided experimentation and ClosureBench’s emphasis on evidence-supported conclusions. We address the concerns about the crystallization example, experimental controls, and evaluator validation below.


📝 W1 / Q1 Clarification: Crystallization Examples from Different Tasks

Thank you for your careful reading and valuable feedback. The two passages refer to different tasks. Appendix C.5 presents fic_task_h, whereas the scientific report on pages 36–37 concerns fic_task_y. Both tasks are based on the same classical PDE system and share the same public task description, but have entirely different hidden target closures.

Specifically, the crystallinity-dependent factor in fic_task_h is (1−ψ)1.5(1-\psi)^{1.5}, which decreases monotonically with crystallinity. In contrast, the corresponding factor in fic_task_y is (1−ψ)−ln⁡(1−ψ)(1-\psi)\sqrt{-\ln(1-\psi)}, which increases at low crystallinity, peaks at ψ≈0.393\psi\approx0.393, and then decreases. Thus, the report’s description of early enhancement followed by depletion refers to fic_task_y and is consistent with its reference closure. It does not contradict the closure for fic_task_h presented in Appendix C.5.

We apologize for the confusion and will add explicit task IDs in the revised manuscript to make this distinction clear across the appendices.


📝 W2 / Q2 Contributions of Parallel Conjectures Exploration and Structured Conjectures

Thank you for this valuable suggestion. We further examine the sources of SCOPE’s gains from two perspectives.

(1) Contribution of parallel conjecture exploration. Table R1.1 presents additional experiments and their computational costs. All variants use the same initial observations, 30-call simulation budget, equation-search budget, and LLM-call limit. The Single-Conjecture variant in Figure 2 of the submitted manuscript (V1) retains roundtable discussions and copies the primary structured conjecture to the other two branches at the start of every exploration round. The second Single-Conjecture variant (V2) copies the conjecture only at the start and otherwise follows SCOPE, including roundtable discussions. The Single-Conjecture variant without roundtable discussions (V3) maintains only a single evolving conjecture. Because it lacks the early-stopping mechanism provided by roundtable discussions, it runs until its budget is exhausted. In all three Single-Conjecture variants (V1–V3), the primary structured conjecture continues to be explored independently, without intervention. Full SCOPE (V0) outperforms all three variants, suggesting that the independent, parallel development of multiple conjectures benefits equation discovery.

(2) Contribution of structured conjectures. We additionally compare SCOPE with a parallel equation-search baseline without structured conjectures (V4). In this variant, the structured conjecture HH is reduced to candidate equation families F\mathcal F, while the other memory mechanisms and tools remain unchanged. Parallel searches begin from three sets of LLM-proposed candidate equation families, with question formulation, active experimentation, and roundtable discussions based solely on the candidate equations. The equation accuracy of this parallel equation-search baseline drops to a level close to that of the Single-Conjecture variant without roundtable discussions (V3). More importantly, without maintained structured conjectures to guide the investigation, scientific reporting is limited to post hoc analysis of the best equation. The resulting reports are of poor quality, particularly in terms of Evidence Faithfulness (EF). We will provide further experimental details and analysis of these findings in the revised manuscript.

We also report LLM calls and token consumption. Some variants incur greater computational expenditure without improving performance. These results further suggest that SCOPE’s gains are not solely due to greater computational expenditure, but are closely related to its research process guided by structured conjectures.

Table R1.1: Discovery performance and LLM computational costs across variants.

MethodID Acc@1% (%) ↑OOD Acc@1% (%) ↑MI ↑EF ↑LLM Calls / TaskInput Tokens / Task (M)Output Tokens / Task (M)
V0: SCOPE41.6725.000.3780.573109.54.4520.248
V1: Single-Conjecture (copied every round; with roundtable discussions)33.8918.890.2820.51195.74.1560.240
V2: Single-Conjecture (copied only at the start; with roundtable discussions)35.5621.670.3240.527103.14.3250.256
V3: Single-Conjecture (without roundtable discussions)28.3317.780.2430.48093.33.9450.221
V4: Parallel Equation Search (without structured conjectures)30.5618.330.2160.363118.45.2380.238
AuthorsOpenReview ↗

Author Response to Reviewer Hem5 (2/2)

📝 Q3 Clarification on Evaluator Validation and Mechanism Insight (MI)

(1) Scope of human validation.

Our human validation was limited to preselected claims and claim–fact pairs. It did not cover the complete pipeline of claim extraction, deduplication, and matching against reference facts. We nevertheless took steps to support the reliability of this pipeline.

Our procedure draws on FACT in DeepResearch Bench (ICLR 2026) and the claim extraction and deduplication procedure in Towards Personalized Deep Research (ICLR 2026). A fixed LLM evaluator first extracts verifiable scientific claims from the complete scientific report and merges semantically duplicate claims within the report to avoid counting the same conclusion multiple times. For Evidence Faithfulness (EF), the LLM judges whether each claim is supported by evidence collected during the investigation. For Mechanism Insight (MI), it assesses the semantic correspondence between report claims and reference mechanism facts frozen in advance, assigning scores of 1 (full match), 0.5 (partial match), or 0 (no match).

Once the LLM has scored the candidate claim–fact pairs, a program selects the matching with the highest total score, subject to each report claim matching at most one reference fact and each reference fact being matched at most once. This prevents duplicate credit. The sum of the selected scores is denoted by WW and is used to calculate MI according to the formula in §4.3. Within this procedure, we assessed agreement with human annotations for the core EF and MI semantic judgments on the preselected items. We will clarify the scope of this human validation in the revised manuscript.

[1] DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents — ICLR 2026
[2] Towards Personalized Deep Research: Benchmarks and Evaluations — ICLR 2026

(2) MI and information provided in the public task description.

MI excludes properties already provided in the task description and assesses only those learned during the investigation.

Drawing on the public task descriptions in LLM-SRBench, ClosureBench’s public task cards provide neutral information such as the scientific context, discovery target, physical meanings of variables, and controllable experimental conditions. They do not disclose the hidden closure’s specific mathematical form, parameters, or mechanistic properties to be discovered.

In contrast, the reference mechanism facts for MI are constructed from each task’s hidden closure and its physical properties, reviewed by researchers, and frozen before evaluation. They assess whether the scientific report correctly recovers specific mechanistic properties rather than merely restating the public task description.

For example, for fic_task_h in Appendix C.5, the public task description specifies the discovery target as the local crystallization rate Rψ=f(ψ,T,g)R_\psi=f(\psi,T,g) and provides the relevant variables and experimental controls. The MI reference facts instead assess specific hidden mechanistic properties, such as the crystallinity dependence (1−ψ)3/2(1-\psi)^{3/2} and the shear enhancement factor 1+g1+g. An example public task card for this task is provided below.

Public Task Card: fic_task_h

Scientific context. A polymer melt flows through a cooled two-dimensional channel. The velocity field is solved beforehand and held fixed, while crystallinity and temperature evolve over time. The governing PDE structure is known, but the local crystallization-rate closure is unknown.

Discovery target. Recover the functional structure and numerical coefficients of the local crystallization rate from observations and executable experiments:

Rψ=f(ψ,T,g)R_\psi=f(\psi,T,g)

Variables.

SymbolPhysical meaning
ψ\psiLocal crystallinity (dimensionless)
TTLocal absolute temperature (K)
ggNormalized local shear rate, g=γ˙/(10 s−1)g=\dot{\gamma}/(10\,\mathrm{s}^{-1})
RψR_\psiUnknown local crystallization rate (s−1\mathrm{s}^{-1}; target output)

Experimental controls.

ControlAllowed range
Inlet velocity uinu_{\mathrm{in}}0.012–0.048 m/s
Seed crystallinity ψseed\psi_{\mathrm{seed}} (initial and inlet conditions)0.06–0.35
Wall temperature TwT_w330–390 K

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We have added experiments and provided detailed responses to address the points you raised. With the discussion period coming to a close, we would be grateful if you could take another look when your schedule allows. Please let us know if any concerns remain or if further clarification would be helpful.

We appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear Area Chair, Senior Area Chairs, and Reviewers,

Thank you for your constructive feedback, which helped strengthen our experiments and clarify the evaluation scope. Reviewer watv raised their score from 3 to 4, reporting increased confidence and recognizing the remaining mechanistic indistinguishability concern as a fundamental issue likely beyond this paper’s scope. Reviewer KX2J raised their score from 2 to 3 following the additional experiments.

We summarize the key results below. Paired accuracy values denote ID/OOD Acc@1%.

Main ConcernReviewersKey Results and Clarifications
Ablation of structured conjecturesHem5SCOPE outperforms parallel equation search without structured conjectures: 41.67%/25.00% vs. 30.56%/18.33% accuracy, and 0.573 vs. 0.363 EF, while using fewer tokens (Table R1.1).
Comparison using the same dataKX2JAfter giving PySR, LLM-SR, and SR-Scientist the same data collected by SCOPE, SCOPE retains its advantage: 41.67%/25.00% vs. 28.33%/20.00% for the strongest baseline (Table R3.1).
Token budgets and tool costsHem5, KX2JSCOPE leads under shared 3M and 5M token caps and common tool-call limits. Even at 3M, it outperforms ReAct and Reflexion-Exp at 5M, using fewer SR-tool calls and less SR-tool CPU time (Table R3.6).
Evaluator reliabilityHem5, watvMistral Medium 3.5 achieves 90.5% EF / 89.3% MI agreement with humans, and SCOPE’s advantages hold under both evaluators. Human validation covers 200 EF claims and 150 MI pairs (Tables R2.1–R2.2).
Crystallization exampleHem5Appendix C.5 and Appendix J concern different tasks with distinct closures; the report’s nonmonotonic behavior matches its corresponding closure. Explicit task IDs will clarify this distinction.
Mechanistic indistinguishabilitywatvEF evaluates evidential support, MI assesses recovered hidden closure properties, and EE tracks accuracy improvement. These metrics do not establish a unique mechanism when explanations are observationally indistinguishable; we will explicitly discuss this fundamental limitation.
Contamination and open-source evaluationKX2JSCOPE achieves 35.56%/26.11% accuracy with OLMo-3.1-32B-Instruct. We also added Feynman/ClosureBench convergence comparisons and acknowledged future contamination risks (Tables R3.4–R3.5).
Heterogeneous backboneswatvCombining three backbones improves OOD accuracy over Qwen alone from 25.00% to 35.56%, but does not consistently outperform the strongest single-backbone configuration (Table R2.3).

We will incorporate these results and clarifications into the revised manuscript. Thank you again for your time and consideration. We look forward to further discussion with Reviewer Hem5 on the final day of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.