- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a two-phase framework for LLM-based scientific equation discovery that adapts both the tool library and the model’s tool-use policy using the same fit-error signal. In Phase A (harness evolution), GPT-5.5 solves problems from LLM-SRBench with an initial library of 25 generic numerical tools. A second GPT-5.5 instance classifies each failure as a missing capability (it then writes a new tool) or a misused one (it revises the tool interface or selection rule), and an edit is accepted only if all previously solved problems remain solved. In Phase B, the evolved 33-tool harness is frozen and Qwen3.6-35B-A3B is fine-tuned with LoRA on GPT-5.5 trajectories from synthetic tasks screened for overlap with the evaluation formulas. The system is evaluated on LSR-Synth, LSR-Transform, SRSD-Feynman, and CP3-28, and reports higher accuracy than published baselines. Ablations attribute most of the gain to the evolved library.
Critical strengths
Simple, practical recipe. Phase B is plain supervised fine-tuning, with LoRA on about 72 tasks for 120 steps. At deployment the system needs only a frozen 33-tool library and a ~3B-active open model, with no frontier-model calls at inference. That makes it cheap to run and straightforward to reproduce once the library and adapter are released.
Critical weaknesses
Limited novelty. The method combines two existing ideas, harness/tool evolution and trajectory fine-tuning, in a single sequential pass with no feedback between phases. Both have direct precedent in SR (SR-Scientist and KeplerAgent for tool use; A-SR and DecAEvolve for model adaptation). Table 3 also shows that the evolved library without any model nearly matches the full system. Saturating metrics. Under trimmed-5% scoring, LSR-Synth accuracy reaches 96.1/95.4 (Acc@10⁻²/10⁻³), leaving little room to separate methods. Results at stricter tolerances, together with NMSE and symbolic accuracy (both standard in LLM-SRBench), would make the comparison more informative. Missing baselines. Closely related SR systems the paper itself cites, DecAEvolve and KeplerAgent, are not compared. Neither are general evolutionary frameworks such as AlphaEvolve (via OpenEvolve) or ThetaEvolve applied to the same panels.
Questions for authors
How sensitive is Phase A to the initial toolset? The 25 seed tools already reach 41.1/30.2 on LSR-Synth under deterministic traversal (Table 3), roughly matching SR-Scientist’s 63.6 under trimmed scoring (Fig. 3). Would a smaller or differently composed seed set (e.g., only basic fitting and verification) lead to a comparable final library and accuracy?
4 comments
Response 2/4
Reviewer's concern:
"Saturating metrics. Under trimmed-5% scoring, LSR-Synth accuracy reaches 96.1/95.4 (Acc@10⁻²/10⁻³), leaving little room to separate methods. Results at stricter tolerances, together with NMSE and symbolic accuracy (both standard in LLM-SRBench), would make the comparison more informative."
We would like to clarify that the 5% pruning convention is used only for comparison with historical baselines. To directly address the ceiling effect concern, we also report stricter all-points metrics that demonstrate clear room for differentiation. Under full-sample evaluation, our method reaches 71.3% at Acc@ on LSR-Synth, while the best published baseline (SR-Scientist) reaches about 50% even under the more lenient trimmed protocol. There is still clear room for differentiation.
The differentiation is even more evident on harder benchmarks. On SRSD-240, our method achieves 58.0% on hard tiers versus 44.6% for uDSR. The multi-tolerance breakdown is Acc@ = 59.6%, Acc@ = 55.4%, and Acc@ = 52.1%. The trained model also achieves 33% lower median NMSE than the tools-only baseline (0.000186 vs. 0.000278), showing a clear improvement from policy training.
We provide complete NMSE, multi-tolerance, and symbolic accuracy results below.
Trained Model + Tools (240/240 tasks):
- Median test NMSE: 0.000186
- Median test : 0.9998
Multi-tolerance breakdown:
- Acc@ (=0.1): 143/240 = 59.6%
- Acc@ (=0.01): 133/240 = 55.4%
- Acc@ (=0.001): 125/240 = 52.1%
T+ baseline: Median NMSE 0.000278, Acc@=51.3%. Trained model achieves 33% lower median NMSE.
Following LLM-SRBench's equivalence judgment protocol, we evaluated symbolic accuracy on all tasks using GPT-6.1-sol. Symbolic accuracy is a more demanding metric: it requires the expression to match the ground truth in structure, not only in numerical behavior.
| Dataset | Symbolic Accuracy | Numerical Acc@ |
|---|---|---|
| LSR-Synth-129 | 28.68% (37/129) | 71.3% |
| LSRT-111 | 44.14% (49/111) | 63.1% |
| SRSD-240 | 31.25% (75/240) | 52.1% |
Lower symbolic accuracy is expected because the criterion is stricter. For example, our predicted expression achieves NMSE = and against the ground truth , yet the LLM judges the two as symbolically non-equivalent because of the additional linear term.
We will add these complete multi-tolerance, NMSE, and symbolic accuracy results to the revised manuscript.
Response 3/4
Reviewer's concern:
"Missing baselines. Closely related SR systems the paper itself cites, DecAEvolve and KeplerAgent, are not compared. Neither are general evolutionary frameworks such as AlphaEvolve (via OpenEvolve) or ThetaEvolve applied to the same panels."
We would like to clarify that direct comparison is not feasible because these methods evolve fundamentally different objects.
AlphaEvolve / OpenEvolve / ThetaEvolve evolve the "solution" itself. They are designed for per-problem search: given a problem and a scoring function (e.g., accuracy), they evolve and select candidates. Each problem requires running the full evolution process (including LLM sampling) from scratch. DecAEvolve evolves symbolic expressions through test-time RL adaptation and term decomposition. Each problem system triggers per-system RL fine-tuning (GRPO/LoRA), making it a per-problem adaptive method rather than a reusable capability builder. The LLM adapts to each specific system during inference. Test-time cost is therefore high (evolution itself is inference-time search), making them unsuitable for symbolic regression tasks.
Our method evolves the "harness,". The evolution target is the executable tool library (harness): 33 tools optimized via failure-driven diagnosis. Evolution occurs only during training. "Solving" occurs at test time: a small model orchestrates the evolved tools to fit equations. The evolution product is a reusable capability asset across problems, not a specific solution.
Here is a comparison of what each method evolves and when:
| Method | Evolution Target | Model Training | Tool Evolution | Evolution Timing |
|---|---|---|---|---|
| OpenEvolve | Candidate programs | ✗ | ✗ | Inference |
| ThetaEvolve | Program generation strategy | ✓ | ✗ | Training + Inference |
| DecAEvolve | Hypothesis generation strategy | ✓ (Test-time) | ✗ | Inference |
| Ours | Tool library + Strategy | ✓ (LoRA) | ✓ | Training (Coupled) |
For KeplerAgent, we thank the reviewer for the suggestion. Since they have not released their code or reported results on additional datasets, we provide experimental comparison on the LSR-Transform dataset:
| Method | Mean NMSE |
|---|---|
| Tool-only (ours) | 0.496 |
| Trained model + tools (ours) | 0.121 |
| KeplerAgent @1 | 1.50 |
| KeplerAgent @3 | 1.21 |
Our trained system reaches a mean NMSE roughly an order of magnitude lower than KeplerAgent at both @1 and @3.
We will add to the revised manuscript: (1) Expanded Related Work clarifying that these methods address different evolution targets, (2) KeplerAgent comparison on LSR-Transform.
Response 4/4
Reviewer's question:
"Phase A starts with 25 seed tools achieving 41.1%/30.2%. How sensitive is the final outcome to initialization? Would a smaller or differently composed seed (e.g., only basic fitting and validation) converge to comparable capability?"
We would like to clarify that the 25-tool seed encodes generic SR knowledge by design. The initial library reflects standard SR operations: basic fitters, diagnostics, verification, structure exploration. This is intentional: evolution's value is measured against a strong baseline, not a trivial one.
Would a smaller seed converge to similar capability? Even if convergence were possible, it would be much slower and lack consistency. We attempted starting with only 6 basic tools. However, facing minimal input information, the tool attribution API pointed to 3 seemingly plausible but distinct fixes across 3 invocations, and none of them generalized to the neighboring 10 cases.
This does not affect our core performance or claims. Any evolution scheme has its operational boundaries within which it demonstrates effectiveness and stability. As observed in A-SR as well, effective iterative self-improvement only occurs in moderate difficulty regimes. Running lengthy and complete/unstable self-evolution for different initial set sizes is extremely expensive and time-consuming, and such boundaries will shift significantly with model capability improvements. While interesting, exploring these boundaries is beyond the scope of this work. Therefore, to better address the scientific problem, we leverage human scientific knowledge to manually design and create the initial tools.
We appreciate the reviewer's constructive feedback, which has helped us better articulate the strengths and clarify the positioning of our work. The comments have been invaluable in highlighting key distinctions and ensuring the paper emphasizes its core contributions more clearly. We will incorporate all the above clarifications and additional experimental details into the revised manuscript and will release the code to facilitate reproduction and further research.
Response 1/4
We would like to clarify that our work addresses a fundamental methodological bottleneck in the domain: existing approaches evolve either the toolset OR the model, but not both.
Trained agents (SR-Scientist) train models on fixed toolsets. Self-evolving search (A-SR) adapts the search process within a run but does not create or modify executable tools. Symbolic evolution (DecAEvolve) evolves expressions, not executable search procedures. Fixed orchestration (KeplerAgent) provides no tool evolution and relies on physics priors.
However, one-sided adaptation creates an uncorrectable bottleneck when the problem lies on the other side. For example, in trained-model methods, if the hand-made toolset itself is inadequate or encounters new problem types, no amount of training can compensate.
Our contribution is coupled evolution of tools and policy. This represents methodological progress. Unlike previous methods that explore model potential via reinforcement learning on fixed toolsets, our approach enables both components to evolve. The framework introduces (1) failure-driven tool evolution (Algorithm 1, Phase A), (2) trajectory distillation that trains a smaller model to effectively organize evolved tools (Phase B), and (3) achieves 79.1%/71.3% on LSR-Synth-129 and median NMSE 0.000186 on SRSD-240 across diverse domains (Tables 1-2).
We will clarify these distinctions in the revised Related Work (Section 2) and add a comparison table in the appendix.