acceptodds
Under review as a conference paper at ICLR 2027

Learning to Discover Equations by Evolving Search Algorithms

Abstract

Symbolic regression (SR), the problem of discovering interpretable equations that explain observed data, is traditionally done with evolutionary search using hand-designed heuristics. We introduce a meta-evolutionary method that uses LLMs to discover and improve symbolic regression algorithms. Our method addresses stochastic evaluation of search algorithms with a reevaluation strategy integrated into the evolutionary loop, reducing selection based on chance successes. We apply meta-evolution to evolve symbolic regression algorithms from a minimal genetic algorithm skeleton and improve an existing popular, high-performance SR algorithm, PySR. The resulting algorithms are reusable across new problems and require no LLM calls at inference time. The evolved PySR algorithm improves ground truth equation recovery from 50.6% to 58.6% on SRBench, and improvements transfer to naturalistic equation discovery tasks (EmpiricalBench) and to a mechanistic interpretability application. We also show how meta-evolution can be used to fine-tune algorithms for specialized domains, showing increased performance for neuroscientific and mechanistic interpretability equation recovery, even with limited training data.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
evolution, symbolic regression, search, learning, meta-learning, equation discovery, scientific discovery, LLM-evolution, program synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rLudOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an LLM-guided framework for evolving symbolic regression algorithms (mainly for PySR). The LLM modifies components of PySR and BasicSR, including mutation, selection, survival, and loss functions, using performance across training tasks as feedback. Repeated evaluations reduce the influence of stochastic search outcomes during algorithm selection. The resulting algorithms are evolved and evaluated on SRBench.

Critical strengths
  1. The research problem is meaningful. Symbolic regression is an important and active research area.

  2. The experiments cover a broad range of settings. The paper investigates multiple training objectives, noise levels, search budgets, and application domains, and examines the effects of reevaluation and individual algorithm components.

Critical weaknesses

1. The comparisons do not establish competitiveness with recent symbolic regression methods.

Demonstrating improvements over the original PySR and several traditional methods is insufficient. Recent methods—including MOD-SR[1], LLM-SR[2], RESTART[3], SR-Scientist[4], GenSR[5], PiT-PO[6] and so on—represent substantial developments in symbolic regression, and several report improvements over PySR under their respective evaluation settings. The final evolved algorithms should therefore be compared directly with representative recent methods, not only traditional methods in Fig 2 and Tab 1.

The recovery improvement decreases from 50.6% → 58.6% in Table 1 to 47.0% → 51.5% after excluding training and validation tasks. On black-box tasks, the best Evolved PySR achieves an average $R^2& of 0.811, below the original PySR’s 0.814. For context, GenSR reports 0.8416 (which is the work in ICLR 2026, not the latest method), compared with 0.817 for the best evolved algorithm in this paper. Different task subsets and budgets prevent a direct ranking, but reinforce the need for a comparison with recent methods. Moreover, MDLFormer’s Appendix Table 8 reports black-box results, whereas its corresponding entry in this paper’s Table 1 is left blank.

Without demonstrating competitive performance over contemporary methods, the results primarily establish an incremental improvement to PySR, rather than a substantive advance over the current state of symbolic regression.

2. The methodological novelty is limited, and a closely related method is omitted.

Using an LLM to modify evolutionary search operators through task-level performance feedback is already explored in prior work. In particular, the paper omits LLM-Meta-SR[7], which uses an LLM to generate selection-operator code, evolves that code in an outer loop, runs symbolic regression in an inner loop, and uses performance across multiple tasks to guide subsequent modifications. This substantially overlaps with the central design of the present framework.

Modifying additional components and incorporating reevaluation provide extensions, but the paper does not establish a sufficiently distinct algorithmic contribution. The analysis also attributes much of the improvement to affine scaling in the loss and subtree duplication, both of which the paper acknowledges have precedents in earlier symbolic regression research.

3.Training on SRBench leaves generalization beyond familiar equation families insufficiently established.

An LLM may possess prior knowledge of public benchmarks and common physical equations. When optimizing search algorithms on SRBench, it may generate rules particularly suited to these familiar structures. Such specialization can be encoded in the resulting algorithm even when no LLM is used at inference time.

4.The main recovery results include tasks used to develop the algorithms.

Table 1 evaluates recovery on all 130 tasks, including the 20 training tasks and 20 validation tasks used for algorithm development and selection. Consequently, the headline results mix development performance with generalization performance.

The main comparison in Tab. 1 should evaluate all methods on the same strictly held-out tasks. This matters quantitatively, because the reported improvement falls from 8.0 percentage points to 4.5 percentage points after excluding training and validation tasks.

5. The improvements are strongly objective-dependent, and evidence of broader practical gains is limited.

The evolved PySR selected for its best recovery performance achieves a black-box R2R^2 of only 0.659, compared with 0.814 for the original PySR. Even the mixed-objective version reaches only 0.811. Thus, improved exact recovery does not translate into improved predictive performance and can come with a substantial trade-off.

The empirical-data results also provide limited evidence of a broad improvement: successful trials increase from 72/90 to 74/90, with the recovery difference confined to the Rydberg task. These findings support useful specialization for particular objectives, but provide weaker support for the broader claim of a generally improved symbolic regression algorithm.

[1] MOD-SR: Unifying Multimodal Learning and Direct Optimization with Gradient-Guided Diffusion Model for Symbolic Regression, ICML 2026.
[2] LLM-SR: Scientific Equation Discovery via Programming with Large Language Models, ICLR 2025.
[3] Robust Equation Structure Learning with Adaptive Refinement, ICLR 2026.
[4] SR-Scientist: Scientific Equation Discovery With Agentic AI, ICLR 2026.
[5] GenSR: Symbolic Regression Based on Equation Generative Space, ICLR 2026.
[6] LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization, KDD 2026.
[7] LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression, IEEE Transactions on Evolutionary Computation,2026

Questions for authors

No

1 comment

Thank you for the feedback! Our response to each point is below.

W1: The comparisons do not establish competitiveness with recent SR methods

We now compare with all methods you list. Evolved PySR is competitive with LLM-based methods while making no LLM calls at inference. Full details are in Appendix G.

  • MOD-SR: we ran the authors' code on SRBench GT and found recovery of 7.9% versus 54.3% for evolved PySR.
  • LLM-SR: on the four problems evaluated in LLM-SR, evolved PySR matches or beats LLM-SR's held-out NRMSE on 3 of 4 problems.
  • GenSR: on its 57 black-box tasks and 2,700-second search budget, evolved PySR achieves an R2R^2 of 0.855 compared to 0.841 for GenSR, and achieves an R2R^2 of 0.837 even with a 90 second budget, using the affine-rescaling for evolved PySR discussed in W5 response below.
  • MDLFormer: we've added its black-box score (0.626) to Table 1.

We evaluated base and evolved PySR on LSR-Synth for 30 minutes (single CPU) or 1 hour (8 cores) and calculated metrics matching each paper's evaluation.

Method (backbone)MetricProblemsTheir scorePySR (1 core, 30 min)Evolved PySR (1 core, 30 min)PySR (8 cores, 1 h)Evolved PySR (8 cores, 1 h)
SR-Scientist (GPT-OSS-120B)95%-of-points Acc, τ=0.01all 12963.642.6 ± 2.359.4 ± 1.260.2 ± 1.673.1 ± 3.6
SR-Scientist (Qwen3-Coder-480B)95%-of-points Acc, τ=0.01all 12949.142.6 ± 2.359.4 ± 1.260.2 ± 1.673.1 ± 3.6
PiT-PO (Llama-3.1-8B)strict Acc, τ=0.1all 12966.737.0 ± 3.151.2 ± 1.354.3 ± 1.365.6 ± 2.6
RESTART (Qwen3-8B)95%-of-points Acc, τ=0.193, no Chem86.065.2 ± 0.679.6 ± 1.178.9 ± 2.290.3 ± 1.1

Evolved PySR improves over base PySR by 11–17 points. At the larger budget, it generally outperforms SR-Scientist and RESTART and roughly matches PiT-PO. These methods call an LLM throughout the search (1,000+ queries per problem, and almost 40× that for PiT-PO), whereas evolved PySR runs a standard evolutionary search on CPU with no LLM calls.

W2: The methodological novelty is limited, and LLM-Meta-SR is omitted

Thanks for the pointer; it is now cited and discussed in Related Work. We agree that our work shares its overall framework and do not claim the meta-evolution outer loop as a novel contribution. What's new relative to LLM-Meta-SR:

  1. Designing reevaluation strategies for evolving stochastic search algorithms, which is relevant beyond SR
  2. Evolving four operators jointly rather than one, applied to a mature, widely used SR library
  3. Fine-tuning SR algorithms to specialized domains
  4. Evaluation across several domains and comparison to leading methods.

W3: An LLM may know SRBench and common physical equations

We consistently find that the evolved algorithm generalizes to new problem families:

  1. On MIPS, a mechanistic-interpretability domain with very different operators and target equations, evolved PySR solves 230/390 trials versus 203/390 for base PySR.
  2. EmpiricalBench contains real-world problems substantially different from SRBench. A manual audit of EmpiricalBench scoring discovered 6 false negatives for evolved PySR (see response to Reviewer Ycuz Q4). Evolved PySR now solves 80/90 trials while base PySR stays at 72/90.
  3. On LSR-Synth, a benchmark designed so LLMs cannot rely on memorized equations, evolved PySR improves over base PySR by 11–17 points under every metric and compute budget (table above).

W4: The main recovery results include tasks used to develop the algorithms

We've revised all headline numbers in the abstract and tables to report only held-out results, which preserve the method's advantage (54.3% vs. 50.2% for base PySR).

W5: The improvements are strongly objective-dependent

The drop in black-box R2R^2 in the GT-evolved PySR is due to the built-in affine-scaling of the evolved loss function. This doesn’t affect GT recovery, which accepts a constant offset, but it can leave coefficients uncalibrated for R2R^2. Fitting a scale and offset to equations on the training data before evaluating R2R^2 improves black-box score to 0.823.

The question of generalization of the evolved algorithm has already been discussed in answers to W3 and W1. The new LSR-Synth results are the most convincing evidence of discovering a generally improved SR algorithm: LSR-Synth measures numeric accuracy on held-out data rather than symbolic recovery, and evolved PySR improves on every metric and budget.

Reviewer_WpCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a meta‑evolutionary method​ that uses Large Language Models (LLMs) to discover and improve symbolic regression (SR) search algorithms. The outer loop employs LLM‑guided program evolution to mutate and refine the source code of SR algorithms (e.g., mutation operators, selection rules, loss functions), while the inner loop executes the evolved algorithms to search for equations. A key technical contribution is the treatment of stochastic algorithm evaluation​ via reevaluation strategies (uniform and Top‑Two Thompson Sampling, TTTS) to mitigate “winner’s curse” in selection. Starting from either a minimal genetic algorithm skeleton (BasicSR) or the high‑performance PySR, the method evolves algorithms that achieve 50.6% → 58.6% ground‑truth equation recovery on SRBench​ (under a 90‑second budget) and transfer to EmpiricalBench, neuroscientific equation discovery (NeuronBench), and mechanistic interpretability (MIPS). Crucially, no LLM calls are required at inference time.

Critical strengths
  1. Practical performance gains on standard benchmarks​ – Evolved PySR consistently outperforms the original PySR and competitive baselines (HPO, Autoresearch, MDLFormer) on SRBench ground‑truth recovery, with improvements persisting under longer budgets and random restarts.

  2. Addressing winner’s curse via TTTS​ – The paper identifies high‑variance fitness estimates as a key bottleneck in meta‑evolution and shows that TTTS‑based reevaluation outperforms both uniform reevaluation and simply increasing the number of initial seeds (Figure 5). This is a solid, empirically backed contribution relevant beyond SR.

  3. Inference‑time efficiency and transferability​ – Evolved algorithms are pure Python code with no neural network in the loop, making them cheap to run and easy to fine‑tune to specialized domains (e.g., neuron equations, RNN interpretability) with limited data. Comprehensive experimental scope​ – Evaluations span SRBench (130 ground‑truth tasks, 122 black‑box tasks, multiple noise levels, 10 seeds), EmpiricalBench (9 real‑world problems, 1‑hour budget), and two specialized scientific domains, with clear ablation of reevaluation strategies.

  4. Code simplification phase​ – The evolution includes a final simplification stage that reduces the best algorithm’s code size (e.g., 454 → 394 LOC for PySR), aiding interpretability and analysis.

Critical weaknesses
  1. Incremental novelty; no SR‑specific search insights​ – The core framework (LLM‑guided program evolution over algorithm code) directly follows prior work (FunSearch, EoH, LLaMEA). The evolved operators (affine‑scaled NRMSE loss, subtree duplication mutation) are refinements of existing heuristics (Keijzer 2003, Hoai et al. 2005), not novel SR mechanisms. The paper does not demonstrate what new structural understanding of SR search has been gained.

  2. Lack of isolation ablations​ – It is impossible to tell how much of the gain comes from (a) the LLM mutation engine, (b) the TTTS reevaluation, or (c) the specific evolved operators. Ablating any one of these (e.g., evolving without TTTS, or replacing LLM mutations with random edits) is missing, weakening the method’s claimed contributions.

  3. Metric overfitting and poor black‑box trade‑off​ – Training with the ground‑truth (GT) objective dramatically improves GT recovery but causes black‑box R² to drop from 0.814 (PySR) to 0.659 (Evolved PySR). Even when excluding training/validation tasks, the GT recovery falls to 51.5%, suggesting the improvement is partly benchmark‑specific. The paper does not adequately discuss this trade‑off. Weak narrative in Section 5​ – The experimental section reads as a table dump. Key questions (e.g., “Why does GT training hurt black‑box performance?”, “How do we know the gain isn’t just from affine scaling?”) are not posed before presenting results, making the section hard to follow.

  4. Limited comparison to closest prior work​ – While LLaMEA is cited, there is no direct comparison (e.g., running LLaMEA on the same SR algorithm skeleton). This makes it difficult to judge whether the gains are due to the specific LLM‑evolution implementation or the general paradigm.

Questions for authors

1, Ablation of TTTS vs. pure code evolution: If you remove the TTTS reevaluation (or replace it with a fixed 10‑seed evaluation), how much does the final SRBench GT recovery drop? This would isolate the value of the reevaluation strategy.

  1. Manual heuristic baseline: Could the same performance be achieved by simply adding affine‑scaled loss and subtree duplication to PySR manually, without any LLM‑driven evolution? If so, the LLM’s role is merely automation, not discovery.

  2. Black‑box performance collapse: Why does optimizing for GT recovery cause such a severe drop in black‑box R²? Is there a way to jointly optimize both metrics (e.g., multi‑objective training)?

  3. Training task sensitivity: With only 20 training problems, how robust is the evolved algorithm to the choice of those tasks? Would a different random split yield similar gains?

2 comments

Thank you for the feedback! Our response to each point is below.

Q1 / W2: Ablation of TTTS: if you remove it, how much does GT recovery drop?

We took the controlled ablation runs from Figure 5 (15 generations, 3 runs per strategy) and evaluated each run on the 90 held-out SRBench tasks. These runs were evolved with zero noise for fewer generations, so the numbers aren't comparable to Table 1.

Reevaluation strategyHeld-out solve rate (%), zero noise
Base PySR (no evolution)52.6
Fixed, 3 seeds50.9 ± 1.5
Fixed, 10 seeds50.9 ± 2.5
Uniform reevaluation (3 + 7 seeds)50.7 ± 1.9
TTTS54.2 ± 2.3

On these runs, only TTTS beats base PySR. Without it, winner's curse prevents evolution progress (see also Figure 5 for more direct evidence of winner's curse). However, with three runs per strategy, this is suggestive rather than conclusive.

Q2 / W2: Could adding affine-scaled loss and subtree duplication to PySR by hand do the same?

We tried this. Manually implementing an affine-scaled loss (Keijzer, 2003) and subtree duplication mutation (Hoai et al., 2005) only solves 40.0% of held-out SRBench tasks, below base PySR (50.2%).

We find that the manually implemented baseline raises performance on tasks with zero noise (+7.4 points) but hurts on tasks with noise (–30.7 points when noise = 0.1). The affine-scaled loss alone never learns any scale and offset, so these must be fitted after search (without this, performance is even worse). But with noisy targets, the fitted scale and offset can both come out wrong, and SRBench’s GT checker tolerates a constant factor or offset but not both. The evolved operator's loss, in contrast, is an extension which includes a small raw-error term that breaks ties in favor of expressions with calibrated constants, which keeps the search building constants into the expression. This is a case where a subtle detail found by the LLM determines whether the operator helps or hurts.

Q3 / W3: Why does GT training hurt black-box R²? Can both be optimized jointly?

The drop in black-box R2R^2 in the GT-evolved PySR is due to the built-in affine-scaling of the evolved loss function. This doesn’t affect GT recovery, which accepts a constant factor or offset, but it can leave coefficients uncalibrated for R2R^2. Fitting a scale and offset to equations on the training data before evaluating R2R^2 improves black-box score to 0.823.

Table 1 already includes results when evolving for both GT and R2R^2 objectives simultaneously. This combined objective yielded the best result when evolving from BasicSR, but did not work as well for evolving PySR.

Q4: How sensitive is the method to the 20 training tasks?

We found the choice of training tasks to be important for evolution effectiveness. In early experiments, random splits limited reward signal: many tasks are either always or never solved. To ameliorate this, we sampled a set of training tasks from a band where base PySR goes from consistently solving to consistently failing. This is now described in the paper.

In a preliminary check with two random 20-task splits, one run did slightly better than our curated split and one did considerably worse. However, with a single run per split, we can’t yet draw firm conclusions about which selection strategy is best. We will add a more systematic study in the revision.

AuthorsOpenReview ↗

Response continued

W1: Incremental novelty; no SR-specific search insights

We agree the LLM-evolution outer loop is not novel and do not claim it as a contribution. Our contributions are:

  1. Designing reevaluation strategies for evolving stochastic search algorithms, which is relevant beyond SR;
  2. Improving a mature, widely used SR library rather than a toy skeleton, with no LLM calls at inference;
  3. Fine-tuning SR algorithms to specialized domains, which to our knowledge has not been shown before.

We also evaluate across several domains and optimization objectives.

For SR-specific insight, see Q2 response.

W2: Lack of isolation ablations

  • Reevaluation ablations: TTTS ablation on held-out tasks is discussed in response to Q1.
  • LLM evolution loop ablation: We now additionally compare with LLaMEA (see W4 response).
  • Evolved operator ablations: These ablations are in Figure 17. We observe a synergistic effect, where replacing only one of the four operators of the base algorithm improves performance slightly, while replacing all four yields a large jump.

W3: Metric overfitting; weak narrative in Section 5

We agree the evolved algorithm overfits to its training tasks (Table 2). Due to the cost of meta-evolution's inner evaluation loop, evolution on a small number of train tasks is necessary, and some overfitting is inevitable; the question is whether any gains transfer. The evolved algorithm maintains consistent performance improvements across all held-out domains: MIPS, EmpiricalBench, and new results on LSR-Synth in response to Reviewer rLud, where evolved PySR improves over base PySR by 11–17 points across metrics and compute budgets (e.g., 73.1% vs. 60.2% at 8 cores / 1 hour).

We will rewrite Section 5 to improve the narrative, and feature more prominently the key questions raised by this and other reviewers.

W4: No direct comparison with LLaMEA

We reimplemented LLaMEA’s algorithm to evolve PySR using its LLM-evolution loop, keeping everything else (models, train set, evaluation protocol, compute budget) the same. On held-out SRBench tasks and a roughly matched compute budget (150 inner loop evaluations for LLaMEA and 120 for meta-evolution), LLaMEA scores 51.9% versus 53.0% for our approach.

Reviewer_YcuzOpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses LLM-guided meta-evolution to discover symbolic regression algorithms. The outer loop prompts an LLM to rewrite one operator (mutation, selection, survival, loss) of an SR program. The inner loop runs the program on a set of training tasks. Fitness is stochastic, so the authors study reevaluation strategies to reduce the winner's curse, including uniform reevaluation and top-two Thompson sampling. They evolve both a minimal GA skeleton and PySR. The evolved PySR improves SRBench ground-truth recovery from 50.6% to 58.6% at a 90-second budget, and the gain persists at longer budgets and on held-out tasks. They also fine-tune the evolved algorithm on NeuronBench and a mechanistic interpretability benchmark.

Critical strengths
  1. The framing is clean. The LLM is expensive, so use it once to write a search program, then run that program cheaply on new problems. This is a real amortization story, and it avoids the cost and latency of LLM calls at inference. The nested GA is a natural way to implement it.

  2. The transfer results are nontrivial. The SRBench-evolved PySR improves MIPS from 203 to 230 solved SR problems out of 390, and fine-tuning on a single NeuronBench task recovers 23 of 25 held-out trials. If these numbers hold, they show that meta-evolution can produce domain-specialized search heuristics, which is a useful practical result.

Critical weaknesses
  1. The headline number is not held-out. The 50.6→58.6 comparison includes 40 tasks used for training or validation. The held-out comparison is 47.0→51.5. The abstract does not mention this. This matters because the paper's claim is about discovering algorithms that generalize to new problems, not about fitting the benchmark.

  2. The MIPS fine-tuned result is trained on the test problems. The "fine-tuned" PySR is trained on the 39 unsolved SR problems and then evaluated on the same 39. The improvement from 230 to 245 is in-sample. The legitimate transfer result is the SRBench-evolved model. The paper presents the in-sample number as evidence of domain fine-tuning without saying so.

Questions for authors
  1. Is the MIPS fine-tuned model trained on the 39 evaluation problems? If so, the 245 vs 230 comparison is in-sample. Can you hold out a subset of problems and report the transfer?

  2. How much compute did the meta-evolution use (LLM calls, inner-loop evaluations, wall-clock)? How does that compare with the 60-experiment, 14-hour Autoresearch run? The conclusion that LLM-evolution beats autoresearch needs an equal-budget comparison.

  3. EmpiricalBench is 74 vs 72 out of 90. What is the variance over tasks and seeds? Is the difference significant?

  4. The EmpiricalBench evaluator is an LLM. What is its agreement with human review on a held-out subset?

1 comment

Thank you for the feedback! Our response to each point is below.

W1: The headline number is not held-out

We've revised all numbers in the abstract and main text to report only held-out results, which preserve the method's advantage (54.3% vs. 50.2% for evolved vs. base PySR). Additionally, to augment our existing held-out evaluation on EmpiricalBench and MIPS problems, we provide new results testing on LSR-Synth (Shojaee et al., 2025), a third held-out domain with 129 problems, where evolved PySR improves over base PySR by 11–17 points (see our response to Reviewer rLud).

W2 / Q1: The MIPS fine-tuned result is trained on the test problems

This is a fair point. For discovery (as in the original MIPS work), using all tasks is appropriate: the goal is to recover unknown programs, and each program is verified against its task. For generalization, we now report held-out results after fine-tuning on a random subset of the 39 problems and testing on the rest (10 seeds each):

Trials solvedFine-tune on 8, test on 31Fine-tune on 20, test on 19
Base PySR143/31091/190
Evolved on SRBench152/310104/190
Fine-tuned on subset (held-out)165/310109/190
Fine-tuned on all problems (in-sample)172/310111/190

Most of the fine-tuning gain holds up on unseen problems.

Q2: How does compute compare with Autoresearch? Needs an equal-budget comparison.

Meta-evolution used 450 evaluations and $49 in API cost over 17 hours. We reran autoresearch with Opus 5, the strongest model in our ensemble. Our autoresearch run spent $63 for 123 evaluations over 45 hours. Comparing held-out SRBench scores:

  • Base PySR: 50.2%
  • Autoresearch: 51.9%
  • (≈\approx equal evaluation budget) Meta-evolution checkpoint at 120 evaluations: 53.0%
  • Meta-evolution, full run: 54.3%

Examining autoresearch experiments, we find that models tend to limit themselves to small changes to the algorithm such as tweaking mutation weights, and do this increasingly over time, despite efforts to prompt models otherwise.

Q4: What is agreement with human-review on EmpiricalBench?

We audited 180 EmpiricalBench trials (blind to method) and agreed with the LLM judge on 174/180. All six disagreements were Rydberg trials where the LLM gave false negatives: equations such as $ \log(\frac{1}{n_1^2} - \frac{1.0004}{n_2^2}) $ were graded by the LLM as failures. These were caused by an overly strict spec given to the LLM which said constants had to match exactly. Using SRBench's convention of rounding constants to three decimals, evolved PySR now solves 80/90 trials while base PySR stays at 72/90.

Q3: What is variance over tasks and seeds, and is the difference significant?

With the newly audited EmpiricalBench results, the difference is significant:

  • 1 hour of search: Evolved PySR solves 8.0 ± 0.0, base PySR solves 7.2 ± 0.4, p < 0.001
  • 30 seconds of search: Evolved PySR solves 6.8 ± 0.4, base PySR solves 5.8 ± 0.9, p = 0.008

That said, evidence of improvement remains limited due to saturation of EmpiricalBench by base PySR. The new LSR-Synth results (129 problems) are stronger evidence of held-out improvement: 73.1% accuracy for evolved PySR vs. 60.2% for base PySR.

Comments on the paper

1 comment

Summary of revisions

We thank all reviewers and the AC for engaging with our work and for the constructive comments. The discussion has made the paper substantially stronger. Here is a summary of our main changes:

  1. Held-out reporting. Headline SRBench GT numbers now report only held-out results. Our method keeps its advantage (54.3% versus 50.2% GT for evolved PySR versus base PySR). (Ycuz, rLud)
  2. New held-out benchmark (LSR-Synth, 129 problems). Evolved PySR solves 73% of problems compared to 60% for the original PySR algorithm, providing a third example of transfer to new domains. (rLud)
  3. Comparisons with recent SR methods. We compared with MOD-SR, LLM-SR, GenSR, SR-Scientist, PiT-PO, and RESTART. Evolved PySR is competitive with or better than all methods despite making no LLM calls at inference. (rLud)
  4. Updated baselines. We added equal-budget comparisons with autoresearch and the evolution framework of LLaMEA. Meta-evolution outperforms both. (Ycuz, WpCH)
  5. Ablations. We strengthened the ablation of reevaluation strategies and confirmed that TTTS performs best. A "manual implementation" of the heuristics used in the evolved operator performed worse than base PySR, showing that subtle details the LLM found matter. (WpCH)
  6. Rigor of evaluation. We added held-out MIPS fine-tuning splits and find most of the gain persists on unseen problems. We also conducted a human audit of EmpiricalBench grading and found six false negatives; after correction, evolved PySR solves 80/90 versus 72/90 for base PySR. (Ycuz)
  7. Recovering black-box R2R^2. We explain how the drop in black-box R2R^2 for GT-evolved PySR is due to the affine-scaling of the evolved loss function, and how an inexpensive refitting of scale and offset recovers performance. (WpCH, rLud)

We hope these additions address the reviewers' concerns. If they do, we would be grateful if reviewers would consider updating their scores. We are happy to answer any remaining questions before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.