Learning to Discover Equations by Evolving Search Algorithms

Symbolic regression (SR), the problem of discovering interpretable equations that explain observed data, is traditionally done with evolutionary search using hand-designed heuristics. We introduce a meta-evolutionary method that uses LLMs to discover and improve symbolic regression algorithms. Our method addresses stochastic evaluation of search algorithms with a reevaluation strategy integrated into the evolutionary loop, reducing selection based on chance successes. We apply meta-evolution to evolve symbolic regression algorithms from a minimal genetic algorithm skeleton and improve an existing popular, high-performance SR algorithm, PySR. The resulting algorithms are reusable across new problems and require no LLM calls at inference time. The evolved PySR algorithm improves ground truth equation recovery from 50.6% to 58.6% on SRBench, and improvements transfer to naturalistic equation discovery tasks (EmpiricalBench) and to a mechanistic interpretability application. We also show how meta-evolution can be used to fine-tune algorithms for specialized domains, showing increased performance for neuroscientific and mechanistic interpretability equation recovery, even with limited training data.