acceptodds
Under review as a conference paper at ICLR 2027

Outside the Dictionary: LLM-Guided Equation Discovery Is Limited by What It Proposes

Abstract

Symbolic regression, including its recent language-model-guided variants, searches a fixed dictionary of primitives, and fails when the governing law needs a primitive outside that dictionary or a quantity that was never measured. Such failures are invisible to in-distribution evaluation: we prove that on a bounded observed domain, at every sample size and every positive noise level, some in-dictionary expression is statistically indistinguishable from the true law. We therefore score hypotheses by extrapolation, and use this to measure what limits a standard discovery loop in which a language model proposes closed-form laws that are then fitted, selected, and tested by active probing. We contribute Crisis-60, 60 procedurally generated environments in which a low-order polynomial fits the observed data almost perfectly but a law counts as recovered only if it extrapolates to a disjoint region with R^2 > 0.999; its control category, whose law is itself a polynomial, is solved at 100.0% by polynomial regression against 0.0% elsewhere. We also contribute the Epicycle Index, a ratio of extrapolation error to description length anchored at the incumbent theory, and a decomposition of recovery into proposal coverage, whether a correct law was ever proposed, and selection efficiency, whether it was then chosen. Selection is nearly saturated: it returns a correct law on 98.0% of the instances where one was proposed, so a perfect selector would gain only 1.1 points, whereas on 3 of 15 structural families no correct law is ever proposed. This accounts for why the loop's parsimony penalty and active probing contribute nothing measurable (95% upper limits 0.6 and 6.4 points), and the same limit holds for LLM-SR (Shojaee et al., 2025), whose representation and search differ from ours. Under observational noise the limit moves to selection, whose efficiency falls to 14.3%. The loop recovers 53.3% of laws (95% CI 42.8 to 63.9) against 23.9% for genetic symbolic regression and 31.1% for LLM-SR at matched budget; with five times the candidates LLM-SR reaches 51.7%, so the loop's advantage is efficiency rather than attainable recovery. Because the decomposition needs only the candidates a search already scores, it tells the builder of any such system which half is worth improving: here, the proposal distribution. Code, benchmark, and all logged runs are available at https://anonymous.4open.science/r/outside-the-basis-1867.

Primary area:
datasets and benchmarks
Keywords:
symbolic regression, out-of-domain generalization, benchmark, large language models, hypothesis discovery, description length, scientific law recovery, epicycle index, LLM-SR, genetic programming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4eihOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of LLM-guided symbolic regression in discovering equations beyond an initial function basis. It introduces CRISIS-60, a benchmark designed to distinguish in-distribution fitting from out-of-distribution extrapolation, along with the Epicycle Index and a proposal–selection decomposition. Experiments show that the discovery loop achieves 53.3% recovery, with proposal coverage rather than selection efficiency being the primary bottleneck in the noise-free setting.

Critical strengths

The following two points offer some new perspectives, though I believe they are flawed—a matter I will address later.

  1. Benchmark design. CRISIS-60 deliberately constructs scenarios where incorrect equations achieve excellent in-distribution fitting but fail on OOD data, providing a controlled setting for investigating extrapolation failures in symbolic regression.

  2. Systematic failure analysis. The proposal–selection decomposition provides useful insights into where equation discovery fails. The authors support their findings with extensive ablation studies, including negative results for active probing and complexity regularization, which help identify promising directions for future improvements.

Critical weaknesses
  • Benchmark Limitations. Existing benchmarks such as LLM-SRBench have already evaluated OOD generalization and symbolic accuracy. Although CRISIS-60 deliberately constructs tasks with high ID accuracy but poor OOD performance, its additional value over existing benchmarks remains unclear. Moreover, the benchmark mainly contains simple, low-dimensional equations.
  • Proposal Coverage. Metric Proposal Coverage is defined using an OOD R2R^2 threshold rather than Symbolic Accuracy (SA). Since the goal is to assess whether LLMs can propose correct equation structures, an SA-based metric would be more appropriate. The current metric cannot distinguish between failures to generate correct structures and failures in subsequent coefficient optimization, potentially misrepresenting the actual proposal capability.
  • Limited Evaluation of Agentic SR Approache. Although the authors investigate different Qwen model sizes and iterative feedback, they do not sufficiently evaluate different model families, prompting strategies, or modern agentic SR approaches. Existing mechanisms such as reflection, repair, and adaptive agent workflows could potentially improve proposal coverage. Even relatively simple prompt modifications might alleviate the identified bottleneck, making it unclear whether the conclusions generalize beyond the specific discovery loop studied.

Overall, this paper doesn't really feel like the product of "auto/vibe" research, so I gave it a score of 2.

Questions for authors
  1. I suggest replicating the Coverage–Selection analysis on existing benchmarks, such as LLM-SRBench. The conclusions would be more compelling if the same phenomenon were observed across other benchmarks—especially since LLM-SRBench also distinguishes between in-distribution (ID) and out-of-distribution (OOD) data.
  2. The authors have already compared their approach against Best-of-N, Residual Feedback, Iteration, and LLM-SR, so it cannot be said that ablation studies are entirely absent. However, further testing could be conducted, including:
  • Prompts that explicitly encourage structural diversity (e.g., by requesting solutions that differ significantly in structure from those already in the existing set).
  • Agent-based approaches involving reflection or repair (I have encountered relevant literature on this, for instance, in ICML proceedings).
  • Evaluations using different LLM families—beyond just Qwen—potentially including some closed-source models. Note that CRISIS-60 was specifically designed to feature a scenario where performance is nearly optimal for ID data but fails significantly for OOD data.
  1. I suggest adjusting the scope of ID observations and the distance of OOD extrapolation to examine the following: Does the "Coverage" metric change significantly? Does the assessment of bottlenecks within the "Proposal–Selection" process shift? Are there specific observation ranges where structural discovery fails particularly severely? Such an analysis would further demonstrate that the benchmark’s conclusions are not merely artifacts of the specific data ranges chosen by the researchers.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4eih (1/2)

Thank you for listing the experiments that would change your view. We ran all of them.

W1. What CRISIS-60 adds over LLM-SRBench, and simple equations. We ran LLM-SRBench and compared the two benchmarks directly (Appendix L.4). A polynomial fits the observed data with R2R^2 above 0.99 on all 60 CRISIS-60 environments, but on only 23 of 158 LLM-SRBench tasks. The median R2R^2 is 0.9996 for ours and 0.8142 for theirs. A near-perfect fit inside the data range followed by failure outside it happens on 48 of 60 of our environments and on 15 of 158 of theirs. So on LLM-SRBench a method can fail because the data is hard to fit, or because the right building blocks are missing. On CRISIS-60 only the second reason applies, because every environment is easy to fit inside the observed range. This is also why the polynomial solves our in-basis control 100% of the time and every other category 0% of the time. We see the two benchmarks as complementary. You are right that our environments are small, with at most three variables, and the Limitations section says so. That is one reason we also ran LLM-SRBench.

W2. Coverage should use symbolic accuracy. You are right. The old coverage could not tell a formula that was never proposed from one that was proposed and badly fitted. We now report four kinds of coverage (Table 2, Appendix L.1). For the full loop, coverage after fitting is 54.4%. After refitting every proposed formula with 256 restarts it is 57.8%. Symbolic coverage is 48.3%. Coverage by any formula that could express the law at all is 60.0%. Recovery is 53.3%. So of the 46.7 points lost, 42.2 are formulas never proposed, 3.4 are formulas proposed but badly fitted, and 1.1 are selection mistakes. With your symbolic criterion, selection never fails. Whenever a symbolically correct law was in the pool, the method returned a symbolically correct law. That held on 87 of 87 instances for the full loop, 84 of 84 for the iterated loop, 53 of 53 for best-of-N and 48 of 48 for LLM-SR.

W3 and Q2. Other models, prompting and agentic methods. We ran all three of your suggestions (Section 7.4, Appendix L.3).

  • We added Gemma-3-27B-it, Mistral-Small-3.2-24B, Llama-3.3-70B-Instruct and the closed Gemini-2.5-Flash alongside Qwen3.5-35B-A3B. Recovery ranges from 46.7% to 61.7%, and selection efficiency from 96.6% to 98.2%. The five models rank in the same order on recovery and on coverage. So a better model helps by proposing more correct formulas. The full loop beats best-of-N for every model, by 15.6 to 23.3 points. No model ever proposed the image-charge law.
  • Our original prompt already asked for at least one structurally different idea in each round. We added a stronger version that forces each new formula to differ clearly from all earlier ones. It raised recovery by 3.9 points, which is not significant (p=0.118p = 0.118).
  • Adding a reflection step after each round, which looks at where the current best formula fails and suggests fixes, raised recovery by 5.6 points (p=0.021p = 0.021). An agent pipeline with separate explorer, refiner and checker roles, built on the design of SR-Scientist and DrSR, raised it by 5.0 points (p=0.035p = 0.035). We built this pipeline ourselves and did not run their released code. The paper says so.

In all three cases recovery rose by 0.6 points less than coverage, and selection efficiency stayed at 97.2%. So these methods help by proposing more correct formulas, as the decomposition predicts, and about 40% of the benchmark is still unsolved. If you had a specific reflection or repair method from ICML in mind, we would be glad to add it.

AuthorsOpenReview ↗

Response to Reviewer 4eih (2/2)

Q1. The same analysis on LLM-SRBench. We ran the full loop, best-of-N and LLM-SR on all 128 LSR-Synth problems and on 30 LSR-Transform problems (Appendix L.4). On LLM-SRBench's own test split, selection efficiency on LSR-Synth is 92.6% for the full loop, 95.0% for best-of-N and 92.0% for LLM-SR. On LSR-Transform it is 92.9% and 92.3%. So the main finding also holds on another benchmark. To check our setup, our LLM-SR run reaches an Acc0.1\mathrm{Acc}_{0.1} of 58.6% on LSR-Synth. The LLM-SRBench paper reports LLM-SR results with Llama-3.1-8B that average 58.4% across the four domains. Our symbolic accuracy uses an exact SymPy check, which is different from LLM-SRBench's judged symbolic accuracy, so those two numbers should not be compared.

Q3. Size of the observed range and distance to the test region. We made each environment's observed window smaller or larger by a factor ss, and moved the test region closer or further by a factor gg (Appendix L.5).

  • A wider window helps. Recovery goes from 41.7% at s=0.5s = 0.5 to 48.3%, 53.3% and 58.3% at s=1.5s = 1.5, a significant trend (p=0.009p = 0.009).
  • The distance to the test region makes no measurable difference. Recovery is 55.0% at g=0.5g = 0.5 and 51.7% at g=2g = 2 (p=0.238p = 0.238).
  • Selection efficiency stays between 96.0% and 97.2% at every setting. So whatever the window size, recovery is still limited by proposals.
  • At every setting the polynomial still solves the in-basis control and nothing else.
  • dark_halo and rel_mass are only recovered once the window is widened to s=1.5s = 1.5, because their telltale features sit near the edge of the window. image_sphere and van_der_waals are not recovered at any setting.

The original benchmark tells the same story. Across its 13 out-of-basis families, the distance to the test region ranges from 0.01 to 1.43 window widths, and it does not predict lower recovery (ρ=+0.42\rho = +0.42, p=0.15p = 0.15). In fact the families tested furthest out, the saturating kinetics, are among the ones recovered most often.

AuthorsOpenReview ↗

Reply to Reviewer 4eih

Thank you again for your review. You wrote that the conclusions would be more convincing if the same result appeared on other benchmarks, with other model families, and with stronger prompting and agent methods. We ran each of these, and the result held every time.

  • Another benchmark (your Q1). On LLM-SRBench, selection picks a correct formula 92% to 95% of the time when one has been proposed, close to the 98% on our benchmark. Recovery there is limited by proposals too (Appendix L.4).
  • Other model families, including a closed model (your Q2). With Gemma, Mistral, Llama, Gemini-2.5-Flash and Qwen, selection efficiency stays between 96.6% and 98.2%. The five models rank in the same order on recovery and on coverage, so a better model helps only by proposing more correct formulas (Section 7.4).
  • Diversity prompting, reflection and agent methods (your Q2). Reflection raises recovery by 5.6 points (p = 0.021) and an agent pipeline by 5.0 (p = 0.035). Diversity prompting adds 3.9, which is not significant. In every case the gain comes from proposing more correct formulas, which is what the paper predicts (Appendix L.3).
  • Symbolic accuracy instead of fitted R² (your W2). Under your criterion, selection picks the correct law on 87 of 87 instances (Appendix L.1).
  • Observed range and test distance (your Q3). The main finding holds at every window size and every distance we tried (Appendix L.5).

All of these are in the revised PDF. You described your soundness concerns as minor or fixable, and we believe these results now address them. If so, we would be very grateful if you would consider raising your score. If anything is still missing, please tell us today and we will answer before the deadline.

Thank you for the detailed response and the additional experiments. The new analysis clarifies the roles of proposal generation and fitting, and the results on LLM-SRBench are helpful. I still have some reservations about the benchmark’s contribution beyond existing evaluations and how broadly the conclusions apply. On balance, I’m keeping my original rating.

Reviewer_4pfMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors make the claim that an issue with SR methods is not their ability to identify correct forms that generalize OOD, but their ability to propose the correct form in the first place during the search. They construct (in a somewhat unclear manner) a dataset, Crisis-60, to examine this question. They test several baselines and SR systems, including some LM-guided approaches, on the dataset and find that when a correct answer is identified, it is almost always selected using a simple, complexity-based rule. However SR methods often do not suggest the correct form. Their results (Table 1) suggest this is robust across several approaches and point to improving the proposal side of SR rather than the selection.

Critical strengths
  1. It seems useful to separate candidate proposals and candidate selection.
  2. The release of the accompanying code was very helpful in evaluating the work.
Critical weaknesses

To be clear, my biggest issue with the paper is clarity and presentation but there are other concerns.

  1. The paper is very unclear and poorly presented.
    1. Please see my reading comments for more examples here. The abstract does not describe the experiments, there are not enough examples, the work is not well motivated, etc...
  2. The dataset construction is not clearly described and important details are missing, even from the supplementary materials. This makes it difficult to fully assess the soundness of the work. I am not convinced that the dataset is very meaningful.
    1. Several of the examples (I wish I could post a picture) show essentially what looks fairly quadratic for the train/validation region, and then are asked to guess a bump that only appears in the test region. This is not even possible to do without priors.
    2. I spot checked these and many seemed off. I would guess that only about 60% seem possible. The rest feel very difficult or totally unreasonable given the minimal priors of the fitting.
    3. The rationale for choosing these equations and sampling regions is unclear.
  3. The theoretical contribution feels disconnected and not relevant to the main results.
  4. The proposer method is not adequately differentiated from the LLM-SR baseline presented. What drives its success? It is stated that there is no novelty in various places (including on this loop) but then what is the novel work?
    1. It is strange to me that the LLM proposer and LLM-SR is given the initial form whereas no analogous information is given to PySR.
    2. I am concerned that the improved results seen by the LM methods are just because the LM has priors for LM-generated equation structures (if that is how the dataset was created). Not because they are actually doing something smarter when it comes to traversing a more general space of possibilities.
  5. A critical check for their main claim requires knowing if a model proposed the correct form but coverage for matching is done after constant fitting so a correct equation with incorrectly fitted constants is counted as an unsuccessful proposal.
    1. This may or may not have mattered in the results but it seems prone to issues.
    2. Supplying the exact correct form for seed 42, environment 14 to their scorer fails with six fitting restarts but succeeds with 64, so it is definitely possible that the counts could be off.

Below are additional notes from my reading of the paper. These are not critical but the authors may find them helpful.

  • The introduction does not do a very good job of explaining what is being done. Every paragraph after the first is just very unclear, introducing terminology and jargon without clear purpose.
    • “Ptolemaic failure mode” and “Epicycle Index”
    • Why begin “We make four contributions” with a long aside about what those contributions are not. Weird choice.
    • Describing your dataset as “60 procedurally generated environments over 15 structural families” means nothing if you don’t even say what the dataset contains. Nobody knows what an environment or structural family is with much specificity at this point in the paper.
    • By the end of the introduction, all I have learned is that there is some theorem that tries to say distinguishing parameter interpolation from basis extension (whatever that means) is difficult (impossible?), there is going to be a dataset of something, there is some weird metric to be introduced, and there will be some argument that “proposal coverage” is what can be improved. I would prefer to have learned more.
    • I do not understand what the purpose of this distinction between parameter interpolation and basis extension is.
  • In section 4, now we are saying Crisis-60 has “60 environments in 5 categories, each with 2 to 4 structural families” which is a confusing presentation. I thought there were 15 “structural families” (whatever that is, we have not been told yet).
    • How is this dataset generated? How were these categories selected? Examples of each category would be very helpful.
    • Results are presented in this section (4) but the methods are never introduced. I see that one is a polynomial baseline that does very well. After this it says “[...] pooling every method in this study [...]”, but we have not been told the methods yet. I should not need to go to the appendix to understand numbers quoted in the main text.
  • Table 1 refers to PySR with niterations 200 as “5x budget” but Table 11 refers to it as “high budget”. I had to go to the appendix to find the niterations used as it is not listed in the main text or the Table 1 caption.
  • Figure 1 could use some work or should be moved later. The numbers presented are only understandable to people who have already read most of the paper.
  • There doesn’t seem to be any information on how the equations for the dataset were chosen. I assume they were either selected from textbooks or generated by AI or something but this severely limits my ability to evaluate the construction of the dataset and therefore the results. This is a critical piece of methodology that should have been in the abstract.
  • Without priors, OOD fitting is totally impossible. What are the priors here? They are not clearly stated. Any function can do anything after a certain point (e.g., imagine cutting a sigmoid at or before the inflection point). How do we know the data chosen was meaningful?
    • The most relevant (explicitly stated) prior I see is the complexity/error selection rule, which assumes that a simpler equation (in terms of tree size) on the train/validation region will be a better fit on the test region.
Questions for authors
  1. How was the dataset constructed and why is it meaningful?
  2. It seems like coverage is measured after fitting the constants and could have led to miscounts. Please comment on the exact methodology and robustness of results.
  3. The LLM methods get the incumbent equation while PySR does not. How do we know the advantage observed is due to the search rather than this additional information?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4pfM (1/3)

Thank you for reading the paper closely and for running our code. Two of your findings were correct, and we changed the paper because of them. We answer the soundness questions first and the presentation points after.

W5 and Q2. Coverage measured after fitting. You are right, and your example was a real case. Seed 42, environment 14 is a dark_halo instance. We gave the fitter the true formula for every instance. With six restarts it reaches the criterion on 168 of 180 instances. It misses 6 of 12 dark_halo and 6 of 18 rel_mass. With 64 restarts it reaches all 180 (Appendix L.2). So we saved every formula the model proposed and refitted each one with 256 restarts. Recovery is now split into three parts. Was a correct formula proposed, was it fitted, and was it chosen (Section 7.2, Table 2)?

  • Fitting does lose some correct formulas, but it is the smallest of the three losses. Fit efficiency is at least 94.2% in every setup.
  • With 64 restarts instead of 6, recovery rises from 53.3% to 56.7% (+3.3+3.3 points, p=0.070p = 0.070), almost all of it in dark_halo and rel_mass. With 256 restarts it is 57.2%, and the most fitting could ever reach is 57.8%. So more restarts beyond 64 gain almost nothing.
  • The submission said no correct law was ever proposed for dark_halo. That was wrong. The correct formula was proposed on 4 of its 12 instances and then lost in fitting. The revised paper says so.
  • For image_sphere and van_der_waals, no formula proposed on any instance could express the law at all. Those really are proposal failures.

W2.1 and W2.2. Instances that look impossible. Every one of the 180 instances can be solved from the observed data alone. If we give the fitter the true formula with free constants and 256 restarts, it recovers the law from the 80 training points, with test R2R^2 above 0.999999 on every instance (Appendix L.2). Take the bump family dark_halo. The bump's peak is between 5 and 6, and the observed window ends at 4.5. But the bump has width one, so at the edge of the window it has already risen to between 10% and 78% of its peak. Its rising side is in the data. The fitter finds it, but only with enough restarts, which is the same issue as your W5. If you tell us which environments looked impossible in your spot check, we will check each one and report back here.

W2, W2.3, N6 and Q1. How the dataset was built and why it is meaningful. Section 4 now defines the terms. An environment is one law with randomly drawn constants, for example Michaelis-Menten kinetics with random VmV_m and KK. A structural family is the shape of that law. The 15 laws are textbook laws. They include Yukawa screening, Michaelis-Menten and Hill kinetics, relativistic velocity addition and relativistic mass, the van der Waals equation, and the image charge outside a grounded sphere. Table 4 lists every family with its true formula, the old theory it replaces, and its observed and test regions. Every family meets three conditions.

  • There is an established old theory that fits part of the data and fails elsewhere.
  • A low-order polynomial fits the observed data with R2R^2 of at least 0.99.
  • The test region is where the law's key feature shows up, such as a cutoff, a limit it approaches, or a pole.

Constants are drawn uniformly from a fixed range for each family. Each instance has 80 training points and 60 validation points drawn from the observed region, and 150 test points on a grid. The benchmark's control shows it measures what we intend. Polynomial regression solves the in-basis category 100% of the time and every other category 0% of the time, even though it fits the observed data of all five categories equally well.

N7. What the priors are. We agree that you cannot extrapolate without priors. That is exactly what Theorem 1 says. Our priors are the allowed operations, the selection rule of Equation 3 that prefers short formulas, and the old theory. The real question is whether these priors pick out the true law, so we measured it. On 173 of 180 instances, the true law has a lower description length on the observed data than every wrong formula tried for that instance anywhere in the study. The median margin is 124.6 nats (Appendix L.2). The seven exceptions are four dark_halo instances and three vel_add instances.

AuthorsOpenReview ↗

Response to Reviewer 4pfM (2/3)

W4.1 and Q3. The language models get the old theory but PySR does not. We tested this both ways on all 180 instances (Appendix L.3).

  • When we remove the old theory from the loop's prompt and candidate pool, recovery drops from 53.3% to 47.8%, a loss of 5.6 points (p=0.031p = 0.031). So the old theory does help, and the paper now says so.
  • When we give the old theory to PySR as a starting guess, its recovery changes by only 1.7 points, to 25.6% (p=0.453p = 0.453). At five times the budget it reaches 26.7%.

Even without the old theory, the loop is 22.2 points ahead of PySR with it. So the extra information does not explain the gap between the loop and PySR.

W4.2. Does the model just recognise familiar formulas? We renamed every variable to x0,x1,…x_0, x_1, \dots and removed the sentence saying the data comes from a physical system. Recovery dropped by 4.4 points, from 53.3% to 48.9%. That drop is not significant (p=0.077p = 0.077), and the loop stays 25.0 points ahead of PySR. Two checks in the original paper point the same way. In the probe of Appendix E, the model is asked for the law without any data, and it gets laws right in only one of five categories. In the random-formula control, the same loop is fed random formulas instead of model proposals, and it recovers only 6.1%. So familiarity with textbook formulas helps a little, as the renaming test shows, but it does not explain the results.

W4. What makes the loop work, and what is new. We do not claim the loop is new. The revised introduction now says this after listing the contributions instead of before. The contributions are the benchmark, the Epicycle Index, and the decomposition with its measurements. Section 7.1 measures what makes the loop work by removing one part at a time.

  • The language model adds 47.2 points over random formulas in the same loop.
  • Telling the model what failed in earlier rounds adds 19.4 points over asking for all proposals at once.
  • The simplicity penalty and active probing add nothing we can measure.

W3. The theory feels disconnected. We have now tied it to a measured result (Section 7.5, Appendix L.6). The selection rule picks a wrong formula when that formula fits the validation data almost as well as the right one and is shorter. How well the two can be told apart depends on how far apart they are, compared with the noise. That is the same quantity Theorem 1 bounds. Using only the measured gaps, we predict selection efficiency of 98.5%, 96.2%, 91.8%, 74.5% and 16.1% at the five noise levels. The measured values are 97.1%, 96.9%, 93.4%, 76.1% and 14.3%. So the theorem predicts the paper's noise result.

AuthorsOpenReview ↗

Response to Reviewer 4pfM (3/3)

W1 and the reading notes. Presentation. We agree with you. The revision changes the following.

  • The introduction now starts with an example. A polynomial fits Michaelis-Menten data as well as the true law inside the observed range, but the two behave differently outside it. We define parameter interpolation and basis extension using this example.
  • The contributions paragraph now starts with the four contributions. The note that the loop itself is not new comes after them.
  • Environments and families are defined where they first appear, in the introduction and again in Section 4, with an example.
  • The mismatch between 13 and 15 families came from an out-of-date appendix table, which we have now rebuilt. There are 15 families in five categories, with two to four families in each.
  • The sentence that pools results from every method moved from Section 4 to Section 6, after the methods are introduced.
  • The Table 1 caption now gives PySR's budgets, niterations 40 by default and 200 for the five times larger budget. The settings table uses the same name.
  • The second and third paragraphs of the introduction are rewritten in plainer words. We now explain "Ptolemaic failure mode" where we first use it. It means a fit that captures where the data happened to be sampled rather than how the system works, like the extra circles once used to fit planet positions. A new sentence explains why the question matters. Systems such as LLM-SR report their biggest gains outside the training range, and knowing whether they are held back by what they propose or by what they choose tells builders where to put their effort.
  • Figure 1 is redrawn. Panel (a) is now the running example, drawn from a real CRISIS-60 environment. A polynomial fits the observed data with R2=1.0000R^2 = 1.0000 and fails outside it, while the true law keeps working. Panel (b) keeps the coverage and recovery bars, and the caption now explains both, so the figure can be read before Section 7.
Reviewer_92qKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the out-of-dictionary problem in symbolic regression, where the selected equations tend to generalize poorly in out-of-distribution(OOD) settings. The authors use an iterative LLM-based framework and test it on the proposed CRISIS-60 dataset to evaluate the performance using the Epicycle Index and recovery rate, and compare it against several existing methods. This paper finds that, in noise-free settings equation recovery is primarily limited by the generation of successful candidate equations rather than their final selection.

Critical strengths

1, The paper addresses an important problem in symbolic regression, where discovered equations can fit in-distribution (ID) data well but fail to generalize to out-of-distribution (OOD) settings. The paper also identifies the limitations of a fixed primitive dictionary as a possible cause of this failure.

2, The paper introduces CRISIS-60, an additional benchmark designed to evaluate equation discovery under challenging extrapolation settings.

Critical weaknesses

1, The paper defines the distinction between outside-the-dictionary discovery and OOD extrapolation, but the experiments do not sufficiently demonstrate this difference. Could the authors clarify how this experiment setting fundamentally differs from conventional OOD generalization in symbolic regression?

2, The proposed coverage metric measures whether at least one fitted candidate achieves the required OOD accuracy, rather than whether the LLM generates the correct equation structure. This makes it difficult to conclude that the primary bottleneck lies in generating correct equation structures rather than fitting their parameters.

3, The novelty and insight of the proposal coverage and selection efficiency decomposition is unclear. First, coverage is based on fitted OOD R^2 rather than structural correctness. Second, near-perfect selection efficiency in noise-free settings and its degradation under noise are largely expected. More importantly, the recommendation to prioritize proposal generation offers limited novelty, as existing LLM-based symbolic regression methods already focus heavily on improving candidate generation. The authors should clarify what additional insights this decomposition provides beyond these observations.

Minor:

The widely used LLM-SRBench is not included in the evaluation. The authors should explain why this benchmark was excluded or why it is not suitable for this study.

Questions for authors

See Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 92qK

Thank you. Your main objection is correct, and we have redone the measurement.

W2 and W3a. Coverage counts a fitted formula, not a correct structure. The revision now separates the two (Section 7.2, Table 2, Appendix L.1). We saved every formula the model proposed and refitted each one with 256 restarts. Recovery is now the product of three parts. Was a correct formula proposed, was it fitted, and was it chosen?

  • For the full loop, structural coverage is 57.8%, fit efficiency 94.2% and selection efficiency 98.0%.
  • Across all eight setups, fit efficiency is at least 94.2%, and selection efficiency is between 93.9% and 100%.
  • With a symbolic criterion, every noise-free setup that records its candidates returns a correct law whenever one is in the pool. For the full loop that is 87 of 87 instances.

So the main bottleneck is still proposing correct formulas, and this now rests on correct structure, not on fitted test accuracy. One correction came out of this. For dark_halo, which the submission said was never proposed, the correct formula was proposed and then lost in fitting. The revised paper corrects this.

W1. How this differs from ordinary OOD generalisation. Three measurements separate them.

  • The in-basis control is built the same way as every other category, with an observed region and a separate test region. The polynomial solves it 100% of the time and every other category 0% of the time. So testing outside the data range does not by itself cause failure. A law outside the polynomial's building blocks does.
  • In the original benchmark, the distance to the test region ranges from 0.01 to 1.43 window widths across the 13 out-of-basis families. Larger distance does not mean lower recovery (ρ=+0.42\rho = +0.42, p=0.15p = 0.15). The families tested furthest out are among the ones recovered most often.
  • Moving the test region twice as far or half as far has no measurable effect on recovery (p=0.238p = 0.238, Appendix L.5).

If this were ordinary OOD generalisation, distance would be what makes it hard. Here, what makes it hard is whether the law can be built from the available operations.

W3b. High selection without noise, and its drop under noise, are expected. We think the first part is less obvious than it seems. Selection only sees the observed data, and Theorem 1 says that with noise no test on that data can tell the true law from a close imitation. In work on repeated sampling for reasoning, the opposite is found, and choosing the right answer is the hard part. Without noise, selection works because a correct formula fits the validation data far better than any imitation, and noise is exactly what removes that advantage. The revision now measures this. Predicting selection efficiency from the measured gaps between formulas matches the measured values to within 1.8 points at all five noise levels, including the drop from 97.1% to 14.3% (Section 7.5, Appendix L.6). So the drop is not just expected, it is predicted by the theorem.

W3c. Focusing on proposals is not new. We agree that many methods work on generating better formulas. But many also work on choosing, with simplicity penalties, verifiers, stopping rules and selection objectives. The decomposition tells you which side is worth improving in which setting, and the revision tests this in a way that could have failed (Appendix L.3).

  • Without noise, three proposal improvements raise recovery by 3.9 to 5.6 points, each within 0.6 points of its coverage gain. Six selection rules run on the same candidates differ by only 1.7 points.
  • At σ=10−1\sigma = 10^{-1}, the selection rules differ by 14.2 points, and the best one reaches 15.9% against 2.5% for ours. The best proposal improvement gains only 0.8 points there.

A builder can tell which situation their system is in from the candidates it already records, at no extra cost.

Minor. LLM-SRBench. We now run it (Appendix L.4). On its test split, the full loop and LLM-SR choose the right formula 92.0% to 92.9% of the time when one is available, and best-of-N 95.0%. So the finding holds there too. The same appendix shows how the two benchmarks differ. A polynomial fits the observed data with R2R^2 above 0.99 on all 60 of our environments but on only 23 of 158 LLM-SRBench tasks. On LLM-SRBench a method can fail because the data is hard to fit, as well as because the right building blocks are missing. CRISIS-60 is built to isolate the second case.

Thank you for the detailed response. The rebuttal have addressed most of my concerns. I have therefore raised my rating to 3. However, while my soundness concerns have been largely resolved, I still find the novelty of the decomposition and its implications for method design somewhat limited. This prevents me from assigning a higher rating.

AuthorsOpenReview ↗

Thank you for the updated rating

Thank you for reading our rebuttal and raising your rating. We are glad our revision resolved your soundness concerns, and we appreciate your constructive feedback throughout the review process.

AuthorsOpenReview ↗

Reply to Reviewer 92qK

Thank you for raising your score, and for saying clearly what holds it back. We would like to make the implications for method design concrete. Each point below is a measured result in the paper, not a suggestion.

  1. Do not tune the selection step when data are clean. Six different selection rules applied to the same candidates differ by only 1.7 points, and a perfect selector would add only 1.1. The simplicity penalty and active probing, two common ingredients, add nothing measurable.
  2. Spend effort on proposals, and on feedback rather than repetition. Every change that raised recovery in our study did so by proposing more correct formulas. This held for stronger models, reflection and an agent pipeline. Asking for proposals over several rounds without telling the model what failed is no better than asking once (10.0 points worse on 30 instances). Telling it what failed adds 26.7 points (p = 0.039).
  3. Bigger buffers are not the lever. At the same budget, LLM-SR's experience buffer does no better than simply sampling the same number of formulas and keeping the best one. To match our loop it needs 160 formulas and 74 model calls per problem, against 27 and 7.
  4. Feedback only pays above a model size. With a 4B model the feedback loop is 8.3 points worse than plain sampling. At 35B it is 25.0 points better.
  5. With noise, the advice flips. At the highest noise level, selection rules differ by 14.2 points, while the best proposal improvement adds only 0.8.
  6. Any system can tell which case it is in, at no cost. The two numbers come from candidates it already scores, so they can be reported next to recovery in any SR paper.

None of these follows from the decomposition being true in principle. Each was measured, and items 1, 2 and 5 test a prediction that could have failed. We hope this shows the decomposition changes what one should build, and we would be grateful if you would consider whether it supports a higher score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

Thank you to all three reviewers. All of you raised the same main concern. We measured coverage after fitting the constants, so a correct formula that was badly fitted looked the same as a formula that was never proposed. You are right. We ran new experiments to separate the two, and they show that one claim in the submission was wrong. We have uploaded a revised PDF. It uses the extra tenth page, and the new experiments are in Section 7 and Appendix L. Here is what changed.

  1. We now measure proposing, fitting and choosing separately. We saved every formula the model proposed and refitted each one with 256 restarts. Recovery is now the product of three parts. Did the model propose a correct formula, did the fit work, and did selection pick it (Section 7.2, Table 2, Appendix L.1)? Fitting loses the least. Fit efficiency is at least 94.2% in every setup, and selection efficiency is between 93.9% and 100%. So recovery is still limited by what gets proposed. The correction concerns dark_halo. The submission said its correct formula was never proposed. In fact it was proposed on 4 of 12 instances and then lost because six restarts were not enough to fit it. The revised paper says this in Section 7.2 and Appendix D.

  2. Every instance can be solved from the observed data. If we give the fitter the true formula with free constants, it recovers the law on all 180 instances using the training data alone. Using the paper's own selection rule (Equation 3), the true law is also the shortest good description on 173 of 180 instances (Section 4, Appendix L.2).

  3. The result holds more widely. We added four more model families, including the closed model Gemini-2.5-Flash. We also added three ways of improving proposals, runs without the old theory in the prompt, runs with renamed variables, LLM-SRBench, and a sweep over the size of the observed window and the distance to the test region (Section 7.4, Table 3, Appendix L.3 to L.5). Selection efficiency stays between 92.0% and 98.2% in all of them.

  4. We tested a prediction that could have failed. Without noise, better proposals should raise recovery by about as much as they raise coverage, and the choice of selection rule should hardly matter. With noise, it should be the other way round. Both held. Three proposal improvements raised recovery to within 0.6 points of their coverage gain. Six selection rules run on the same candidates differed by only 1.7 points without noise, but by 14.2 points at σ=10−1\sigma = 10^{-1} (Appendix L.3).

  5. The theorem now predicts a result. From the measured gap between competing formulas, we predict how often selection picks the right one at each noise level. The prediction matches the measured drop from 97.1% to 14.3% to within 1.8 points at all five noise levels (Section 7.5, Appendix L.6).

  6. The paper is easier to read. Figure 1 now shows a real example from the benchmark. The introduction starts from that example and uses plainer words. The contributions come first in their paragraph. Section 4 explains what an environment and a family are and how the families were chosen. Methods are introduced before results, and PySR's budgets are in the Table 1 caption.

  7. We fixed an out-of-date table. The family table in the appendix still listed a family we had replaced and was missing two we had added. That is why it showed 13 families while the text said 15. It is now built from the families we actually ran, and it shows each family's observed and test regions (Table 4, Appendix B).

To make room, the compute analysis moved to Appendix C.1. Its main result is still stated in Section 7.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.