- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of LLM-guided symbolic regression in discovering equations beyond an initial function basis. It introduces CRISIS-60, a benchmark designed to distinguish in-distribution fitting from out-of-distribution extrapolation, along with the Epicycle Index and a proposal–selection decomposition. Experiments show that the discovery loop achieves 53.3% recovery, with proposal coverage rather than selection efficiency being the primary bottleneck in the noise-free setting.
Critical strengths
The following two points offer some new perspectives, though I believe they are flawed—a matter I will address later.
-
Benchmark design. CRISIS-60 deliberately constructs scenarios where incorrect equations achieve excellent in-distribution fitting but fail on OOD data, providing a controlled setting for investigating extrapolation failures in symbolic regression.
-
Systematic failure analysis. The proposal–selection decomposition provides useful insights into where equation discovery fails. The authors support their findings with extensive ablation studies, including negative results for active probing and complexity regularization, which help identify promising directions for future improvements.
Critical weaknesses
- Benchmark Limitations. Existing benchmarks such as LLM-SRBench have already evaluated OOD generalization and symbolic accuracy. Although CRISIS-60 deliberately constructs tasks with high ID accuracy but poor OOD performance, its additional value over existing benchmarks remains unclear. Moreover, the benchmark mainly contains simple, low-dimensional equations.
- Proposal Coverage. Metric Proposal Coverage is defined using an OOD threshold rather than Symbolic Accuracy (SA). Since the goal is to assess whether LLMs can propose correct equation structures, an SA-based metric would be more appropriate. The current metric cannot distinguish between failures to generate correct structures and failures in subsequent coefficient optimization, potentially misrepresenting the actual proposal capability.
- Limited Evaluation of Agentic SR Approache. Although the authors investigate different Qwen model sizes and iterative feedback, they do not sufficiently evaluate different model families, prompting strategies, or modern agentic SR approaches. Existing mechanisms such as reflection, repair, and adaptive agent workflows could potentially improve proposal coverage. Even relatively simple prompt modifications might alleviate the identified bottleneck, making it unclear whether the conclusions generalize beyond the specific discovery loop studied.
Overall, this paper doesn't really feel like the product of "auto/vibe" research, so I gave it a score of 2.
Questions for authors
- I suggest replicating the Coverage–Selection analysis on existing benchmarks, such as LLM-SRBench. The conclusions would be more compelling if the same phenomenon were observed across other benchmarks—especially since LLM-SRBench also distinguishes between in-distribution (ID) and out-of-distribution (OOD) data.
- The authors have already compared their approach against Best-of-N, Residual Feedback, Iteration, and LLM-SR, so it cannot be said that ablation studies are entirely absent. However, further testing could be conducted, including:
- Prompts that explicitly encourage structural diversity (e.g., by requesting solutions that differ significantly in structure from those already in the existing set).
- Agent-based approaches involving reflection or repair (I have encountered relevant literature on this, for instance, in ICML proceedings).
- Evaluations using different LLM families—beyond just Qwen—potentially including some closed-source models. Note that CRISIS-60 was specifically designed to feature a scenario where performance is nearly optimal for ID data but fails significantly for OOD data.
- I suggest adjusting the scope of ID observations and the distance of OOD extrapolation to examine the following: Does the "Coverage" metric change significantly? Does the assessment of bottlenecks within the "Proposal–Selection" process shift? Are there specific observation ranges where structural discovery fails particularly severely? Such an analysis would further demonstrate that the benchmark’s conclusions are not merely artifacts of the specific data ranges chosen by the researchers.
4 comments
Response to Reviewer 4eih (2/2)
Q1. The same analysis on LLM-SRBench. We ran the full loop, best-of-N and LLM-SR on all 128 LSR-Synth problems and on 30 LSR-Transform problems (Appendix L.4). On LLM-SRBench's own test split, selection efficiency on LSR-Synth is 92.6% for the full loop, 95.0% for best-of-N and 92.0% for LLM-SR. On LSR-Transform it is 92.9% and 92.3%. So the main finding also holds on another benchmark. To check our setup, our LLM-SR run reaches an of 58.6% on LSR-Synth. The LLM-SRBench paper reports LLM-SR results with Llama-3.1-8B that average 58.4% across the four domains. Our symbolic accuracy uses an exact SymPy check, which is different from LLM-SRBench's judged symbolic accuracy, so those two numbers should not be compared.
Q3. Size of the observed range and distance to the test region. We made each environment's observed window smaller or larger by a factor , and moved the test region closer or further by a factor (Appendix L.5).
- A wider window helps. Recovery goes from 41.7% at to 48.3%, 53.3% and 58.3% at , a significant trend ().
- The distance to the test region makes no measurable difference. Recovery is 55.0% at and 51.7% at ().
- Selection efficiency stays between 96.0% and 97.2% at every setting. So whatever the window size, recovery is still limited by proposals.
- At every setting the polynomial still solves the in-basis control and nothing else.
dark_haloandrel_massare only recovered once the window is widened to , because their telltale features sit near the edge of the window.image_sphereandvan_der_waalsare not recovered at any setting.
The original benchmark tells the same story. Across its 13 out-of-basis families, the distance to the test region ranges from 0.01 to 1.43 window widths, and it does not predict lower recovery (, ). In fact the families tested furthest out, the saturating kinetics, are among the ones recovered most often.
Reply to Reviewer 4eih
Thank you again for your review. You wrote that the conclusions would be more convincing if the same result appeared on other benchmarks, with other model families, and with stronger prompting and agent methods. We ran each of these, and the result held every time.
- Another benchmark (your Q1). On LLM-SRBench, selection picks a correct formula 92% to 95% of the time when one has been proposed, close to the 98% on our benchmark. Recovery there is limited by proposals too (Appendix L.4).
- Other model families, including a closed model (your Q2). With Gemma, Mistral, Llama, Gemini-2.5-Flash and Qwen, selection efficiency stays between 96.6% and 98.2%. The five models rank in the same order on recovery and on coverage, so a better model helps only by proposing more correct formulas (Section 7.4).
- Diversity prompting, reflection and agent methods (your Q2). Reflection raises recovery by 5.6 points (p = 0.021) and an agent pipeline by 5.0 (p = 0.035). Diversity prompting adds 3.9, which is not significant. In every case the gain comes from proposing more correct formulas, which is what the paper predicts (Appendix L.3).
- Symbolic accuracy instead of fitted R² (your W2). Under your criterion, selection picks the correct law on 87 of 87 instances (Appendix L.1).
- Observed range and test distance (your Q3). The main finding holds at every window size and every distance we tried (Appendix L.5).
All of these are in the revised PDF. You described your soundness concerns as minor or fixable, and we believe these results now address them. If so, we would be very grateful if you would consider raising your score. If anything is still missing, please tell us today and we will answer before the deadline.
Thank you for the detailed response and the additional experiments. The new analysis clarifies the roles of proposal generation and fitting, and the results on LLM-SRBench are helpful. I still have some reservations about the benchmark’s contribution beyond existing evaluations and how broadly the conclusions apply. On balance, I’m keeping my original rating.
Response to Reviewer 4eih (1/2)
Thank you for listing the experiments that would change your view. We ran all of them.
W1. What CRISIS-60 adds over LLM-SRBench, and simple equations. We ran LLM-SRBench and compared the two benchmarks directly (Appendix L.4). A polynomial fits the observed data with R2 above 0.99 on all 60 CRISIS-60 environments, but on only 23 of 158 LLM-SRBench tasks. The median R2 is 0.9996 for ours and 0.8142 for theirs. A near-perfect fit inside the data range followed by failure outside it happens on 48 of 60 of our environments and on 15 of 158 of theirs. So on LLM-SRBench a method can fail because the data is hard to fit, or because the right building blocks are missing. On CRISIS-60 only the second reason applies, because every environment is easy to fit inside the observed range. This is also why the polynomial solves our in-basis control 100% of the time and every other category 0% of the time. We see the two benchmarks as complementary. You are right that our environments are small, with at most three variables, and the Limitations section says so. That is one reason we also ran LLM-SRBench.
W2. Coverage should use symbolic accuracy. You are right. The old coverage could not tell a formula that was never proposed from one that was proposed and badly fitted. We now report four kinds of coverage (Table 2, Appendix L.1). For the full loop, coverage after fitting is 54.4%. After refitting every proposed formula with 256 restarts it is 57.8%. Symbolic coverage is 48.3%. Coverage by any formula that could express the law at all is 60.0%. Recovery is 53.3%. So of the 46.7 points lost, 42.2 are formulas never proposed, 3.4 are formulas proposed but badly fitted, and 1.1 are selection mistakes. With your symbolic criterion, selection never fails. Whenever a symbolically correct law was in the pool, the method returned a symbolically correct law. That held on 87 of 87 instances for the full loop, 84 of 84 for the iterated loop, 53 of 53 for best-of-N and 48 of 48 for LLM-SR.
W3 and Q2. Other models, prompting and agentic methods. We ran all three of your suggestions (Section 7.4, Appendix L.3).
In all three cases recovery rose by 0.6 points less than coverage, and selection efficiency stayed at 97.2%. So these methods help by proposing more correct formulas, as the decomposition predicts, and about 40% of the benchmark is still unsolved. If you had a specific reflection or repair method from ICML in mind, we would be glad to add it.