acceptodds
Under review as a conference paper at ICLR 2027

L2G-SR: Learning to Guide LLM-based Symbolic Regression

Abstract

Symbolic regression searches for an equation that explains observed data. Numerical error is commonly used to guide the search, but it alone can favor an accurate surrogate over a candidate with the right structure. We propose L2G-SR, a framework for LLM-based symbolic regression that guides search with a learned structural verifier. We construct structural supervision signal using tree edit distance (TED), accounting for commutative operators and alternative expression forms. A verifier first aligns numerical observations with the formulas, then learns to score a candidate's structural similarity to the target formula. At search time, the target formula is hidden: the verifier complements numerical error in parent selection. An analyst then uses verifier feedback and diagnostic tools to propose revisions, which a generator converts into refined expressions. On LLM-SRBench, L2G-SR achieves state-of-the-art numerical fit, equation level accuracy and TED across the evaluated LLM backbones and domains. L2G-SR demonstrates the value of learned structural guidance for LLM-based symbolic regression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Symbolic Regression, Large Language Models, LLM Agents, Scientific Discovery, Structual Guidance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vrZ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes L2G-SR, a method that uses a learned structural verifier to guide large language models for symbolic regression. The authors point out that traditional symbolic regression mainly uses numerical fitting error to guide search, but a lower prediction error does not mean that the candidate expression has the correct mathematical structure. To solve this problem, this paper first designs an improved Tree Edit Distance (TED), which provides supervision signals for structural differences between candidate and target expressions through constant abstraction, commutativity handling, and multiple algebraic representations. Then, the authors use large-scale synthetic data to train a multimodal verifier, enabling it to predict the structural distance between a candidate expression and the unknown target expression based on observed data, candidate expressions, and auxiliary regression features. Finally, the authors integrate the verifier into an iterative search framework composed of two LLM roles, Analyst and Generator, alternately using numerical fitting error and predicted TED to guide candidate selection, formula modification, and search.

Critical strengths

This paper focuses on the inconsistency between numerical fitting and structure recovery in symbolic regression, with a clear and practically meaningful research motivation. L2G-SR constructs structural supervision signals by improving TED, and combines formula tree addresses and auxiliary regression features to train a structural verifier, providing structural feedback beyond numerical error for search.

Critical weaknesses
  1. Existing work, Symbolic Regression via MDLformer-Guided Search: From Minimizing Prediction Error to Minimizing Description Length, has already used a learned model to estimate description length and used it as search guidance to improve the structural recovery ability of symbolic regression. L2G-SR directly predicts the TED between a candidate expression and the unknown target expression, and integrates it into an LLM-driven iterative search framework. Although the two differ in structural scoring methods and search algorithms, the core idea of both is to use learned structure-related signals to compensate for the insufficiency of numerical fitting error. Therefore, the paper has not yet fully explained what substantive methodological innovation direct TED prediction brings compared with the existing MDLformer. In addition, the paper lacks a direct comparison with MDLformer, making it difficult to determine whether the performance improvement comes from the new structural guidance or from other mechanisms such as LLM search, formula diagnosis, and post-processing. The authors are suggested to compare on common benchmarks and matched computational budgets, and further analyze the advantages and disadvantages of the two structural guidance methods to clarify the incremental contribution of this paper.

  2. The core goal of symbolic regression is not only to obtain a low numerical fitting error, but also to recover the mathematical structure of the target expression. However, this paper mainly uses NMSE, equation-level accuracy (Acco-01), and true normalized tree edit distance (True TED) to evaluate algorithm performance, and does not directly report the expression recovery rate.

Questions for authors

See Weaknesses

Reviewer_sMxUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a learned verifier that guides the search in LLM-based symbolic regression (LLM-SR). The basic idea is to provide a better guidance measure than plain goodness-of-fit measures such as MSE to traverse the large search space more efficiently. The learned verifier is a structural score, a kind of surrogate of the true recovery based on a modified tree edit distance (TED).

Critical strengths

The paper introduces a structure-based distance metric to guide the search in a vast search space. It can be seen as a variant or extension of the distance metric introduced by MDLFormer.

Critical weaknesses

Major points

  • Methodology: The authors explicitly remove identical formulas from the training set. But the training and test data are still sampled from the same distribution. This also holds for the structural verifier which is effectively trained on a data distribution that resembles the test set.

  • Evaluation: The evaluation is incomplete. The authors do not report OOD/extrapolation performance, complexity of the retrieved models as well as structural metrics such as symbolic recovery rate. Moreover, I am missing two baselines: What is the performance of an unguided baseline like e.g. PySR? What is the performance of a method that uses the edit distance predicted by MDLFormer's approach? To validate the approach, the authors should demonstrate that L2G-SR outperforms the edit distance predicted by concurrent guidance functions as, e.g. MDLFormer.

  • Robustness: Transformer-based methods have been shown to suffer strongly from domain overfitting on SR tasks. Therefore, reporting robustness under noise as well as domain shifts and/or moving sampling distributions is a must for SR methods, such that practitioners can get an insight into the reliability and trustworthiness of such a method in practice.

  • Parts of the submission seem to be AI generated. For example, the case study "Physics 11" is incomprehensible.

Minor points

  • The architecture shown in Figure 1 is at no point explained to the reader. For example, it remains mostly unclear what the 'Correlations' are.

  • Several symbols are not properly explained (e.g. EDE_D, ERE_R, etc.)

  • "Structural verifier" is misleading terminology since typically one associates formal verification methods with the notion of "verification".

Questions for authors

In my view, minimal requirements to improve future versions of the paper are

  1. Comparison with MDLFormer and PySR.
  2. OOD performance.
Reviewer_W56aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes L2G-SR, an LLM-based symbolic regression framework that introduces learned structural guidance beyond numerical fitting errors. Its key innovation is a structural verifier trained to predict a modified tree edit distance (TED) between candidate expressions and unknown target equations, enabling structure-aware search and refinement. Experiments on 240 equations across five domains demonstrate strong numerical accuracy and structural recovery, outperforming existing LLM-based symbolic regression methods in most evaluated settings.

Critical strengths
  1. Novel Structural Guidance: The paper introduces a learned structural verifier to complement numerical fitting errors, addressing an important limitation of existing symbolic regression methods.

  2. Strong Empirical Performance: L2G-SR achieves substantial improvements across multiple benchmarks and LLM backbones, with a matched ablation providing initial evidence for the benefits of structural guidance.

Critical weaknesses
  1. Insufficient Justification and Validation of Structural Guidance. The proposed structural metric and learned verifier lack convincing evidence of their effectiveness. The modified TED can assign zero distance to scientifically distinct expressions (e.g., x2x^2 and x3x^3), raising concerns about its ability to capture meaningful structural differences. Moreover, the verifier exhibits relatively weak correlations with true TED on actual search candidates (Pearson r=0.25–0.58), while its ability to rank promising candidates during search is not directly evaluated. Although the authors report downstream improvements, they do not establish whether these gains arise from meaningful structural guidance or simply from introducing an additional search criterion. Comparisons with alternative guidance signals, such as complexity or diversity-based scores, are necessary to substantiate the central methodological claim.

  2. Insufficient Ablations and Unclear Attribution of Performance Gains. L2G-SR integrates multiple mechanisms, including a pretrained verifier, dual-score selection, multi-agent refinement, and specialized expression simplification. However, the key matched ablation is restricted to the Physics domain, leaving the contributions of individual components and their generalizability insufficiently examined. Furthermore, comparisons with existing methods involve differences in search strategies, computational budgets, and post-processing procedures. For example, the reproduced RESTART baseline does not use the same post-fitting expression-pruning procedure as L2G-SR. Without more comprehensive component-wise ablations and controlled comparisons, it is difficult to determine whether the substantial reported improvements originate from the proposed structural guidance or other implementation differences. This uncertainty in performance attribution is a major concern underlying my Weak Reject recommendation.

Questions for authors

Q1. Validity and Reliability of Structural Guidance

The proposed TED can assign zero distance to scientifically distinct expressions (e.g., x2x^2 and x3x^3), while the learned verifier exhibits relatively weak correlations with true TED (Pearson r=0.25r=0.25–0.580.58). Furthermore, Eq. (9) does not preserve relative distances when dTED>1d_{\mathrm{TED}}>1. How do the authors justify that this metric and its predictor provide reliable structural guidance? Can the authors report candidate-level ranking accuracy during actual search and evaluate the sensitivity to different TED definitions?

Q2. Generalizability of the Key Ablation

Table 4 demonstrates the benefits of structural guidance only on the Physics domain with one LLM backbone. Can the authors provide matched MSE-only versus MSE+TED comparisons across the remaining domains and backbones? This is essential to establish whether structural guidance consistently contributes to the reported improvements.

Q3. Necessity of Learned Structural Guidance

The current ablation does not establish whether the improvements specifically originate from learned structural similarity or simply from introducing an additional search criterion. Can the authors compare TED guidance against alternative signals, such as expression complexity, structural novelty, or diversity-based selection, under identical budgets? Without these controls, the necessity of the proposed learned verifier remains unclear.

Q4. Attribution of Performance Gains

L2G-SR introduces multiple mechanisms, including a pretrained verifier, dual-score selection and retention, multi-agent refinement, and expression simplification. Can the authors provide component-wise downstream ablations to isolate their contributions? In particular, how much improvement comes from TED-based parent selection, candidate retention, and analyst feedback individually?

Q5. Fairness of Baseline Comparisons

Comparing symbolic regression methods fairly is challenging due to differences in search strategies, computational budgets, constant optimization, and post-processing. For example, the reproduced RESTART baseline does not use the same post-fitting expression-pruning procedure as L2G-SR. Can the authors clarify whether the reported improvements persist under more closely matched optimization and post-processing settings? Additionally, how do the comparisons change when accounting for LLM token consumption, verifier inference, and other computational overhead?

Q6. Generalization and Numerical Dependence of the Verifier

The verifier is pretrained on a large synthetic dataset generated using structural priors informed by existing benchmarks and search trajectories. How well does it generalize to unseen equation families or structural distributions? Furthermore, while Table 5 shows that numerical inputs improve TED prediction on synthetic validation data, can the authors evaluate the formula-only verifier in downstream search to demonstrate that the improvements genuinely depend on numerical-symbolic alignment rather than learned formula priors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.