acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Fitting Error in Symbolic Regression through Local Taylor Consistency: Theory, Empirical Investigation, and a New Evaluation Framework

Abstract

Symbolic regression aims to discover underlying mathematical expressions from data, and most existing methods rely on fitting error to evaluate candidate expressions and guide the search. However, we observe that symbolic regression algorithms often produce structurally diverse but incorrect expressions with low fitting errors. This observation motivates us to investigate whether structurally distinct low-error candidates are prevalent and how they affect fitting-error-driven search. Theoretically, we derive an upper bound on fitting error and show that structurally different candidate expressions can achieve low fitting errors over small sampling domains when they exhibit local Taylor consistency. Furthermore, infinitely many such candidates can be constructed by introducing different higher-order remainder terms, while free constants further increase the attainability of local Taylor consistency. These results suggest that low-error but structurally incorrect candidates may be widespread over small sampling domains. Empirically, we first introduce RDF-SR, a random non-repetitive symbolic regression algorithm, as an unguided-search baseline. For relatively low-complexity expressions, several fitting-error-driven methods achieve lower recovery rates than RDF-SR, suggesting that optimization landscapes induced by fitting error may hinder structural recovery. Our theoretical error bound further indicates that enlarging the sampling domain can reduce low-error spurious solutions. Motivated by this result, we conduct experiments across different sampling domains and show that larger domains improve recovery rates but may weaken fitting-error guidance, making low-error regions harder to reach. Based on these findings, we develop and apply a new evaluation framework comprising new datasets, evaluation metrics, and an unguided-search baseline to systematically assess the search effectiveness, fitting-error guidance, and structural recovery capabilities of symbolic regression algorithms.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Symbolic Regression, Local Taylor Consistency, Error-Guided Search, Evaluation Framework

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2dqsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether fitting error can serve as a reliable search signal for exact structural recovery in symbolic regression (SR) and proposes Random Duplicate-Free Symbolic Regression (RDF-SR) as a structure-agnostic search baseline. RDF-SR is compared against fitting-error-driven methods such as PySR, DSO, and RSRM.

Critical strengths

1.The paper focuses on an important issue that is often overlooked in conventional evaluations of symbolic regression.

2.The paper is generally well organized, and the progression among the research questions is easy to follow.

Critical weaknesses

1.The discussion of related work is insufficient. The paper places “local Taylor consistency” at the center of its title, theoretical analysis, and claimed contributions, yet the Related Work section does not discuss prior symbolic regression methods that have already exploited Taylor- or derivative-based structural information to guide the search.

2.RDF-SR is not truly a “fitting-error-free baseline,” and the paper’s wording is potentially misleading. More precisely, RDF-SR is not an “error-free search” method; it simply does not use fitting error to determine which structure should be explored next. These are two very different claims.

3.The RDF-SR experiments do not isolate the effect of fitting-error guidance, and therefore the paper’s central empirical conclusion is not actually established. The comparison is not an ablation of the same search algorithm with and without fitting-error guidance. Instead, it compares fundamentally different search mechanisms. At most, the results show that “this random duplicate-free enumeration strategy performs better on these relatively simple problems.” They do not justify the stronger conclusion that fitting-error guidance itself causes worse structural recovery.

4.There appear to be several problems with the theoretical analysis. In particular, the inference from Theorem 1 to the claim that “enlarging the sampling domain reduces spurious low-error solutions” does not seem mathematically justified. The argument appears to confuse a looser upper bound with an actual increase in the quantity being bounded. In addition, the theoretical result claiming “infinitely many spurious candidates” is disconnected from the finite symbolic search spaces used in the experiments. The construction with m→∞ necessarily leaves the practical search space as expression complexity grows. Therefore, it provides very limited support for the claim that spurious candidates are abundant within the actual bounded SR search space. Using the infinitude of an unbounded function family to justify practical abundance in a bounded expression space substantially overstates the theoretical significance.

5.Beyond the potential mathematical issues, the theoretical novelty itself appears limited. The main ingredients are largely standard facts from mathematical analysis and linear algebra. More importantly, the theoretical core does not actually resolve the central question suggested by the title, Rethinking Fitting Error. The analysis merely attempts to show that distinct functions can be locally similar and then makes a substantial conceptual leap to the conclusion that fitting error therefore misleads the search.

6.The new benchmark suffers from a clear circularity issue: the test targets are generated using RDF-SR, while RDF-SR is then used as a key baseline on the same benchmark. Unlike the original benchmark, the newly constructed target expressions come from a structural distribution that RDF-SR has already demonstrated it can access. The paper then asks other algorithms to compete with RDF-SR in terms of recovery and structural balance. This benchmark is therefore likely to be inherently biased toward the grammar accessibility and search distribution of RDF-SR, making the comparison potentially unfair.

7.The paper is validated only in highly idealized settings—noise-free, univariate, synthetic benchmarks—yet the conclusions are phrased as if they apply broadly to symbolic regression. In realistic SR problems, some of the most challenging issues arise precisely from noise, multivariate interactions, and other sources of complexity that are absent from the current experiments.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Clarification of the Paper’s Contributions, Theoretical Analysis, and Evaluation Design

We acknowledge that the original manuscript lacked clarity in its organization and presentation, failing to adequately convey the research scope and main contributions of this work. We sincerely apologize for this and would like to clarify our contributions. Our main contribution is not to propose a new SR algorithm or benchmark, but to investigate, through theoretical and experimental analyses, why structurally different expressions can achieve low fitting errors, the existence and structural diversity of low-error expressions, the effectiveness of fitting-error-guided search, and the impact of sampling ranges on search performance. Based on these analyses, we identify two key capabilities of SR algorithms: extracting effective search directions from fitting error and suppressing misleading search directions induced by fitting error. We further highlight the need to study and evaluate these capabilities separately. Finally, we present an illustrative benchmark design to demonstrate how the two capabilities can be independently evaluated.

Studies that use Taylor expansions or derivative information to guide expression search differ from our theoretical analysis in their research questions, objectives, and methods. These two lines of research should therefore be distinguished. We do not use Taylor consistency to guide expression search. Instead, we use it as a theoretical tool to derive a fitting-error upper bound that explains why candidate expressions structurally distinct from the target can still achieve low fitting errors. We further investigate the existence and structural diversity of such expressions.

GP, RL, and MCTS-based SR algorithms typically use fitness, rewards, or node values derived from fitting error to guide search. Since fitting error is integral to their search mechanisms, simply removing it would prevent them from operating as designed. We therefore introduce RDF-SR, a random duplicate-free search baseline that does not use fitting error for search guidance. Under the same search space and budget, we compare its recovery rate with those of fitting-error-driven algorithms. RDF-SR randomly explores unvisited expressions using only a binary criterion to identify the target, whereas fitting-error-driven algorithms additionally use fitting error to guide search. If fitting-error guidance is effective, these algorithms should achieve higher recovery rates. This comparison provides empirical evidence for assessing the effectiveness and limitations of fitting-error guidance.

Regarding the theoretical analysis, first, Theorem 1 does not imply that an increasing error upper bound necessarily leads to higher actual fitting errors. Rather, it shows that the low-error guarantee provided by low-order Taylor consistency weakens as the sampling range expands, so some candidate expressions may no longer achieve low fitting errors. We examine the actual effects of expanding the sampling range through experiments. Second, the infinitude of a search space does not imply that the subset of expressions satisfying specific conditions is also infinite. Even if the search space contains infinitely many expressions, those satisfying low-order Taylor consistency are not necessarily infinite in number. Finally, to connect our theoretical analysis with practical search spaces under finite complexity constraints, we have added Theorem 3, “Existence of Low-Order Taylor-Consistent Expressions in a Finite Constant-Free Search Space,” and Theorem 6, “Existence of Low-Order Taylor-Consistent Expressions in a Finite Search Space with Constants,” to the revised manuscript.

Using RDF-SR for both candidate generation and baseline comparison does not constitute circular reasoning. RDF-SR does not target specific expression structures or use fitting error or other metrics to guide search. Instead, it explores the valid expression space through random branch selection and node exhaustion without revisiting expressions, and thus does not deliberately favor particular structures. In subsequent evaluations, RDF-SR and other algorithms run independently under the same search space and budget. Generating an expression during dataset construction does not guarantee its recovery in later runs, as RDF-SR searches randomly.

The constructed dataset is neither intended to simulate real-world applications nor designed as a general-purpose SR benchmark. Rather, it serves as an illustrative example of an evaluation methodology. By constructing sets of structurally different expressions with similar fitting errors, we demonstrate how to independently evaluate two key SR capabilities: extracting effective search directions from fitting error and suppressing misleading search directions induced by fitting error.

Reviewer_sGfaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present a new approach to symbolic regression. It first provides a theoretical foundation, then introduces RDF-SR, and finally provides a new evaluation framework. The paper is easy to follow, and presentation is generally clear, although the paper does not demonstrate whether the proposed idea and framework is better than the state-of-the-art methods or not.

Critical strengths

A major strength of the paper is that it provides a sound theoretical foundation with proof. While it is not entirely clear whether the theory is 100% percent correct (I can't guarantee this), it at least demonstrates that a theoretical upper bound exist. This is a good point.

Based on the theoretical research, the authors proposed a new SR method and a evaluation framework. Again, they did not demonstrate the proposed method and framework are better than the state-of-the-art methods for SR, particularly those GP approaches, the move the idea a step forward.

Critical weaknesses

A major weakness is that the paper did not show clear evidence that the proposed RDF-SR is better than the state-of-the-art methods for SR problems. This is actually important since many people can propose theories and framework and claim they are new and better but without providing evidence to show the proposed method is better, at least in some aspects.

Another major weakness is that the test beds/SR problems are with all known solutions (expressions). This will not help much on the SR problems with unknown model structure and/or co-effients or with any good generalisation capability.

A third major issue is that the test problems only have a single input variable (x), which does not cover real-world SR problems which often has a number of input variables/features, some with more than 1000 (high-dimensional SR problems). However the conclusions made are over too general.

Questions for authors
  1. Could the proposed RDF-SR perform on the SR problems presented in this paper achieve better results than state-of-the-art SR methods, particularly those modern GP (including GSGP) methods?

  2. Could the proposed RDF-SR perform on SR problems with a single input variable without the target expression/models known in advance?

  3. Could the proposed RDF-SR perform on SR problems with multiple variables without the target expression/models known in advance?

1 comment
AuthorsOpenReview ↗

Clarification of RDF-SR's Role and the Main Contributions of This Work

We introduce RDF-SR not as a new SR algorithm or to demonstrate superior performance over existing methods, but as a random duplicate-free search baseline without fitting-error guidance. It allows us to examine whether fitting-error guidance genuinely improves target expression recovery and to investigate its effectiveness and limitations.

We acknowledge that the original manuscript lacked clarity in its organization and presentation, which obscured the research focus and main contributions of this work. We sincerely apologize for this. We have therefore revised the entire manuscript and uploaded the revised PDF. This paper does not propose a new SR algorithm or benchmark. Rather, it uses theoretical and empirical analyses to explain why structurally different expressions can achieve low fitting errors and to establish the existence and structural diversity of such expressions. It further examines the effectiveness of fitting-error-guided search and the impact of sampling intervals on search performance. Based on these findings, we identify two key search capabilities of SR algorithms: extracting effective search directions from fitting error and suppressing misleading directions induced by fitting error. We also clarify why these two capabilities need to be studied and evaluated separately. Finally, we provide an illustrative benchmark design demonstrating how the two capabilities can be evaluated independently.

Reviewer_gHyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why symbolic-regression methods often produce structurally diverse but incorrect expressions with low fitting error. It derives an upper bound on continuous MSE when a candidate and the target have contact of order p at a point: MSE ≤ (L_p/p!)^2 c^{2p}/(2p+1). It shows that infinitely many locally Taylor-consistent candidates can be constructed via different remainder terms, and that free constants increase the attainability of local Taylor consistency through jet-map rank conditions. It then introduces RDF-SR, a random duplicate-free search baseline, and uses it to argue that fitting-error guidance can be less effective than unguided search for low-complexity targets. Finally, it studies how the sampling domain affects recovery and low-error rates, and proposes a new evaluation framework with 68 expressions in 17 groups, using low-error rate (LER), recovery rate (RR), conditional recovery rate (C-RR), and within-group standard deviation (WSD).

Critical strengths
  1. Important and under-discussed question. The observation that structurally distinct expressions can achieve low fitting error is practically important, and the paper makes a convincing case that fitting error alone is an incomplete search signal. The framing of SR as a discrete decision problem versus a continuous optimization problem is insightful.

  2. Clean theoretical exposition. The order-of-contact argument gives a simple, intuitive explanation for why small sampling domains admit many spurious low-error solutions. The construction of infinitely many locally consistent candidates via remainder terms (Theorems 2–3) is a nice formalization, and the jet-map conditions (Theorems 4–5) clarify the role of free constants.

  3. Useful diagnostic baseline. RDF-SR is a valuable sanity check. The finding that it outperforms several fitting-error-driven methods at complexity ≤12 is provocative and worth reporting, even if the comparison needs more controls.

  4. Sampling-domain study. The empirical trade-off—larger domains reduce spurious low-error solutions but weaken the guidance toward low-error regions—is a useful contribution that connects the theory to practice.

  5. New benchmark design. Grouping expressions with similar fitting errors but distinct structures is a sensible way to test whether a method can escape spurious solutions. C-RR and WSD target important failure modes that aggregate recovery rates hide.

  6. Honest, analysis-oriented framing. The paper does not oversell a new algorithm; it presents a diagnosis and a benchmark. This is refreshing.

Critical weaknesses
  1. The theory is elementary and does not directly address the finite-sample setting. Theorem 1 bounds continuous MSE on an interval. SR observes N discrete points, and the relevant quantity is empirical error on those points. A low continuous MSE implies low empirical error, but the converse is not true; the bound does not explain why a candidate can fit N samples well while being structurally wrong. A finite-sample version (e.g., via covering numbers or a sampling argument) is needed. The paper also does not connect the bound to the search dynamics or to the probability that a search method encounters a spurious candidate.

  2. The uniform-boundedness assumption in Corollary 1 is strong and not justified for the operator set. The corollary assumes Δ^(p) is uniformly bounded independently of p. For operator sets containing exp, log, sin, and division, higher derivatives can grow rapidly, and the bound L_p may itself grow with p. The paper should state precise conditions under which the p→∞ conclusion holds, or give counterexamples. As written, the “infinitely many” result is a statement about function spaces, not about the finite, grammar-constrained search spaces used in SR.

  3. The abundance claim is not quantified. The paper shows that infinitely many functions can be locally Taylor-consistent, but it does not quantify the measure or count of such functions within a realistic grammar, nor the probability that a random expression is spurious. The empirical candidate counts in Table 7 are a start, but they are for one dataset and one search method. A grammar-level counting or sampling analysis would make the claim much stronger.

  4. The RDF-SR comparison needs tighter controls. RDF-SR is described as unguided, but it uses duplicate-free enumeration over valid expressions and BFGS constant fitting; its behavior depends on the enumeration order, grammar, and complexity limit. The paper should report the exact grammar, operator set, complexity limit, and evaluation budget for RDF-SR and each baseline, and provide an iso-budget comparison. It should also test whether the conclusion holds when RDF-SR is given the same constant-fitting budget and the same grammar as each competitor.

  5. The new benchmark is constructed from RDF-SR’s own low-error candidates. This is a clever way to obtain hard negatives, but it introduces selection bias: the benchmark is conditioned on RDF-SR’s search space and may favor methods that share its grammar. The paper should validate the benchmark with an independent construction (e.g., candidates from multiple methods or human-designed near-misses) and report inter-group structural distances. With 68 expressions and 17 groups, the benchmark is also small; confidence intervals and per-group results are essential.

  6. The empirical evaluation is dated. The main comparisons use PySR, DSO, RSRM, uDSR, and iMCTS, but omit recent LLM-based SR methods (LLM-SR, LaSR, SR-Scientist, DIGEST) and modern GP variants (Operon, GP-GOMEA, EGGP). Since the paper proposes a general evaluation framework, the absence of these methods limits its usefulness. At minimum, the authors should explain why they were excluded and add a representative subset.

  7. The practical guidance is limited. “Enlarge the sampling domain” is a known trade-off, and the paper does not propose a method that resolves it. The framework is diagnostic rather than prescriptive. This is acceptable for an analysis paper, but the contribution should be framed accordingly; the current abstract and introduction sometimes read as if a new method is proposed.

  8. Coverage of learned/neural-guided SR is incomplete for a benchmark paper. The empirical study includes a solid conventional set (PySR, DSO, uDSR, RSRM, iMCTS, TPSR, GPlearn, EGGP), but it does not discuss transformer-based joint supervised SR (Li et al., ICLR 2022) or neural-guided dynamic symbolic networks (DySymNet, Li et al., ICML 2024). Since the paper proposes a general evaluation framework, the baseline set should either cover the main learned SR families or justify why the existing neural-guided methods (uDSR, iMCTS, TPSR) are sufficient. I recommend adding DySymNet to the new 68-expression benchmark, using the official implementation, the same grammar/complexity limit, and the same evaluation budget, and reporting C-RR and WSD. If a full run is infeasible, a representative subset plus an explicit justification would be sufficient. The related work should also explain how the paper's diagnosis applies to this family.

Questions for authors
  1. Can you provide a finite-sample analogue of Theorem 1 for N observed points, and bound the number or measure of ε-low-error candidates as a function of N, c, and p?
  2. Is L_p uniformly bounded as p→∞ for your operator set? If not, under what conditions does Corollary 1 hold?
  3. What are the exact grammar, operator set, complexity limit, and evaluation budget for RDF-SR and each baseline? Can you provide an iso-budget and iso-grammar comparison?
  4. How sensitive are the conclusions to the enumeration order and random seed of RDF-SR?
  5. How were the 17 groups constructed, and what is the structural-distance threshold? Are the groups semantically distinct, or only syntactically different?
  6. Why are recent LLM-based SR and modern GP methods excluded from the main comparison?
  7. Does enlarging the domain help because of fewer spurious solutions, or because more data improves constant fitting? Can you disentangle these effects?
  8. Will the code and the new benchmark be released?
  9. Can you add a representative neural-guided method such as DySymNet to the new benchmark at matched budget, or explain why the current baseline set is sufficient for the paper's conclusion?
1 comment
AuthorsOpenReview ↗

Summary of Revisions and Clarification of the Paper's Contributions

We have revised Theorem 1 to express the MSE in terms of the discrete error over NN sampled points. The revised theorem confirms that the original conclusion remains valid: structurally different expressions can still achieve fitting errors approaching zero as the sampling interval shrinks, provided that they satisfy low-order Taylor consistency.

For the operator set considered in this paper, the uniform boundedness of LpL_p as p→∞p\to\infty cannot be guaranteed. Accordingly, we have revised Corollary 1 by removing the convergence result as p→∞p\to\infty and retaining only the result that the fitting error approaches zero as c→0c\to0 for any fixed pp.

In the original experiments, we used a common operator set, O={+,−,×,÷,sin⁡,cos⁡,exp⁡,log⁡}\mathcal{O}=\{+,-,\times,\div,\sin,\cos,\exp,\log\}, a maximum expression complexity of 30, a maximum tree depth of 6, and a search budget of 2×1062\times10^6 per run for all algorithms, wherever supported by their implementations. Detailed parameter configurations are provided in Appendix D.

Since enumeration order affects RDF-SR's results under a finite search budget, we employ level-order expression construction combined with random branch selection and node exhaustion to achieve duplicate-free search. This design aims to minimize biases toward particular expression lengths and structures introduced by fixed traversal orders. A detailed description of RDF-SR is provided in Appendix C.

For GP baselines, we include PySR, EGGP (2025), and the classical method gplearn to cover representative methods from different stages of GP development. Although LLM-based SR has made significant progress, this paper focuses on SR methods that use fitting error rather than external knowledge priors to guide expression search. Since LLM-based methods leverage pretrained mathematical and scientific knowledge to guide expression generation, their performance depends on both knowledge priors and search strategies, making it difficult to isolate the effectiveness of fitting-error-based guidance. DySymNet is an EQL-based neural-symbolic method that uses reinforcement learning to explore network architectures and parameter optimization and sparsification to derive expressions. However, it searches network architectures rather than explicit expression spaces, making its network configurations difficult to match with our expression complexity and tree depth constraints. Thus, strict comparisons under matched search budgets are difficult, and DySymNet is excluded from the main comparison. We additionally tested DySymNet on the Livermore dataset. Although it achieved good numerical fitting performance, its recovery rate (RR) was 0% under our experimental settings.

Appendix H provides all 68 target expressions in the new dataset, together with their data generation procedures, including the sampling ranges, distributions, and rules for determining the number of sampling points. Since the dataset is synthetic, researchers can reproduce the corresponding input-output data by following these instructions without requiring additional data files.

In the original experiments, fixing the sampling density caused the sampling range and the number of sampling points to change simultaneously. To isolate their effects, we conducted additional experiments with a fixed number of sampling points and progressively expanded sampling intervals. We evaluated PySR on the Livermore dataset with 50 independent runs per expression. The results are consistent with the original experiments: RR increases while LER decreases as the sampling range expands. This suggests that a wider sampling range may reduce the misleading influence of spurious solutions while making it more difficult to obtain low-error expressions.

Finally, we acknowledge that the original manuscript lacked clarity in its organization and presentation, which obscured the research focus and main contributions of this work. We sincerely apologize for this. We have therefore revised the entire manuscript and uploaded the revised PDF. This paper does not propose a new SR algorithm or benchmark. Rather, it uses theoretical and empirical analyses to explain why structurally different expressions can achieve low fitting errors and to establish the existence and structural diversity of such expressions. It further examines the effectiveness of fitting-error-guided search and the impact of sampling intervals on search performance. Based on these findings, we identify two key search capabilities of SR algorithms: extracting effective search directions from fitting error and suppressing misleading directions induced by fitting error. We also clarify why these two capabilities need to be studied and evaluated separately. Finally, we provide an illustrative benchmark design demonstrating how the two capabilities can be evaluated independently.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.