acceptodds
Under review as a conference paper at ICLR 2027

Not Every Term Adds New Structure: Sobolev Novelty for Symbolic Regression

Abstract

Symbolic regression (SR) aims to discover compact and meaningful mathematical equations from data, but searching the vast combinatorial space of symbolic structures remains challenging. Existing methods typically guide this process using expression-level objectives, such as fitting error, which assess a candidate equation as a whole but provide little information about whether an individual term contributes genuinely new structure or is largely redundant with the rest of the expression. We introduce Sobolev Novelty, a term-level measure of structural independence for symbolic equations. For each term, we construct an empirical Sobolev signature from its function values and exact derivatives over the observed inputs, and quantify how much of this behavior cannot be reconstructed by the remaining terms. We further derive a theory-calibrated threshold, yielding a principled and tuning-free criterion for identifying structurally novel terms. Using this threshold, 92.6% of terms in benchmark ground-truth equations exhibit sufficient structural novelty, compared with only 38.3% on average for expressions produced by 15 SR methods, revealing a substantial gap between scientific equations and current SR solutions. As a lightweight plug-in, Sobolev Novelty can be incorporated into diverse SR paradigms to support term pruning, search guidance, LLM feedback, and data selection, yielding consistent performance gains and demonstrating broad applicability.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
symbolic regression, equation discovery, Sobolev theory

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_6LS5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sobolev Novelty, a term-level projection residual based on function values and derivatives. It relates the metric to coefficient sensitivity and evaluates applications to symbolic search, pretraining-data filtering, and turbulence modeling.

Critical strengths

The metric requires no target derivatives and has clear projection and coefficient-sensitivity interpretations. Experiments cover multiple SR paradigms, and the value-only ablation supports the usefulness of derivative information.

Critical weaknesses

The threshold follows from an adopted tenfold tolerance on squared coefficient sensitivity relative to the orthogonal baseline. Although the derivation is clear, the robustness of the reported qualification-rate gap to this choice is not established. The GP comparison also changes the search procedure through an archive, recombination, and additional structural proposals (Appendix D.1). A matched control without Sobolev guidance would help isolate its contribution. Appendix D.5.2 describes forward-selection variants with and without Sobolev guidance, while distinguishing this implementation from the original LSTM-based DSRRANS search. The correspondence between these variants and the DSRRANS/SN-DSRRANS rows in Table 1 should be explicit.

Questions for authors
  1. How sensitive is the reported qualification-rate gap to the threshold?
  2. Can SN-GP be compared with the same search components and budget, but without Sobolev guidance?
  3. Please confirm which implementations produced the DSRRANS and SN-DSRRANS rows in Table 1.

My weak rejection is mainly due to uncertainty about the source of the reported gains. Controlled comparisons and implementation clarification could change my assessment.

Reviewer_vKYmOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Sobolev Novelty (SN), a term-level metric for measuring structural independence in symbolic regression (SR). By combining function values and first-order derivatives, SN quantifies the unique behavioral contribution of each term beyond the remaining expression. The authors establish theoretical connections to projection residuals and attribution stability, providing an interpretable criterion for structural novelty. The paper further identifies a novelty gap between expressions discovered by existing SR methods and genuine scientific equations. Extensive experiments demonstrate that SN can effectively guide symbolic search, LLM-guided search, and generative pretraining. Its application to turbulence modeling further validates its practical value in scientific discovery.

Critical strengths
  1. A principled, term-level evaluation criterion that reveals a limitation of existing symbolic regression (SR) methods.

Existing SR methods are typically guided by expression-level objectives, such as fitting error and complexity. However, these objectives rarely indicate whether an individual term makes a genuinely novel structural contribution. The paper introduces Sobolev Novelty (SN) to directly measure this property by quantifying how much of a term's values and gradients cannot be reconstructed by the remaining terms.

  1. Broad applicability, validated on real-world scientific tasks.

SN guides the search process in genetic programming (GP) and Monte Carlo tree search (MCTS), provides natural-language feedback for large language model (LLM)-guided search (IGSR), and filters pretraining data for end-to-end SR (E2ESR), improving the average (R^2) from 0.497 to 0.712. In the turbulence study, the discovered closure structures are embedded into a Reynolds-averaged Navier–Stokes (RANS) solver, and the resulting flow fields are compared against direct numerical simulation (DNS) results. SN-DSRRANS reduces the errors in the reattachment point and recirculation velocity by 72.3% and 84.9%, respectively. Notably, SN-MCTS, despite having no turbulence-specific prior knowledge, still outperforms both RANS and DSRRANS on these quantities.

Critical weaknesses
  1. Two essential aspects of the definition are only specified in the appendix.

Sobolev Novelty is computed based on an additive decomposition of the expression, but the rules for extracting individual terms are only provided in Appendix D.4. Moreover, the gradient block depends on the scale of the input coordinates. Appendix D.4 uses normalized coordinates, whereas Section 3 defines the formulation using gradients with respect to the original inputs, ∇_x. Both choices affect the computed novelty scores, and readers need these details to properly interpret and reproduce the results presented in the main text.

  1. Higher-order Sobolev representations are not evaluated.

Section 3 states that higher-order representations can be obtained by incorporating additional derivative blocks, while the conclusion identifies them as future work. The only derivative ablation study (Appendix F) compares the first-order representation against a value-only representation. The derivative order is a fundamental design choice in the representation, and the claimed straightforward extension to higher orders is not yet supported by empirical evidence.

Questions for authors
  1. Term extraction (see Weakness 1). Were the term extraction rules described in Appendix D.4 applied consistently across all experiments? This includes the analysis of 15 SR methods and ground-truth equations in Section 4.1, as well as the GP, MCTS, IGSR, and turbulence experiments.

  2. Input scaling for the gradient block (see Weakness 1). Were the input coordinates normalized when computing the gradient block in all experiments, or was normalization applied only to E2ESR pretraining in Section 4.3? If different conventions were used across experiments, could the authors clarify which convention was adopted in each setting?

  3. Presentation and clarity. Figure 1 refers to 15 SR methods but displays only 13 labels. Which methods were omitted, and why? Could the caption of Figure 2 direct readers to Tables 3 and 4, which report the corresponding numerical results? The conclusion mentions a deletion-and-refitting interpretation, whereas Section 3 only discusses projection and attribution stability. Where is the deletion-and-refitting result introduced in the main text? Additionally, there is a typo in the proof of Lemma A.7 ("equation equation A.15").

Reviewer_GVsfOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new method for evaluating the novelty of individual terms in mathematical expressions generated by symbolic regression algorithms. They define the structural contribution of a term relative to the current candidate by evaluating the extent to which it cannot be reconstructed by other terms. To achieve this goal, the authors introduce the Sobolev Novelty metric, which uses a Sobolev representation of a term (consisting of the term values and their derivatives evaluated at observed inputs). They further use this metric to derive a theory-grounded threshold for novelty. They compare the novelty of expressions predicted by symbolic regression algorithms with that of ground-truth expressions and conclude that there is a large gap. They later show how this metric can be accommodated in existing SR algorithms to improve their generalization and complexity. Finally, the authors show how the metric could enable scientific discovery through a case study of turbulence modeling.

Critical strengths
  • The paper addresses an important and overlooked problem. Many algorithms focus on the number of terms as the primary metric for optimization, apart from performance. This paper shows the limitations of this view and proposes a novel way of evaluating expressions that improves both the generalization and compactness of expressions.
  • The proposed metric is relatively easy to calculate and can be accommodated in different types of algorithms.
  • The paper is well-written and easy to follow.
  • The evaluations of SR algorithms with respect to novelty and complexity are insightful.
  • The authors include the new metric in several algorithms (including in a pre-trained model) and show meaningful improvements.
Critical weaknesses

The main weakness of this approach is its assumption that the expression is represented as a sum of terms. Of course, this is always true, as each equation can always be represented as a sum of terms. However, that means that this notion of novelty cannot be applied in a meaningful way to many expressions. For instance, if the expression is a square root of a sum of terms. I do not think this is a serious concern, as the method still seems to be useful in practice. However, I did not see this limitation explicitly mentioned in the paper. I think this is an important thing to disclose and discuss potential consequences.

My second concern is a limited comparison with other methods for measuring the novelty of a term. For instance, the authors use the IGSR algorithm, which computes influence scores for each found term. Is the proposed novelty metric highly correlated with such measures? Does it provide an orthogonal assessment? That is something that I would like to better understand.

Questions for authors
  • Could you provide an explicit discussion on the limitations of your approach?
  • Could you provide a comparison of other measures for term importance, like the influence score in IGSR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.