acceptodds
Under review as a conference paper at ICLR 2027

Learning The Vocabulary of Mathematics From Data

Abstract

Symbolic regression (SR) searches for formulas built from a set of functions fixed before the search, which we call its vocabulary. When a law needs a function outside that set, the search can only return a longer expression that approximates the missing function with the functions it has. Yet one rarely knows in advance which functions a law will need. We show that the vocabulary of mathematics can be learned from data. We add to an SR library one learnable operator, a spline or a small neural network that the search can insert anywhere a built-in function can go. All occurrences of the operator share one parameter vector, which is fitted together with the numeric constants. We tested the search on three laws that contain Bessel's , with only elementary functions in the library. The learned operator matched in of runs. In most of these runs the three selected formulas had nodes in total, the same as a search whose library contains . Searching with only the elementary functions, even when given four times as much time, returned formulas of about nodes with higher error. The runs whose operator matched were exactly those with the lowest training loss. We prove that several laws sharing one operator can determine it, up to the scale and offset of its output, even when no single law does. It is enough that the laws together rule out every change of the operator's shape, and that no other set of formulas of the same or smaller size can reproduce how the laws change with their parameters. To test the case in which a needed function is missing, we also delete familiar functions from the library and check whether the operator recovers them. It recovers the square root in about two thirds of runs, and addition in of runs when the operator is a neural network. On recordings of a real pendulum, three of ten runs select the energy law with a cosine-shaped operator. Functions such as and became standard because each describes many different laws compactly, and a search that fits new functions from data could find useful functions that do not yet have names.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic regression, interpretability, equation discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wiEQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an approach to extend symbolic regression (SR) with a single learned operator φθ\varphi_\theta. The search then fits the given dataset while attempting to find short formulas, where φθ\varphi_\theta counts as a single symbol. Experiments show that this setup allows to recover notable functions such as Bessel’s function.

Critical strengths

It is impressive that the proposed method recovers (or gets close to) the desired hidden function in many scenarios. I believe this is a nontrivial finding.

The paper also discusses to what extent it is actually possible to uniquely discover the hidden function, as opposed to variations of it (Section 4). Some theoretical results in this direction are included.

Critical weaknesses

The experimental setup is quite limited. The first set of experiments presented (Section 3) is essentially a single experiment (repeated many times) with data generated by three fixed laws that involve Bessel’s function; while the results are good, it is still just a single experiment.

This is mitigated by the additional experiments in Section 5 (in synthetic an physical scenarios), but the setup is still far from a large-scale test where the deleted function is sampled from a larger pool, including other “complicated” functions.

Questions for authors

Have you considered expanding the set of experiments substantially? It would be desirable to test the proposed method on a dataset where the hidden function is sampled from a large pool.

3 comments

Thank you for the positive assessment and the concrete suggestion.

Have you considered expanding the set of experiments substantially? It would be desirable to test the proposed method on a dataset where the hidden function is sampled from a large pool.

We ran the experiment you propose (general response, item 1). Ten special functions that are not in the library are each hidden in three laws drawn at random from six templates, with 10 searches per function.

  • The method identifies every function in the pool, eight at default settings and two (Airy and E1E_1) with finer knots placed 0.5 apart.
  • The finer knots also find Dawson's integral from the submission, in 9/10 runs against 0/20 before (so thanks for this suggestion).
  • With the default spline, the held-out error is lower than with the elementary library alone in 96/100 runs. The median error is 41 to 69,000 times lower, depending on the function.

Together with J0J_0 and Dawson's integral, the paper now tests twelve functions that the library never had, instead of two. The table is now in the main text (Table 1).

Thanks for the addition. The new experiments are very helpful and confirm my positive assessment of this paper. I'll maintain my current score.

Thanks for your reply and the helpful review.

Reviewer_xg1yOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether symbolic regression (SR) can automatically learn mathematical operators from data, rather than relying on a fixed vocabulary of predefined functions. The authors propose extending the SR grammar with a learnable operator, parameterized by either a B-spline (BOP) or a small neural network (NOP). The parameters of this operator are shared across multiple expressions and jointly optimized with numerical constants during evolutionary symbolic search.

The method is evaluated on synthetic equations involving Bessel functions, square roots, and arithmetic operations, as well as several real and simulated physical systems. The paper further provides theoretical conditions under which multiple mathematical laws can identify a shared operator up to an affine transformation of its output.

Critical strengths
  1. Interesting problem formulation and potentially significant contribution. The paper addresses a fundamental limitation of existing symbolic regression methods: their dependence on a predefined set of mathematical operators. Allowing the search to discover reusable mathematical primitives is an interesting extension of conventional symbolic regression and could open new directions for automated scientific discovery. The emphasis on discovering mathematical vocabulary, rather than merely improving equation fitting, makes the problem particularly compelling.

  2. Well-motivated framework combining symbolic search and differentiable optimization. The method combines evolutionary symbolic regression with a shared learnable operator, jointly optimizing operator parameters and numerical constants through gradient-based methods. Sharing the operator across multiple equations introduces constraints that help prevent it from simply fitting individual equations.

Critical weaknesses
  1. Although the results on Bessel's J0 are encouraging, the MLP-based operator fails to recover J0 in all 20 runs, while the B-spline succeeds in 13 runs. Conversely, for binary addition and subtraction, the MLP achieves 21/50 recoveries whereas the B-spline succeeds in only 3/50 runs. These results indicate substantial sensitivity to the chosen operator representation. This raises concerns about the robustness and general applicability of the proposed framework. Since the intended setting involves discovering functions whose mathematical forms are unknown in advance, it is important to understand how an appropriate learnable operator can be selected without prior knowledge of the target function.

  2. Most successful operator-recovery results are obtained from synthetic equations constructed using known functions. On the real pendulum recordings, only 3 out of 10 runs recover the expected energy-law structure with a cosine-shaped operator. Moreover, none of these runs achieves an affine R² of 0.99 over the entire pooled set of pendulum tasks, and the learned operators are not shown to match cosine accurately over its full period. For the physical tasks involving deleted binary operations, no run successfully recovers the missing operation. Consequently, the empirical evidence currently supports the feasibility of learning missing operators in controlled settings more strongly than reliable discovery in realistic scientific applications.

Questions for authors
  1. Could the authors provide further insight into why the B-spline operator successfully recovers J0 while the MLP operator fails in all 20 runs, despite their comparable numbers of parameters in the unary setting? In particular, how sensitive are the results to network initialization, architecture, spline resolution, and optimization settings? A limited sensitivity analysis would help establish whether the observed failures are fundamental to the operator family or primarily due to the current implementation.

2.The real-world results suggest that learning missing operators is considerably more difficult than in the synthetic experiments. Could the authors clarify the main causes of these failures and distinguish difficulties in recovering the correct symbolic structure from difficulties in optimizing the shared operator?

3 comments

Thank you for the detailed review and the precise questions. We have provided new tables in the general response (and the updated submission).

Since the intended setting involves discovering functions whose mathematical forms are unknown in advance, it is important to understand how an appropriate learnable operator can be selected without prior knowledge of the target function.

Suppose you run both families (i.e., MLP and spline) and keep the one with the lower training loss, which needs no knowledge of the target. For J0J_0 this then picks the spline (the 10 lowest-loss runs of all 40 are splines, and all 10 identify J0J_0), and for ++ and −- it picks the MLP in 8 of the 10 deletion sets. Keeping the lowest-loss of three spline and three MLP runs finds J0J_0 97% of the time, so this simple selection method seems to work well. This is now a paragraph in Section 3.

Could the authors provide further insight into why the B-spline operator successfully recovers J0 while the MLP operator fails in all 20 runs, despite their comparable numbers of parameters in the unary setting? In particular, how sensitive are the results to network initialization, architecture, spline resolution, and optimization settings?

We ran the analysis you suggest (general response, item 2). No MLP setting learns J0J_0 over the laws' input range, and every ReLU variant learns ++ and −- in at least 9/10 runs. The spline finds J0J_0 at 15, 30 and 60 knots, and with twice the optimizer iterations it finds J0J_0 in all 10 runs, including the four it missed at the default settings. So the spline's misses came from the optimization budget, while the MLP fails at every budget we tried.

As for why, the parameter counts are similar (33 and 49), but the two parametrizations have different biases. A ReLU MLP is piecewise linear, so a sum is easy for it and an oscillating function is hard (with tanh, its ++ and −- recoveries drop from 10/10 to 4/10). The new functions show the same pattern. On the five that oscillate, the MLP's operator reaches R2≥0.99R^2 \ge 0.99 in 6/50 runs, against 37/50 for the spline, and on the other five the two are close (17/50 and 22/50). This is in line with a known spectral bias of neural networks (Rahaman et al., 2019), which we now mention in Section 3. It seems like the splines don't have this problem since each control point models a small part of the input range.

The real-world results suggest that learning missing operators is considerably more difficult than in the synthetic experiments. Could the authors clarify the main causes of these failures and distinguish difficulties in recovering the correct symbolic structure from difficulties in optimizing the shared operator?

We agree that the evidence is stronger in the controlled settings, and have revised the conclusion accordingly. We believe the failures have two different causes (Section 5.5, Appendix Table 25).

  • For the pendulum it appears related to the search. With cos in the library, the search finds the energy law in 3/10 runs, and with cos learned, in 5/20. The learned operator does about as well as the true function would. It seems like SR in general is struggling here.
  • For the air track and geometry conditions, the data looks like the issue. With the full library, the search finds three of these laws in every run, one in each group. With ++ or ×\times deleted, the learned operator saw only 11 to 43% of its input plane (78 to 91% on the synthetic sets), so the data cannot pin it down. On the simulated air track, for example, 8/10 runs write the correct acceleration law with the learned operator in place of ++, but that operator is not addition across the group's inputs.

[...] the learned operators are not shown to match cosine accurately over its full period.

The recorded angles span only about ±1.4\pm 1.4 rad, where cos and its quartic Taylor polynomial agree to affine R2R^2 0.99997. No method could identify cos over its full period from these recordings.

We hope these results answer your questions, and we would be happy to clarify anything further.

Thank you for your additional clarifications. They have helped me better understand your work and have increased my confidence in my assessment.

Hi, just checking whether you had a chance to see our response before the discussion closes. If it addresses your concerns, we’d appreciate you reconsidering your score. Thanks!

Reviewer_7YSiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A method for expanding the function vocublarily in SR challenges is proposed. A learnable function is introduced into the standard SR algorithm, but then its parameters are optimised together with the rest of the function coefficients. Using this method the authors show they can recover (or discover) functions which were removed from the library - thus potentially discovering unseen patterns in the data or even new special functions.

Critical strengths
  • A simple and elegant concept, that is easy to check and to incorporate into existing SR workflows.

  • Fairly well written and motivated.

Critical weaknesses
  • Why are there so few experiments? You use synthetic data, so you are not bounded by physical limitations - then why only 5 seeds? Only 20 runs for J_0? Most importantly, why only 3 (?) functions tested for recovery?
    Also, most of the functions in Table 4 are quite basic. If all the expressions containing cos, sin, exp, and log failed, you would still get the ~60% success rate - but it would show the method to be highly brittle.
    I believe your approach to be conceptually sound, just needs to be expanded and substantiated.

  • I'm not entirely certain what is the practical applicability of this method. The motivation of finding new, useful named funcitons is great - but to actually do that, wouldn't we need at least several different systems/dataset, which are materially different (multiple recordings of the same pendulum wouldn't help), and in all of them this new hypothetical function is a central piece? How would scientists look for such collections?

Questions for authors
  • Can you show examples from real physics/chemistry applications? How hard would that be to test out? Clearly a real discovery in a dataset would be a very strong claim in the paper.

  • Does the MLP architecture make any difference in your testing? Is that a limit on the complexity of the unknown function? Could a simple Tailor series be used instead of the B Spline? Do you need that expressive power? With the examples you've shown so far I'd guess not, but any data that could illuminate these points would be helpful.

(won't affect my assesment:)
Perhaps also cite arxiv.org/abs/2205.09751 - which focuses on Taylor series generation?
Pages 4-6 are very verbose - perhaps bring up some of the tables/graphs from the appendix up to the main text to "show and not tell"?

3 comments

Thank you for the review. You mentioned that you believe the approach is "conceptually sound, just needs to be expanded and substantiated" and so we have expanded it. The general response summarizes these new results.

Why are there so few experiments? [...] why only 5 seeds? Only 20 runs for J_0? Most importantly, why only 3 (?) functions tested for recovery?

You are right that runs are cheap. Thanks to your push, we added 540 searches (670 to 1,210), with 10 seeds for every new condition. The main addition is ten new hidden special functions (general response, item 1). The method identifies all ten, eight at default settings and two with a finer spline. The finer spline also finds Dawson's integral from the submission, which was a nice improvement. With the six targets already in the original submission (⋅\sqrt{\cdot}, ++ and −-, sin⁡/cos⁡\sin/\cos, ×\times, J0J_0 and Dawson's integral), the paper now tests sixteen.

Also, most of the functions in Table 4 are quite basic. If all the expressions containing cos, sin, exp, and log failed, you would still get the ~60% success rate - but it would show the method to be highly brittle.

Recovery is lower on the three sets that also use cos, log or exp, but it doesn't collapse. The MLP recovers ⋅\sqrt{\cdot} there in 9/15 runs and the spline in 5/15 (compared to 23/35 and 28/35 on the basic sets). To make this more clear Table 2 now reports the two groups separately. The new experiments hopefully address your concerns regarding brittleness, since the hidden functions now include many special functions (e.g., Bessel, Airy, and the exponential integral) in laws drawn at random.

[...] wouldn't we need at least several different systems/dataset, which are materially different [...] How would scientists look for such collections?

Materially different systems aren't necessarily required. Section 4 asks for laws that use the function in different ways, and different observables of one system can do this. J0J_0 itself is identified from three laws over the same two inputs. Once learned, the operator can be reused like a named function (Appendix E). Natural places to look are symbolic distillation of trained networks, closure terms in coarse-grained fluid simulations, and correlators in statistical physics. We were actually motivated to work on this method because we thought it was strange that MLPs worked so well for fluid simulations but SR couldn't find the expressions being used. We thought it could be related to needing the right operator.

Can you show examples from real physics/chemistry applications? [...] Clearly a real discovery in a dataset would be a very strong claim in the paper.

The paper already uses real pendulum and air-track recordings. On the pendulum, the search finds the energy law about as rarely with cos in the library as with cos learned, so the issue there seems like the search, not the operator (see our reply to xg1y). A real discovery would be the strongest result, and the conclusion now names the settings above as next steps.

Does the MLP architecture make any difference in your testing?

Not for J0J_0. No width (8 to 64), activation, initialization or optimizer budget lets the MLP learn J0J_0 over the laws' range, while every ReLU variant learns ++ and −- (general response, item 2). We think this is related to the MLP's trouble with higher frequency functions, which is a known spectral bias of neural networks (Rahaman et al., 2019). On the five new functions that oscillate, its operator reaches R2≥0.99R^2 \ge 0.99 in 6/50 runs, against 37/50 for the spline (17/50 and 22/50 on the other five). Since size made no difference, we kept the MLP small and fast.

Could a simple Tailor series be used instead of the B Spline? Do you need that expressive power?

We tested this. A degree-10 polynomial operator identifies J0J_0 in 0/10 runs, against 6/10 for the spline, although a degree-8 polynomial can fit J0J_0 on this range to R2R^2 0.999. Spline control points let the fit adjust one part of the curve without changing much elsewhere. This may make the joint fit more stable than using a polynomial, whose coefficients affect the whole curve. We also avoided bases built from known functions, because learning sin or cos from a Fourier basis would not tell us anything too surprising.

Perhaps also cite arxiv.org/abs/2205.09751 [...] Pages 4-6 are very verbose - perhaps bring up some of the tables/graphs from the appendix up to the main text

We now cite TaylorGP (He et al., 2022), and we shortened Section 4 to bring the new results table into the main text, as you suggest.

Hopefully the new experiments give the substance you were looking for. We would be grateful if you would take them into account in your assessment.

Thank you for your comments and additional experiments. There seem to be still many open questions, but the current data is convincing enough that the method has potential. I will raise my assessment to 3.

Hi, just checking whether you had a chance to see our response before the discussion closes. If it addresses your concerns, we’d appreciate you reconsidering your score. Thanks!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Summary of new results and changes

We thank the reviewers for their careful reading and their positive comments. All three find the idea sound, and their questions are largely about the experiments. Namely how broad the experiments are, how much the results depend on how the operator is represented, and how well the method works on real data. We ran two new experiments for the first two questions while the individual replies discuss the real-data performance. The results below have been incorporated into the revised PDF with new (or changed) text in blue.

1. Ten more hidden functions in randomly drawn laws.

Each of ten new special functions appears in three laws drawn at random from six templates such as f(x1)+x2f(x_1) + x_2 and x1f(x2)x_1 f(x_2). We ran 10 searches per function with a library containing (⋅\sqrt{\cdot}, exp⁡\exp, log⁡\log, sin⁡\sin, cos⁡\cos, ++, −-, ×\times, //) and applied the paper's J0J_0 identification rule (affine R2≥0.999R^2 \ge 0.999, and ten times closer than any elementary look-alike).

Hidden functionIdentified (of 10 runs)Lower held-out error than elementary library (of 10)
Sine integral Si910
Bessel J2J_288
Bessel J1J_1710
Digamma ψ\psi510
Cosine integral Ci49
Log-gamma ln⁡Γ\ln \Gamma410
erf310
Scaled complementary error function erfcx29
Airy Ai0 (9 with a finer spline)10
Exponential integral E1E_10 (4 with a finer spline)10

The method identifies all ten and eight are found with the default settings. For Airy and E1E_1, no spline with knots 1 apart fits the function to R2R^2 0.999 over its input range but with knots 0.5 apart, the method finds them in 9/10 and 4/10 runs. It now also finds Dawson's integral from the submission in 9/10 runs (0/20 before). With the default spline, the held-out error beats the elementary library in 96/100 runs.

2. Sensitivity to the operator's representation.

We changed one setting at a time and ran 10 searches for each, on the paper's J0J_0 laws and on one deletion set with ++ and −- removed.

OperatorJ0J_0 found++ or −- found
Spline, 30 knots (default)65
Spline, 15 knots68
Spline, 60 knots42
Spline, 40 optimizer iterations instead of 2010not run
One polynomial of degree 100not run
MLP, width 8 / 16 (default) / 32 / 640 / 0 / 0 / 09 / 10 / 10 / 10
MLP, new random initialization per seed010
MLP, tanh instead of ReLU04
MLP, 40 optimizer iterations instead of 200not run

The polynomial takes one input, so it cannot learn ++ or −-, and we ran the optimizer check on J0J_0 only.

No MLP setting learns J0J_0 over the laws' range, and every ReLU MLP learns ++ or −-. The spline learns J0J_0 at every knot count, and in all 10 runs with more optimizer iterations.

3. Choosing the operator without knowing the target. (requested by xg1y)

Suppose you run both families and keep the one with the lower training loss. Then this picks the spline for J0J_0 and all 10 new functions, as well as the MLP for ++ and −- in 8 of the 10 deletion sets. For J0J_0, keeping the lowest-loss of three spline and three MLP runs finds J0J_0 97% of the time. So this simple selection method seems to work well.

Changes in the revised PDF

  • The table of 10 new functions is in the main text (Section 3, Table 1).
  • A new paragraph in Section 3 explains how to choose the operator family by training loss, and why the two families suit different functions.
  • The deletion table (now Table 2) is split into the basic sets and the sets that also contain cos, log or exp.
  • The J0J_0 table (Table 19) has an equal-wall-time elementary baseline. The spline's error is 318 times lower.
  • A new Appendix G gives the full tables for the new functions, the sensitivity tables, and the analysis of the real-data failures, which Section 5.5 now explains.
  • We added the TaylorGP citation and shortened Section 4.

The paper now reports 1,210 searches, up from 670, plus a rerun of all 450 deletion searches (Appendix G.3).

We ask the reviewers to consider updating their scores if they are happy with these improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.