acceptodds
Under review as a conference paper at ICLR 2027

Synthesizing Physics Formulae with Transformers

Abstract

Finding a compact formula that fits a set of input-output pairs and predicts outputs on unseen inputs is a fundamental problem in science. Symbolic regression automates the search for such formulae: search-based methods explore the space of possible formulae directly, while transformers pre-trained on synthetic data produce formulae of comparable quality substantially faster. Existing transformers, however, are prone to overfitting — they find formulae that fit the training data well but do not extrapolate to input ranges unseen during training. We address this by shaping the set of formulae used to train a transformer, and show that the resulting formulae extrapolate substantially better. Fine-tuning the transformer on data with noise-corrupted target values further makes the synthesized formulae robust to noise in the observations. On SRBench and LLM-SRBench our transformer synthesizes a formula in about ten seconds and extrapolates better than all evaluated methods at a comparable budget. Search-based methods surpass our accuracy only when given one to three orders of magnitude more time.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic regression, extrapolation, transformer, generative AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZaS3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Pre-trained symbolic regression (SR: a method in which a Transformer trained on synthetic data solves the problem of predicting the expression itself from given data points) is fast at inference but tends to generate complex expressions that do not hold true outside the input range of the given data points (Voigt et al. 2025). To address this, the authors standardize the form of the expressions generated for training. Specifically, they use a generator based on E2E (Kamienny et al. 2022; a representative pre-trained SR that outputs the skeleton and constants of an expression in a single step) does not include the affine constants placed at every addition position (e.g., generating sin⁡(v1)\sin(v_1) as c0+sin⁡(c1+c2v1)c_0+\sin(c_1+c_2 v_1)), and by converting the generated expression into a proprietary normal form (flattening chains of addition and multiplication, combining like terms, rearranging terms, and renumbering variables, etc.) before using it as a teacher.

The authors train a 145M-parameter encoder-decoder and perform additional training using target values with added noise. Inference, like in E2E, involves beam search for each subset of data points and fitting constants using BFGS (a quasi-Newton method). Evaluation is based on the proportion ρk\rho_k (k≤128k \le 128) of tasks that satisfy a coefficient of determination R2≥0.99R^2 \ge 0.99 within a range shifted by kk times the width of the input range of data points provided for each task. The authors used the 99 Feynman physics equation problems from SRBench (the standard benchmark for symbolic regression) and two categories from LLM-SRBench (LSR-Transform: 111 problems in which the roles of the variables in the Feynman equations were swapped, LSR-Synth: synthetic problems with inputs involving physical unit scales) from LLM-SRBench.

The authors claim that their proposed method achieves an SRBench score of ρ128=0.57\rho_{128}=0.57 and an LSR-Transform score of 0.45 in approximately 11–12 seconds per equation, and is the most robust to extrapolation among methods that run in less than one minute. The authors themselves report that the proposed method does not match techniques that perform 1 to 3 orders of magnitude longer searches such as running PySR for 30 minutes and significantly underperforms LSR-Synth. In a comparison of the four combinations of presence/absence of affine constants × presence/absence of simplification using an 89M model (Fig. 4), no affine constants, with simplification yielded the best results.

Critical strengths
  1. The authors demonstrated a significant improvement in the extrapolation collapse of pre-trained SR through a broad and fair comparison with extrapolation as the primary evaluation metric. On SRBench, with no noise and k=128k=128, E2E achieves nearly 0. TPSR (E2E), a method that derives the decoding of the E2E model via Monte Carlo tree search, achieves 0.38; in contrast, the proposed method achieves 0.57 in approximately 11 seconds while maintaining noise at 0.01. The authors openly report that their method is outperformed by the 30-minute version of PySR and LSR-Synth, ensuring that the scope of their claims is clear to the reader. This paper provides a positive answer, backed by concrete numerical results, to the practical question of whether fast pre-trained SR models can be used as prediction models.

  2. The proposed method achieves results through generator design, an intervention that is easily transferable to other pre-trained SR models. The effect of eliminating the affine constant is evident in LSR-Transform regardless of whether reduction is applied (without reduction, ρ128\rho_{128} goes from approximately 0.19 to 0.28), while in Feynman, it is evident when combined with reduction (approximately 0.44 → 0.54). This observation is valuable as it calls into question the conventions surrounding generators that have prevailed since E2E. The fact that extrapolation changes based solely on the training distribution without altering the architecture provides insights directly applicable to future training data design.

Critical weaknesses
  1. The central claim, “Training with simplified expressions yields expressions that are robust to extrapolation,” is not sufficiently substantiated. In the comparison of 89M models in Fig. 4, for the Feynman model with and without simplification but without affine terms, ρ128\rho_{128} was approximately 0.54 and 0.42, respectively (compared to approximately 0.57 for the base model). While the effect of simplification is visible, the difference is larger for k=0k=0 (within the range of the provided data, approximately 0.75 and 0.53), and the decline from within the range is actually greater for the model with simplification (approximately 0.21 and 0.11). To the reviewer, it appears as though the improvement in accuracy within the range is carried over to extrapolation. Although the accuracy within the range is nearly the same with reduction (approximately 0.75 and 0.72), the gap widens at k=128k=128 (approximately 0.54 and 0.44); this difference is actually due to the presence or absence of the affine constant. In LSR-Transform, the difference between with and without reduction is small (approximately 0.30 and 0.28 at k=128k=128, and approximately 0.47 and 0.42 even at k=0k=0), and the majority of the difference from the main model (approximately 0.45) may be due to the main model’s settings. According to the submitted code’s checkpoints and README, the settings differ not only in scale but also in maximum expression length, handling of constants, and additional training with noise (there is also no explanation for why the version with affine terms and reduction can only achieve a difference of approximately 0.05 even at k=0k=0, which affects the reliability of the ablation; see “Questions for Authors 1”). Furthermore, the authors did not measure the overlap between the training distribution and the evaluation function. The generator, excluding the affine constants, produces many Feynman-type expressions with few additive constants, and since reduction always assigns the same training sequence to the same function, if the model observes expressions equivalent to the evaluation function during training, it may become more likely to generate them. While the overlap should be quantifiable using normal forms, the paper reports neither the number of such cases nor the symbolic accuracy rate; therefore, it is unclear to what extent the improvement in extrapolation is attributable to this overlap (Questions for Authors 3 and 4). The effect of reduction, which §1 and §7 place at the center of their contributions, does not prevent a decline in extrapolation accuracy beyond the range where Feynman improves it, and in the LSR-Transform, it is confounded with differences in settings relative to the main model. Since overlap with the evaluation function is not measured in either case, it is impossible to determine whether this is an effect specific to extrapolation.

  2. The differentiation from the closest prior research is insufficient. Saegert & Köthe’s (ICML 2026, arXiv version from February 2026) Flash-ANSR uses SimpliPy, a fast reducer based on rewriting rules, to normalize the training expressions and selects the representative with the minimum length and number of constants as the teacher. Although this paper cites that work in §3, the differentiation presented there (“one formula per equivalence class, no redundant constants”) is virtually identical to this selection method. What is unique to this paper is that it demonstrates through ablation experiments that affine constants impair extrapolation and that it includes an evaluation of extrapolation; however, the authors do not compare whether differences in normal forms affect extrapolation, and the results of the SimpliPy training path included in the submitted code are not presented in the main text. The substance of the novelty required of an ICLR method paper cannot be assessed without a comparison to the closest method.

Questions for authors
  1. [Clarification] Please provide a table showing the differences between the main model (ours) in Fig. 4 and the four ablation models in terms of scale, maximum expression length, treatment of constants (arbitrary real numbers or a set of symbolic constants), operator weights (as in Table 1 or uniform), additional training with noise, and the amount of training. Are there any results comparing the presence or absence of simplification while maintaining the same scale and training settings as the main model? How do you explain why the model with affine constants and simplification can only solve approximately 0.05 of the problems even with k=0k=0 in LSR-Transform? (Critical Weaknesses 1)

  2. [Clarification] What were the results for the 145M model trained using simplification with SimpliPy, as included in the submitted code? Alternatively, can it be compared using the same ρk\rho_k as Flash-ANSR (Saegert & Köthe 2026)? What is the basis for claiming that your custom normal form is advantageous for extrapolation? (Critical Weaknesses 2)

  3. [Clarification] If we count based on the correspondence of normal forms (since this is not full standardization, the lower bound is sufficient), it should be possible to measure how frequently the generator produces expressions equivalent to the respective SRBench Feynman and LSR-Transform expressions. Even the most closely related work by Saegert & Köthe (2026) trains the model excluding expressions that are equivalent to the evaluation expression. Can you show the proportion of such expressions and the corresponding ρk\rho_k values, broken down by tasks where equivalent expressions appeared during training versus those where they did not? How do you verify that the improvement in extrapolation is not due to overlap with the evaluation expression? (Critical Weaknesses 1)

  4. [Clarification] Can the authors report the symbolic accuracy rate (the submitted code includes a calculation script) and the structural similarity to the correct expression (e.g., edit distance of expression trees) for each method? I would like to confirm whether the superiority of ρk\rho_k at large kk corresponds to correctly identifying the correct expression. (Critical Weaknesses 1)

  5. [Clarification] Does the computation time for this method refer to the time taken to process multiple tasks in batch on a GPU, or is it the wall-clock time for a single task? The authors measured PySR on a single CPU core, including approximately 40 seconds for launching the computational component (Julia); however, does the claim that it is the best among methods taking less than one minute still hold true when excluding the startup time but using the same computational resources? If it does not hold, the authors should revise the statement in the Abstract that their method offers better extrapolation than all other methods at a comparable budget.

  6. [Clarification] Please indicate the actual number of seeds and the number of tasks for each curve by method. While the main text states that all methods use 10 seeds and 99 questions, the plotting script in the submitted code shows that many of the SRBench curves were calculated using 96 questions and the Llama-8B version of LLM-SR (a search method where a large language model generates and refines candidate formulas) appears to use only 1 seed. Furthermore, SRBench’s Feynman set contains 119 problems; according to a comment in the evaluation script of the submitted code, the 20 excluded problems are from the AI Feynman “Bonus” set (for which the correct equations are publicly available), which contradicts the description in §5.1 stating “99 problems with known correct equations.” If the denominator differs, the interpretation of the rankings among methods changes.

  7. [Clarification] For expressions that truly require additive offset, phase, and scale constants (e.g., c0+sin⁡(c1+c2v)c_0+\sin(c_1+c_2 v), a e−bv+ca\,e^{-b v}+c), how accurately can this method that excludes affine constants from the training distribution derive the expressions? Can the authors present a ρk\rho_k that specifically isolates expressions containing such constants? If the performance is significantly inferior, the authors should state this as a limitation of the method’s scope in the main text.

  8. [Does not affect the proposal or evaluation] The extrapolation in this paper is defined by a translation of the input range. If the authors were to investigate generalization in cases where the structure of the expression lies outside the training distribution (expressions longer than the upper limit during training, or expressions containing operators that are rare in the training data), the scope of “strong extrapolation” would become clearer.

  9. [Does not affect the proposal or evaluation] If the authors were to test, on an unknown task, which of the following to use—the default version or the substitution version (ours (unscaled): expressions with whitened inputs into which the mean and standard deviation are substituted to restore the original variables, §4.3)—the authors could clarify the scope of applicability to inputs on the scale of physical quantities (a scenario where this method significantly underperforms on LSR-Synth) by testing a procedure that automatically selects, based on validation data, whether to use the default version or the substitution version, or by attempting the diversification of the input range during training as previewed in §6.

  10. [Does not affect the proposal or evaluation] If the authors were to verify the explanation that the proposed framework works because the structure of the learned expression is close to that of a physical expression (§7) by examining the number of constants in the output expression or the relationship between the similarity of the learning distribution to the structure of the Feynman expression and the extrapolation performance, they could replace conjecture with empirical evidence.

  11. [Does not affect the proposal or evaluation] If extrapolation is treated according to the definition in this paper, the authors should evaluate it using a Feynman system benchmark in which the input sampling range has been adjusted to physically valid values; this would make the significance of shifting the range clearer.

Reviewer_NoTCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the issue of extrapolation failure in symbolic regression (SR). Specifically, the authors observe that pre-trained SR models tend to overfit the training interval, meaning that the given formulas fit well within the training data range but quickly break down once the input range is shifted outside. The core assertion of the paper is that the root of the problem lies in the "shape" of the training objectives. Therefore, improving extrapolation can be achieved by modifying the formula set used to train the Transformer.

Critical strengths

The experiments conducted in this paper are comprehensive, utilizing a variety of comparative methods

Critical weaknesses
  1. The author states in the experiment that the biggest advantage of the method presented in this paper is its "fastest and best performance within the same budget". However, the model in this paper runs on an H100 GPU, while PySR runs on a single CPU core. The comparison between CPU-seconds and GPU-seconds is not based on the same unit, which seems somewhat unfair.

  2. There are some confusing aspects in the writing of this paper. In Section 4.1, it is stated that the whitened data xi = (xi−µ)/σ will be used in the "Data points generation" step, but Section 4.3 explicitly argues that the substitution vᵢ→(vᵢ−μᵢ)/σᵢ should not be made, which is confusing.

Questions for authors

The principle stated in this paper that improving extrapolation by normalizing the shape of formulas is not very clear. In my opinion, scaling the data is an effective method to improve extrapolation, while normalizing the shape of formulas can merge equivalent formulas to avoid one-to-many situations during training, thereby improving training, but it does not seem to directly contribute to extrapolation performance. Experiments on LSR-Synth also seem to illustrate this point. The author needs to further elaborate and explain the principle of improving extrapolation by the method proposed in this paper.

Reviewer_mFidOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about using synthetic regression for finding a compact formula give a set of input-output data. They aim to improve extrapolation performance by shaping the set of formulae used in the training. Various numerical results show the advantage of the method over the competitions.

Critical strengths

The authors showed superior performance of this approach over a different types of benchmark cases (adding noise, out of distribution, etc) and the list of other methods is long enough.

Critical weaknesses

The application of this method when runtime is not limited is a weak point, as is acknowledged by the authors.

The authors haven't investigated why simpler formula lead to better generalization. This may be a well understood behavior, but is there a limit to which increasing simplicity will no longer enhance generalization or vice versa.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.