acceptodds
Under review as a conference paper at ICLR 2027

Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment

Abstract

Symbolic regression is the problem of finding an expression that fits a dataset. Solving it requires searching a combinatorial space of candidate expressions. Neural symbolic regression can amortize that search by training a model once on synthetic data and then predicting expressions for a new dataset. But existing models still underperform genetic programming on real, noisy data. We argue that the deficit lies in the training rather than in the paradigm, and present RASR (Reinforced Amortized Symbolic Regression), a method with two key changes to the training recipe. First, we design a synthetic prior that narrows the gap with real-world data: inputs come from the prior of a tabular foundation model; constants are sampled bottom-up, conditioned on the data; targets carry heteroscedastic and structured noise; and everything is standardized. Second, we split training into two stages. Supervised pre-training maximizes the token-level likelihood of a reference expression. This is followed by reinforcement-learning post-training that aligns the model with its inference-time use by rewarding held-out fit and penalizing expression size. On the black-box track of SRBench 2.0, RASR achieves the best mean test R² rank of all evaluated methods. Three RASR variants, trained with different size penalties, make up most of the accuracy-complexity Pareto front. On Strogatz, our method recovers more ground-truth expressions than any baseline, and on Feynman it is competitive with the best-performing methods.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic regression, neural symbolic regression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2ntMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a symbolic regression framework that consists of three main stages: (1) a data generation method designed to produce training samples that more closely resemble real-world data distributions, (2) pre-training of a deep learning model on this synthetically generated data, and (3) expression inference via reinforcement learning. The central hypothesis is that the poor performance of pre-trained symbolic regression models on black-box datasets stems primarily from a mismatch between the synthetic training data and real data distributions. By addressing this data gap, the authors aim to enhance the generalization capability of pre-trained models, particularly in high-dimensional settings where pure search-based methods become computationally intractable.

Critical strengths

The authors correctly identify that the limited success of pre-trained symbolic regression models on black-box data is largely due to deficiencies in the training data distribution. This observation is both novel and valuable. If effectively addressed, it could substantially improve the practical utility of pre-trained models, especially as input dimensionality grows and search-based methods struggle to maintain accuracy.

Critical weaknesses
  1. Incomplete Experimental Comparison:​ While the proposed method demonstrates superior performance over PySR on 12 black-box models, it notably omits comparisons on the Strogatz and Feynman datasets. Given PySR’s strong performance on these benchmarks, such comparisons are essential for a fair and comprehensive evaluation. Although SRBench2.0 may not include these datasets, the authors should still provide results to contextualize their method’s effectiveness.

  2. Insufficient Black-Box Dataset Diversity:​ The paper’s core motivation is to improve pre-trained model performance on black-box data; however, it evaluates on only 12 black-box datasets from SRBench2.0. The PMLB repository contains a much larger collection of black-box problems, and a more extensive evaluation across diverse datasets is necessary to convincingly validate the method’s generalizability.

  3. Limited Methodological Innovation:​ The proposed approach combines pre-training with reinforcement learning, which has been explored in prior works such as SNR[1], DGSR[2], and uDSR[3]. The distinction from TPSR (pre-training + Monte Carlo Tree Search) is noted, but the overall framework closely resembles GSR, with the only notable difference being the use of refined constant encoding (which itself was introduced in E2E). Consequently, the paper lacks significant methodological novelty.

  4. Inadequate Validation of Data Generation Contribution:​ The primary claimed contribution is the generation of more realistic data, yet evidence supporting this claim is weak. The method is only evaluated on 12 black-box datasets, with no ablation studies comparing performance against pre-trained models trained on the same amount of data without the proposed generation method. This raises concerns that performance gains may simply result from using larger training datasets rather than improved data quality. Furthermore, the paper does not provide any analysis (e.g., distribution similarity metrics, comparison with real-world expression statistics) to demonstrate that the generated data truly aligns better with real-world data distributions.

[1] Deep generative symbolic regression, 2023, ICLR

[2] SNR: Symbolic network-based rectifiable learning framework for symbolic regression, 2023, Neural Networks

[3] A unified framework for deep symbolic regression, 2022, NeurIPS

Questions for authors
  1. Could you provide experimental comparisons with PySR on the Strogatz and Feynman datasets to more thoroughly benchmark your method against state-of-the-art symbolic regression tools?

  2. Why were only 12 black-box datasets used? Would it be feasible to include additional black-box problems from PMLB to strengthen the empirical claims?

  3. Given the similarities between your framework and existing methods like DGSR and SNR, what specific novel components does your method introduce beyond constant encoding and RL fine-tuning?

  4. Can you present ablation studies that isolate the effect of your data generation method, controlling for training data size? Additionally, how do you quantitatively validate that your generated data distribution more closely matches real-world data distributions compared to existing synthetic data generation approaches?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2ntM (Part 1)

We thank the reviewer for the thoughtful review and for recognizing the relevance of the training-data mismatch we identify for pre-trained symbolic regression models. We respond to each weakness and question below. Section, figure and table numbers refer to the revised PDF.

W1/Q1: PySR on Strogatz and Feynman

Could you provide experimental comparisons with PySR on the Strogatz and Feynman datasets to more thoroughly benchmark your method against state-of-the-art symbolic regression tools?

We appreciate the reviewer pointing out this gap. The Feynman and Strogatz tracks come from the original SRBench, whose published results we used and extended with Flash-ANSR and SNIP. PySR only joined the benchmark with SRBench 2.0, so it was missing from these tracks. We have now run PySR on both tracks with its SRBench 2.0 configuration and added it to all Feynman and Strogatz results (Section 4.3, Figure 5, Appendix L).

As the reviewer anticipated, PySR is a strong baseline on these tracks, particularly for recovering the exact law. RASR, in turn, is ahead of PySR in accuracy on both tracks and also recovers the exact law more often on Strogatz. Since accuracy alone does not show whether the law was found, the revision now also reports the symbolic solution rate using SRBench's own check. It also adds RASR-94M-L+, our main model with an optional pruning step that removes near-redundant constants using only the training data (Section 3.4, Appendix G).

  • Accuracy solution rate (R2>0.999R^2 > 0.999, Figure 5): averaged over noise levels, RASR-94M-L ranks ahead of PySR on both tracks.
  • Symbolic solution rate (Figure L.1): averaged over noise levels, RASR-94M-L+ ranks first on Strogatz and PySR second. Without noise, RASR-94M-L+ recovers the law in 61.4% of runs against 42.1% for PySR, while PySR is ahead at the two highest noise levels. On Feynman, PySR ranks first, followed by Flash-ANSR and RASR-94M-L+.

W2/Q2: Number of black-box datasets

Why were only 12 black-box datasets used? Would it be feasible to include additional black-box problems from PMLB to strengthen the empirical claims?

We agree that twelve datasets are few, and we list this as a limitation (Section 5). The small number is, however, a deliberate choice of SRBench 2.0. The black-box track of the original SRBench comprised 122 PMLB datasets, but this larger pool distorted the results in two ways. About half of the datasets are variants of the Friedman benchmark, which skewed aggregate results towards this single family of synthetic problems. In addition, many datasets are solved almost perfectly by every method, so they add little information about how methods differ.

SRBench 2.0 therefore purposely reduced the track to twelve datasets (Aldeia et al., 2025, Section 3.1). It first removed every dataset on which all previously benchmarked methods, or plain linear regression, reach R2>0.99R^2 > 0.99. It then clustered the remaining datasets by size, dimensionality and the performance of these methods, and kept the dataset closest to each of twelve cluster centres, with Friedman datasets capped at 25% of the selection. Adding further PMLB datasets would risk reintroducing exactly these biases. Following the benchmark's protocol also keeps our results directly comparable to the published runs of all 25 methods it evaluates. Constructing a better black-box benchmark is important, but beyond the scope of this work.

AuthorsOpenReview ↗

Response to Reviewer 2ntM (Part 2)

W3/Q3: Novelty with respect to DGSR, SNR and uDSR

Given the similarities between your framework and existing methods like DGSR and SNR, what specific novel components does your method introduce beyond constant encoding and RL fine-tuning?

We suspect there may be a misunderstanding about how reinforcement learning is used. The review summarizes our third stage as "expression inference via reinforcement learning", which tells us that we should have explained this more clearly. We apply reinforcement learning only once, on synthetic data, before the model is ever used on a real dataset. After this post-training stage, the model is frozen. Given a new dataset, RASR encodes it, samples candidate expressions, refines their constants and selects among them, without any further training.

We fully agree that reinforcement learning has been used in neural symbolic regression before, and we thank the reviewer for pointing us to DGSR, SNR and uDSR, which our initial submission did not discuss. We have added all three to the related work (Section 2 and Appendix A). Throughout the paper, for example in the abstract and Section 3.3, we now state more explicitly that post-training runs once on synthetic data.

The key difference lies in what happens at inference. Each of the three mentioned methods continues training on every new dataset, so each one requires an additional, time-consuming search:

  • DGSR pre-trains with a policy gradient on fit and then continues training the decoder on each test dataset with RL and genetic programming, for up to 2M expression evaluations.
  • uDSR optionally pre-trains with supervised learning or RL and then fine-tunes on each test dataset with a risk-seeking policy gradient, combined with genetic programming and other components.
  • SNR pre-trains a policy network and then rectifies its predicted structure on each test dataset with a risk-seeking policy gradient.

RASR instead moves this effort into training. It keeps cross-entropy as the scalable pre-training objective, and then adds a much smaller RL stage that rewards held-out fit and compact expressions. This post-training even makes inference faster. At the 94M scale, it cuts the mean fitting time on the black-box track from 762 to 133 seconds while also improving accuracy (Appendix N).

Beyond post-training, we also see our synthetic prior as a methodological contribution in its own right. As the reviewer notes, deficiencies in the training data distribution are a main reason why pre-trained models struggle on black-box data. Our prior is our answer to exactly this problem. As for constants, we agree that emitting them builds on E2E (Appendix A), and we do not claim this as a main contribution. Our variant uses the emitted constants only as an initialization for refinement, which improves over a skeleton-only variant (9.51 vs. 12.09, Table O.3).

AuthorsOpenReview ↗

Response to Reviewer 2ntM (Part 3)

W4/Q4: Validating the data-generation contribution

Can you present ablation studies that isolate the effect of your data generation method, controlling for training data size?

Table O.1 (Table N.1 in the initial submission) isolates the prior in this way. Both 24M models share the architecture, the pre-training budget (250k steps, about 16M datasets) and the post-training, and differ only in the used data-generation method. The simple prior follows earlier work, with independent uniform inputs, constants from fixed ranges and no standardization. With the same amount of data, our prior lowers mean improvability from 26.56 to 9.36 before post-training, and from 18.31 to 9.51 after it.

This raises concerns that performance gains may simply result from using larger training datasets rather than improved data quality.

RASR is also not trained on more data than earlier pre-trained models. The table lists the number of dataset–expression pairs seen during pre-training, as reported in the respective papers:

ModelPre-training pairs
NeSymReS≈225M
E2E≈150M
SNIP≥60M
Flash-ANSR (120M)≈512M
RASR-94M≈62M
RASR-24M≈16M

Our 24M model, trained on only about 16M datasets, already ranks ahead of all baselines on the black-box track (Figure K.2).

How do you quantitatively validate that your generated data distribution more closely matches real-world data distributions?

We agree that a direct measure of similarity to real-world data would be valuable. Such a measure is, however, difficult to define. Our prior is a distribution over datasets of varying size and dimensionality, while real data comes as a small number of heterogeneous datasets. The field of prior-data fitted networks (PFNs; Müller et al., 2022), which includes tabular foundation models such as TabPFN and TabICL, faces the same problem. There, the quality of a prior is commonly judged by how well the trained model performs on real data, and we follow this practice. The revision adds an ablation that evaluates each component of our prior in exactly this way, by removing it and measuring the effect on the real-world datasets. Without the TabICLv2 inputs, the bottom-up constants or the noise, the model is significantly worse on four or five of the real-world black-box datasets, and better on none (Table O.2).

We hope these clarifications and additions address the reviewer's concerns, and we are happy to discuss any remaining questions.

References

Guilherme Seidyo Imai Aldeia, Hengzhe Zhang, Geoffrey F. Bomarito, Miles D. Cranmer, Alcides Fonseca, Bogdan Burlacu, William G. La Cava, and Fabrício Olivetti de França. Call for action: towards the next generation of symbolic regression benchmark. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO), 2025.

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian Inference. In International Conference on Learning Representations (ICLR), 2022.

Reviewer_HPPMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RASR, which improves neural symbolic regression through a more realistic synthetic prior and reinforcement-learning post-training. It achieves the best mean test R2R^2 rank on the 12 black-box datasets in SRBench 2.0 and performs particularly well on Strogatz.

Critical strengths

Strengths:

  1. The method is coherent and directly optimizes both predictive fit and expression complexity during post-training.

  2. The evaluation covers both black-box regression and ground-truth equation recovery.

  3. The accuracy–complexity Pareto results are compelling, and the implementation details are thorough.

Critical weaknesses
  1. The prior ablation changes the input distribution, noise, constant sampling, and standardization simultaneously, so the contribution of each component remains unclear.

  2. Evidence for the accuracy benefit of post-training is mixed: at the 24M scale, improvability changes from 9.36 to 9.51, while RS-GRPO and standard GRPO are very close at large sampling budgets. Repeated post-training runs with uncertainty estimates would strengthen the claim.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HPPM

We thank the reviewer for the careful review and for the two concrete suggestions. We respond to both weaknesses below. Section, figure and table numbers refer to the revised PDF.

W1: Ablating the prior components

The prior ablation changes the input distribution, noise, constant sampling, and standardization simultaneously, so the contribution of each component remains unclear.

We thank the reviewer for raising this point. Since the simple prior of the initial submission differed from ours in all four components at once, the gap between the two could not be attributed to any single change. We have therefore extended our 24M ablations with a component ablation (Table O.2), which removes the TabICLv2 inputs, the bottom-up constants and the noise from our prior one at a time, and adds our standardization to the simple prior. Without TabICLv2 inputs, all columns are drawn independently and uniformly, as in Flash-ANSR, and without bottom-up constants, every constant is drawn from N(0,52)\mathcal{N}(0, 5^2) independently of the data. Each variant is pre- and post-trained on its own prior. Besides mean improvability, we count the datasets on which RASR-24M-L has a significantly lower or higher test RMSE, separately for the nine real-world and three synthetic Friedman datasets of the black-box track.

PriorImprovability (%)Real-world (9)Friedman (3)
Our prior (RASR-24M-L)9.51––
− TabICLv2 inputs8.564 / 00 / 2
− bottom-up constants13.514 / 03 / 0
− noise11.355 / 01 / 1
Simple prior18.315 / 03 / 0
+ standardization16.966 / 03 / 0

The last two columns count the datasets on which RASR-24M-L is significantly better / worse than the variant.

TabICLv2 inputs, bottom-up constants and noise each help on real data. Without any one of them, the model is significantly worse on four or five of the nine real-world datasets and better on none. Adding standardization to the simple prior, in contrast, closes little of its gap to our prior, so standardization alone does not account for our prior's advantage.

The variant without TabICLv2 inputs nonetheless reaches a slightly lower mean improvability than our prior, 8.56 against 9.51. This difference comes entirely from two Friedman datasets, fri_c0_250_5 and fri_c2_1000_10, whose features largely behave like the independent uniform draws of that variant's prior. The weakness is specific to the 24M scale. On these two datasets, post-training worsens our 24M model but improves our 94M model, which points to a lack of capacity at the smaller scale rather than a deficit of our prior (see W2).

W2: Accuracy benefit of post-training

Evidence for the accuracy benefit of post-training is mixed [...]. Repeated post-training runs with uncertainty estimates would strengthen the claim.

We thank the reviewer for this suggestion. We have post-trained RASR-94M-L twice more from the same pre-trained checkpoint with different random seeds, and evaluated both new runs with the same protocol (Table N.2). For compute reasons, we did this only for the main model, on which the paper's claims rest. Run 1 is the main model reported throughout the paper. It was trained before the other two, so it was not selected among them.

ModelImprovability (%)Expression sizeFitting time (s)
Pre-trained base10.0437.4762
Post-training run 1 (main model)7.4427.5133
Post-training run 27.9328.5159
Post-training run 37.8523.7122
Mean ± std over runs7.74 ± 0.2726.6 ± 2.6138 ± 19

All three runs improve on the base model on all three axes, by a margin far larger than the variation between runs.

at the 24M scale, improvability changes from 9.36 to 9.51

At 24M, this flat mean hides two opposing effects. Post-training lowers improvability on the nine real-world datasets from 6.52 to 4.60, but raises it on the two Friedman datasets discussed under W1 from 4.38 to 15.52. At 94M, it improves both, from 7.54 to 4.80 and from 4.09 to 1.50, which again points to the limited capacity of the smaller model. Even at 24M, post-training shortens the returned expressions from 39.3 to 28.5 nodes and cuts the fitting time from 646 to 131 seconds (Table N.1).

RS-GRPO and standard GRPO are very close at large sampling budgets.

We agree that the two objectives end up close, and the paper does not claim a separation between them at large budgets (Appendix N). Both, however, improve on the base model at nearly every budget (Figure N.2).

We hope these additions address the reviewer's concerns, and we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer HPPM

Dear Reviewer HPPM,

Thank you again for your review and suggestions. As the discussion period ends on November 18, we would be grateful for your feedback on our responses and the additional experiments. Please let us know if any questions remain.

Thank you to the author for the patient response; my concerns have been addressed, and I will maintain my positive rating.

Reviewer_PjvCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RASR, a neural symbolic regression method trained on a diverse synthetic prior and refined through reinforcement-learning post-training. The prior introduces correlated inputs, data-conditioned constant sampling, noise, and standardization. Post-training rewards held-out predictive fit and penalizes expression size. At inference, the model generates candidate expressions and numerically refines their constants. Experiments cover SRBench 2.0 and the Feynman and Strogatz benchmarks, with analyses of training choices, expression size, and fitting cost.

Critical strengths

1.The training changes address concrete problems. Correlated inputs, noise, and data-conditioned constants make the training distribution more relevant to practical regression. Evaluating the post-training reward on observations held out from coefficient fitting is also a sensible choice.

2.The method shows useful gains beyond a single accuracy number. At the 94M scale, post-training improves predictive performance while producing smaller expressions and reducing per-task fitting time. The comparisons across sampling budgets and size penalties help explain the practical trade-offs.

Critical weaknesses

1.The equation-recovery claims need direct recovery experiments.

Section 4.3 measures success by test R2 > 0.999, and Figure 5 presents these results as ground-truth recovery. This establishes high numerical accuracy on the test distribution, but does not check whether the returned expression represents the underlying law. Node count measures compactness and does not resolve this issue.

The authors should report symbolic recovery rates on Feynman and Strogatz for the pre-trained model, the post-trained model, and the main baselines. The recovery rule should allow algebraically equivalent expressions and clearly explain the treatment of fitted constants and noise. This comparison is particularly important because post-training rewards fit and size without using the reference equation. We need to know whether it improves equation recovery or mainly produces better compact approximations. I consider this a necessary addition to support the paper’s scientific-discovery claims.

2.The prior ablation does not identify which changes matter.

Appendix N changes the input distribution, constant sampling, and standardization together. The simple-prior baseline receives no standardization, so the large performance gap cannot be attributed specifically to the more distinctive prior design. A simple-prior baseline with the same standardization would make the comparison much more informative. Targeted ablations of correlated inputs and bottom-up constant sampling would then show which parts of the recipe are worth adopting.

Questions for authors

1.Can you report symbolic recovery rates on Feynman and Strogatz using the same criterion for RASR and the main baselines?

2.How does equation recovery change before and after post-training?

3.Among runs with test R2 > 0.999, how many actually recover an equivalent ground-truth expression?

4.How will the recovery criterion handle algebraic equivalence, fitted coefficients, and noisy observations?

5.How does the simple-prior baseline perform with the same standardization as RASR?

6.What are the separate contributions of correlated inputs and bottom-up constant sampling?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PjvC (Part 1)

We thank the reviewer for the careful review and the concrete suggestions, which we have addressed in the revision. We respond to each weakness and question below. Section, figure and table numbers refer to the revised PDF.

W1/Q1: Symbolic recovery on Feynman and Strogatz

The equation-recovery claims need direct recovery experiments.

The initial submission scored Feynman and Strogatz only by the fraction of runs reaching test R2>0.999R^2 > 0.999. This measures accuracy, not whether the returned expression is the law, and we did not make this distinction clear enough. We have clarified it throughout the revised paper. Since recovering the law is what ultimately matters in scientific discovery, we have added SRBench's symbolic solution rate for all methods, including PySR, which we now also run on these tracks (Section 4.3, Figure L.1).

Without noise, plain RASR-94M-L recovers the law in only 13.6% of the Strogatz and 24.2% of the Feynman runs. This is because our prior places constants at nearly every operator argument, so that one skeleton can realize many different functions. Refinement fits all these constants, so even an otherwise correct expression keeps small artifacts, such as a negligible extra term or a coefficient of 0.998 where the law has 1, and no longer simplifies to the law.

We therefore added an optional expression pruning step (Section 3.4, Appendix G) and call the main model with it RASR-94M-L+. Pruning simplifies parts of the selected expression that are already close to a simpler form, for example by dropping a negligible summand or rounding 0.998 to 1, and keeps each simplification only if the predictions on the training inputs change by at most 1%. Like the rest of inference, it uses only the training data. On Feynman, pruning shrinks the expressions from 34 to 21 nodes, while the share of runs reaching R2>0.999R^2 > 0.999 drops by less than one percentage point at any noise level.

  • Strogatz: without noise, RASR-94M-L+ recovers the law in 61.4% of runs against 42.1% for PySR, the strongest baseline, while PySR is ahead at the two highest noise levels. Averaged over noise levels, RASR-94M-L+ ranks first.
  • Feynman: without noise, RASR-94M-L+ recovers the law in 41.7% of runs, behind AIFeynman, Flash-ANSR and PySR, and it stays between 36% and 43% at every noise level. Averaged over noise levels, it ranks third, behind PySR and Flash-ANSR.

Q4: Recovery criterion

How will the recovery criterion handle algebraic equivalence, fitted coefficients, and noisy observations?

We use SRBench's symbolic check without changes (Appendix I). The released results include it for the original baselines, and we apply the same code to the methods we ran ourselves, so all methods are evaluated under the same criterion.

  • Algebraic equivalence: the difference and the ratio between the true law and the returned expression are simplified with SymPy, and a run counts as solved if either is constant. Any equivalent form therefore passes, as does one that differs from the law by an additive constant or a constant factor.
  • Fitted coefficients: constants are rounded to three decimals before the check, and those below 10−410^{-4} in magnitude are set to zero. Apart from a global offset and scale, coefficients must therefore match the law to this precision. The check applies only to expressions with test R2>0.5R^2 > 0.5.
  • Noisy observations: as in SRBench, noise is added only to the training targets, and every run is compared with the noise-free law. Noise thus leaves the criterion unchanged but makes it harder to meet.

Q3: Accurate runs that recover the law

Among runs with test R2 > 0.999, how many actually recover an equivalent ground-truth expression?

Of the noise-free runs that reach R2>0.999R^2 > 0.999, RASR-94M-L+ recovers the law in 62.3% on Strogatz and 53.0% on Feynman, against 13.7% and 30.7% for plain RASR-94M-L and 68.6% and 67.8% for PySR. Since pruning only removes parts that change the training predictions by at most 1%, its gain over plain RASR-94M-L shows that many accurate expressions are the law up to near-redundant parts.

Q2: Recovery before and after post-training

How does equation recovery change before and after post-training?

We agree that this is an interesting question, since post-training rewards fit and size without ever seeing a reference expression. So far, we have evaluated only the post-trained model on Feynman and Strogatz, and we have now started the same evaluation for the pre-trained base model, with and without pruning. With 130 problems at four noise levels and ten seeds each, and a base model that fits several times more slowly than the post-trained one (Appendix N), these runs take considerable time, and we cannot yet tell for sure whether they will finish within the discussion period. If they do, we will post the results here, and otherwise we will add them to the final version.

AuthorsOpenReview ↗

Response to Reviewer PjvC (Part 2)

W2/Q5/Q6: Ablating the prior components

The prior ablation does not identify which changes matter.

We thank the reviewer for this suggestion. Since the simple prior in our initial ablation differed from ours in inputs, constants, noise and standardization at once, the gap could not be attributed to any single change. We have therefore extended our 24M ablations with a component ablation (Table O.2), which removes one component of our prior at a time and adds our standardization to the simple prior. Each variant is pre- and post-trained on its own prior. Besides mean improvability, we count the datasets on which RASR-24M-L has a significantly lower or higher test RMSE, separately for the nine real-world and three synthetic Friedman datasets of the black-box track.

PriorImprovability (%)Real-world (9)Friedman (3)
Our prior (RASR-24M-L)9.51––
− TabICLv2 inputs8.564 / 00 / 2
− bottom-up constants13.514 / 03 / 0
− noise11.355 / 01 / 1
Simple prior18.315 / 03 / 0
+ standardization16.966 / 03 / 0

The last two columns count the datasets on which RASR-24M-L is significantly better / worse than the variant.

How does the simple-prior baseline perform with the same standardization as RASR?

Standardization lowers the simple prior's improvability only from 18.31 to 16.96, far from the 9.51 of our prior. Our prior remains significantly better on nine of the twelve black-box datasets and worse on none, so standardization alone does not account for its advantage.

What are the separate contributions of correlated inputs and bottom-up constant sampling?

Both help on real data. Drawing every constant from N(0,52)\mathcal{N}(0, 5^2) independently of the data, instead of bottom-up, raises improvability to 13.51, with RASR-24M-L being significantly better on four real-world and all three Friedman datasets. Drawing all columns independently and uniformly, as in Flash-ANSR, instead of from the TabICLv2 prior makes the model significantly worse on four real-world datasets and better on none. Removing the noise has a similar effect.

Without TabICLv2 inputs, the mean improvability is nonetheless slightly lower, 8.56 against 9.51. This comes entirely from two Friedman datasets, fri_c0_250_5 and fri_c2_1000_10, whose features largely behave like the independent uniform draws of that variant's prior. The weakness is specific to the 24M scale. There, post-training worsens improvability on these two datasets from 4.38 to 15.52 while improving it on the real-world datasets from 6.52 to 4.60, whereas at 94M it improves both, from 4.09 to 1.50 and from 7.54 to 4.80. This suggests that the smaller model lacks the capacity to serve both input regimes, rather than that our prior is unsuited to them.

We hope these additions address the reviewer's concerns, and we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer PjvC

Dear Reviewer PjvC,

Thank you again for your review and suggestions. As the discussion period ends on November 18, we would be grateful for your feedback on our responses and the additional experiments. Please let us know if any questions remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.