- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a symbolic regression framework that consists of three main stages: (1) a data generation method designed to produce training samples that more closely resemble real-world data distributions, (2) pre-training of a deep learning model on this synthetically generated data, and (3) expression inference via reinforcement learning. The central hypothesis is that the poor performance of pre-trained symbolic regression models on black-box datasets stems primarily from a mismatch between the synthetic training data and real data distributions. By addressing this data gap, the authors aim to enhance the generalization capability of pre-trained models, particularly in high-dimensional settings where pure search-based methods become computationally intractable.
Critical strengths
The authors correctly identify that the limited success of pre-trained symbolic regression models on black-box data is largely due to deficiencies in the training data distribution. This observation is both novel and valuable. If effectively addressed, it could substantially improve the practical utility of pre-trained models, especially as input dimensionality grows and search-based methods struggle to maintain accuracy.
Critical weaknesses
-
Incomplete Experimental Comparison: While the proposed method demonstrates superior performance over PySR on 12 black-box models, it notably omits comparisons on the Strogatz and Feynman datasets. Given PySR’s strong performance on these benchmarks, such comparisons are essential for a fair and comprehensive evaluation. Although SRBench2.0 may not include these datasets, the authors should still provide results to contextualize their method’s effectiveness.
-
Insufficient Black-Box Dataset Diversity: The paper’s core motivation is to improve pre-trained model performance on black-box data; however, it evaluates on only 12 black-box datasets from SRBench2.0. The PMLB repository contains a much larger collection of black-box problems, and a more extensive evaluation across diverse datasets is necessary to convincingly validate the method’s generalizability.
-
Limited Methodological Innovation: The proposed approach combines pre-training with reinforcement learning, which has been explored in prior works such as SNR[1], DGSR[2], and uDSR[3]. The distinction from TPSR (pre-training + Monte Carlo Tree Search) is noted, but the overall framework closely resembles GSR, with the only notable difference being the use of refined constant encoding (which itself was introduced in E2E). Consequently, the paper lacks significant methodological novelty.
-
Inadequate Validation of Data Generation Contribution: The primary claimed contribution is the generation of more realistic data, yet evidence supporting this claim is weak. The method is only evaluated on 12 black-box datasets, with no ablation studies comparing performance against pre-trained models trained on the same amount of data without the proposed generation method. This raises concerns that performance gains may simply result from using larger training datasets rather than improved data quality. Furthermore, the paper does not provide any analysis (e.g., distribution similarity metrics, comparison with real-world expression statistics) to demonstrate that the generated data truly aligns better with real-world data distributions.
[1] Deep generative symbolic regression, 2023, ICLR
[2] SNR: Symbolic network-based rectifiable learning framework for symbolic regression, 2023, Neural Networks
[3] A unified framework for deep symbolic regression, 2022, NeurIPS
Questions for authors
-
Could you provide experimental comparisons with PySR on the Strogatz and Feynman datasets to more thoroughly benchmark your method against state-of-the-art symbolic regression tools?
-
Why were only 12 black-box datasets used? Would it be feasible to include additional black-box problems from PMLB to strengthen the empirical claims?
-
Given the similarities between your framework and existing methods like DGSR and SNR, what specific novel components does your method introduce beyond constant encoding and RL fine-tuning?
-
Can you present ablation studies that isolate the effect of your data generation method, controlling for training data size? Additionally, how do you quantitatively validate that your generated data distribution more closely matches real-world data distributions compared to existing synthetic data generation approaches?
7 comments
Response to Reviewer 2ntM (Part 2)
W3/Q3: Novelty with respect to DGSR, SNR and uDSR
Given the similarities between your framework and existing methods like DGSR and SNR, what specific novel components does your method introduce beyond constant encoding and RL fine-tuning?
We suspect there may be a misunderstanding about how reinforcement learning is used. The review summarizes our third stage as "expression inference via reinforcement learning", which tells us that we should have explained this more clearly. We apply reinforcement learning only once, on synthetic data, before the model is ever used on a real dataset. After this post-training stage, the model is frozen. Given a new dataset, RASR encodes it, samples candidate expressions, refines their constants and selects among them, without any further training.
We fully agree that reinforcement learning has been used in neural symbolic regression before, and we thank the reviewer for pointing us to DGSR, SNR and uDSR, which our initial submission did not discuss. We have added all three to the related work (Section 2 and Appendix A). Throughout the paper, for example in the abstract and Section 3.3, we now state more explicitly that post-training runs once on synthetic data.
The key difference lies in what happens at inference. Each of the three mentioned methods continues training on every new dataset, so each one requires an additional, time-consuming search:
- DGSR pre-trains with a policy gradient on fit and then continues training the decoder on each test dataset with RL and genetic programming, for up to 2M expression evaluations.
- uDSR optionally pre-trains with supervised learning or RL and then fine-tunes on each test dataset with a risk-seeking policy gradient, combined with genetic programming and other components.
- SNR pre-trains a policy network and then rectifies its predicted structure on each test dataset with a risk-seeking policy gradient.
RASR instead moves this effort into training. It keeps cross-entropy as the scalable pre-training objective, and then adds a much smaller RL stage that rewards held-out fit and compact expressions. This post-training even makes inference faster. At the 94M scale, it cuts the mean fitting time on the black-box track from 762 to 133 seconds while also improving accuracy (Appendix N).
Beyond post-training, we also see our synthetic prior as a methodological contribution in its own right. As the reviewer notes, deficiencies in the training data distribution are a main reason why pre-trained models struggle on black-box data. Our prior is our answer to exactly this problem. As for constants, we agree that emitting them builds on E2E (Appendix A), and we do not claim this as a main contribution. Our variant uses the emitted constants only as an initialization for refinement, which improves over a skeleton-only variant (9.51 vs. 12.09, Table O.3).
Response to Reviewer 2ntM (Part 3)
W4/Q4: Validating the data-generation contribution
Can you present ablation studies that isolate the effect of your data generation method, controlling for training data size?
Table O.1 (Table N.1 in the initial submission) isolates the prior in this way. Both 24M models share the architecture, the pre-training budget (250k steps, about 16M datasets) and the post-training, and differ only in the used data-generation method. The simple prior follows earlier work, with independent uniform inputs, constants from fixed ranges and no standardization. With the same amount of data, our prior lowers mean improvability from 26.56 to 9.36 before post-training, and from 18.31 to 9.51 after it.
This raises concerns that performance gains may simply result from using larger training datasets rather than improved data quality.
RASR is also not trained on more data than earlier pre-trained models. The table lists the number of dataset–expression pairs seen during pre-training, as reported in the respective papers:
| Model | Pre-training pairs |
|---|---|
| NeSymReS | ≈225M |
| E2E | ≈150M |
| SNIP | ≥60M |
| Flash-ANSR (120M) | ≈512M |
| RASR-94M | ≈62M |
| RASR-24M | ≈16M |
Our 24M model, trained on only about 16M datasets, already ranks ahead of all baselines on the black-box track (Figure K.2).
How do you quantitatively validate that your generated data distribution more closely matches real-world data distributions?
We agree that a direct measure of similarity to real-world data would be valuable. Such a measure is, however, difficult to define. Our prior is a distribution over datasets of varying size and dimensionality, while real data comes as a small number of heterogeneous datasets. The field of prior-data fitted networks (PFNs; Müller et al., 2022), which includes tabular foundation models such as TabPFN and TabICL, faces the same problem. There, the quality of a prior is commonly judged by how well the trained model performs on real data, and we follow this practice. The revision adds an ablation that evaluates each component of our prior in exactly this way, by removing it and measuring the effect on the real-world datasets. Without the TabICLv2 inputs, the bottom-up constants or the noise, the model is significantly worse on four or five of the real-world black-box datasets, and better on none (Table O.2).
We hope these clarifications and additions address the reviewer's concerns, and we are happy to discuss any remaining questions.
References
Guilherme Seidyo Imai Aldeia, Hengzhe Zhang, Geoffrey F. Bomarito, Miles D. Cranmer, Alcides Fonseca, Bogdan Burlacu, William G. La Cava, and Fabrício Olivetti de França. Call for action: towards the next generation of symbolic regression benchmark. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO), 2025.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do Bayesian Inference. In International Conference on Learning Representations (ICLR), 2022.
Response to Reviewer 2ntM (Part 1)
We thank the reviewer for the thoughtful review and for recognizing the relevance of the training-data mismatch we identify for pre-trained symbolic regression models. We respond to each weakness and question below. Section, figure and table numbers refer to the revised PDF.
W1/Q1: PySR on Strogatz and Feynman
We appreciate the reviewer pointing out this gap. The Feynman and Strogatz tracks come from the original SRBench, whose published results we used and extended with Flash-ANSR and SNIP. PySR only joined the benchmark with SRBench 2.0, so it was missing from these tracks. We have now run PySR on both tracks with its SRBench 2.0 configuration and added it to all Feynman and Strogatz results (Section 4.3, Figure 5, Appendix L).
As the reviewer anticipated, PySR is a strong baseline on these tracks, particularly for recovering the exact law. RASR, in turn, is ahead of PySR in accuracy on both tracks and also recovers the exact law more often on Strogatz. Since accuracy alone does not show whether the law was found, the revision now also reports the symbolic solution rate using SRBench's own check. It also adds RASR-94M-L+, our main model with an optional pruning step that removes near-redundant constants using only the training data (Section 3.4, Appendix G).
W2/Q2: Number of black-box datasets
We agree that twelve datasets are few, and we list this as a limitation (Section 5). The small number is, however, a deliberate choice of SRBench 2.0. The black-box track of the original SRBench comprised 122 PMLB datasets, but this larger pool distorted the results in two ways. About half of the datasets are variants of the Friedman benchmark, which skewed aggregate results towards this single family of synthetic problems. In addition, many datasets are solved almost perfectly by every method, so they add little information about how methods differ.
SRBench 2.0 therefore purposely reduced the track to twelve datasets (Aldeia et al., 2025, Section 3.1). It first removed every dataset on which all previously benchmarked methods, or plain linear regression, reach R2>0.99. It then clustered the remaining datasets by size, dimensionality and the performance of these methods, and kept the dataset closest to each of twelve cluster centres, with Friedman datasets capped at 25% of the selection. Adding further PMLB datasets would risk reintroducing exactly these biases. Following the benchmark's protocol also keeps our results directly comparable to the published runs of all 25 methods it evaluates. Constructing a better black-box benchmark is important, but beyond the scope of this work.