- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The paper studies how symbolic-regression rankings change with the evaluation metric, aggregation over runs, ranking method, and dataset composition.
- It analyzes 23 methods on 21 SRBench datasets using 15 MCDM ranking schemes and proposes ranking-scheme and dataset-composition robustness indicators.
- The main conclusion is that EQL and PS-TREE are strongest on real-world data, while QLattice and Operon are strongest on first-principles data. Five selected methods are also tested on five external datasets.
Critical strengths
- The paper asks an important practical question: whether an SR “winner” remains strong under reasonable changes to the evaluation design. The separation between real-world and first-principles datasets is especially useful (some version of this was already done by SRBench).
- The analysis covers many metrics, aggregation rules, and ranking schemes, and the release of scores and analysis artifacts should help reproducibility.
Critical weaknesses
- The originality is limited. The paper does not introduce a new symbolic-regression method; it mainly applies a ranking-robustness framework to existing SRBench results. The evaluation question is useful, but the paper should clarify what is methodologically new beyond the prior robustness framework and the underlying benchmark.
- The paper misses an opportunity to rerun all methods under the same hardware and software conditions. Instead, it relies on previously reported scores and excludes GP-GOMEA and uDSR because some results are missing. This makes the comparison incomplete and may confound the training-time rankings.
- “Robustness” is based on rank variance and top-10 membership, but no threshold for declaring a method robust is defined. The top-10 cutoff includes almost half of the 23 methods, and its sensitivity is not reported. Dataset-composition robustness also uses only three random subsamples.
Questions for authors
- Why did the authors exclude GP-GOMEA and uDSR instead of running the missing method-dataset combinations themselves? Please also cite the primary publication for every method named in the paper. In particular, uDSR should cite Landajuela et al. (2022), A Unified Framework for Deep Symbolic Regression. Petersen et al. (2021), Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients, is also relevant to the discussion of neural SR methods.
- Were all training times measured under the same hardware, software environment, and resource budget? If not, the relative timing results may not be fair. Running the methods under one controlled setup would provide a more accurate comparison.
- Can the authors report sensitivity to the top- cutoff and use more dataset subsamples? Because the external validation compares only five preselected methods, please either validate against all feasible methods or limit the claim to the relative ordering of those five.
5 comments
Response to Weakness 3 and Questions 1-2
“Robustness” is based on rank variance and top-10 membership, but no threshold for declaring a method robust is defined. The top-10 cutoff includes almost half of the 23 methods, and its sensitivity is not reported. Dataset-composition robustness also uses only three random subsamples.
For us, a method is robust when its rank does not change with the evaluation design. RS and DS measure this continuously, as rank variance across schemes and across compositions, and cross-consistency counts how often a method stays in the top-η across metrics or aggregations. Each captures a different aspect, so we report all three. We agree, however, that the text used "robust" and "fully robust" without a threshold. The revision defines a method as RS-stable (DS-stable) when its rank variance across schemes (compositions) is at most 5, and as robustly top-ranked when it is stable under both and in the top-η under every scheme and composition. This also separates stability from quality: a method that is always ranked last is stable but not robustly top-ranked. With 26 methods, η = 10 covers 38% of the field rather than almost half, and we now also report η = 5 (Q3) in Appendix K, showing that the top-ranked methods, under each regime, are almost positioned the same way even when switching to η = 5. A sensitivity on the correlation threshold with 0.7 and 0.8 is already provided in Appendix G along with sensitivity of ranking schemes and dataset compositions in Appendix H (computed with 9 random seeds this time).
Why did the authors exclude GP-GOMEA and uDSR instead of running the missing method-dataset combinations themselves? Please also cite the primary publication for every method named in the paper. In particular, uDSR should cite Landajuela et al. (2022), A Unified Framework for Deep Symbolic Regression. Petersen et al. (2021), Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients, is also relevant to the discussion of neural SR methods.
Thanks for your comment. All scores come from SRBench's own runs. Our hardware differs from SRBench's cluster, so results we produced for the missing pairs would not be comparable, above all for training time, which is one of our criteria. To include both methods without mixing sources, the revision adds a complete matrix of 28 methods on the 10 datasets where both have results (Appendix J). The revision cites the primary publication of every method in Section 4, including Landajuela et al. (2022) for uDSR, and cites Petersen et al. (2021) in the discussion of neural SR methods. Thank you again for providing us these references.
Were all training times measured under the same hardware, software environment, and resource budget? If not, the relative timing results may not be fair. Running the methods under one controlled setup would provide a more accurate comparison.
Thanks for your comment. All times are those recorded by SRBench 2025: every method ran in its own container on the same SLURM cluster with the same budget (1 hour per fit, 8 hours per job, 10 GB of memory) and the same 30 seeds. The five methods in the external validation were all re-run by us on one machine (32GB RAM, intel core ultra 9, 16-cores) with the same budget, so their times are directly comparable.
Response to Question 3
Can the authors report sensitivity to the top-(\eta) cutoff and use more dataset subsamples? Because the external validation compares only five preselected methods, please either validate against all feasible methods or limit the claim to the relative ordering of those five.
Thank you for flagging this. The analysis you mention is highly relevant and interesting. Eta parameter: every analysis in the main paper is now reported for η = 5 in Appendix K, showing that the top-ranked methods, under each regime, are almost positioned the same way even when switching to η = 5. Subsamples: the number of distinct subsamples is the product of the cluster sizes, so drawing more seeds eventually repeats subsamples and over-weights them. More compositions with other random seeds are presented in Appendix L. Each subsample has between 17 and 19. datasets, depending on which metric is taken into account. This means that the 22 datasets are clustered into 17 to 19 clusters depending on which metric is taken into account, where most of the clusters contain a single dataset, while only a few contain, occasionally, 2 to 4 datasets. This is actually expected since the next-generation SRBench was already designed to cover datasets that are already uncorrelated and diverse from each other. We nonetheless included the analysis on dataset compositions, in line with the text embedding benchmark, to check sensitivity w.r.t. changes in the number of datasets under consideration. This is why our initial submissions only included three random seeds for dataset subsampling. We now computed up to nine random seeds for dataset subsampling for better stability. We believe that more random seeds are not needed given the very small size of the clusters, since they would lead to many overlapping dataset compositions. Our dendrograms show the first three random seeds for space availability reasons, while the robustness indicators (RS and DS), along with the metric- and aggregation-agnostic measures, are computed by accounting for all 9 random seeds. Also, Kendall’s W in Appendix H has been recomputed across the 9 random seeds. Our analysis shows that the overall-top-ranked methods stay robust regardless of the random seed employed, while Kendall’s W (0.95-1.00) confirms that rankings remain highly consistent regardless of the random seed used for dataset subsampling, even when 9 random seeds are considered instead of only 3. External validation: we agree and now limit the claim. The five methods were selected as the overall top-ranked ones, so the experiment tests whether their relative ordering is preserved on held-out data, not whether other methods would enter the core. The text now says exactly that.
Response to Weakness 1-2
The originality is limited. The paper does not introduce a new symbolic-regression method; it mainly applies a ranking-robustness framework to existing SRBench results. The evaluation question is useful, but the paper should clarify what is methodologically new beyond the prior robustness framework and the underlying benchmark.
The paper misses an opportunity to rerun all methods under the same hardware and software conditions. Instead, it relies on previously reported scores and excludes GP-GOMEA and uDSR because some results are missing. This makes the comparison incomplete and may confound the training-time rankings.