acceptodds
Under review as a conference paper at ICLR 2027

Who Really Wins? Stress-Testing Symbolic Regression Benchmarks Across Metrics, Aggregations, and Datasets

Abstract

Symbolic regression (SR) benchmarks such as SRBench rank methods by averaging a single accuracy metric over repeated runs across evaluated datasets. Yet, the conclusions drawn from such leaderboards depend on implicit design choices, i.e., which error metric is reported, how repeated runs are aggregated, and which datasets are included. These design choices are rarely stress-tested. In this paper, we present a meta-study about the robustness of the SR ranking built on the recently proposed next-generation SRBench. Using each performance metric (MSE, MAE, , and training time) and each run-aggregation rule (mean, median, max, min, standard deviation, and four random single-run picks), we generate model rankings with a portfolio of multi-criteria decision-making (MCDM) schemes and introduce two robustness indicators: ranking-scheme robustness (sensitivity of rankings to both aggregation methodology and chosen metric) and dataset-composition robustness (sensitivity to correlation-aware dataset subsampling). Then, we perform metric-agnostic and aggregation-agnostic cross-consistency analysis. In addition, we complement our robustness analysis by isolating real-world (black-box) from physical-law-based (first-principles) datasets. Across SR methods and datasets, we find that no single method dominates every regime, but a small core—most consistently EQL and PS-Tree—remains robust across metrics, aggregations, and dataset compositions on real-world data, while QLattice and Operon rise to the top on first-principles problems. We validate five among the overall-top-ranked methods on five held-out external, well-established datasets. Our findings support the call for action of the SR community for a living benchmark and show that robustness indicators, rather than single averaged scores, should accompany future SR leaderboards. We publicly release all model scores and derived rankings as reusable benchmark artifacts, enabling new benchmarking studies without rerunning the original experiments.

Primary area:
datasets and benchmarks
Keywords:
symbolic regression, genetic programming, machine learning, supervised learning, benchmark, sensitivity, multi-criteria decision-making

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EAkKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies how symbolic-regression rankings change with the evaluation metric, aggregation over runs, ranking method, and dataset composition.
  • It analyzes 23 methods on 21 SRBench datasets using 15 MCDM ranking schemes and proposes ranking-scheme and dataset-composition robustness indicators.
  • The main conclusion is that EQL and PS-TREE are strongest on real-world data, while QLattice and Operon are strongest on first-principles data. Five selected methods are also tested on five external datasets.
Critical strengths
  1. The paper asks an important practical question: whether an SR “winner” remains strong under reasonable changes to the evaluation design. The separation between real-world and first-principles datasets is especially useful (some version of this was already done by SRBench).
  2. The analysis covers many metrics, aggregation rules, and ranking schemes, and the release of scores and analysis artifacts should help reproducibility.
Critical weaknesses
  1. The originality is limited. The paper does not introduce a new symbolic-regression method; it mainly applies a ranking-robustness framework to existing SRBench results. The evaluation question is useful, but the paper should clarify what is methodologically new beyond the prior robustness framework and the underlying benchmark.
  2. The paper misses an opportunity to rerun all methods under the same hardware and software conditions. Instead, it relies on previously reported scores and excludes GP-GOMEA and uDSR because some results are missing. This makes the comparison incomplete and may confound the training-time rankings.
  3. “Robustness” is based on rank variance and top-10 membership, but no threshold for declaring a method robust is defined. The top-10 cutoff includes almost half of the 23 methods, and its sensitivity is not reported. Dataset-composition robustness also uses only three random subsamples.
Questions for authors
  1. Why did the authors exclude GP-GOMEA and uDSR instead of running the missing method-dataset combinations themselves? Please also cite the primary publication for every method named in the paper. In particular, uDSR should cite Landajuela et al. (2022), A Unified Framework for Deep Symbolic Regression. Petersen et al. (2021), Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients, is also relevant to the discussion of neural SR methods.
  2. Were all training times measured under the same hardware, software environment, and resource budget? If not, the relative timing results may not be fair. Running the methods under one controlled setup would provide a more accurate comparison.
  3. Can the authors report sensitivity to the top-η\eta cutoff and use more dataset subsamples? Because the external validation compares only five preselected methods, please either validate against all feasible methods or limit the claim to the relative ordering of those five.
5 comments
AuthorsOpenReview ↗

Response to Weakness 1-2

The originality is limited. The paper does not introduce a new symbolic-regression method; it mainly applies a ranking-robustness framework to existing SRBench results. The evaluation question is useful, but the paper should clarify what is methodologically new beyond the prior robustness framework and the underlying benchmark.

We agree that the paper proposes neither a new SR method nor a new robustness indicator, and the revision says so plainly. The revision presents the original framework as an adaptation to a new domain. The Introduction and Contributions now state that the RS/DS indicators and the cross-consistency analysis follow Gjorgjevikj et al. (2026). Our contribution is the first systematic stress test of an SR leaderboard, which required axes that are not taken into account in the study of text-embedding benchmarks: Run aggregation: SR methods are stochastic, with 30 runs per dataset, and we sweep nine rules (mean, median, max, min, standard deviation, four single runs). The evaluation metric: error, training time, and now model size, handled as minimization criteria. The real-world vs. law-based split, and external validation with methods we re-ran ourselves. The findings are SR-specific: for example, QLattice and Operon lead on law-based data, Bingo under the min of MSE, its best run, on the pooled benchmark (except for some CRITIC regimes), PS-Tree under the mean, and lightweight methods under training time. Its practical message is that a method can look best only because the evaluation design favours it. On the pooled benchmark, PS-Tree is top-3 under mean MSE but ranks 11–17 under the best run in most schemes, and Operon leads the law-based track while it exhibits extreme variable ranking on the real-world one depending upon the metric-aggregation-rule combination. A practitioner reading a single leaderboard would be misled in either case, which is why we argue that new SR methods should be reported together with such robust evidence, and we want to raise awareness about that.

The paper misses an opportunity to rerun all methods under the same hardware and software conditions. Instead, it relies on previously reported scores and excludes GP-GOMEA and uDSR because some results are missing. This makes the comparison incomplete and may confound the training-time rankings.

We would have liked to, but re-running 28 methods × 25 datasets × 30 runs, with hyperparameter tuning in every run and up to 8 hours per job in SRBench's protocol, is beyond our resources. It would also change the object of study: we stress-test the published leaderboard, not a new one. Re-running only the missing GP-GOMEA and uDSR pairs on our hardware would make them incomparable with everyone else's since all SRBench methods were executed under the same HW environment with timeout on training, so we did not do that either. Instead, the revision includes both methods through a reduced complete matrix (see Q1). The method is not tied to this snapshot: when SRBench adds or re-runs methods, the same code produces updated indicators. Also, as we said, our goal is to stress-test the existing leaderboard rather than to crown an absolute winner and update the benchmark itself, which, according to the authors, was created under the same conditions and environment. In Appendix J, we report the analysis on the reduced matrix including all methods, also accounting for GP-GOMEA and uDSR. The apparently high performance of GP-GOMEA was a result of SRBench’s evaluation choice, since our rankings show that this method is not robust across different ranking schemes and metrics, with a rank that varies from 5 to 14. Particularly, GP-GOMEA is favoured by a few lucky runs that boost its aggregated metric (in the original SRBench, the best-run values were shown). We believe that this result additionally motivates why our study is important for benchmarking practices.

AuthorsOpenReview ↗

Response to Weakness 3 and Questions 1-2

“Robustness” is based on rank variance and top-10 membership, but no threshold for declaring a method robust is defined. The top-10 cutoff includes almost half of the 23 methods, and its sensitivity is not reported. Dataset-composition robustness also uses only three random subsamples.

For us, a method is robust when its rank does not change with the evaluation design. RS and DS measure this continuously, as rank variance across schemes and across compositions, and cross-consistency counts how often a method stays in the top-η across metrics or aggregations. Each captures a different aspect, so we report all three. We agree, however, that the text used "robust" and "fully robust" without a threshold. The revision defines a method as RS-stable (DS-stable) when its rank variance across schemes (compositions) is at most 5, and as robustly top-ranked when it is stable under both and in the top-η under every scheme and composition. This also separates stability from quality: a method that is always ranked last is stable but not robustly top-ranked. With 26 methods, η = 10 covers 38% of the field rather than almost half, and we now also report η = 5 (Q3) in Appendix K, showing that the top-ranked methods, under each regime, are almost positioned the same way even when switching to η = 5. A sensitivity on the correlation threshold with 0.7 and 0.8 is already provided in Appendix G along with sensitivity of ranking schemes and dataset compositions in Appendix H (computed with 9 random seeds this time).

Why did the authors exclude GP-GOMEA and uDSR instead of running the missing method-dataset combinations themselves? Please also cite the primary publication for every method named in the paper. In particular, uDSR should cite Landajuela et al. (2022), A Unified Framework for Deep Symbolic Regression. Petersen et al. (2021), Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients, is also relevant to the discussion of neural SR methods.

Thanks for your comment. All scores come from SRBench's own runs. Our hardware differs from SRBench's cluster, so results we produced for the missing pairs would not be comparable, above all for training time, which is one of our criteria. To include both methods without mixing sources, the revision adds a complete matrix of 28 methods on the 10 datasets where both have results (Appendix J). The revision cites the primary publication of every method in Section 4, including Landajuela et al. (2022) for uDSR, and cites Petersen et al. (2021) in the discussion of neural SR methods. Thank you again for providing us these references.

Were all training times measured under the same hardware, software environment, and resource budget? If not, the relative timing results may not be fair. Running the methods under one controlled setup would provide a more accurate comparison.

Thanks for your comment. All times are those recorded by SRBench 2025: every method ran in its own container on the same SLURM cluster with the same budget (1 hour per fit, 8 hours per job, 10 GB of memory) and the same 30 seeds. The five methods in the external validation were all re-run by us on one machine (32GB RAM, intel core ultra 9, 16-cores) with the same budget, so their times are directly comparable.

AuthorsOpenReview ↗

Response to Question 3

Can the authors report sensitivity to the top-(\eta) cutoff and use more dataset subsamples? Because the external validation compares only five preselected methods, please either validate against all feasible methods or limit the claim to the relative ordering of those five.

Thank you for flagging this. The analysis you mention is highly relevant and interesting. Eta parameter: every analysis in the main paper is now reported for η = 5 in Appendix K, showing that the top-ranked methods, under each regime, are almost positioned the same way even when switching to η = 5. Subsamples: the number of distinct subsamples is the product of the cluster sizes, so drawing more seeds eventually repeats subsamples and over-weights them. More compositions with other random seeds are presented in Appendix L. Each subsample has between 17 and 19. datasets, depending on which metric is taken into account. This means that the 22 datasets are clustered into 17 to 19 clusters depending on which metric is taken into account, where most of the clusters contain a single dataset, while only a few contain, occasionally, 2 to 4 datasets. This is actually expected since the next-generation SRBench was already designed to cover datasets that are already uncorrelated and diverse from each other. We nonetheless included the analysis on dataset compositions, in line with the text embedding benchmark, to check sensitivity w.r.t. changes in the number of datasets under consideration. This is why our initial submissions only included three random seeds for dataset subsampling. We now computed up to nine random seeds for dataset subsampling for better stability. We believe that more random seeds are not needed given the very small size of the clusters, since they would lead to many overlapping dataset compositions. Our dendrograms show the first three random seeds for space availability reasons, while the robustness indicators (RS and DS), along with the metric- and aggregation-agnostic measures, are computed by accounting for all 9 random seeds. Also, Kendall’s W in Appendix H has been recomputed across the 9 random seeds. Our analysis shows that the overall-top-ranked methods stay robust regardless of the random seed employed, while Kendall’s W (0.95-1.00) confirms that rankings remain highly consistent regardless of the random seed used for dataset subsampling, even when 9 random seeds are considered instead of only 3. External validation: we agree and now limit the claim. The five methods were selected as the overall top-ranked ones, so the experiment tests whether their relative ordering is preserved on held-out data, not whether other methods would enter the core. The text now says exactly that.

Reviewer_QupqOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper re-analyzes the per-run scores of the next-generation SRBench to test whether SR rankings are robust to changes in the metric, run-aggregation rule, and dataset composition. Using 15 MCDM ranking schemes and correlation-aware dataset subsampling, it reports two rank-variance indicators and top-10 membership counts, separately for real-world and law-based tracks.

Critical strengths
  1. The authors are right to point out that existing SR evaluations are less than ideal.
  2. The run-aggregation sweep shows the best-of-30 view used by the call-for-action paper and the mean or median views disagree materially for several methods, which is useful evidence for a living benchmark.
Critical weaknesses
  1. The chosen metrics do not reflect the core of SR tasks, where identifying the underlying mathematical formula is of utmost importance. On datasets with ground-truth data generating function, metrics that rely solely on (yi,y^i)(y_i, \hat{y}_i), such as R2R^2, MSE, MAE, can be misleading since many expressions can lead to low errors yet mathematically incorrect and unnecessarily complex. If the SR community only cares about high R2R^2, but not the complexity or correctness of the inferred equations, why bother with SR methods? Practitioners could've just reached for other regression techniques that outperform SR methods in accuracy. When no ground truth is available, standard error metrics can serve as surrogates, but model complexity should also be compared.
  2. The proposed benchmark is built on top of the "next-generation SRBench," where the synthetic dataset portion only includes low-dimensional datasets with low-to-none observation errors and mostly contains only the relevant predictors. In real-world applications, datasets often have a lower signal-to-noise ratio and may contain many irrelevant predictors, which the paper does not address.
  3. The framework is not novel and is a direct port of Gjorgjevikj et al. 2026, down to leftover "task-agnostic" and "language-agnostic" labels in Section 3.
  4. On the full benchmark, the aggregation-agnostic core is QLattice, Operon, RILS-ROLS, and PS-Tree falls from top-3 under the mean to rank 16 under the max. Yet the paper states that mean and median give "almost identical top clusters", calls EQL "the clearest example" of a fully robust method, and names EQL and PS-Tree as the most consistently robust pair.
Questions for authors

See weaknesses above. Also,

  1. How are failed runs or negative R2R^2 values handled? Which normalization is applied per dataset?
  2. How many datasets does each subsample at threshold 0.9 contain?
  3. Appendix E reports the real-world and law-based tracks under the mean only. Please show both tracks under the median and the max. In particular, does Operon's rank of 13 to 15 on the real-world track (Figure 5) survive the median, given that SRBench 2021 found Operon to be the most accurate method on black-box problems?
  4. Section 4 lists 23 methods with a single Genetic Engine entry, but the figures contain 26 rows, including four Genetic Engine variants (1P1, HC, RS, and plain). Which count is correct, and were the variants ranked as separate competitors?
  5. For the standard-deviation aggregation, is R2R^2 still treated as a maximization criterion? If so, that ranking rewards run-to-run instability rather than stability.
  6. GP-GOMEA was among the strongest black-box methods in the call-for-action paper. Why were GP-GOMEA and uDSR dropped entirely instead of being handled with rank aggregation over the available method–dataset pairs?
  7. The dendrograms are extremely hard to read. Please use other visualization techniques.
10 comments
AuthorsOpenReview ↗

Response to Weakness 1-2

The chosen metrics do not reflect the core of SR tasks, where identifying the underlying mathematical formula is of utmost importance. On datasets with ground-truth data generating function, metrics that rely solely on , such as , MSE, MAE, can be misleading since many expressions can lead to low errors yet mathematically incorrect and unnecessarily complex. If the SR community only cares about high , but not the complexity or correctness of the inferred equations, why bother with SR methods? Practitioners could've just reached for other regression techniques that outperform SR methods in accuracy. When no ground truth is available, standard error metrics can serve as surrogates, but model complexity should also be compared.

Thanks for the comment. We agree that accuracy alone cannot tell the whole story, and the revision adds model size as a fifth criterion on the reduced matrix. Since model size is missing for many method–dataset pairs, we did not include that in the main analysis. We now build a reduced matrix of 28 methods × 10 datasets (we also included GP-GOMEA and uDSR) in which model size is available for every configuration, and rank on it with the same 15 schemes. The analysis shows that GP-GOMEA is not robust to variation of evaluation design choice, thereby contrasting with the original claim from SRBench of GP-GOMEA being state-of-the-art, while the Genetic Engine family (representing among the less accurate methods) discovers the smallest models. On ground-truth data, exact recovery of the generating formula is the most informative criterion. Error metrics are only a surrogate, which the revised text now says explicitly in Section 3 and in Section 6 (limitations); these problems serve as controlled testbeds of how well a method approximates a known function. On real-world data, no true equation exists, and SR is used to find a compact empirical model, so accuracy and size are the natural pair of criteria, which is what we report. We do not add an interpretability score, because there is no agreed scalar measure: what counts as interpretable depends on the user and the domain. Any such score, or a symbolic-recovery indicator once one is released for every method, can enter our framework as one more criterion.

The proposed benchmark is built on top of the "next-generation SRBench," where the synthetic dataset portion only includes low-dimensional datasets with low-to-none observation errors and mostly contains only the relevant predictors. In real-world applications, datasets often have a lower signal-to-noise ratio and may contain many irrelevant predictors, which the paper does not address.

We agree that this limits what any analysis of SRBench 2025 can say about noisy, high-dimensional problems with irrelevant features, and the Limitations now say so. Extending the benchmark is outside our scope. Our aim is to show that conclusions drawn from a fixed evaluation design should be stress-tested, and we took the most recent SRBench edition as the reference case. The method applies unchanged when such datasets are added, and the dataset-composition indicator (DS) is exactly the tool that measures how much the ranking moves when they are. A natural next target is the benchmark of de França et al. (2025), which varies noise and adds irrelevant features.

AuthorsOpenReview ↗

Response to Weakness 3-4

The framework is not novel and is a direct port of Gjorgjevikj et al. 2026, down to leftover "task-agnostic" and "language-agnostic" labels in Section 3.

We agree that the framework is not new, and the revision presents it as an adaptation to a new domain. The labels "task-agnostic" and "language-agnostic" were left over from that work and are removed; the Introduction and Contributions now state that the RS/DS indicators and the cross-consistency analysis follow Gjorgjevikj et al. (2026). Our contribution is the first systematic stress test of an SR leaderboard, which required axes that are not taken into account in the study of text-embedding benchmarks: Run aggregation: SR methods are stochastic, with 30 runs per dataset, and we sweep nine rules (mean, median, max, min, standard deviation, four single runs). The evaluation metric: error, training time, and now model size, handled as minimization criteria. The real-world vs. law-based split, and external validation with methods we re-ran ourselves. The findings are SR-specific: for example, QLattice and Operon lead on law-based data, Bingo under the min of MSE, its best run, on the pooled benchmark (except for some CRITIC regimes), PS-Tree under the mean, and lightweight methods under training time.

On the full benchmark, the aggregation-agnostic core is QLattice, Operon, RILS-ROLS, and PS-Tree falls from top-3 under the mean to rank 16 under the max. Yet the paper states that mean and median give "almost identical top clusters", calls EQL "the clearest example" of a fully robust method, and names EQL and PS-Tree as the most consistently robust pair.

Thank you; you read the figures correctly. Some claims that hold only on the real-world track were stated without that qualifier, and one interpretation was wrong. Corrections: Pooled benchmark: the aggregation-agnostic core is Operon, QLattice and RILS-ROLS. EQL and PS-Tree are not so robust to the aggregation rule there. Real-world track: EQL ranks first under nearly every scheme and composition and stays in the top-10 under most aggregation rules for MSE and MAE, but only in the real-world track. We revised both the abstract and discussion section accordingly. Mean vs. median: the top clusters overlap (QLattice, Rils-Rols and Operon lead under both) but are not "almost identical". PS-Tree falls from 1–4 under the mean to 13–17 under the median and to 11–17 under the best run. Our explanation of the PS-Tree pattern was wrong. A method that "occasionally finds extremely good solutions" would gain under the best run, not lose. The pattern instead indicates that PS-Tree rarely produces a poor run, so the mean rewards it, while the occasional poor runs that depress other methods' means do not affect their medians or best runs. This is now better explained in the Results and Discussion sections. Robustness is now defined operationally and used consistently.

AuthorsOpenReview ↗

Response to Questions 1-3

How are failed runs or negative values handled? Which normalization is applied per dataset?

We use the per-run test scores released by SRBench 2025 without modification, so that our rankings stay comparable with the official ones. In SRBench's protocol, features and targets are standardized for training, and predictions are mapped back to the original scale before MSE, MAE, and R² are computed on the test split, so MSE and MAE are in each dataset's own units. After aggregating the runs, we therefore apply per-dataset log-min–max normalization (log-scale values followed by min-max normalization, to avoid occasionally large values to narrow the scaling range) to each column of P, and the criterion direction is then set inside each MCDM method. Negative R² values are kept, with values below −1 clipped to −1 so that one diverging run cannot dominate. R² does not undergo log-scaling prior to min-max normalization since its value range is controlled. The other metrics have no negative values. We removed R2 from the main analysis since it is a linear transformation of MSE, and we thereby kept metrics that are all minimization criteria so that the min-max aggregation rules correspond exactly to best and worst run scenarios. Failed/missing runs are not considered; we need a full performance matrix with all method-dataset combinations available for our ranking scheme to work. This is why we dropped GP-GOMEA and uDSR from the main analysis. Section 4 now states all of this.

How many datasets does each subsample at threshold 0.9 contain?

Thanks for the interesting question. Each subsample has between 17 and 19 datasets. This means that the 22 datasets are clustered into 17 to 19 clusters depending on which metric is taken into account, where most of the clusters contain a single dataset, while only a few contain, occasionally, 2 to 4 datasets. This is actually expected since the next-generation SRBench was designed to cover datasets that are already uncorrelated and diverse from each other. We nonetheless included the analysis on dataset compositions, in line with the text embedding benchmark, to check sensitivity w.r.t. changes in the number of datasets under consideration. This is why our initial submissions only included three random seeds for dataset subsampling. However, we now computed up to nine random seeds for dataset subsampling for better stability. We believe that more random seeds are not needed given the very small size of the clusters, since they would lead to many overlapping dataset compositions. Our dendrograms show the first three random seeds for space availability reasons, while the robustness indicators (RS and DS), along with the metric- and aggregation-agnostic measures, are computed by accounting for all 9 random seeds. The Kendall’s W in Appendix H has been recomputed across the 9 random seeds. Our analysis shows that the overall top-ranked methods stay robust regardless of the random seed employed, while Kendall’s W (0.95-1.00) confirms that rankings remain highly consistent regardless of the random seed used for dataset subsampling, even when 9 random seeds are considered instead of only 3.

Appendix E reports the real-world and law-based tracks under the mean only. Please show both tracks under the median and the max. In particular, does Operon's rank of 13 to 15 on the real-world track (Figure 5) survive the median, given that SRBench 2021 found Operon to be the most accurate method on black-box problems?

We thank the reviewer for the comment. We added the requested plots to Appendix E, specifically, we added plots for median, max (pessimistic for error metrics), and min (optimistic for error metrics), and we would like to take the opportunity to additionally motivate our work by showing that even Operon, which is top-ranked in the old version of SRBench, can exhibit variable ranking depending on the evaluation scenario, as shown by our analysis in Appendix E, highlighting the importance of adopting robust benchmarking practices. On the real-world track and for MSE, Operon is ranked bad under mean, however, it is ranked good (but not the best) under median and min (optimistic scenario), it is ranked 16-18 under max (pessimistic scenario), meaning that it occasionally exhibits very good runs and, at the same time, it has really bad worst runs. This also explains why in the original leaderboard Operon is ranked good, since in that case, authors employed the best-run as run aggregation rule. Interestingly, always on the real-world track, Operon is ranked good for mean MAE, but its rank varies heavily when switching to the other aggregation rules (median, min, max), going from top-5 in median to 11-14 under min. We also summarized this finding in the Discussion section.

Reviewer_KtfJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the robustness of symbolic regression benchmark rankings by varying the performance metric, run aggregation, ranking method and dataset composition. The authors find that no algorithm dominates across all settings and only a few methods keep consistently high ranks. They therefore suggest reporting ranking robustness alongside traditional leaderboards.

Critical strengths
  1. The study varies several evaluation choices at once, which gives a systematic view of how sensitive benchmark rankings are. This matters because many symbolic regression papers rely on a single SRBench ranking.
  2. The finding that real-world and physics-based datasets favour different algorithms is useful for practitioners choosing a method.
Critical weaknesses
  1. The new knowledge beyond SRBench is unclear. The paper appears to be a post-hoc analysis of existing SRBench results without new experiments, so its contribution is limited for ICLR. A benchmarking venue may be a better fit.
  2. The robustness analysis has design flaws that weaken its conclusions. Page 3, line 153, states that "max and min are optimistic and pessimistic stress tests", which holds for R2 but is reversed for MSE. Line 148 states that "MCDM directions are set accordingly", but the direction for standard deviation should be inverted. Moreover, MSE, MAE and R2 are highly correlated, and MSE and R2 theoretically give identical rankings on each dataset because R2 is a linear transformation of MSE, so the reported stability across metrics is expected rather than informative.
Questions for authors
  1. What does this paper reveal that the original SRBench analysis does not, and does it run any new experiments?
  2. How are the max and min aggregations and the MCDM direction handled for MSE and standard deviation?
  3. Would the rankings remain stable under different levels of label noise?
4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Questions 1

The new knowledge beyond SRBench is unclear. The paper appears to be a post-hoc analysis of existing SRBench results without new experiments, so its contribution is limited for ICLR. A benchmarking venue may be a better fit.

We comment upon this in the answer to question 1.

What does this paper reveal that the original SRBench analysis does not, and does it run any new experiments?

Thank you for the questions. We have proposed a new methodology for benchmarking by stress-testing results of SR methods. By using our proposed methodology on the original SRBench analysis, we have shown that the original SRBench outcomes are artifacts of the employed design choices of the authors. Our empirical results identified the most robust methods under changes to the evaluation design choices. We identify the mean/median discrepancy of the PS-Tree method and the ranking of Operon, which is not a result in the original study. In addition, in our original ICLR submission, we have shown the robustness of the SR methods under changes to the evaluation design choices separately for black-box and first-principles datasets (analysis not presented in the original SRBench). To validate our results about robust SR methods, we have run new experiments on five external datasets to see if the outcomes generalize (experiments presented in the original ICLR submission). In the revised manuscript, we have also included GP-GOMEA and uDSR on a reduced matrix of 28 SR methods on 10 datasets, so that all methods from SRBench are considered. We report the outcome in Appendix J, showing that the apparently high performance of GP-GOMEA was a result of SRBench’s evaluation choice, since our rankings show that this method is not robust across different ranking schemes and metrics, with a rank that varies from 5 to 14. Particularly, GP-GOMEA is favoured by a few lucky runs that boost its aggregated metric (in the original SRBench, the best-run values were shown). We believe that this result additionally motivates why our study is important for benchmarking practices. We also did a sensitivity analysis on top-eta (Appendix K) with 5. A sensitivity analysis on the correlation threshold with 0.7 and 0.8 is already provided in Appendix G, along with sensitivity of ranking schemes and dataset compositions in Appendix H (computed with 9 random seeds this time). The submission is in ICLR's datasets and benchmarks primary area, and its point applies to ML leaderboards generally: a ranking is only meaningful once it survives reasonable changes in the evaluation design. SR is a demanding test case because its methods are stochastic (30 runs each), are judged on several metrics, and are evaluated on both real-world and law-based data.

AuthorsOpenReview ↗

Response to Weakness 2 and Questions 2-3

The robustness analysis has design flaws that weaken its conclusions. Page 3, line 153, states that "max and min are optimistic and pessimistic stress tests", which holds for R2 but is reversed for MSE. Line 148 states that "MCDM directions are set accordingly", but the direction for standard deviation should be inverted. Moreover, MSE, MAE and R2 are highly correlated, and MSE and R2 theoretically give identical rankings on each dataset because R2 is a linear transformation of MSE, so the reported stability across metrics is expected rather than informative.

We comment upon this in the answer to question 2.

How are the max and min aggregations and the MCDM direction handled for MSE and standard deviation?

We agree that the sentence on max-min with directions was not clear for the error metrics. Our implementation takes the literal maximum and minimum over the runs for every metric, and the MCDM direction always follows the metric (R² maximized; MSE, MAE, and training time minimized), regardless of the aggregation rule. Standard deviation is an exception: it is always minimized regardless of the metric to capture run-to-run stability (we now clarify this in Section 3). Each per-(metric, aggregation) analysis is therefore internally correct, but "max" is the optimistic view only for R²: for MSE, MAE, and training time it is the pessimistic one, and the reverse holds for "min". In the revision, both to simplify the readability of the analysis and taking into account that, after all, R2 is a linear transformation of MSE, we decided not to include R2 in the main analysis and focus on MSE, MAE, and training time, which are all minimization metrics and, most importantly, are more diverse from each other. We only keep a summary R2 dendrogram in Appendix B as a consistency check with MSE, and all the plots with R2 in the main text are replaced with MSE plots. Moreover, all plots with R2 are removed from Appendix C on to streamline the text. Section 3 now identifies, for each metric, the best run as the min aggregation-rule and the worst run as the max aggregation-rule. We keep the literal operators on purpose. The metric-agnostic analysis for min (or max) asks whether a method stays top-ranked when the metric changes but the reported operator does not, thereby accounting for the best-run (or worst-run). The revised text describes the analysis this way, having min and max as plain optimistic and pessimistic stress tests. The informative contrasts on the metric axis are accuracy versus training time and, new in the revision, accuracy versus model size in Appendix J. Just like the other metrics of the main analysis, model size is a minimization metric, so optimistic-pessimistic views hold, also for metric-agnostic analysis. Our findings on model size show that all methods from the Genetic Engine family are the ones discovering the smallest methods consistently, while the accurate methods are typically low-ranked w.r.t. the model size, as it usually happens in SR and ML in general, since this represents the typical accuracy-complexity trade-off. We additionally stress that, in case a maximization metric is added into the analysis, it is always possible to still treat min and max as optimistic and pessimistic scenarios by just inverting the sign of the maximization metric.

Would the rankings remain stable under different levels of label noise?

Thanks for the comment. We read this as asking whether the rankings would change if Gaussian noise were added to the target y, as SRBench 2021 did for its ground-truth problems (noise levels 0, 0.001, 0.01 and 0.1). The SRBench 2025 runs we analyze add no target noise, so noise level is not a dimension we can vary without re-running every method. Our framework would handle it directly: noise level becomes one more axis next to metric and aggregation, and RS, DS, and cross-consistency apply unchanged. The closest evidence in the paper is the track split. The real-world track carries measurement noise, and the law-based track carries none, and their leaders differ substantially (Figs. 5–6). The revision states this as a limitation and as future work in Section 6.

Thank you for the clarification. I think this is a borderline paper. On the positive side, it points out an issue that the symbolic regression community should pay attention to. On the negative side, it does not give practical advice to the symbolic regression community. Specifically, the paper lacks guidance on how to evaluate symbolic regression algorithms in a way that better matches real-world applications.

AuthorsOpenReview ↗

Practical Guidelines for Symbolic Regression Benchmarking and a New Joint Accuracy–Complexity Ranking

We sincerely thank the reviewer for taking the time to follow up, and for recognising that our paper raises an issue the SR community should pay attention to. We appreciate this concern, and we realise we may not have presented the practical side of the paper clearly enough. We would like to take this opportunity to clarify it.

1. Practical reporting guidelines (Section 6, “Implications for SR benchmarking”). Our aim was to identify the problem and also to offer guidance that authors can apply directly. Based on our results, we propose three recommendations:

  • Reporting more than one aggregation view. Mean and median rankings largely agree, but standard-deviation and training-time rankings can differ substantially. We therefore suggest that leaderboards report at least one central-tendency ranking and one stability/efficiency ranking, alongside the usual averaged accuracy.
  • Keeping the real-world (black-box) and first-principles tracks separate. The two tracks favour different methods: EQL on real-world data, QLattice and Operon on law-based problems. Reporting them separately helps avoid misrepresenting methods whose strengths depend on the regime.
  • Attaching robustness indicators to reported rankings. The ranking-scheme (RS) and dataset-composition (DS) robustness indicators can show whether a state-of-the-art claim holds up under reasonable variations in metric, aggregation rule and dataset mixture. To make this easy to adopt, we release all model scores and derived rankings, so authors can compute these indicators without rerunning any experiments.

2. New joint ranking experiment for selecting across metrics (Appendix J.1, summarised in Section 5). Following the reviewers’ helpful comments, we added an experiment in the revised manuscript that addresses how practitioners might choose a method when several criteria matter. Ranking model size separately from accuracy does not fully capture the accuracy–complexity trade-off, so we extended our MCDM methodology into three steps:

  1. Rank methods jointly on MSEtest\mathrm{MSE}_{\mathrm{test}} and model size within each dataset.
  2. Apply correlation-aware dataset subsampling to these joint rankings.
  3. Aggregate across datasets.

We believe this experiment brings out several useful insights:

  • PySR emerges as the most consistent compromise. It ranks first under mean and median aggregation, second under max, and in the top three under standard deviation, even though it ranks near the bottom on MSE alone under median aggregation. RILS-ROLS and AFP-EHC complete the top three under mean, median and max.
  • Methods that lead on a single criterion tend to lose their top positions in the joint ranking. These are the MSE-only leaders (GPZGD, EQL, ITEA, PS-Tree) and the size-only leaders (GeneticEngine variants, uDSR).
  • GP-GOMEA does not enter the joint top-10, which suggests that its strong standing in the next-generation SRBench depends on the chosen metric and aggregation rule.
  • The picture changes notably only under the optimistic best-run (min) aggregation, which further supports reporting the aggregations practitioners typically use.

The procedure does not depend on the metric, so users can include the criteria most relevant to their own needs (e.g., accuracy, complexity, runtime or energy) and obtain a joint ranking together with its robustness.

3. Controlled benchmarking and real-world evaluation. We fully agree that evaluation better aligned with real-world applications is an important goal. We see our work as a step toward it. Real-world uses of SR vary considerably, and practitioners often weigh accuracy, interpretability, model size and runtime differently, so real-world evaluation may well need protocols tailored to specific user preferences. In our view, such protocols are best built on sound and consistent benchmarking practices in controlled settings, which is where academic comparisons of SR algorithms are mostly carried out today. With this paper we hope to contribute to that foundation. The joint ranking experiment also illustrates how our framework could support preference-based selection, and we hope it will be useful for future application-oriented work, which we consider a valuable direction beyond the scope of the current paper.

In line with the reviewer’s suggestion, we would be glad to make these contributions more visible in the final version. We could summarise the practical guidelines as a short checklist at the end of Section 6 and give the joint ranking results more prominence in the main text.

Thank you once again for the thoughtful and constructive feedback, which has helped us improve the paper.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for handling our submission. We are grateful to the three reviewers (KtfJ, Qupq and EAkK) for their careful and constructive feedback, which has substantially improved the paper. Below is a summary of the main concerns and how we addressed them.

  1. Practical guidance for the SR community (KtfJ, Qupq)

In the follow-up discussion, Reviewer KtfJ described the paper as borderline. They agreed it raises an issue the SR community should pay attention to, but noted that it lacked practical advice. We took this seriously, and the second revision adds two concrete contributions: Reporting guidelines (Section 6, “Implications for SR benchmarking”). We give three recommendations that authors can apply directly: Report at least one central-tendency ranking and one stability or efficiency ranking, alongside averaged accuracy. Keep the real-world and first-principles tracks separate, because they favour different methods. Attach robustness indicators (RS and DS) to reported rankings, together with the correlation threshold and subsample sizes. We release all model scores and derived rankings, so authors can compute these indicators without rerunning any experiments. Joint ranking for selecting a method across criteria (Appendix J.1, summarized in Section 5). We extended the methodology to rank methods jointly on accuracy and model size. Under every commonly reported aggregation rule, PySR comes out as the most consistent compromise, followed by RILS-ROLS and AFP-EHC. Methods that lead on a single criterion lose their top positions. The procedure does not depend on the metric, so practitioners can plug in the criteria that matter for their application (accuracy, complexity, runtime or energy) and get a joint ranking together with its robustness. For the final version, we offer to condense the guidelines into a short checklist at the end of Section 6 and to give the joint-ranking results more prominence in the main text.

  1. Contribution and novelty (KtfJ, EAkK, Qupq)

We agree that the paper does not propose a new SR method. Our contribution is the first systematic stress test of an SR leaderboard. This needed SR-specific axes: aggregation over stochastic runs (nine rules over 30 runs), several metrics, and the split between real-world and law-based tracks. It also produced findings that are not visible in the original results. For example, PS-Tree ranks 1st–4th under mean MSE but 13th–17th under the median, and Operon falls to 16th–18th on the real-world track under worst-run MSE. The submission is in the datasets and benchmarks area, and we hope the work is read as a contribution to benchmarking practice.

  1. Soundness of the metric and aggregation design (KtfJ, Qupq)

We clarified the MCDM directions. Standard deviation is always minimized. Because R² is a linear transformation of MSE, we moved it to an appendix as a consistency check. The main analysis now uses only minimization metrics (MSE, MAE, training time and model size), so min and max correspond exactly to the best-run and worst-run scenarios.

  1. Corrections to claims (Qupq)

Reviewer Qupq correctly pointed out that some claims held only on the real-world track, and that our reading of the PS-Tree pattern was wrong. We corrected the abstract, results and discussion. We also gave “robustness” an operational definition and added a sensitivity analysis of the threshold (Appendix M).

  1. Missing methods and re-running experiments (EAkK, Qupq)

GP-GOMEA and uDSR are included through a complete reduced matrix of 28 methods on 10 datasets (Appendix J). Re-running the missing combinations on our own hardware would make them incomparable with the published SRBench runs, especially for training time. A full re-run is beyond our resources. We state this, and the non-random missingness of the reduced matrix, explicitly in the Limitations. We also softened the GP-GOMEA claim to what the evidence supports: its standing depends on the aggregation rule and the metric.

  1. Sensitivity and additional analyses (EAkK, Qupq)

We added: a top-η sensitivity analysis with η = 5 (Appendix K) nine subsampling seeds instead of three (Appendix H, Kendall’s W of 0.95–1.00) additional compositions (Appendix L) extra track-wise plots under median, min and max (Appendix E) primary citations for all methods, including Landajuela et al. (2022) and Petersen et al. (2021) redrawn figures for readability We also limited the external-validation claim to the relative ordering of the five selected methods.

  1. Scope limitations (Qupq, KtfJ)

Benchmarks with noise and irrelevant predictors, and exact equation recovery, are outside what the SRBench 2025 data allows. We now acknowledge them as limitations and as the main direction for future work. Our framework does not depend on the metric, so it applies to such benchmarks without modification.

We hope this summary helps your assessment. Best Regards, Authors of Submission 31390

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.