- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DIGEST, which uses a single decoder block from Qwen2.5-0.5B to seed GP populations and then updates the policy online using high-quality expressions discovered by GP. It reports strong results on Nguyen, Livermore, and 122 SRBench datasets.
Critical strengths
Strengths:
-
The algorithm is clearly presented, with detailed implementation settings and per-equation results.
-
The single-block policy provides a favorable accuracy-efficiency trade-off.
-
The evaluation is reasonably extensive: 25 runs per ground-truth equation and 10 runs per SRBench dataset.
Critical weaknesses
Main concerns:
-
Insufficient attribution of the gains. NGGP already combines a neural generator, GP search, and feedback-based generator updates. DIGEST mainly replaces the generator with a pretrained decoder block and a symbolic interface. Without controls using a randomly initialized Transformer, shuffled pretrained weights, or random symbolic embeddings, it is unclear whether the gains come from language-model pretraining, the architecture, or GP.
-
The LLM baselines are evaluated without scientific context and with different method-specific candidate limits, whereas DIGEST uses a two-million-expression evaluation budget. Timing boundaries also differ across methods. Therefore, the reported recovery and runtime advantages are not strictly compute-matched.
-
SRBench results are primarily cross-dataset medians, without per-dataset results, confidence intervals, or significance tests. The robustness of the improvement is therefore difficult to assess.
Questions for authors
Please refer to the weaknesses.
8 comments
Response to Reviewer 5bmT (2/3)
W2: Search budgets, timing boundaries, and comparison with LLM-based methods.
(1) Under the 2M evaluation cap, DIGEST takes only 49.2/118.8 seconds on average.
The 49.2 seconds on Nguyen and 118.8 seconds on Livermore in Figure 4 are the mean times for DIGEST's actual search under the 2M expression-evaluation cap, including LLM-based generation, GP search, and online training. The cap covers LLM-generated expressions and GP evaluations together. Recovery-based early stopping is enabled, so not every run exhausts the cap.
Unlike prompt-based LLM baselines, DIGEST directly generates symbolic actions through a single decoder block. Symbolic prediction counts, full-model generation requests, and text-token counts therefore do not represent equivalent computational costs.
Under the reported configurations, DIGEST achieves higher recovery in less recorded search time than the compared LLM-based methods. This demonstrates the practical search efficiency of our approach. We summarize the original results from Table 3 and Figure 4 below; values separated by "/" denote Nguyen / Livermore.
| Method | Recovery (%) ↑ | Median test NMSE ↓ | Mean search time (s) ↓ |
|---|---|---|---|
| DIGEST | 93.33 / 74.00 | / | 49.2 / 118.8 |
| ICSR | 14.00 / 16.73 | / | 187.8 / 234.0 |
| LLM-SR | 3.33 / 1.27 | / | 702.0 / 710.4 |
| LaSR | 29.67 / 19.09 | / | 684.0 / 681.6 |
| RESTART | 4.00 / 2.18 | / | 463.2 / 418.8 |
The reported search times exclude initial LLM loading (Appendix C.3); other setup costs within the search interval follow each implementation.
(2) LaSR evaluated within DIGEST's mean search times.
We further evaluated LaSR, the strongest LLM-based baseline by recovery in Table 3, using only expressions available within 49.2 seconds on Nguyen and 118.8 seconds on Livermore, corresponding to DIGEST's reported mean search times. Only complete model snapshots saved before each time limit were evaluated, without additional coefficient fitting afterward.
For comparison, we also include the original results reported in Table 3 and Figure 4.
| Benchmark | Method / setting | Recovery (%) ↑ | Median test NMSE ↓ | Search time (s) |
|---|---|---|---|---|
| Nguyen | DIGEST | 93.33 | 49.2 (mean) | |
| Nguyen | LaSR (reported) | 29.67 | 684.0 (mean) | |
| Nguyen | LaSR (time-limited) | 16.67 | 49.2 (limit) | |
| Livermore | DIGEST | 74.00 | 118.8 (mean) | |
| Livermore | LaSR (reported) | 19.09 | 681.6 (mean) | |
| Livermore | LaSR (time-limited) | 9.09 | 118.8 (limit) |
Under time limits corresponding to DIGEST's mean search times, LaSR achieves lower recovery and higher NMSE on both benchmarks. Even with its substantially longer originally reported search times, LaSR still falls behind DIGEST in both metrics. These results further demonstrate DIGEST's practical search efficiency.
In the revised paper: Section 4.2; Appendix F.3, Table 16.
(3) DIGEST remains faster when recorded loading and startup costs are included.
We also examined broader elapsed times from the existing run logs, including the recorded model-loading and startup overheads:
| Method | Nguyen: elapsed time including loading/startup (s) ↓ | Livermore: elapsed time including loading/startup (s) ↓ |
|---|---|---|
| DIGEST | 60.17 | 133.51 |
| ICSR | 282.48 | 326.70 |
| LLM-SR | 820.02 | 870.09 |
| LaSR | 824.28 | 836.28 |
| RESTART | 593.13 | 488.39 |
DIGEST retains its runtime advantage on both benchmarks after these overheads are included. Its shorter reported search time is therefore not simply a consequence of excluding initial model loading.
Regarding scientific context, our evaluation focuses on symbolic regression from numerical observations, without task-specific scientific descriptions. We retain the LLM baselines' generic mathematical instructions and original search mechanisms (Appendix C.2).
Response to Reviewer 5bmT (3/3)
W3: Per-dataset results, confidence intervals, and significance tests on SRBench.
Following SRBench's released BlackBox evaluation and plotting implementation [1], our Figure 5 reports cross-dataset median test and median model size, aggregating ten runs per dataset. We further clarify the confidence intervals and provide dataset-level results and significance tests below.
(1) Confidence intervals in Figure 5.
Figure 5 already includes 95% bootstrap confidence intervals, following SRBench's released BlackBox plotting implementation. Some error bars are difficult to distinguish because they overlap with the point markers at the current plotting scale. We have clarified the 95% CI notation in the revised figure caption and will further improve their visibility in the final version.
For DIGEST, the corresponding values are:
| Metric | Cross-dataset median | 95% bootstrap CI |
|---|---|---|
| Test | 0.946 | [0.931, 0.950] |
| Model size | 55.5 | [55, 56] |
In the revised paper: Figure 5 caption.
(2) Per-dataset results.
We will release the complete per-dataset and per-seed results for all 122 SRBench datasets, including ten runs per dataset, alongside our code on GitHub upon acceptance. Baseline results and their sources will also be provided.
As examples, the following table shows the individual test NMSE values across ten seeds for two datasets.
| Seed | 579_fri_c0_250_5 | 589_fri_c2_1000_25 |
|---|---|---|
| 860 | 0.046775 | 0.049669 |
| 4426 | 0.041583 | 0.058696 |
| 5390 | 0.070646 | 0.022618 |
| 14423 | 0.054226 | 0.045748 |
| 15795 | 0.044196 | 0.103239 |
| 16850 | 0.043856 | 0.048193 |
| 21962 | 0.069794 | 0.072140 |
| 23654 | 0.070254 | 0.034119 |
| 28020 | 0.053526 | 0.057794 |
| 29910 | 0.059515 | 0.078760 |
(3) Significance tests.
We additionally conducted dataset-level significance tests against existing baselines. The results show that DIGEST achieves statistically significant improvements over compared baselines, while maintaining the highest cross-dataset median test .
| Method | Median test ↑ | p-value vs. DIGEST |
|---|---|---|
| DIGEST | 0.946 | — |
| Operon | 0.934 | 0.117230 |
| SBP-GP | 0.908 | 0.030774 |
| FEAT | 0.895 | 6.733e-06 |
| EPLEX | 0.868 | 7.566e-10 |
| XGB | 0.854 | 1.404e-04 |
| LGBM | 0.821 | 3.691e-09 |
| GP-GOMEA | 0.816 | 6.167e-12 |
| AdaBoost | 0.757 | 1.835e-16 |
| RandomForest | 0.748 | 1.591e-12 |
| ITEA | 0.713 | 3.635e-12 |
| AFP_FE | 0.706 | 4.625e-19 |
| SR4MDL | 0.696 | 0.020345 |
| AFP | 0.692 | 4.719e-19 |
| FFX | 0.688 | 0.002926 |
| KernelRidge | 0.686 | 8.051e-11 |
| DSR | 0.590 | 6.909e-27 |
| MRGP | 0.588 | 3.465e-12 |
| gplearn | 0.582 | 4.957e-22 |
| MLP | 0.576 | 4.040e-14 |
| Linear | 0.380 | 2.021e-21 |
| BSR | 0.110 | 1.153e-32 |
| AIFeynman | -0.437 | 0.002620 |
In the revised paper: Section 4.3; Appendix F.7, Table 21.
References
[1] La Cava, William, et al. “Contemporary Symbolic Regression Methods and their Relative Performance.” NeurIPS Datasets and Benchmarks, 2021.
Response to Reviewer 5bmT (1/3)
We thank the reviewer for the constructive feedback and comments, all of which are helpful to further improve our paper. See below for our responses to the “Critical Weaknesses” (W). We have carefully revised the manuscript in response to the review, with all additions and revisions highlighted in blue for ease of reference. The relevant manuscript revisions and their locations are indicated below.
Following the reviewer's suggestion, we conducted three additional initialization ablations to assess the contribution of pretrained weights beyond the generator architecture, keeping the policy architecture and training/search hyperparameters unchanged:
All variants retain GP refinement and online adaptation.
Parentheses indicate changes relative to DIGEST.
All three ablations yield lower recovery on both benchmarks, supporting the contribution of pretrained initialization beyond the generator architecture alone, even when online adaptation remains enabled.
These results complement our original random-seed GP and frozen-policy ablations (Table 4). The former assesses the benefit of policy-based population seeding while retaining the same GP refinement procedure; the latter isolates the incremental effect of online updates. Together, these comparisons provide separate evidence for the contributions of pretrained initialization, population seeding, and online adaptation.
In the revised paper: Section 5, Table 5.