acceptodds
Under review as a conference paper at ICLR 2027

DIGEST: Discovery-Guided LLM Tuning for Symbolic Regression

Abstract

Symbolic regression (SR) seeks to discover mathematical expressions directly from data, without requiring task-specific scientific descriptions. Despite tremendous progress in both conventional search-based methods and recent LLM-based methods, two critical limitations largely remain. First (Limited Generation Flexibility), conventional search-based methods rely on either fixed search rules or specialized learned generators, severely limiting adaptive candidate generation across diverse expression structures. Second (Underutilized Search Discoveries), most LLM-based methods cannot reliably recover ground-truth expressions because they fall short in effectively leveraging the LLM itself to thoroughly digest past expressions or adaptively guide future search. To address these limitations, we propose DIGEST. To render greater flexibility in generation, DIGEST repurposes a pretrained decoder block from a lightweight LLM into a symbolic policy for Genetic Programming (GP) population seeding, thereby leveraging the broader modeling capacity of the LLM. To more thoroughly utilize search discoveries, high-quality expressions discovered by GP are fed back to update the LLM-derived policy online, thereby enabling the LLM to progressively internalize past search discoveries and translate them into improved subsequent population generation. Experiments show that on two ground-truth benchmarks, DIGEST outperforms the strongest competitor by up to 2.59% in recovery. Among LLM-based baselines, it improves recovery by at least 54.91% while using at most 17.4% of the strongest competitor's mean search time. On 122 BlackBox datasets, DIGEST achieves the highest accuracy while maintaining a moderate model complexity.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic regression, large language models, genetic programming, online adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5bmTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DIGEST, which uses a single decoder block from Qwen2.5-0.5B to seed GP populations and then updates the policy online using high-quality expressions discovered by GP. It reports strong results on Nguyen, Livermore, and 122 SRBench datasets.

Critical strengths

Strengths:

  1. The algorithm is clearly presented, with detailed implementation settings and per-equation results.

  2. The single-block policy provides a favorable accuracy-efficiency trade-off.

  3. The evaluation is reasonably extensive: 25 runs per ground-truth equation and 10 runs per SRBench dataset.

Critical weaknesses

Main concerns:

  1. Insufficient attribution of the gains. NGGP already combines a neural generator, GP search, and feedback-based generator updates. DIGEST mainly replaces the generator with a pretrained decoder block and a symbolic interface. Without controls using a randomly initialized Transformer, shuffled pretrained weights, or random symbolic embeddings, it is unclear whether the gains come from language-model pretraining, the architecture, or GP.

  2. The LLM baselines are evaluated without scientific context and with different method-specific candidate limits, whereas DIGEST uses a two-million-expression evaluation budget. Timing boundaries also differ across methods. Therefore, the reported recovery and runtime advantages are not strictly compute-matched.

  3. SRBench results are primarily cross-dataset medians, without per-dataset results, confidence intervals, or significance tests. The robustness of the improvement is therefore difficult to assess.

Questions for authors

Please refer to the weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 5bmT (1/3)

We thank the reviewer for the constructive feedback and comments, all of which are helpful to further improve our paper. See below for our responses to the “Critical Weaknesses” (W). We have carefully revised the manuscript in response to the review, with all additions and revisions highlighted in blue for ease of reference. The relevant manuscript revisions and their locations are indicated below.

W1: Attribution of gains to pretrained initialization, architecture, and GP.

Following the reviewer's suggestion, we conducted three additional initialization ablations to assess the contribution of pretrained weights beyond the generator architecture, keeping the policy architecture and training/search hyperparameters unchanged:

  • Random decoder block: Randomly reinitialize the selected decoder block.
  • Shuffled pretrained weights: Shuffle pretrained weights within the selected decoder block.
  • Random symbolic input embeddings: Randomly initialize the parent–sibling symbolic input embeddings.

All variants retain GP refinement and online adaptation.

InitializationNguyen recovery (%) ↑Livermore recovery (%) ↑
DIGEST: pretrained initialization93.3374.00
Random decoder block91.33 (−2.00 pp)71.27 (−2.73 pp)
Shuffled pretrained weights91.33 (−2.00 pp)72.18 (−1.82 pp)
Random symbolic input embeddings88.33 (−5.00 pp)71.82 (−2.18 pp)

Parentheses indicate changes relative to DIGEST.

All three ablations yield lower recovery on both benchmarks, supporting the contribution of pretrained initialization beyond the generator architecture alone, even when online adaptation remains enabled.

These results complement our original random-seed GP and frozen-policy ablations (Table 4). The former assesses the benefit of policy-based population seeding while retaining the same GP refinement procedure; the latter isolates the incremental effect of online updates. Together, these comparisons provide separate evidence for the contributions of pretrained initialization, population seeding, and online adaptation.

In the revised paper: Section 5, Table 5.

AuthorsOpenReview ↗

Response to Reviewer 5bmT (2/3)

W2: Search budgets, timing boundaries, and comparison with LLM-based methods.

(1) Under the 2M evaluation cap, DIGEST takes only 49.2/118.8 seconds on average.

The 49.2 seconds on Nguyen and 118.8 seconds on Livermore in Figure 4 are the mean times for DIGEST's actual search under the 2M expression-evaluation cap, including LLM-based generation, GP search, and online training. The cap covers LLM-generated expressions and GP evaluations together. Recovery-based early stopping is enabled, so not every run exhausts the cap.

Unlike prompt-based LLM baselines, DIGEST directly generates symbolic actions through a single decoder block. Symbolic prediction counts, full-model generation requests, and text-token counts therefore do not represent equivalent computational costs.

Under the reported configurations, DIGEST achieves higher recovery in less recorded search time than the compared LLM-based methods. This demonstrates the practical search efficiency of our approach. We summarize the original results from Table 3 and Figure 4 below; values separated by "/" denote Nguyen / Livermore.

MethodRecovery (%) ↑Median test NMSE ↓Mean search time (s) ↓
DIGEST93.33 / 74.001.50×10−321.50\times10^{-32} / 5.20×10−325.20\times10^{-32}49.2 / 118.8
ICSR14.00 / 16.733.5×10−43.5\times10^{-4} / 6.2×10−46.2\times10^{-4}187.8 / 234.0
LLM-SR3.33 / 1.279.5×10−69.5\times10^{-6} / 1.8×10−41.8\times10^{-4}702.0 / 710.4
LaSR29.67 / 19.092.3×10−52.3\times10^{-5} / 5.7×10−55.7\times10^{-5}684.0 / 681.6
RESTART4.00 / 2.181.1×10−41.1\times10^{-4} / 9.7×10−49.7\times10^{-4}463.2 / 418.8

The reported search times exclude initial LLM loading (Appendix C.3); other setup costs within the search interval follow each implementation.

(2) LaSR evaluated within DIGEST's mean search times.

We further evaluated LaSR, the strongest LLM-based baseline by recovery in Table 3, using only expressions available within 49.2 seconds on Nguyen and 118.8 seconds on Livermore, corresponding to DIGEST's reported mean search times. Only complete model snapshots saved before each time limit were evaluated, without additional coefficient fitting afterward.

For comparison, we also include the original results reported in Table 3 and Figure 4.

BenchmarkMethod / settingRecovery (%) ↑Median test NMSE ↓Search time (s)
NguyenDIGEST93.331.50×10−321.50\times10^{-32}49.2 (mean)
NguyenLaSR (reported)29.672.3×10−52.3\times10^{-5}684.0 (mean)
NguyenLaSR (time-limited)16.672.28×10−32.28\times10^{-3}49.2 (limit)
LivermoreDIGEST74.005.20×10−325.20\times10^{-32}118.8 (mean)
LivermoreLaSR (reported)19.095.7×10−55.7\times10^{-5}681.6 (mean)
LivermoreLaSR (time-limited)9.093.57×10−43.57\times10^{-4}118.8 (limit)

Under time limits corresponding to DIGEST's mean search times, LaSR achieves lower recovery and higher NMSE on both benchmarks. Even with its substantially longer originally reported search times, LaSR still falls behind DIGEST in both metrics. These results further demonstrate DIGEST's practical search efficiency.

In the revised paper: Section 4.2; Appendix F.3, Table 16.

(3) DIGEST remains faster when recorded loading and startup costs are included.

We also examined broader elapsed times from the existing run logs, including the recorded model-loading and startup overheads:

MethodNguyen: elapsed time including loading/startup (s) ↓Livermore: elapsed time including loading/startup (s) ↓
DIGEST60.17133.51
ICSR282.48326.70
LLM-SR820.02870.09
LaSR824.28836.28
RESTART593.13488.39

DIGEST retains its runtime advantage on both benchmarks after these overheads are included. Its shorter reported search time is therefore not simply a consequence of excluding initial model loading.

Regarding scientific context, our evaluation focuses on symbolic regression from numerical observations, without task-specific scientific descriptions. We retain the LLM baselines' generic mathematical instructions and original search mechanisms (Appendix C.2).

AuthorsOpenReview ↗

Response to Reviewer 5bmT (3/3)

W3: Per-dataset results, confidence intervals, and significance tests on SRBench.

Following SRBench's released BlackBox evaluation and plotting implementation [1], our Figure 5 reports cross-dataset median test R2R^2 and median model size, aggregating ten runs per dataset. We further clarify the confidence intervals and provide dataset-level results and significance tests below.

(1) Confidence intervals in Figure 5.

Figure 5 already includes 95% bootstrap confidence intervals, following SRBench's released BlackBox plotting implementation. Some error bars are difficult to distinguish because they overlap with the point markers at the current plotting scale. We have clarified the 95% CI notation in the revised figure caption and will further improve their visibility in the final version.

For DIGEST, the corresponding values are:

MetricCross-dataset median95% bootstrap CI
Test R2R^20.946[0.931, 0.950]
Model size55.5[55, 56]

In the revised paper: Figure 5 caption.

(2) Per-dataset results.

We will release the complete per-dataset and per-seed results for all 122 SRBench datasets, including ten runs per dataset, alongside our code on GitHub upon acceptance. Baseline results and their sources will also be provided.

As examples, the following table shows the individual test NMSE values across ten seeds for two datasets.

Seed579_fri_c0_250_5589_fri_c2_1000_25
8600.0467750.049669
44260.0415830.058696
53900.0706460.022618
144230.0542260.045748
157950.0441960.103239
168500.0438560.048193
219620.0697940.072140
236540.0702540.034119
280200.0535260.057794
299100.0595150.078760

(3) Significance tests.

We additionally conducted dataset-level significance tests against existing baselines. The results show that DIGEST achieves statistically significant improvements over compared baselines, while maintaining the highest cross-dataset median test R2R^2.

MethodMedian test R2R^2 ↑p-value vs. DIGEST
DIGEST0.946—
Operon0.9340.117230
SBP-GP0.9080.030774
FEAT0.8956.733e-06
EPLEX0.8687.566e-10
XGB0.8541.404e-04
LGBM0.8213.691e-09
GP-GOMEA0.8166.167e-12
AdaBoost0.7571.835e-16
RandomForest0.7481.591e-12
ITEA0.7133.635e-12
AFP_FE0.7064.625e-19
SR4MDL0.6960.020345
AFP0.6924.719e-19
FFX0.6880.002926
KernelRidge0.6868.051e-11
DSR0.5906.909e-27
MRGP0.5883.465e-12
gplearn0.5824.957e-22
MLP0.5764.040e-14
Linear0.3802.021e-21
BSR0.1101.153e-32
AIFeynman-0.4370.002620

In the revised paper: Section 4.3; Appendix F.7, Table 21.

References

[1] La Cava, William, et al. “Contemporary Symbolic Regression Methods and their Relative Performance.” NeurIPS Datasets and Benchmarks, 2021.

Reviewer_yxo7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses symbolic regression from numerical observations without requiring task-specific scientific descriptions. The method repurposes a pretrained decoder block from a lightweight language model into a symbolic generation policy that uses expression-tree context to seed genetic programming (GP) populations. GP then searches for high-quality expressions, which provide online training signals to update the policy, forming an iterative loop of expression generation, evolutionary search, and policy adaptation. The central idea is to internalize search discoveries through parameter updates rather than reuse them solely as external context. Experiments on Nguyen, Livermore, and 122 SRBench BlackBox datasets evaluate the method’s expression recovery and predictive performance.

Critical strengths
  1. The paper has a clear motivation and a coherent design. It examines how expressions discovered during genetic programming can help improve subsequent candidate generation. These expressions are used to update an LLM-derived symbolic policy during search, allowing the generator to learn from results on the current task. The generation, search, and update stages have clear roles, and the overall procedure is easy to follow.
  2. The experiments provide useful evidence for the main design choices. The evaluation covers equation recovery and black-box regression, with results on predictive accuracy, expression complexity, and search time. The ablations examine online adaptation, decoder depth, and population seeding. These comparisons help clarify the contribution of policy updates and show the trade-off between recovery performance and computational cost when using a single decoder block.
Critical weaknesses
  1. The added value of online generator training remains unclear. NGGP already combines GP search with generator updates, including PQT [1]. Work on subexpression reuse [2] also shows that search discoveries can be used effectively without training a neural policy. This raises a question that DIGEST does not sufficiently address: what additional advantage does training provide when useful structures can already be retained and reused directly? The ablations show some improvements from adaptation, but provide limited insight into the additional search capability gained through parameter updates.
  2. The equation-recovery evaluation is limited in scope. Recovery is evaluated on 12 Nguyen and 22 Livermore targets, without free-constant tokens. The BlackBox results provide useful predictive evidence but cannot establish recovery of the underlying equations. A broader known-equation evaluation, such as Feynman [3] with fitted coefficients, would better support the claimed flexibility across diverse expressions. The discussion should also distinguish the numerical recovery criterion from symbolic equivalence.

References: [1] Mundhenk et al. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding. NeurIPS, 2021. arXiv:2111.00053. [2] Wu et al. Progressive Subexpression Reuse in Symbolic Regression: Insights from RL-based Search and a Genetic Programming Realization. IJCAI, 2026. [3] La Cava et al. Contemporary Symbolic Regression Methods and their Relative Performance. NeurIPS Datasets and Benchmarks, 2021.

Questions for authors

1.What does DIGEST’s training mechanism add beyond the existing NGGP–PQT loop?

2.What measurable benefit does the trained generator provide over direct subexpression retention and reuse?

3.How does DIGEST compare with explicit-reuse methods under the same search budget?

4.How much of the gain depends on pretrained weights rather than the generator architecture?

5.Do the recovery gains extend to Feynman equations with fitted coefficients and symbolic-equivalence checks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yxo7 (1/2)

We thank the reviewer for the constructive feedback and comments, all of which are helpful to further improve our paper. See below for our responses to the “Critical Weaknesses” (W) and “Questions For Authors” (Q). We have carefully revised the manuscript in response to the review, with all additions and revisions highlighted in blue for ease of reference. The relevant manuscript revisions and their locations are indicated below.

W1, Q2, Q3: Added value of online adaptation versus direct subexpression retention and reuse.

We thank the reviewer for highlighting the work on progressive subexpression reuse by Wu et al. [1], which introduces Reinforcement Genetic Programming (RGP) for explicit subexpression retention and reuse. Following this suggestion, we evaluated RGP on the same Nguyen and Livermore benchmarks, using the same nominal 2M expression-evaluation budget and numerical recovery criterion as in our original experiments (Section 4.1 and Appendix B.1).

BenchmarkMethodRecovery (%) ↑Mean search time (s) ↓
NguyenRGP84.3376.8
NguyenDIGEST93.33 (+9.00 pp)49.2
LivermoreRGP54.36174.6
LivermoreDIGEST74.00 (+19.64 pp)118.8

DIGEST achieves 9.00 and 19.64 percentage points higher recovery on Nguyen and Livermore, respectively, with shorter recorded mean search times. These results provide a direct comparison with the explicit-reuse method identified by the reviewer.

Importantly, DIGEST already builds on GP's structural retention and reuse, rather than replacing them. Selection favors promising individuals, and subtree crossover recombines existing subexpressions. Our random-seed GP control retains these operations; adding DIGEST's adaptive population seeding increases recovery from 91.00% to 93.33% on Nguyen and from 66.91% to 74.00% on Livermore (Table 4). Thus, the observed benefit is obtained on top of a search procedure that already reuses structures.

The key distinction from RGP is how search discoveries guide subsequent generation. RGP explicitly retains and reinserts useful subexpressions into new populations, whereas DIGEST uses high-quality expressions to update a context-conditioned symbolic policy, allowing past discoveries to influence subsequent symbol-generation probabilities. Thus, explicit retention/reuse and parameter adaptation are not mutually exclusive.

To isolate the contribution of parameter updates, our frozen-policy ablation (Table 4) shows that online adaptation further improves recovery from 90.67% to 93.33% on Nguyen and from 72.91% to 74.00% on Livermore, with the same GP refinement procedure. The RGP comparison evaluates the overall approach against explicit reuse, while this ablation separately quantifies the incremental benefit of online adaptation.

In the revised paper: Sections 4.1 and 6; Appendix F.1, Table 14.

Q1, W1: Contributions beyond the NGGP–PQT loop.

DIGEST contributes a pretrained symbolic interface and a single-block execution strategy, turning a general-purpose pretrained language model into an efficient symbolic policy that learns from GP discoveries.

(1) Pretrained symbolic interface. Unlike NGGP's SR-specific recurrent generator [2], DIGEST repurposes a general-purpose pretrained decoder through a symbolic input/output interface. We initialize symbolic input embeddings and the output projection from pretrained text-token representations, and encode parent–sibling tree contexts as an interleaved causal sequence. Unlike most recent LLM-based symbolic regression methods, which primarily rely on prompt-based candidate generation or refinement, DIGEST directly repurposes a pretrained decoder as a symbolic generation policy for GP population seeding, with online adaptation from search feedback.

(2) Single-block execution and adaptation. DIGEST executes and updates only one selected pretrained decoder block, together with the symbolic interface. In contrast, full-depth LoRA still executes the entire decoder stack. Despite similar trainable parameter counts (14.94M vs. 14.87M), DIGEST achieves competitive recovery with substantially lower search cost: 93.33%/74.00% recovery on Nguyen/Livermore versus 93.67%/79.27% for full-depth LoRA, with 8.1×/10.2× speedups in mean search time (Table 4). At the system level, DIGEST also achieves higher recovery than LaSR while using only 7.2%/17.4% of its recorded mean search time under the reported configurations (Figure 4).

AuthorsOpenReview ↗

Response to Reviewer yxo7 (2/2)

W2, Q5: Feynman evaluation with fitted coefficients and symbolic recovery.

(1) Additional Feynman evaluation. We thank the reviewer for this suggestion. We additionally evaluated DIGEST on all Feynman equations in our SRBench evaluation suite, enabling free-constant tokens and coefficient fitting. We report the original SRBench symbolic solution rate [3] as the equation-recovery metric.

MethodSRBench symbolic solution rate (%) ↑
DIGEST56.25
AIFeynman55.36
AFP_FE31.03
GP-GOMEA24.14
ITEA22.41
DSR20.69
AFP18.10
gplearn12.93
SBP-GP12.07
EPLEX7.76
Operon7.76
BSR1.72
FEAT0.00
FFX0.00
MRGP0.00

These additional results provide evidence of DIGEST's ability to recover known equations with fitted coefficients, extending the original Nguyen/Livermore evaluation beyond a search vocabulary without free constants.

(2) Numerical recovery versus symbolic equivalence. Our original Nguyen and Livermore experiments follow the numerical recovery criterion adopted from the released NGGP/DSO implementation [2] (test NMSE < 10−1210^{-12}, Appendix B.1). This measures predictive agreement on noiseless test observations, rather than algebraic identity.

For Feynman, we adopt the original SRBench Symbolic Solution criterion (Definition 4.1), which is the same criterion used to evaluate Feynman equations in the original SRBench study [3]. This criterion checks symbolic relationships while allowing constant offsets or nonzero multiplicative factors. We therefore report the symbolic solution rate used in SRBench's original Feynman evaluation, distinguishing it from our numerical recovery criterion and strict algebraic identity.

In the revised paper: Section 4.2; Appendix F.2, Table 15.

Q4: Contribution of pretrained weights versus the generator architecture.

We conducted three additional initialization ablations, keeping the policy architecture and training/search hyperparameters unchanged:

  • Random decoder block: Randomly reinitialize the selected decoder block.
  • Shuffled pretrained weights: Shuffle pretrained weights within the selected decoder block.
  • Random symbolic input embeddings: Randomly initialize the parent–sibling symbolic input embeddings.

All variants retain GP refinement and online adaptation.

InitializationNguyen recovery (%) ↑Livermore recovery (%) ↑
DIGEST: pretrained initialization93.3374.00
Random decoder block91.33 (−2.00 pp)71.27 (−2.73 pp)
Shuffled pretrained weights91.33 (−2.00 pp)72.18 (−1.82 pp)
Random symbolic input embeddings88.33 (−5.00 pp)71.82 (−2.18 pp)

Parentheses indicate changes relative to DIGEST.

All three ablations yield lower recovery on both benchmarks, supporting the contribution of pretrained initialization beyond the generator architecture alone, even when online adaptation remains enabled.

In the revised paper: Section 5, Table 5.

References

[1] Wu, Xiangdong, et al. “Progressive Subexpression Reuse in Symbolic Regression: Insights from RL-based Search and a Genetic Programming Realization.” IJCAI, 2026.

[2] Mundhenk, T. Nathan, et al. “Symbolic Regression via Neural-Guided Genetic Programming Population Seeding.” NeurIPS, 2021.

[3] La Cava, William, et al. “Contemporary Symbolic Regression Methods and their Relative Performance.” NeurIPS Datasets and Benchmarks, 2021.

AuthorsOpenReview ↗

Looking Forward to Your Feedback and Further Discussion

Dear Reviewer yxo7,

Thank you again for your thoughtful and constructive review. We have carefully considered your feedback, conducted additional experiments and analyses to address your concerns, and revised the manuscript accordingly.

As the discussion period draws to a close, we would sincerely appreciate your feedback on whether our responses have addressed your main concerns. We would be happy to discuss and clarify any remaining questions.

Thank you for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

A Gentle Follow-up on Our Responses

Dear Reviewer yxo7,

We apologize for reaching out once more, especially during this busy period. As the discussion period is coming to a close, we wanted to kindly follow up on our earlier responses and revised manuscript.

If our responses have addressed your main concerns, we would be very grateful if you could confirm this or consider updating your assessment. If anything remains unclear, we would be happy to provide further clarification in the time remaining.

Thank you again for your time and thoughtful feedback, which has helped us improve the paper.

Best regards,

The Authors

Reviewer_SzG1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DIGEST repurposes a single decoder block from a lightweight LLM (Qwen2.5-0.5B) as a symbolic policy. Expressions are serialized as parent–sibling context sequences; the policy generates a population of expressions that seeds genetic programming (GP). GP refines the population for 25 generations, and high-reward expressions are stored in a persistent priority queue. The policy is updated online with priority-queue training plus entropy regularization, and the adapted policy seeds the next GP restart. On Nguyen and Livermore, DIGEST reports 93.33% and 74.00% recovery, respectively, with mean search times of 49.2 s and 118.8 s. On 122 SRBench BlackBox datasets, it reports the highest median test R² (0.956 vs. 0.945 for TPSR) with a smaller mean model size (53.44 vs. 95.71).

Critical strengths
  1. Clever and lightweight use of pretrained language models. Repurposing a single decoder block, initializing symbolic embeddings from text-token embeddings, and using parent–sibling context is an elegant way to obtain a symbolic prior without prompting a full LLM. The design is much cheaper than LLM-based prompting pipelines.

  2. Strong efficiency. The reported 49.2 s and 118.8 s mean search times are 7.2% and 17.4% of the strongest competitor’s time, respectively. This is a genuine practical advantage, and the paper reports the comparison transparently.

  3. Competitive recovery. 93.33% on Nguyen and 74.00% on Livermore are strong numbers, and the SRBench median R² with a much smaller model size is a useful result. The method is competitive with iMCTS on Livermore (+2.59 points) and essentially tied on Nguyen (+0.08).

  4. Informative ablations. The comparison between random-seed GP (66.91/91.00), frozen single-block policy (72.91/90.67), interface-only updates (72.18/90.67), and full DIGEST (74.00/93.33) is helpful. The full-depth LoRA result (79.27/93.67 at 1213.8 s) is a useful cost–accuracy reference.

  5. Clear algorithm and good exposition. The three-stage procedure (generation, GP refinement, adaptation) is easy to follow, and the appendix provides the implementation details.

Critical weaknesses
  1. The headline gain over the strongest conventional baseline is very small on Nguyen. DIGEST reports 93.33% vs. iMCTS’s 93.25%, a 0.08-point difference on 12 equations. Even with 25 seeds per equation, this is almost certainly within run-to-run noise. The abstract’s “outperforms the strongest competitor by up to 2.59%” is technically true because of Livermore, but the Nguyen result does not establish superiority. The paper should report confidence intervals and a paired test, and avoid claiming a win on Nguyen without evidence.

  2. The comparison with LLM-based baselines conflates budget with method quality. The paper states that conventional baselines and DIGEST use a 2×10^6 evaluation budget, while LLM-based baselines use “bounded, method-specific search budgets.” A 54.91-point gap over LaSR therefore does not show that DIGEST is a better use of LLMs; it mainly shows that LaSR was run with a much smaller budget. An iso-budget comparison (same number of LLM calls/tokens and the same GP evaluations) is needed before the “54.91% improvement over LLM-based baselines” claim can be interpreted.

  3. GP does most of the work. Random-seed GP already achieves 66.91/91.00, and the full method achieves 74.00/93.33. The incremental gain from the LLM policy is 7.09/2.33 points, and from online adaptation 1.09/2.66 points. The framing “LLM tuning for symbolic regression” overstates the LLM’s role; the method is better described as a GP search with a learned seeding distribution and a light adaptation step. A stronger GP baseline with more generations or a better seeding strategy might close much of the gap.

  4. The “LLM” is a single 0.5B decoder block, and the pretrained-prior claim is not isolated. The frozen-policy ablation (72.91/90.67) still uses pretrained weights; there is no randomly initialized decoder with the same architecture and interface. Without that control, it is unclear how much of the gain comes from pretraining versus the additional parameters and the parent–sibling serialization. The interface-only ablation (72.18/90.67) is close to the frozen policy, suggesting that most of the benefit may come from the architecture/serialization rather than pretrained knowledge.

  5. Online adaptation is extremely light. The policy receives one PQT step per restart and one queue sample; “progressively internalize past search discoveries” is a strong description for such a small number of updates. The paper should report the number of updates per run, the diversity of the queue, and whether the policy distribution changes measurably. A comparison with a frozen policy given more GP generations would help isolate the effect of adaptation from the effect of extra search.

  6. Limited ground-truth evaluation. The recovery results use 34 equations (12 Nguyen + 22 Livermore) with a strict NMSE < 1e-12 threshold. This is a narrow benchmark; the SRBench evaluation uses one seed per dataset and reports no variance. The conclusions about recovery and model size should be tempered accordingly.

  7. Model-size comparison is approximate. The 53.44 vs. 95.71 comparison depends on tokenization, simplification, and constant handling; it is not a fair measure of functional complexity. The paper should report the distribution of expression sizes and the simplification procedure, not only the mean.

  8. The closest learned-proposal baselines are missing. DIGEST sits in the hybrid family that learns where to propose expressions and then refines them with a numerical search procedure. The related work mentions MCTS with learned guidance in passing, but the experiments compare only with conventional SR (DSR, GEGL, NGGP, PySR, iMCTS) and LLM-based methods (ICSR, LLM-SR, LaSR, RESTART). Two closely related lines are absent: transformer-based joint supervised SR (Li et al., ICLR 2022) and neural-guided dynamic symbolic networks (DySymNet, Li et al., ICML 2024). This matters for the central claim that the learned policy contributes beyond GP: DySymNet is a direct example of a learned proposal mechanism refined by search, and a comparison at matched evaluation budget would test whether DIGEST's gain comes from the LLM prior, the online adaptation, or simply from adding a learned seeding distribution. I recommend adding DySymNet on Nguyen and Livermore, and on a subset of SRBench if feasible, with the same time and model-size metrics. At minimum, the related work should explain the difference between the two approaches.

Questions for authors
  1. Please report mean ± std over the 25 seeds and a paired test for DIGEST vs. iMCTS on both Nguyen and Livermore. Is the 0.08-point Nguyen difference significant?
  2. What are the exact LLM-call/token and evaluation budgets for each LLM-based baseline? Can you run them at the same budget as DIGEST?
  3. Can you add a randomly initialized decoder block with the same architecture and interface as a control, to isolate the effect of pretrained weights?
  4. How many PQT updates does the policy receive per run, and how much does the policy distribution change? Does adaptation still help if the frozen policy is given more GP generations?
  5. Can you compare against a strong GP baseline (Operon, GP-GOMEA, or PySR) at the same evaluation budget?
  6. How are constants fitted and expressions simplified, and how does that affect the model-size comparison?
  7. Will the code and the trained policy be released?
  8. Can you compare DIGEST with DySymNet at the same evaluation budget on Nguyen and Livermore? If not, can you explain how the two learned-proposal mechanisms differ and why the comparison would not be informative?
6 comments
AuthorsOpenReview ↗

Response to Reviewer SzG1 (1/4)

We thank the reviewer for the constructive feedback and comments, all of which are helpful to further improve our paper. See below for our responses to the “Critical Weaknesses” (W) and “Questions For Authors” (Q). We have carefully revised the manuscript in response to the review, with all additions and revisions highlighted in blue for ease of reference. The relevant manuscript revisions and their locations are indicated below.

Q1, W1: Performance comparison with iMCTS across benchmarks.

DIGEST demonstrates its main performance advantages over iMCTS on Livermore and SRBench BlackBox. On Livermore, DIGEST achieves 74.00% recovery versus 71.41% for iMCTS, an improvement of 2.59 percentage points (Table 3). On SRBench BlackBox, DIGEST achieves the highest cross-dataset median test R2R^2, ranking first among the compared methods, while iMCTS ranks third (Figure 5).

We additionally report 95% confidence intervals for DIGEST's mean recovery across 25 seeds:

BenchmarkMean recovery (%) ↑95% CI (%)
Nguyen93.33[91.11, 95.55]
Livermore74.00[72.08, 75.92]

On Nguyen, DIGEST achieves 93.33% recovery versus 93.25% for iMCTS, demonstrating comparable performance.

W2, Q2: Search budgets, timing boundaries, and comparison with LLM-based methods.

(1) Under the 2M evaluation cap, DIGEST takes only 49.2/118.8 seconds on average.

The 49.2 seconds on Nguyen and 118.8 seconds on Livermore in Figure 4 are the mean times for DIGEST's actual search under the 2M expression-evaluation cap, including LLM-based generation, GP search, and online training. The cap covers LLM-generated expressions and GP evaluations together. Recovery-based early stopping is enabled, so not every run exhausts the cap.

Unlike prompt-based LLM baselines, DIGEST directly generates symbolic actions through a single decoder block. Symbolic prediction counts, full-model generation requests, and text-token counts therefore do not represent equivalent computational costs.

Under the reported configurations, DIGEST achieves higher recovery in less recorded search time than the compared LLM-based methods. This demonstrates the practical search efficiency of our approach. We summarize the original results from Table 3 and Figure 4 below; values separated by "/" denote Nguyen / Livermore.

MethodRecovery (%) ↑Mean search time (s) ↓
DIGEST93.33 / 74.0049.2 / 118.8
ICSR14.00 / 16.73187.8 / 234.0
LLM-SR3.33 / 1.27702.0 / 710.4
LaSR29.67 / 19.09684.0 / 681.6
RESTART4.00 / 2.18463.2 / 418.8

The reported search times exclude initial LLM loading (Appendix C.3); other setup costs within the search interval follow each implementation.

(2) LaSR evaluated within DIGEST's mean search times.

We further evaluated LaSR, the strongest LLM-based baseline by recovery in Table 3, using only expressions available within 49.2 seconds on Nguyen and 118.8 seconds on Livermore, corresponding to DIGEST's reported mean search times. Only complete model snapshots saved before each time limit were evaluated, without additional coefficient fitting afterward.

For comparison, we also include the original results reported in Table 3 and Figure 4.

BenchmarkMethod / settingRecovery (%) ↑Median test NMSE ↓Search time (s)
NguyenDIGEST93.331.50×10−321.50\times10^{-32}49.2 (mean)
NguyenLaSR (reported)29.672.3×10−52.3\times10^{-5}684.0 (mean)
NguyenLaSR (time-limited)16.672.28×10−32.28\times10^{-3}49.2 (limit)
LivermoreDIGEST74.005.20×10−325.20\times10^{-32}118.8 (mean)
LivermoreLaSR (reported)19.095.7×10−55.7\times10^{-5}681.6 (mean)
LivermoreLaSR (time-limited)9.093.57×10−43.57\times10^{-4}118.8 (limit)

Under time limits corresponding to DIGEST's mean search times, LaSR achieves lower recovery and higher NMSE on both benchmarks. Even with its substantially longer originally reported search times, LaSR still falls behind DIGEST in both metrics. These results further demonstrate DIGEST's practical search efficiency.

In the revised paper: Section 4.2; Appendix F.3, Table 16.

(3) DIGEST remains faster when recorded loading and startup costs are included.

We also examined broader elapsed times, including the recorded model-loading and startup overheads:

MethodNguyen: elapsed (s) ↓Livermore: elapsed (s) ↓
DIGEST60.17133.51
ICSR282.48326.70
LLM-SR820.02870.09
LaSR824.28836.28
RESTART593.13488.39

DIGEST retains its runtime advantage on both benchmarks after these overheads are included. Its shorter reported search time is therefore not simply a consequence of excluding initial model loading.

AuthorsOpenReview ↗

Response to Reviewer SzG1 (2/4)

W3, Q5: Contribution beyond GP and comparison with strong GP baselines.

(1) Comparisons with strong GP baselines are already included.

Our evaluation already includes all three GP-based baselines mentioned by the reviewer: PySR, Operon, and GP-GOMEA. Table 3 compares DIGEST with PySR on Nguyen and Livermore under the same 2M expression-evaluation budget:

BenchmarkPySR recovery (%)DIGEST recovery (%)
Nguyen74.4193.33
Livermore46.1474.00

Figure 5 additionally compares DIGEST with Operon and GP-GOMEA on SRBench BlackBox. DIGEST achieves the highest observed cross-dataset median test R2R^2, while Operon and GP-GOMEA rank second and ninth, respectively, among the methods shown.

(2) The generation policy contributes on top of GP refinement.

GP performs the evolutionary search in DIGEST; our contribution is to improve population seeding through a pretrained symbolic policy that adapts to search feedback. With the same GP refinement procedure and evaluation budget, DIGEST improves recovery over random-seed GP from 91.00% to 93.33% on Nguyen and from 66.91% to 74.00% on Livermore (Table 4).

The frozen-policy control separately measures the incremental benefit of online updates, with recovery increasing from 90.67% to 93.33% on Nguyen and from 72.91% to 74.00% on Livermore. These results quantify the generator's contribution without attributing the entire search performance to the language model. Our additional frozen-policy experiment with more GP generations is discussed under W5/Q4 below.

W4, Q3: Contribution of pretrained weights beyond the generator architecture.

Following the reviewer's suggestion, we conducted three additional initialization ablations, keeping the policy architecture and training/search hyperparameters unchanged:

  • Random decoder block: Randomly reinitialize the selected decoder block.
  • Shuffled pretrained weights: Shuffle pretrained weights within the selected decoder block.
  • Random symbolic input embeddings: Randomly initialize the parent–sibling symbolic input embeddings.

All variants retain GP refinement and online adaptation.

InitializationNguyen recovery (%) ↑Livermore recovery (%) ↑
DIGEST: pretrained initialization93.3374.00
Random decoder block91.33 (−2.00 pp)71.27 (−2.73 pp)
Shuffled pretrained weights91.33 (−2.00 pp)72.18 (−1.82 pp)
Random symbolic input embeddings88.33 (−5.00 pp)71.82 (−2.18 pp)

Parentheses indicate changes relative to DIGEST.

All three ablations yield lower recovery on both benchmarks, supporting the contribution of pretrained weights beyond the generator architecture and symbolic interface alone, even when online adaptation remains enabled.

In the revised paper: Section 5, Table 5.

W5, Q4; W3 (GP search depth): Online policy updates and deeper GP search.

(1) Update frequency and training feedback.

The policy receives an average of 26.8 PQT updates per complete search on Nguyen and 77.4 on Livermore. Each GP restart is followed by one update.

In the additional queue analysis, an average of 17.7 distinct expression token sequences enter the queue during a complete search, and 13.3 are used for training. Thus, online training incorporates multiple expressions discovered during search.

(2) Changes in the generation policy.

For the same fixed expression prefixes, we compare the probabilities assigned to the next symbol before and after online training. The average total-variation distance is 0.096, compared with 0 for the frozen policy, showing a measurable change in the generation distribution.

For example, in a Nguyen-12* search, the generator's candidates improve before GP refinement:

MetricFirst ten restartsLast ten restarts
Invalid-expression rate ↓62.40%22.64%
Mean batch-best training reward ↑0.5610.837

This example shows fewer invalid proposals and higher batch-best training rewards in later stages of the search.

(3) Frozen policy with more GP generations.

We compared DIGEST with online policy updates and 25 GP generations per restart (Online + 25) against a frozen policy with 50 GP generations per restart (Frozen + 50).

ConfigurationNguyen recovery (%) ↑Livermore recovery (%) ↑
DIGEST (Online + 25, Table 4)93.3374.00
Frozen + 50 (additional experiment)90.0068.18

Frozen + 50 still achieves lower recovery than DIGEST on both benchmarks, suggesting that increasing GP generations alone does not necessarily reproduce the benefits of online adaptation.

In the revised paper: Section 5; Appendix F.4, Tables 17–18.

AuthorsOpenReview ↗

Response to Reviewer SzG1 (3/4)

W6: Ground-truth evaluation coverage and repeated runs on SRBench.

(1) SRBench already uses ten runs per dataset, not one.

Our SRBench BlackBox evaluation uses ten independent runs per dataset, as stated in Appendix B.2.

Figure 5 already includes 95% bootstrap confidence intervals for the cross-dataset medians, following SRBench's released BlackBox plotting implementation [1]. Some error bars overlap with the point markers at the current plotting scale and are therefore difficult to distinguish. We have clarified the 95% CI notation in the revised caption and will further improve their visibility in the final version.

In the revised paper: Figure 5 caption.

(2) Expanded ground-truth evaluation on Feynman.

We additionally evaluated DIGEST on all Feynman equations in our SRBench evaluation suite, enabling free-constant tokens and coefficient fitting. We report the original SRBench symbolic solution rate [1] as the equation-recovery metric.

MethodSRBench symbolic solution rate (%) ↑
DIGEST56.25
AIFeynman55.36
AFP_FE31.03
GP-GOMEA24.14
ITEA22.41
DSR20.69
AFP18.10
gplearn12.93
SBP-GP12.07
EPLEX7.76
Operon7.76
BSR1.72
FEAT0.00
FFX0.00
MRGP0.00

These results extend the ground-truth evaluation beyond Nguyen and Livermore and provide evidence of symbolic recovery with fitted coefficients. The interpretation of model size is clarified separately in our response to W7/Q6 below.

In the revised paper: Section 4.2; Appendix F.2, Table 15.

W7, Q6: Model-size distributions, constant fitting, and simplification.

(1) Model size counts expression-tree nodes.

DIGEST measures model size by expression-tree node count, not language-model tokens or character length. Each operator, variable occurrence, and constant occurrence counts as one node; the number of digits in a constant does not affect its size.

(2) Constant fitting and simplification.

DIGEST fits free constants using training data only, minimizing the negative training reward with SciPy. Fitting changes coefficient values, not the node count of a fixed tree. Model size is recorded from the native expression tree; simplification of the separately exported SymPy expression does not change this count.

TPSR [2] uses BFGS coefficient refinement, and its returned-tree count includes nodes introduced by reversing input scaling. Thus, the reported sizes refer to each method's returned representation, rather than expressions reduced to a common simplified form.

(3) Additional model-size distributions.

We supplement the mean sizes reported in Table 13 with the following distributional statistics:

MethodMeanMedian25th–75th percentilesMin–Max
DIGEST53.445551–5839–60
TPSR95.719675.5–11524–149

DIGEST achieves smaller returned expression trees than TPSR on average, as reflected in the mean (53.44 vs. 95.71), median (55 vs. 96), and interquartile ranges (51–58 vs. 75.5–115).

In the revised paper: Section 4.3; Appendix F.5, Table 19.

W8, Q8: Comparison with other learned-proposal methods.

(1) Additional comparison with DySymNet.

We thank the reviewer for highlighting DySymNet [3]. We evaluated its released implementation on our Nguyen and Livermore benchmarks, using the same numerical recovery criterion.

BenchmarkMethodRecovery (%) ↑Median test NMSE ↓Mean search time (s) ↓
NguyenDIGEST93.331.50×10−321.50\times10^{-32}49.2
NguyenDySymNet0.008.737×10−38.737\times10^{-3}230.93
LivermoreDIGEST74.005.20×10−325.20\times10^{-32}118.8
LivermoreDySymNet0.003.589×10−33.589\times10^{-3}258.18

DIGEST achieves higher recovery and lower median test NMSE on both benchmarks, with shorter mean search times in the reported experiments.

In the revised paper: Section 4.1; Appendix F.6, Table 20.

(2) Differences from related learned-proposal methods.

DySymNet [3] uses an RNN trained by reinforcement learning to propose symbolic-network architectures, then optimizes and prunes these networks to obtain expressions. The joint-supervised Transformer [4] learns a mapping from numerical observations to expression skeletons using synthetic training data, followed by coefficient fitting. Unlike these SR-specific neural proposal methods, DIGEST repurposes a decoder block from a general-purpose pretrained language model to generate expression trees for GP population seeding, and adapts the policy online through PQT using GP discoveries.

In the revised paper: Section 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.