acceptodds
Under review as a conference paper at ICLR 2027

RISR: Residual-Informed Scientific Equation Discovery with Large Language Models

Abstract

Symbolic regression combines structural search with numerical fitting, but aggregate fit scores do not describe how the remaining error varies across inputs. We introduce RISR, a residual-informed method that uses these error patterns to guide formula discovery and learn which corrections are worth fitting. A residual encoder compresses aligned inputs, targets, current predictions, and residuals into continuous tokens that condition a language model to propose formulas. For subsequent refinement, a dual-view relational encoder uses additive and regularized multiplicative residuals to predict the post-fit utility of candidate corrections. We evaluate RISR on scientific tasks from the LLM-SRBench. RISR achieves 63.57% and 38.50% ID accuracy at the 1% and 0.1% pointwise relative-error tolerances, respectively. The corresponding OOD accuracies are 56.07% and 38.24%. RISR outperforms the reported baselines using the same backbone. The results show that our residual-informed approach can improve numerical equation recovery.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Symbolic Regression, LLMs, Scientific Equation Discovery, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P2iNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RISR, which uses residual encoding to guide a frozen LLM in generating candidate equations, followed by structural correction and further refinement based on predicted post-fit utility. The authors evaluate accuracy and symbolic recovery on LSR-Synth, achieving competitive results.

Critical strengths

The method jointly encodes inputs, current predictions, and pointwise residuals to guide candidate generation, and selects corrections based on post-fit utility. Compared with relying solely on aggregate fitting errors, this design provides more detailed numerical information. The complete system achieves competitive ID/OOD numerical accuracy across multiple scientific domains, outperforming some baselines that use the same LLM backbone.

Critical weaknesses
  • The necessity and independent contribution of the additional Residual Encoder have not been sufficiently validated. The current experiments do not adequately distinguish the effects of the dedicated Encoder, additional supervised training, and Structural Editor. Key comparisons with direct input of the same numerical information and LLM fine-tuning without the Encoder are missing.
  • The experiments primarily report accuracy, with limited analysis of how individual components contribute to symbolic structure recovery. The presented case study also does not sufficiently illustrate the structural discovery process or its associated complexity cost.
Questions for authors
  • The paper introduces an additional Residual Encoder to jointly process inputs, predictions, and residuals. However, the LLM itself can also model relationships among these inputs through its attention mechanism. What advantages does the dedicated numerical encoder offer over directly providing the same information to the LLM? Two additional ablations would be valuable: removing the Encoder and providing the same information as text to the frozen LLM; and removing the Encoder and fine-tuning the LLM using the same synthetic training data.
  • LSR-Synth uses newly constructed scientific equations to reduce reliance on LLM memorization. However, RISR introduces additional supervised training on synthetic equations, which may bring some test equation structures back into the model’s familiar training distribution. To what extent are the training and test equations mathematically equivalent or structurally overlapping? Results after excluding closely related templates would be valuable.
  • In Figure 3, the LLM Proposal already contains all the target structural terms, while the additional term could potentially be eliminated by setting its coefficient to zero. Thus, the Proposal may already satisfy the paper’s symbolic recovery criterion. More diagnostic case studies would be helpful, such as an initial Proposal that lacks key structural terms but subsequent steps successfully recover the correct structure. Please also provide the complete search trajectory from the Initial Seed to the final equation. Representative failure cases would help distinguish the actual contributions of the LLM Proposal, Structural Editor, and coefficient fitting.
  • RESTART also uses residual information to guide structural corrections of equations. Please clarify the key differences between the two methods in residual representation, candidate generation, and refinement mechanisms. A direct comparison using a consistent LLM backbone, search budget, and evaluation protocol would also be valuable.
Reviewer_pMTVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISR, a residual-informed framework that first encodes input observations, predictions, and residuals into continuous tokens to guide an LLM in generating candidate equations. It then employs a dual-view encoder to capture additive and multiplicative residual patterns, guiding iterative equation refinement through learned correction selection.

Critical strengths
  1. RISR utilizes point-wise residuals as conditional information for structural search—a well-founded motivation that is directly relevant to the challenge of iterative refinement in symbolic regression.
  2. The two-stage design offers a degree of methodological innovation. It is more targeted than a simple ranking based on residual correlations.
  3. The design of synthetic training data and supervised objectives is quite comprehensive.
Critical weaknesses
  1. How much does each component—the Residual Encoder and the Structural-edit Generator—contribute? Table 2 only shows the performance gains of the complete Stage A model compared to an LLM-only baseline; it does not include an experiment where the Structural-edit Generator is removed.
  2. The refinement stage is limited to 4 correction attempts. It is unclear how the number of refinement iterations affects numerical accuracy, out-of-domain generalization, and symbolic recovery. Could the authors provide a sensitivity analysis with different iteration budgets (e.g., 1, 2, 8, and 16) to investigate whether additional refinement consistently improves performance or leads to overfitting?
  3. The proposed method only considers additive and multiplicative residual views, without sufficiently demonstrating their complementarity. An ablation comparing additive-only, multiplicative-only, and dual-view configurations, along with an exploration of alternative residual representations, would strengthen the justification for this design.
Questions for authors

Q1: The primary concerns in Weaknesses 1–3.

Q2: Some mathematical notations and presentation details could be clarified. In Figure 4, the dimension (h) is not explicitly defined. Eq. (7) does not specify how relative errors are handled when the ground-truth value is zero. There are also minor citation and formatting issues.

Reviewer_a66YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RISR, a two-stage method for LLM-based symbolic regression. In the first stage, a residual encoder trained on synthetic data converts the current equation’s errors into soft tokens that condition a frozen LLM during an LLM-SR-style search, alongside a learned structural editor. In the second stage, the selected equation is refit and extended with a few correction terms chosen by a learned utility scorer, with no further LLM calls. On LSR-Synth, RISR reports higher accuracy than the reported baselines with the same 30B backbone.

Critical strengths
  • Using the structure of the residual instead of a single fit score is a well-motivated idea, and the paper turns it into concrete learnable components with sensible design choices, such as the additive and multiplicative residual views and a utility target measured on held-out points.

  • The results are strong for a small backbone and a small search budget, and they are consistent across seeds. The evaluation also covers OOD accuracy and symbolic accuracy.

Critical weaknesses
  • It is hard to tell how much of the improvement comes from the residual-informed components. The largest gain in the ablation comes from refitting constants under a relative-error loss that closely matches the evaluation metric, while the baselines fit constants with MSE. The correction terms also add fitted parameters beyond the limits used for the baselines. The main comparison therefore mixes the method’s contribution with differences in fitting objective and model capacity.

  • The core claim of the first stage, that the residual tokens guide the LLM’s proposals, is not directly supported. The ablations do not isolate the tokens from the structural editor, and no evidence is given that the encoder learns a useful representation from its relatively small synthetic training set.

  • The evaluation is limited in scope and statistical support: a single benchmark, single-seed ablations, and a symbolic-accuracy protocol that differs from the one used for the baselines.

Questions for authors
  1. Could the same relative-error refit be applied to the final equations of LLM-SR and SR-Scientist? This would separate the effect of the fitting objective from the effect of the search.

  2. How many free parameters and what expression size do the final RISR equations have compared to the baselines? it would help to see results under a parameter budget matched to the baselines.

  3. Could the authors report Stage A without residual tokens, with tokens computed from an unrelated residual, and without the structural editor? Encoder validation results would also help.

  4. What does the encoder’s training prompt contain? In particular, is the parent program given as text, and are constants written as parameters or literals?

  5. Residuals are normalized by the target’s standard deviation, so they become very small once the parent fits well. Does the encoder still provide useful signal late in the search? Relatedly, training uses 256 points while search uses up to 4,000 points split into attention chunks; was this mismatch examined?

  6. Does each of the 64 slots correspond to one LLM request, and are all four continuations evaluated? Is the “LLM-only” row in Table 2 run under the same budget? Results at a matched budget in either direction would make the efficiency claim clearer.

  7. On the benchmark, how does the learned scorer compare to residual-correlation ranking, or to trying all valid actions and keeping the best? Ablations of the ranking and paired-state losses, and multiple seeds for Table 2, would show which choices matter.

  8. Could the baseline equations be re-judged for symbolic accuracy using the same judge and protocol? How many of the positive cases already hold before Stage B?

  9. Some synthetic training families (onset, hardening and softening, damage gates) resemble the benchmark’s materials and dynamics domains. How were these families chosen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.