- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PostDiag, an external diagnostic module for symbolic regression. Its central idea is to move beyond using a single error metric such as MSE as feedback. Instead, it analyzes the local behavior of candidate expressions and diagnoses them along two dimensions, namely expressiveness and stability, to determine whether a candidate should be preserved, repaired, simplified, or rewritten. An LLM is then used to modify the symbolic structure accordingly.
Critical strengths
-
Moving from scalar fitness to actionable candidate-level diagnosis is a practically meaningful direction. Scalar fitness functions are effective for ranking candidate expressions, but provide very limited information about where a candidate fails, why it fails, or what kind of structural modification should be made. Moving from purely scalar evaluation toward actionable candidate-level diagnosis is therefore a useful research direction.
-
The overall framework is reasonably modular, and the experimental evaluation covers a relatively broad range of solvers and settings.
Critical weaknesses
-
The paper presents structured diagnosis as one of its main methodological contributions. However, it does not discuss several highly relevant works, including Symbolic Regression with Physics-Informed Residual and Structural Sensitivity Pruning (RSSP, IEEE TEVC 2026), StruSR: Structure-Aware Symbolic Regression with Physics-Informed Taylor Guidance (arXiv 2025), and An Interpretable Approach to PDE Solution Discovery via Structural Experience Distillation (arXiv 2026). In particular, as one of the earliest works exploring this direction, RSSP already performs fine-grained evaluation of candidate substructures through residual sensitivity and structural sensitivity, and uses such information to guide pruning, mutation, and crossover. The authors should cite and systematically compare against RSSP, while also discussing the relationship between PostDiag and the other closely related structure-aware methods. Such a comparison is necessary to appropriately narrow the paper’s claims that existing structured feedback is generally tightly coupled to specific search processes, as well as its novelty claims regarding solver-independent diagnosis. The absence of these comparisons makes the current related-work discussion incomplete and makes it difficult to determine the actual boundary of the paper’s contribution.
-
Limited methodological novelty is one of the main concerns. The underlying problem itself was not first identified by this paper. Moreover, the Level, Trend, and Roughness features used in PFP essentially correspond to a local mean, a local linear trend, and a local nonlinear residual, respectively. These are fairly standard local statistical or local regression descriptors. The paper does not sufficiently demonstrate that this particular combination offers a clear advantage over simpler alternatives such as residual maps, local gradients, or curvature-based diagnostics.
-
The current experiments do not sufficiently establish that the mechanisms emphasized by the paper, namely PFP, DRP, the four quadrant formulation, and search independent diagnosis, are individually necessary, correct, or substantively novel. As a result, the paper has not yet convincingly shown that its claimed core diagnostic mechanism, rather than other components of the overall pipeline, is responsible for the observed improvements.
-
Interpreting high expressiveness risk directly as “underfitting” and high stability risk as “overfitting,” and then mapping these cases to repair, simplification, or rewriting, is intuitive but lacks sufficient theoretical justification. High instability may also result from parameter non-identifiability, optimization failure, or insufficient data, rather than structural redundancy. The current method does not adequately distinguish among these possibilities.
-
High Bias does not necessarily imply insufficient skeleton expressiveness, and high Var does not necessarily imply overfitting or redundancy. The labels “Underfit,” “Overfit,” and “Mismatched” are therefore better understood as heuristic semantic interpretations of the two risk measures rather than validated structural diagnoses.
-
The paper also has several noticeable writing issues. The Introduction tends to conflate two distinct issues, namely solver specific diagnostic feedback and the limitations of scalar fitness, thereby overstating the gap between prior work and the proposed method. The term “search independent” is also too strong, since PostDiag still affects subsequent search trajectories through candidate replacement and memory updates. In addition, the methodology currently reads more like a collection of modules than a mechanism driven design. The experimental section should be reorganized around clearly stated claims and the corresponding evidence required to support them. Finally, several conclusions should be stated more cautiously to avoid implying a level of general applicability that exceeds the scope of the current empirical validation.
Questions for authors
See Weaknesses.
6 comments
Author Response to Reviewer CFBY 2/4
📝 W2 / W6 Motivation, Novelty, and the Choice of PFP Features
Thank you for the feedback. We clarify the connection between our motivation and the resulting design.
1. The Introduction presents a connected motivation. Scalar fitness provides limited diagnostic information, as prior work has recognized. Several methods address this by designing richer feedback together with a new search procedure. We investigate a complementary question: can one diagnostic plug-in improve different solvers while preserving their distinctive search mechanisms? Since different solvers employ distinct internal search mechanisms, diagnosis must rely solely on candidate skeletons and observations, with lightweight adapter interfaces enabling integration across solvers. This motivates extracting richer information about the candidate's behavior. Inspired by the clinical workflow of examination → diagnosis → intervention, we use PFP to examine local behavioral patterns, DRP to assess discrepancy and refitting stability, and an LLM editor to execute the prescribed interventions. We will clarify this process more explicitly in the Introduction and Method sections of the revised manuscript.
2. Why level, trend, and roughness? These descriptors address three complementary questions:
- Level: is the local output magnitude correct? Derivative-based features alone cannot detect an additive offset.
- Trend: does the candidate vary in the right directions and at the right rates? The local least-squares slope provides a patch-level estimate of the gradient.
- Roughness: how much behavior remains unexplained by the local affine fit? It summarizes nonlinear variation without estimating a full Hessian.
This gives an ordered description of constant behavior, linear variation, and remaining nonlinear variation, estimated from groups of observations. Curvature is a useful higher-order alternative for the third role; it does not replace level or trend. Our contribution is the use of these patch profiles for a shared diagnostic mechanism, jointly comparing fitted behavior with observations and across bootstrap refits.
3. Relationship to the suggested alternatives. Figure 2b of the submitted text already compares PFP-based feedback with residual-based NMSE feedback through NMSE-only and NMSE+DRP. A gradient estimated by the same local linear fit coincides with our trend feature, making an additional comparison unnecessary. We add an ablation study (Table R3.2) replacing roughness with local curvature while retaining level and trend, to assess whether additional second-order information improves editing performance. Replacing Roughness with Curvature neither improves performance nor causes a noticeable decline. However, using Curvature alone leads to a substantial performance drop. We believe this is because Level and Trend are more direct and informative features, enabling more accurate DRP localization and evidence maps.
Table R3.2. Replacing roughness with local curvature.
| Solver | Patch features | ID [email protected] (%) ↑ | ID NMSE ↓ |
|---|---|---|---|
| LLM-SR + PostDiag | Level + Trend + Roughness | 31.0 | 2.12E-04 |
| LLM-SR + PostDiag | Level + Trend + Curvature | 29.5 | 3.28E-04 |
| LLM-SR + PostDiag | Curvature only | 26.4 | 7.53E-04 |
| PySR + PostDiag | Level + Trend + Roughness | 28.7 | 1.34E-03 |
| PySR + PostDiag | Level + Trend + Curvature | 27.6 | 2.06E-03 |
| PySR + PostDiag | Curvature only | 25.3 | 2.41E-03 |
4. Meaning of “search-independent.” The term distinguishes the diagnostic core from feedback that requires a particular search procedure. With the candidate, data, fitting routine, and random draws fixed, PostDiag computes the same report regardless of which solver supplied the candidate. The adapter returns accepted revisions to the solvers, allowing them to improve subsequent search. Independence concerns how diagnosis is computed; this intended downstream influence is the purpose of integration. We will clarify the scope and definition of this claim in the revised manuscript.
Author Response to Reviewer CFBY 3/4
📝 W3 Why the Diagnostic Components Are Needed
Thank you for raising this concern. We explain the design rationale, the need for search-independent diagnosis, and the contribution of each diagnostic component.
1. A connected examination–diagnosis–intervention workflow. Inspired by clinical practice, we designed three stages with distinct, connected roles:
- PFP performs the examination. NMSE aggregates pointwise errors into a single global score, providing limited information about where and how a candidate's behavior differs from the observations. PFP instead examines local level, trend, and roughness across input-space patches and their stability across bootstrap refits.
- DRP provides diagnostic localization. It organizes behavioral discrepancy and refitting instability into a two-dimensional risk plane. Their combination distinguishes four risk patterns and assigns an explicit priority: preservation, repair, simplification, or rewriting.
- The LLM editor performs the intervention. It uses this priority and the patch-wise evidence to revise the skeleton; the revision must pass the audit-error check before returning to the solver.
The stages therefore form a connected progression: behavioral evidence → intervention priority → validated revision.
2. Why diagnosis should be search-independent. Our objective is to equip existing SR methods with richer diagnostic feedback while preserving their distinctive search mechanisms. These methods use different internal states and candidate-generation operators, but share a common interface: a candidate skeleton and observed data. We therefore compute diagnostics through this interface without relying on solver-specific search statistics, and use lightweight adapters to integrate PostDiag with different solvers. This allows the same examination–diagnosis–intervention pipeline to serve multiple solvers. Table 1 of the submitted text demonstrates this reusability across seven LLM-based and GP-based methods.
3. Ablations test the contribution of the components. Within each solver, all variants use the same native search budget, LLM editor, and validation procedure. NMSE-only supplies scalar error; NMSE+DRP builds risk guidance from pointwise errors; Evidence-only supplies PFP evidence without DRP guidance; Full PostDiag combines PFP and DRP. Var-only routing and Bias-only routing are ablations of the two risk axes. In these variants, the evidence map contains only Var or Bias information, and risk diagnosis relies solely on the corresponding axis. For example, under Bias-only routing, both low-discrepancy/low-instability and low-discrepancy/high-instability cases are assigned to Preserve, while both high-discrepancy/low-instability and high-discrepancy/high-instability cases are assigned to Repair. As shown in Table R3.3, Full PostDiag achieves the highest in-distribution accuracy across all three solvers. The comparisons with NMSE + DRP and Evidence-only support combining local PFP evidence with DRP guidance for editing. Full PostDiag also consistently outperforms both single-axis variants, supporting the use of both behavioral discrepancy and instability across refits in diagnosis.
Table R3.3. Component ablation on LSR-Synth. ID [email protected] (%), Qwen3.5-27B.
| Feedback variant | LLM-SR ↑ | SR-Scientist ↑ | PySR ↑ |
|---|---|---|---|
| NMSE-only | 22.5 | 26.1 | 24.0 |
| NMSE + DRP | 24.8 | 28.2 | 25.3 |
| Evidence-only | 27.1 | 31.5 | 25.1 |
| Var-only routing | 23.0 | 25.8 | 24.3 |
| Bias-only routing | 25.1 | 28.7 | 27.4 |
| Full PostDiag | 31.0 | 35.7 | 28.7 |
Author Response to Reviewer CFBY 1/4
We thank the reviewer for recognizing the practical value of actionable diagnosis and the framework's modularity. We address the relationship to prior work, the evidence for each component, and the interpretation of the diagnostic risks below.
📝 W1 / W6 Relationship to Structure-Aware SR and Cross-Solver Compatibility
Thank you for highlighting these relevant works. We will cite RSSP (Gong Y, IEEE TEVC 2026), StruSR (Gong Y, arXiv 2025), and SED-MCTS (Gong Y, arXiv 2026) and clarify their relationship to PostDiag. SED-MCTS's arXiv preprint was posted on October 8, 2026, after the September 25 submission deadline; we will discuss it as well.
1. The distinction concerns what is diagnosed and how the evidence guides intervention.
Thus, inspired by the clinical process of examination, diagnosis, and intervention, PostDiag introduces a dedicated diagnostic pipeline for symbolic regression, comprising patch-wise behavioral evidence examination, risk diagnosis and localization, and evidence-guided intervention to provide richer feedback for different solvers. Since PostDiag's diagnosis is independent of the underlying search process, it can be integrated with GP-based, LLM-based, and agent-based methods through lightweight adapters. This integration improves performance across diverse solvers, enabling them to achieve performance comparable to or even better than higher-budget searches with fewer search resources.
2. Feasible integration with these methods.
For RSSP and StruSR, PostDiag can examine candidates before population selection, propose edits within the host's expression grammar, and return validated candidates for native evaluation. Their sensitivity-guided operators remain active. For SED-MCTS, PostDiag can examine completed expressions and submit revised candidates for reevaluation. Each revision should receive its own valid grammar derivation and attribution before entering the archive or updating structural experience.
3. Direct performance comparison on LSR-Synth.
We compare RSSP, PySR, and PySR + PostDiag on the same LSR-Synth tasks and data splits, with accuracy aggregated over three random seeds. We adapt the RSSP to the non-PDE setting by evaluating candidate fit against the observed responses and estimating the structural reference using only the training data. Both PySR configurations use 65 GP iterations. PySR + PostDiag uses Qwen3.5-27B for editing.
As shown in Table R3.1, adding PostDiag to PySR improves both in-domain and out-of-domain accuracy. PySR + PostDiag achieves the highest accuracy on both metrics among the three methods.
Table R3.1. Comparison with RSSP on LSR-Synth.