- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper investigates how overlap among acceptable solutions affects the usefulness of component-level feedback for LLMs. The authors construct a controlled symbolic regression task in which LLMs select five mathematical terms from a predefined grammar. They vary the overlap among 32 designated acceptable solutions while keeping the numerical data, solution count, and evaluation procedure fixed.
Experiments across 11 models show that component-level feedback becomes more effective as acceptable solutions share more components. Additional experiments using a mechanical reader and SAT problems provide supporting evidence.
Critical strengths
-
The experimental design systematically controls several factors, including the number of acceptable solutions, numerical data, and feedback format.
-
The evaluation includes multiple LLMs, feedback conditions, and a mechanical baseline.
-
The authors distinguish between partial recovery of correct components and exact recovery of an acceptable solution.
Critical weaknesses
- Limited scientific significance of the artificially defined acceptable solutions
My primary concern is the scientific meaning of the acceptable solution sets used in the main experiments.
The authors artificially designate 32 five-term combinations as acceptable solutions, although other combinations can fit the same numerical data equally well. The paper does not provide a scientifically motivated criterion for distinguishing the designated acceptable solutions from other numerically equivalent solutions.
Consequently, the task primarily evaluates whether LLMs can recover elements of a researcher-defined subset rather than whether they can discover scientifically meaningful equations.
This distinction substantially limits the significance of the findings, particularly given the paper's motivation in scientific discovery and symbolic regression.
- The main finding may largely follow from the experimental construction
The component-level feedback is generated directly from the artificially defined acceptable solution sets. When these solutions share more components, the feedback provides information about components that are more frequently useful across acceptable answers.
Therefore, the observation that feedback becomes more effective as solution overlap increases seems largely predictable from the construction itself.
The similar trend observed with the mechanical reader further suggests that the result may primarily reflect the information structure of the task and feedback, rather than revealing a new property of LLM reasoning.
Although the experiments quantify this relationship, it remains unclear what fundamentally new insight is obtained beyond this intuitive expectation.
- Unclear practical implications and generalizability
The paper does not sufficiently demonstrate how the findings would improve feedback mechanisms in realistic scientific discovery or other practical LLM applications.
In real-world symbolic regression, acceptable equations are generally determined by criteria such as predictive accuracy, interpretability, physical consistency, or domain constraints, rather than membership in an arbitrarily selected subset of numerically equivalent expressions.
While the additional SAT experiments provide some evidence beyond the synthetic setting, they do not fully establish the relevance of the main findings to realistic scientific discovery tasks.
Questions for authors
-
What scientifically meaningful criterion distinguishes the 32 designated acceptable solutions from other solutions that fit the numerical data equally well?
-
Given that the feedback is constructed directly from the acceptable solution sets, to what extent is the observed relationship between solution overlap and feedback effectiveness an expected consequence of the experimental design?
-
What new understanding of LLM reasoning or feedback utilization does this study provide beyond what can be explained by a simple mechanical reader?
-
How could these findings inform the design of practical feedback systems when the complete set of acceptable solutions is unknown or cannot be enumerated?
2 comments
Follow-up to Reviewer KY9d: Overlap on Published Benchmarks
Thank you again for your review. We add new evidence on your main concern: the scientific meaning of the acceptable sets (W1, Q1) and the relevance to real discovery tasks (W3). It is in an updated revision (Section 7 and Appendix K.9).
Overlap on published benchmarks. We measured the overlap index on published benchmark problems. There, the benchmark itself defines which answers are acceptable. No model is involved.
| Benchmark | Acceptable answers | Problems measured | Overlap L |
|---|---|---|---|
| AI-Feynman equations | Supports that fit held-out data | 15 of 38 | 1.00 for all |
| Nguyen equations | Supports that fit held-out data | 2 of 10 | 0.05 and 1.00 |
| Published dynamical systems (dysts) | Supports that fit held-out data | 102 of 250 | 1.00 for all |
| Reference causal networks (bnlearn) | Graphs in the Markov equivalence class | 23 of 24 | 0.59 to 1.00; 20 are at 0.9 or above |
| SAT formulas (SATLIB uf20-91) | Satisfying assignments | 1,000 of 1,000 | 0.18 to 1.00 |
This gives three answers to your questions.
- Scientific criteria give high overlap. On the measured AI-Feynman and dynamical-system problems, exactly one support is acceptable (L = 1). Our controlled study includes this level. There the full report gains +0.507 over no report, the largest gain in the study (Table 11).
- Why the sets are authored. Published scientific benchmarks do not vary in overlap. A study of how overlap changes the value of feedback therefore needs sets with controlled overlap. SAT formulas are the one external family we found with a wide range, and the relation holds there (Section 6).
- What the levels mean. Fit alone gives L = 0.03. A parsimony criterion gives 0.71. One true answer gives 1. Our levels (0.00, 0.38, 0.78, and 1.00) cover this range.
Your Q2, stated precisely. We added a test that separates overlap from what the report says. We regress the models' gap on a slot-filling reader's gap and on L. The coefficient on L is +0.010 [−0.122, +0.138], and the coefficient on the reader's gap is +0.88 [+0.66, +1.11] (Section 5, Appendix K.4). So the effect of overlap passes through what the report says, and the models realise most of it.
A shared concern. Your concern about evidence beyond the synthetic task (W3) is close to Reviewer xXyo's second weakness. Our reply to Reviewer xXyo gives the per-model results of the SAT study (Table 6). Reviewer xXyo has since replied and raised their rating, so that thread may also be useful.
We hope this answers your main concern. If any point remains open, we will be glad to answer it before the discussion closes. If the revision resolves your concerns, we would be grateful if you would consider updating your rating.
Thank you for your time and consideration.
Response to Reviewer KY9d
Thank you for the review. We answer your four questions in order. We also revised the presentation.
Q1. What scientifically meaningful criterion distinguishes the 32 acceptable solutions?
The 32 solutions are authored. This keeps the number of answers fixed while the overlap changes. Section 7 now says this.
Simple criteria of the kind used in equation discovery give the same range of overlap. We checked this on the same catalogues, with the same number of answers (Appendix K.7).
So a verifier that checks only the fit gives a component report almost nothing to say. A parsimony criterion gives it a lot to say. Our authored levels (0.00, 0.38, 0.78) test this range under controlled conditions.
Q2. To what extent is the result an expected consequence of the design?
The direction follows from the design. The abstract now says so. The experiment measures the size of the effect. It also shows that a simple rule accounts for the size. A mechanical reader that only fills each reported slot has gaps of −0.018, +0.157 and +0.283. The models' gaps are −0.056, +0.166 and +0.271. The two agree across levels and across families.
Q3. What new understanding does the study give beyond a simple mechanical reader?
It shows which simple rule describes the models, and where a component report stops helping.
Q4. How can this inform practical systems when the acceptable set cannot be enumerated?
Overlap can be estimated from a few known answers (Section 6.3). On 1,000 SAT formulas, we computed the index from five sampled solutions. It correlates 0.90 with the exact overlap, and it gives the same slope (+0.215 against +0.213). When we rank held-out formulas by this estimate, the top half gains +0.221 from the report and the bottom half gains +0.119.
Section 7 gives three design points:
W3. Relevance beyond the synthetic task.
The external evidence uses SAT formulas from SATLIB. It now starts with a study of feedback on models' own answers (Section 6.1). The slope is +0.620 [+0.434, +0.806] against a sham report, and +0.609 [+0.417, +0.815] against no report.
Presentation.
Thank you again. We hope the revised paper is easier to follow.