- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Falsify–Repair–Repeat (FRR), a framework for repairing discovered dynamical equations. FRR follows a constrained repair mechanism: a statistical verifier first identifies localized model failures and represents them as specific types of witnesses. It then generates a set of syntactically valid repair candidates, each restricted to a single modification. Subsequently, either a deterministic algorithm or an LLM-based selector chooses the most suitable candidate, while numerical tools handle parameter refitting, verification, regression testing against previous experiments, and rejection or rollback of unsuccessful repairs.
Critical strengths
- Propose Falsify–Repair–Repeat (FRR), a bounded repair framework designed to systematically audit and locally repair executable dynamical laws.
- Demonstrate that typed structural editing is the primary driver of improvements in behavioral accuracy and structural correctness, while replay validation and abstention mechanisms help preserve previously validated behavior and characterize risk–coverage trade-offs.
- Achieve 85.6% semantic success on the single-edit benchmark. Paired confidence intervals support behavioral improvements at abstract syntax tree (AST) distances of up to three, while independent ordinary differential equation (ODE) rollouts demonstrate improved generalization to unseen initial conditions.
Critical weaknesses
- Some equations and notation are not defined clearly enough. I suggest adding a dedicated Notation section or table to systematically define the key symbols and equations used throughout the paper.
- The main text should explicitly refer readers to the corresponding sections of the supplementary material when presenting Propositions 1 and 2, since their full proofs are provided there.
- The paper does not clearly specify the exact language model used in the LLM-based experiments. This omission makes the Agent-FRR results difficult to reproduce and should be addressed by reporting the model name/version and relevant inference settings.
- The experimental section is somewhat difficult to follow because of the large number of variants, protocols, datasets, and evaluation settings introduced in a relatively compressed manner. The current presentation reads more like a detailed technical report than a focused conference paper; reorganizing and simplifying the main experimental narrative would substantially improve clarity.
Questions for authors
- Equation formatting. Unnumbered equations that are not referenced in the text should use
\begin{align*}...\end{align*}rather than numbered environments. - What does it mean for
10 comments
Response to Reviewer 5Pn7 — Part 2: W3
W3: Which language model, version, and inference settings were used for Agent-FRR?
A3: Agent-FRR used DeepSeek V4 Flash with a fixed inference configuration and access only to the current equation, typed witness, and admissible actions. We report those settings below, together with a 495-task paired comparison that isolates replacement of the selector while keeping numerical refitting and acceptance fixed.
(a) Model and inference settings.
Agent-FRR used DeepSeek V4 Flash (deepseek-v4-flash) at temperature 0, with at most one model call per decision, a 24,000-character input limit, and a 2,400-token output limit. The model received the current equation, a typed witness, and a list of admissible edits; it returned one action ID as a JSON object. It did not receive the sealed audit data. The separate free-rewrite control used a 4,096-token output limit. Deterministic FRR uses no language model.
(b) Matched selector comparison.
On 495 matched tasks, both selectors used the same numerical refitting and acceptance rules.
R3-Table2: Outcomes after replacing the selector on the same 495 tasks.
| Selector | Tasks | Semantic (%) | Exact (%) |
|---|---|---|---|
| Agent-FRR | 495 | 74.7 | 31.3 |
| Deterministic FRR | 495 | 75.6 | 32.1 |
Their decisions agreed as follows (Supplementary Table S20):
R3-Table3: Selector agreement on 495 matched tasks.
| Quantity compared | Tasks on which the selectors agreed |
|---|---|
| Full action sequence | 479/495 (96.8%) |
| Decision to edit or abstain | 489/495 (98.8%) |
Deterministic FRR implements the repair mechanism without language-model calls. Across 495 paired tasks, the selectors agreed on 479 complete action sequences (96.8%) and 489 edit-or-abstain decisions (98.8%). Deterministic FRR achieved 75.6% semantic success and 32.1% exact recovery, compared with 74.7% and 31.3% for Agent-FRR. These matched results support using the deterministic selector for the evaluated tasks while retaining the same numerical refitting and acceptance procedure.
Manuscript updates and evidence locations.
- Appendix C — p. 22, lines 1136–1141: model identifier, temperature, one-call rule, input/output limits, JSON action format, and audit-data exclusion.
- Appendix C — p. 22, lines 1142–1145: free-rewrite output limit.
- Main-text Section 4, Methods — p. 5, lines 257–262: pointer to the model and inference settings.
- Appendix J.2 — p. 38, lines 2000–2006: the 495-task pairing and agreement rates.
- Table S20 — p. 38, lines 2029–2034: stratified selector results.
Response to Reviewer 5Pn7 — Part 3: W4 (a–b)
W4: How do the experiments support the main claims, and how has their presentation been clarified?
A4: We organize the evidence below around four questions: which component drives the gain, how far repair extends, what replay preserves, and whether the repaired equation predicts independent trajectories. This separates the scientific questions from the list of method variants. Main-text Section 4 introduces tasks before methods and endpoints. Sections 5.2–5.4 address recovery radius, transfer, and component attribution; Appendix E.1 supplies the independent rollout statistics.
(a) Component attribution.
The factorial comparison tests edit-space effects; the distance-stratified comparison measures how far the behavioral gains extend. The effects are reported in main-text Table 4 (p. 8, lines 412–425), and the distance-stratified paired gains in S10 (p. 28, lines 1474–1485).
R3-Table4: Typed-edit effects in the one-edit factorial experiment.
| Endpoint | Typed-edit effect (pp) | 95% CI (pp) | Holm-adjusted p |
|---|---|---|---|
| Semantic success | +12.5 | [5.8, 20.2] | 0.006 |
| Exact recovery | +25.6 | [15.0, 36.9] | <0.001 |
Typed editing improves both endpoints across the tested feedback, selector, and replay settings, with the larger effect on exact recovery. This attributes the gain to the edit space while accounting for the other factorial components.
(b) Repair radius.
R3-Table5: FRR recovery by initial AST distance; 192 paired tasks at each distance.
| Initial AST distance | Initial semantic (%) | FRR semantic (%) | Paired gain (pp) | 95% CI (pp) | FRR exact (%) |
|---|---|---|---|---|---|
| 2 | 56.3 | 63.0 | +6.77 | [3.65, 10.42] | 4.2 |
| 3 | 51.0 | 58.9 | +7.81 | [4.17, 11.98] | 0.0 |
Behavioral improvement extends to tasks with initial AST distance three: both paired gain intervals lie above zero. The accompanying exact-recovery results distinguish behavioral improvement from reconstruction of the generating structure. Gains use unrounded outcomes from the sealed audit, withheld from edit selection.
Response to Reviewer 5Pn7 — Part 1: W1 / W2
Thank you for pointing out where the notation, proof references, and experimental organization made FRR difficult to evaluate. For clarity, we distinguish three steps: the selector proposes an edit, the numerical controller decides whether to accept it, and the sealed audit evaluates the returned equation. Below we give the notation, the model settings, and the comparisons supporting the main claims. We address the four weaknesses first, followed by the equation-formatting and AST-distance questions.
Related weaknesses and questions are summarized below.
Changes made in this revision are highlighted in blue in the main text and appendix. All page and line references below refer to the revised combined manuscript (main text and appendix).
W1: Some equations and notation are not defined clearly enough. I suggest adding a dedicated Notation section or table to systematically define the key symbols and equations used throughout the paper.
A1: Appendix B now brings the key symbols together in a dedicated notation table. These definitions distinguish the equation structure, repair and audit data, typed witness, admissible edits, and replay set, so the equations can be read alongside the actual repair procedure.
(a) Dedicated notation table. The opening of Appendix B defines the key structures, data splits, witnesses, admissible actions, replay set, and structural distance. We reproduce the central symbols below.
(b) Operational meaning. For example, a failed lag check restricts the admissible edits to lag changes. The selector proposes one, and the controller refits the candidate, repeats the failed check, and rechecks earlier passes before acceptance. The final audit also counts regressions introduced by any subsequent coefficient refit.
Manuscript locations.
W2: The main text should explicitly refer readers to the corresponding sections of the supplementary material when presenting Propositions 1 and 2, since their full proofs are provided there.
A2: Section 3.3 now explicitly links Proposition 1 to Appendix B.3 and Proposition 2 to Appendix B.4. We also distinguish the two statements below: one concerns selection along a corrective path, and the other concerns run-level false-repair control under its stated assumptions.
(a) Direct main-text proof pointers. Section 3.3 points readers from Proposition 1 to Appendix B.3 and from Proposition 2 to Appendix B.4.
(b) The two statements address distinct questions. Proposition 1 explains when the validation ranking selects the correct enumerated successor. If each estimated score differs from its population value by at most ϵk, the true gap Δk can shrink by at most 2ϵk. Thus Δk>2ϵk preserves that ranking. Recovery along the path requires the correct edits to be enumerated, accepted, and reachable within the run budget. Proposition 2 gives a formal run-level false-repair construction under conditional test validity and error spending. Our experiments evaluate a frozen policy calibrated at marginal thresholds and measure its operating point on held-out data.
Manuscript locations.