acceptodds
Desk rejected at ICLR 2027

Falsify, Repair, Repeat: Auditable Local Repair of Typed Dynamical Equations

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Equation discovery can fit observed trajectories while retaining the wrong term, regime, or delay. We introduce Falsify–Repair–Repeat (FRR), a bounded scientific agent for auditing and locally repairing executable dynamical laws. Statistical verifiers turn failed tests into typed witnesses, a constrained policy selects grammar-valid edits, and numerical tools retain authority over refitting, acceptance, replay, and abstention. The selector is replaceable, but every structural change must survive executable validation and accumulated tests. This separation lets heterogeneous scientific agents share comparable evidence and decision records without exposing unrestricted control over the model. We characterize the finite repair basin, conditional recovery, replay preservation, budgeted termination, and an optional error-spending construction under conditionally valid tests. Across one-edit, compound-error, held-out, and null studies, typed editing supplies the principal behavioral and structural gains, while replay and abstention expose preservation–performance and risk–coverage trade-offs. Deterministic FRR reaches 85.6% semantic success on the one-edit benchmark; paired intervals support behavioral improvement through AST distance three, and independent ordinary-ODE rollouts improve from unseen initial conditions. Distant SINDy candidates instead trigger almost no action, revealing that basin entry remains a discovery problem. The proposal-agnostic interface therefore turns local equation repair, within an explicitly bounded typed neighborhood, into a traceable scientific decision process.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific agents, Equation discovery, Symbolic regression, Dynamical systems, Model repair, Neuro-symbolic reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5Pn7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Falsify–Repair–Repeat (FRR), a framework for repairing discovered dynamical equations. FRR follows a constrained repair mechanism: a statistical verifier first identifies localized model failures and represents them as specific types of witnesses. It then generates a set of syntactically valid repair candidates, each restricted to a single modification. Subsequently, either a deterministic algorithm or an LLM-based selector chooses the most suitable candidate, while numerical tools handle parameter refitting, verification, regression testing against previous experiments, and rejection or rollback of unsuccessful repairs.

Critical strengths
  • Propose Falsify–Repair–Repeat (FRR), a bounded repair framework designed to systematically audit and locally repair executable dynamical laws.
  • Demonstrate that typed structural editing is the primary driver of improvements in behavioral accuracy and structural correctness, while replay validation and abstention mechanisms help preserve previously validated behavior and characterize risk–coverage trade-offs.
  • Achieve 85.6% semantic success on the single-edit benchmark. Paired confidence intervals support behavioral improvements at abstract syntax tree (AST) distances of up to three, while independent ordinary differential equation (ODE) rollouts demonstrate improved generalization to unseen initial conditions.
Critical weaknesses
  • Some equations and notation are not defined clearly enough. I suggest adding a dedicated Notation section or table to systematically define the key symbols and equations used throughout the paper.
  • The main text should explicitly refer readers to the corresponding sections of the supplementary material when presenting Propositions 1 and 2, since their full proofs are provided there.
  • The paper does not clearly specify the exact language model used in the LLM-based experiments. This omission makes the Agent-FRR results difficult to reproduce and should be addressed by reporting the model name/version and relevant inference settings.
  • The experimental section is somewhat difficult to follow because of the large number of variants, protocols, datasets, and evaluation settings introduced in a relatively compressed manner. The current presentation reads more like a detailed technical report than a focused conference paper; reorganizing and simplifying the main experimental narrative would substantially improve clarity.
Questions for authors
  • Equation formatting. Unnumbered equations that are not referenced in the text should use \begin{align*}...\end{align*} rather than numbered environments.
  • What does it mean for dASTd_\text{AST}
10 comments
AuthorsOpenReview ↗

Response to Reviewer 5Pn7 — Part 1: W1 / W2

Thank you for pointing out where the notation, proof references, and experimental organization made FRR difficult to evaluate. For clarity, we distinguish three steps: the selector proposes an edit, the numerical controller decides whether to accept it, and the sealed audit evaluates the returned equation. Below we give the notation, the model settings, and the comparisons supporting the main claims. We address the four weaknesses first, followed by the equation-formatting and AST-distance questions.

Related weaknesses and questions are summarized below.

Changes made in this revision are highlighted in blue in the main text and appendix. All page and line references below refer to the revised combined manuscript (main text and appendix).

W1: Some equations and notation are not defined clearly enough. I suggest adding a dedicated Notation section or table to systematically define the key symbols and equations used throughout the paper.

A1: Appendix B now brings the key symbols together in a dedicated notation table. These definitions distinguish the equation structure, repair and audit data, typed witness, admissible edits, and replay set, so the equations can be read alongside the actual repair procedure.

(a) Dedicated notation table. The opening of Appendix B defines the key structures, data splits, witnesses, admissible actions, replay set, and structural distance. We reproduce the central symbols below.

R3-Table1: Key notation and definitions.

SymbolMeaning
h0,hk,h⋆h_0,h_k,h^\starInitial, current, and target equation structures
Dd,DaD_{\mathrm d},D_{\mathrm a}Repair data and sealed audit data
rt(h),A(r)r_t(h),A(r)Witness from a failed check and its admissible edits
PkP_kPreviously passed checks eligible for replay

(b) Operational meaning. For example, a failed lag check restricts the admissible edits to lag changes. The selector proposes one, and the controller refits the candidate, repeats the failed check, and rechecks earlier passes before acceptance. The final audit also counts regressions introduced by any subsequent coefficient refit.

Manuscript locations.

  • Appendix B — p. 18, lines 920–927: notation table.
  • Main-text Section 3.2 — p. 4, lines 170–198: grammar actions, acceptance and replay.

W2: The main text should explicitly refer readers to the corresponding sections of the supplementary material when presenting Propositions 1 and 2, since their full proofs are provided there.

A2: Section 3.3 now explicitly links Proposition 1 to Appendix B.3 and Proposition 2 to Appendix B.4. We also distinguish the two statements below: one concerns selection along a corrective path, and the other concerns run-level false-repair control under its stated assumptions.

(a) Direct main-text proof pointers. Section 3.3 points readers from Proposition 1 to Appendix B.3 and from Proposition 2 to Appendix B.4.

(b) The two statements address distinct questions. Proposition 1 explains when the validation ranking selects the correct enumerated successor. If each estimated score differs from its population value by at most ϵk\epsilon_k, the true gap Δk\Delta_k can shrink by at most 2ϵk2\epsilon_k. Thus Δk>2ϵk\Delta_k>2\epsilon_k preserves that ranking. Recovery along the path requires the correct edits to be enumerated, accepted, and reachable within the run budget. Proposition 2 gives a formal run-level false-repair construction under conditional test validity and error spending. Our experiments evaluate a frozen policy calibrated at marginal thresholds and measure its operating point on held-out data.

Manuscript locations.

  • Main-text Section 3.3 — p. 5, line 225 and lines 239–240: proof pointers.
  • Appendix B.3 — p. 20, lines 1028–1044: conditional recoverability proof.
  • Appendix B.4 — p. 21, lines 1082–1095: conditional run-level error spending.
AuthorsOpenReview ↗

Response to Reviewer 5Pn7 — Part 2: W3

W3: Which language model, version, and inference settings were used for Agent-FRR?

A3: Agent-FRR used DeepSeek V4 Flash with a fixed inference configuration and access only to the current equation, typed witness, and admissible actions. We report those settings below, together with a 495-task paired comparison that isolates replacement of the selector while keeping numerical refitting and acceptance fixed.

(a) Model and inference settings.

Agent-FRR used DeepSeek V4 Flash (deepseek-v4-flash) at temperature 0, with at most one model call per decision, a 24,000-character input limit, and a 2,400-token output limit. The model received the current equation, a typed witness, and a list of admissible edits; it returned one action ID as a JSON object. It did not receive the sealed audit data. The separate free-rewrite control used a 4,096-token output limit. Deterministic FRR uses no language model.

(b) Matched selector comparison.

On 495 matched tasks, both selectors used the same numerical refitting and acceptance rules.

R3-Table2: Outcomes after replacing the selector on the same 495 tasks.

SelectorTasksSemantic (%)Exact (%)
Agent-FRR49574.731.3
Deterministic FRR49575.632.1

Their decisions agreed as follows (Supplementary Table S20):

R3-Table3: Selector agreement on 495 matched tasks.

Quantity comparedTasks on which the selectors agreed
Full action sequence479/495 (96.8%)
Decision to edit or abstain489/495 (98.8%)

Deterministic FRR implements the repair mechanism without language-model calls. Across 495 paired tasks, the selectors agreed on 479 complete action sequences (96.8%) and 489 edit-or-abstain decisions (98.8%). Deterministic FRR achieved 75.6% semantic success and 32.1% exact recovery, compared with 74.7% and 31.3% for Agent-FRR. These matched results support using the deterministic selector for the evaluated tasks while retaining the same numerical refitting and acceptance procedure.

Manuscript updates and evidence locations.

  • Appendix C — p. 22, lines 1136–1141: model identifier, temperature, one-call rule, input/output limits, JSON action format, and audit-data exclusion.
  • Appendix C — p. 22, lines 1142–1145: free-rewrite output limit.
  • Main-text Section 4, Methods — p. 5, lines 257–262: pointer to the model and inference settings.
  • Appendix J.2 — p. 38, lines 2000–2006: the 495-task pairing and agreement rates.
  • Table S20 — p. 38, lines 2029–2034: stratified selector results.
AuthorsOpenReview ↗

Response to Reviewer 5Pn7 — Part 3: W4 (a–b)

W4: How do the experiments support the main claims, and how has their presentation been clarified?

A4: We organize the evidence below around four questions: which component drives the gain, how far repair extends, what replay preserves, and whether the repaired equation predicts independent trajectories. This separates the scientific questions from the list of method variants. Main-text Section 4 introduces tasks before methods and endpoints. Sections 5.2–5.4 address recovery radius, transfer, and component attribution; Appendix E.1 supplies the independent rollout statistics.

(a) Component attribution.

The factorial comparison tests edit-space effects; the distance-stratified comparison measures how far the behavioral gains extend. The effects are reported in main-text Table 4 (p. 8, lines 412–425), and the distance-stratified paired gains in S10 (p. 28, lines 1474–1485).

R3-Table4: Typed-edit effects in the one-edit factorial experiment.

EndpointTyped-edit effect (pp)95% CI (pp)Holm-adjusted p
Semantic success+12.5[5.8, 20.2]0.006
Exact recovery+25.6[15.0, 36.9]<0.001

Typed editing improves both endpoints across the tested feedback, selector, and replay settings, with the larger effect on exact recovery. This attributes the gain to the edit space while accounting for the other factorial components.

(b) Repair radius.

R3-Table5: FRR recovery by initial AST distance; 192 paired tasks at each distance.

Initial AST distanceInitial semantic (%)FRR semantic (%)Paired gain (pp)95% CI (pp)FRR exact (%)
256.363.0+6.77[3.65, 10.42]4.2
351.058.9+7.81[4.17, 11.98]0.0

Behavioral improvement extends to tasks with initial AST distance three: both paired gain intervals lie above zero. The accompanying exact-recovery results distinguish behavioral improvement from reconstruction of the generating structure. Gains use unrounded outcomes from the sealed audit, withheld from edit selection.

Reviewer_FZk4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a common failure mode in equation discovery: a discovered model may fit trajectories while containing incorrect terms, delays, thresholds, or regime structure. Existing discovery techniques typically generate equations but rarely provide mechanisms for identifying which structural elements are unsupported by evidence.

FRR addresses this by introducing a constrained repair loop: Statistical verifiers detect model deficiencies. Failures are converted into typed witnesses that identify locations and admissible edits. A deterministic or learned selector chooses among available repairs. Numerical refitting validates candidates. Replay verifies previously passed tests. Accepted repairs are logged; otherwise the system abstains.

The framework is intentionally proposal-agnostic: symbolic regression systems, sparse discovery methods, neural-symbolic models, and LLM-based selectors can all be plugged into the same repair interface. Formally, the paper defines repair basins, replay conditions, recovery guarantees under finite-margin assumptions, and an optional error-spending framework.

Experimentally, the authors evaluate: 180 one-edit benchmark tasks 768 compound-error tasks 7 held-out ODEBench systems OOD shifts discovery-generated candidates selective deployment experiments physiological-law transfer studies.

Key empirical findings include: FRR reaches 85.6% semantic success on one-edit tasks. Typed editing provides the largest contribution among tested components. Behavioural gains persist to AST distance 3, although exact recovery sharply declines. Replay improves preservation and regression control but may reduce semantic success. LLM selectors contribute relatively little beyond deterministic selection.

Critical strengths

Clear Novelty Most symbolic-regression literature focuses on equation discovery. This work instead addresses equation repair after discovery, which is a distinct and underexplored problem. The framing as "local constrained repair" is genuinely new.

Strong Experimental Attribution A major strength is the full 2⁴ factorial design that isolates: selector feedback repair space replay rather than relying on ablations that change multiple factors simultaneously.

The factorial results show that typed editing yields: +12.5 percentage points semantic success +25.6 percentage points exact recovery while selector effects are considerably smaller.

Good Separation of Concerns The system design cleanly separates: diagnosis, candidate generation, selection, validation, replay. This improves interpretability and auditability relative to many agentic approaches.

Careful Treatment of LLMs A particularly impressive aspect is that the paper does not overstate the value of language models. The results show: 96.8% action-sequence agreement between Agent-FRR and deterministic FRR, only marginal performance differences, evidence that most gains arise from typed diagnostics rather than the LLM itself. This is a refreshing and credible finding.

Strong Reproducibility Effort The supplement provides: precise budgets, verifier thresholds, complete treatment definitions, statistical procedures, protocol descriptions, audit records, proof sketches. This is stronger than many submissions in symbolic regression and scientific ML.

Thoughtful Safety/Audit Perspective The concept of "auditable repair" is operationally useful. The paper carefully defines auditable as traceability rather than correctness, avoiding exaggerated claims.

Broad Evaluation Suite The evaluation spans: controlled repairs, compound errors, external ODEBench systems, discovery outputs, OOD tests, selective deployment, physiology transfer.

This breadth increases confidence that the observed effects are not benchmark-specific.

Critical weaknesses

Major Weaknesses

Strong Dependence on Basin-Entry Assumptions The paper explicitly assumes that the target model is reachable through a short sequence of grammar-supported edits. This assumption is central to the method's success. The natural-candidate analysis shows median AST distances around 15 for discovery outputs, substantially outside the controlled benchmark regime.

This raises an important question: How often will real equation-discovery pipelines produce candidates inside FRR's repair basin? The paper acknowledges this concern but does not fully quantify it.

External Performance Gains Are Modest Held-out ODEBench results are encouraging but statistically inconclusive after correction. The paper itself reports all three prespecified external contrasts as inconclusive.

Although transparency is appreciated, this limits confidence that the strongest benchmark gains will consistently transfer beyond controlled settings.

Theoretical Guarantees Rely on Strong Conditions Proposition 1 requires: witness localisation enumeration of the correct edit positive validation margin bounded estimation error successful replay acceptance at every repair step.

These conditions are understandable but quite restrictive.

Similarly, Proposition 2 relies on conditional super-uniform p-values, yet the paper explicitly states that the evaluated thresholds do not instantiate this guarantee.

Therefore the theoretical guarantees are informative but somewhat disconnected from the deployed system. Limited Analysis of Grammar Design Repair power depends critically on: witness design, repair operators, grammar coverage.

The paper demonstrates effectiveness with its chosen grammar but provides little understanding of robustness to grammar choice or grammar incompleteness.

Given that grammar coverage is a key determinant of recoverability, this deserves deeper treatment.

False-Repair Behaviour Remains Nontrivial The supplement reports: 15% correct-law false repairs for FRR, 9% for a phase-randomized null control, under marginal thresholding.

The paper appropriately disclaims safety guarantees, but these rates are large enough that stronger calibration evidence would improve confidence for scientific deployment.

Questions for authors

How often are real discovery outputs actually inside the FRR repair basin? How sensitive are results to repair grammar coverage and design? Why should practitioners trust FRR given the observed correct-model false-repair rates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer FZk4 — Part 1: W1 / Q1

Thank you for recognizing local repair as a distinct problem and for asking about its practical use. Below we address basin entry and grammar coverage, then report the model-preservation and held-out calibration evidence. We respond to the weaknesses and questions in the order of your review.

Related weaknesses and questions are summarized below.

Changes made in this revision are highlighted in blue in the main text and appendix. All page and line references below refer to the revised combined manuscript (main text and appendix).

W1 / Q1: How often do discovery pipelines produce candidates inside FRR's repair basin?

A1: We examine practical use through two settings: candidates produced by discovery pipelines and existing models evaluated after a data update. The first characterizes the starting candidates and observed intervention; the second tests whether FRR preserves correct structure when coefficient adaptation already explains the new observations.

(a) Discovery-generated candidates and basin entry.

The original 112 candidates have median initial AST distance 15. The separate 224-candidate extension accepts no structural edits. These cohorts characterize candidate quality and selective intervention in the evaluated discovery pipelines.

(b) Model preservation after a data update.

The model-update study tests preservation after new observations: coefficient adaptation is performed before deciding whether structural intervention is needed.

R2-Table1: Structural intervention and preservation on 80 model-update tasks.

MethodTasksStructurally editedFinal exact models
Local search8080/800/80
Residual-guided8080/800/80
FRR800/8010/80

Coefficient refitting alone met the semantic criterion on all 80 candidates. FRR retained all ten initially exact models, whereas both search controls edited every candidate and returned no exact models. This is the value of selective structural intervention in the model-update setting: once refitting explains the new data, a structural change can be unnecessary. The supplement reports this experiment separately from the natural-candidate stress test; the 224-candidate extension is reported in Supplementary Table S8.

Evidence locations.

  • Appendix D.3 — p. 26, lines 1352–1367: the 112-candidate study.
  • Appendix D.3 and Table S8 — p. 26, lines 1369–1382: the separate 224-candidate extension.
  • Appendix M.4 — p. 46, lines 2432–2444: model-update design; lines 2456–2460: structural-preservation result.
  • Table S32 — p. 46, lines 2461–2467: model-update outcomes.
AuthorsOpenReview ↗

Response to Reviewer FZk4 — Part 2: W2 & W3

W2: How strong is the held-out transfer evidence after statistical correction?

A2: The held-out study evaluates FRR on seven unseen ODEBench systems, reporting behavioral success and exact recovery alongside prespecified paired tests. Together, these results distinguish the observed transfer performance from the statistical evidence for differences between methods.

(a) Held-out evaluation.

The seven unseen ODEBench systems test transfer beyond the controlled task families.

R2-Table2: Held-out ODEBench results.

MethodSemantic (%)Exact (%)
FRR56.526.8
CV-FRR56.025.0

(b) Interpretation of the endpoints.

FRR achieved 26.8% exact recovery and 56.5% semantic success on unseen-system tasks. CV-FRR obtained similar observed rates; all three prespecified contrasts were inconclusive after Holm adjustment (main-paper Table 3 and Supplementary Table S18).

The recovery-radius study distinguishes two useful outcomes: behavioral gains persist through initial AST distance three, while exact reconstruction is concentrated near the generating structure. This separates improved equation behavior from recovery of the generating law.

Evidence locations.

  • Main-text Table 3 — p. 8, lines 398–410: held-out endpoints and prespecified contrasts.
  • Appendix H and Table S18 — p. 35, lines 1838–1850: inference procedure and confirmatory results.
  • Appendix E.1 and Table S10 — p. 28, lines 1474–1485: distance-stratified paired semantic gains.

W3: How do the recovery and false-repair guarantees relate to the implemented method?

A3: The two propositions address different parts of the repair procedure: selecting the correct available successor and controlling false repair across adaptive decisions. We connect each statement to its assumptions and report the implemented policy's measured false-repair rate separately.

(a) Recovery conditions. Proposition 1 states the conditions under which the enumerated correct successor is selected: if each estimated score is within ϵk\epsilon_k of its population value, a true margin Δk>2ϵk\Delta_k>2\epsilon_k preserves the ranking. Recovery along a path also requires the corrective action to be available, accepted after refitting and replay, and reachable within the run budget. These conditions identify the roles of diagnosis, grammar coverage, selection, and numerical acceptance.

(b) Run-level error control. Proposition 2 specifies the conditions for run-level false-repair control: conditionally valid verifier tests and an error-spending budget. Our experiments evaluate a frozen, marginally calibrated policy on held-out data; these marginal thresholds alone do not establish conditional validity. The held-out evaluation measures its practical operating point: 6 false repairs among 300 correct laws (2.00%; 95% Wilson CI [0.92%, 4.29%]). Part 4 (W5 / Q3) gives the calibration protocol and the separate compound-error intervention results.

Evidence locations.

  • Main-text Section 3.3 — p. 5, lines 216–241: propositions and proof pointers.
  • Appendix B.3 — p. 20, lines 1028–1044: conditional recoverability proof.
  • Appendix B.4 — p. 21, lines 1082–1095: conditional run-level error spending.
  • Appendix F — p. 33, lines 1730–1748: empirical selective-policy evaluation.
AuthorsOpenReview ↗

Response to Reviewer FZk4 — Part 3: W4 / Q2 & W5 / Q3

W4 / Q2: How sensitive is FRR to grammar coverage, missing operators, and grammar design?

A4: Grammar coverage determines which corrective actions FRR can propose, while typed diagnosis restricts attention to the relevant edit class. The factorial experiment tests the value of that restriction within the specified grammar; the recovery statement makes the required availability of corrective actions explicit.

(a) Grammar coverage.

The grammar determines which repairs FRR can express. For example, suppose the missing term is a x(t−τ)a\,x(t-\tau), but the edit grammar permits only coefficient changes and additions or deletions of undelayed terms. The correct successor cannot be proposed because the required delayed term is unavailable. A typed witness identifies the relevant edit class; grammar coverage determines whether that class contains the corrective action, and numerical verification determines whether the proposed change is accepted.

(b) Controlled edit-space evidence. The factorial study measures the benefit of typed editing under our chosen grammar: semantic success increases by 12.5 percentage points and exact recovery by 25.6 points. The factorial comparison establishes the contribution of typed editing within the specified grammar.

Evidence locations.

  • Main-text Section 3.2 — p. 4, lines 170–177: grammar-action neighborhood and repair basin.
  • Main-text Table 4 — p. 8, lines 412–425: factorial component estimates.
  • Appendix B.3 — p. 20, lines 1028–1044: availability and acceptance conditions along a repair path.

W5 / Q3: What supports practical use given the observed false-repair rates?

A5: Practical intervention is evaluated with a frozen selective policy on separate correct-law and compound-error test sets. This directly measures unnecessary edits and intervention on erroneous candidates, while replay checks preservation of previously passed tests.

(a) Held-out false-repair measurement.

The held-out calibration test directly measures unnecessary intervention after the selection threshold has been fixed. At the 5% calibration budget, FRR made six false repairs on 300 held-out correct laws.

R2-Table3: FRR false repair on held-out correct laws at the 5% calibration budget.

Test lawsFalse repairsFalse repair (%)95% Wilson CI (%)
30062.00[0.92, 4.29]

On the separate compound-error test set, the same selective policy acted on 188/768 tasks (24.5%). The two cohorts measure unnecessary intervention and intervention on erroneous candidates, respectively (Supplementary Table S16).

(b) Complementary roles of calibration and replay.

Calibration determines when FRR intervenes; replay checks whether an accepted edit preserves previously satisfied tests. The marginal-threshold controls—15/100 correct-law edits and 9/100 phase-null edits—evaluate a different policy setting and separate cohorts.

The held-out rates describe the frozen selective policy on the evaluated distribution.

Evidence locations.

  • Appendix F — p. 33, lines 1730–1740: selective score and frozen-threshold evaluation.
  • Appendix F — p. 33, lines 1737–1740: the separate counts of 6/300 clean-law edits and 188/768 compound interventions.
  • Table S16 — p. 33, lines 1742–1748: selective outcomes.

Thank you for the constructive questions about practical use. The model-update and held-out calibration studies show how FRR can preserve an existing correct structure and intervene selectively when new evidence warrants an edit. We hope this evidence addresses your concerns about when FRR is useful, and we welcome any further feedback.

Dear Reviewer FZk4,

Thank you for recognizing the value of local structural repair and the strengths of our experimental design. We sincerely appreciate your thoughtful comments, which have helped us improve the paper.

As the discussion period draws to a close, may we ask whether our responses have addressed your concerns, and whether you see any remaining questions or opportunities to strengthen the paper? We would appreciate the opportunity to address any outstanding points as fully as possible during the remaining discussion time.

Thank you again for your time and constructive feedback.

Reviewer_oPdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Falsify–Repair–Repeat (FRR), a framework for auditable local structural repair of symbolic dynamical equations. The paper addresses a clearly defined and valuable problem. The separation of typed witnesses, grammar-valid edits, numerical verification, and historical replay provides good modularity and interpretability.

Critical strengths

The paper addresses a meaningful problem and proposes a well-structured methodological framework. In particular, the clear separation between diagnosis, constrained editing, numerical validation, and replay makes the repair process interpretable and auditable.

Critical weaknesses

1. Novelty relative to existing approaches.

Although the framework is well organized, its methodological differences from existing counterexample-guided repair and residual-driven equation optimization methods are not sufficiently clear. These existing approaches appear to be potentially applicable to the problem through relatively straightforward adaptation. The paper does not adequately distinguish FRR's contributions from those of these related approaches.

2. Inconsistencies in the reported experimental results.

Several results in the supplementary material appear inconsistent or insufficiently explained.

  • Tables S26 and S27: The reported results involving Local search and Residual-guided search appear inconsistent across these tables. For example, Table S26 reports 95.9% semantic success for FRR on compound tasks, whereas the paired differences in Table S27 appear to correspond to substantially lower FRR performance.
  • Table S29: The results at the fit ceiling of 128 exhibit substantial changes, particularly in semantic success and exact recovery, despite the reported average number of fits remaining unchanged. The paper does not provide sufficient information to determine the cause of these changes.

3. Definition and accounting of verification costs.

In the compound-error experiments, FRR reports an average of 5.67 verifier calls. In the supplementary one-edit search experiments, FRR reports 4.81 Tests, whereas the compound search-budget experiments report 0.00 Tests. Table S29 further reports zero Tests for FRR across all fit budgets.

The relationship between Tests and verifier calls is not sufficiently explained. Without clarification of their definitions and accounting rules, it is difficult to determine why the compound experiments report zero Tests or whether the reported compound-task results reflect the complete verification and replay procedure.

Questions for authors

1. Methodological novelty.

What are the key methodological differences between FRR and existing counterexample-guided repair or residual-driven equation optimization approaches? Given that these approaches appear potentially applicable through relatively straightforward adaptation, what additional contributions does FRR provide?

2. Experimental inconsistencies in Tables S26, S27, and S29.

  • Can the authors reconcile the results in Tables S26 and S27, particularly the apparent discrepancy between the reported compound-task performance of FRR and its paired differences relative to Local search and Residual-guided search?
  • What explains the substantial changes in semantic success and exact recovery at the fit ceiling of 128 in Table S29, despite the unchanged average number of fits?

3. Verification-cost definitions and experimental completeness.

Are Tests and verifier calls defined and counted consistently across the one-edit and compound-error experiments? If they represent different quantities or follow different accounting rules, could the authors explicitly explain these differences?

In particular, why does FRR report zero Tests across all compound-task fit budgets in Table S29? Do these results reflect the complete verification and replay procedure?

5 comments
AuthorsOpenReview ↗

Response to Reviewer oPda — Part 1: W1 / Q1

Thank you for checking our results and methodology carefully. We address the three grouped concerns in separate replies. Changes in the revised main text and appendix are highlighted in blue; all page and line references below refer to the revised combined manuscript.

W1 / Q1: What distinguishes FRR from counterexample-guided repair and residual-guided equation optimization?

A1: FRR makes the link from a failed check to an admissible structural edit explicit. Within a counterexample-guided propose–check loop, a failed check produces a typed witness that restricts the available edits. For example, a lag witness permits a finite set of lag changes. After selection and coefficient refitting, acceptance requires resolving the triggering check and passing replay of previously satisfied checks. The selector proposes an action; the numerical controller enforces these acceptance conditions.

The methodological question is therefore what these edit restrictions and acceptance conditions contribute within the broader counterexample-guided framework. Our residual-guided control ranks candidates using residual evidence and accepts them by validation error. Two controlled comparisons isolate the relevant effects: typed editing improves exact recovery by 25.6 percentage points, and the full acceptance rule improves exact recovery by 2.60 points when fitted candidates and their order are held fixed.

(a) Component attribution.

The complete 242^4 factorial design evaluates all 16 combinations of feedback, edit space, selector, and replay. This estimates the typed-edit effect across balanced settings of the other components.

R1-Table1: Typed-edit effects in the controlled one-edit factorial experiment.

EndpointEffect (pp)95% CI (pp)Holm-adjusted p
Semantic success+12.5[5.8, 20.2]0.006
Exact recovery+25.6[15.0, 36.9]<0.001

Typed editing improves both endpoints, with its largest effect on exact structural recovery. To separate this edit-space effect from acceptance, the next experiment holds the candidates and their order fixed.

(b) Common-pool acceptance.

In a separate single-step experiment on 768 paired tasks, both rules receive the same fitted candidates in the same order. This removes candidate generation and ordering as explanations for an outcome difference. Only the acceptance rule changes.

R1-Table2: Exact recovery under the two acceptance rules on 768 paired tasks.

Acceptance ruleExact successesExact recovery (%)
Validation-only54/7687.03
FRR74/7689.64

The paired difference is +2.60 percentage points, with a 95% CI of [0.13, 5.08], obtained by bootstrap resampling over 24 system–seed blocks.

FRR returns 20 more exact structures from the same candidate pool. Holding candidate generation and order fixed isolates the contribution of the full acceptance rule, including replay. The two rules emphasize different endpoints: FRR recovers more generating structures, while validation-only accepts more behaviorally successful candidates. The structural-recovery result concerns this single-step comparison, separately from the end-to-end aggregate in Q2(a).

The evidence links FRR's contribution to two measured effects: typed editing increases exact recovery by 25.6 percentage points, and the acceptance rule recovers 20 additional exact structures from the same candidate pool. The common-pool protocol is detailed in Supplementary Table S30; Q3 separately reports numerical fits and verifier calls.

Evidence locations.

  • Main-text Table 4 — p. 8, lines 412–425: factorial effects.
  • Appendix M.3 — p. 45, lines 2404–2408: common-pool protocol; lines 2410–2412: outcome interpretation.
  • Table S30 — p. 45, lines 2414–2422: common-pool results.
AuthorsOpenReview ↗

Response to Reviewer oPda — Part 2: W2 / Q2

W2 / Q2: Are the S26/S27 endpoints and paired differences consistent? Why did S29 change at 128 fits despite unchanged mean fits?

A2: Thank you for checking these comparisons carefully. We verified the existing records for all 768 compound tasks and unified the endpoint, paired-effect, and budget tables. FRR's semantic-success result is 506/768 (65.89%). S26–S29 now refer to the same task-level outcomes.

(a) Endpoint results and paired differences: S26 and S27.

The structural-recovery and cost results show what FRR contributes in this comparison: it recovers 68 generating structures using 5.59 numerical fits per task.

R1-Table3: Generating-law recovery and numerical fitting cost on the same 768 compound tasks, with a common 192-fit ceiling.

MethodExact recoveriesExact recovery (%)Mean numerical fits
Local search0/7680.00174.98
Residual-guided30/7683.91156.45
FRR68/7688.855.59

FRR recovers 38 additional structures relative to Residual-guided search and 68 relative to Local search, while using over 96% fewer fits than either control. Verifier calls are accounted for separately in Q3.

R1-Table4: Paired exact-recovery effects from S27.

ComparatorExact-recovery gain (pp)95% CI (pp)
Local search+8.85[+5.47, +12.50]
Residual-guided+4.95[−0.52, +10.29]

Differences are FRR minus the comparator, in percentage points; 95% intervals use 10,000 bootstrap resamples of 24 system–seed blocks. The gain over Local search has a paired interval entirely above zero. The comparison with Residual-guided yields 38 additional recovered structures, with its paired uncertainty shown above.

The experiment distinguishes generating-law recovery from behavioral agreement. The semantic-success rates are 65.89% for FRR, 85.68% for Local search, and 89.97% for Residual-guided, giving the S27 semantic differences of −19.79 pp (95% CI [−24.22, −15.62]) and −24.09 pp ([−28.78, −19.53]). S26–S29 report both endpoints alongside computation: the structural results above identify FRR's contribution to recovering generating equations with a small fitting budget. S26 and S29 retain one-decimal percentages (65.9%); the same value is 65.89% at two decimals.

Manuscript update.

  • Appendix M.1 — p. 42, lines 2243–2258: compound success counts and their interpretation.
  • Table S26 — p. 43, lines 2268–2275: endpoint results.
  • Appendix M.2 — p. 44, lines 2324–2329: the 24-block bootstrap procedure; Table S27 — lines 2330–2345: paired effects.
  • Table S29 — p. 45, lines 2383–2402: budget outcomes.

(b) Fit-budget sensitivity: S29.

The matched 768-task records show identical outcomes and computation at the 64-, 128-, and 192-fit ceilings. We checked each task's semantic success, exact recovery, fit count, and verifier-call count across all three settings.

The 64-fit ceiling already allowed FRR to complete the same computations as at 128 and 192. All three ceilings yield 506 semantic successes and 68 exact recoveries. S29 now reports these identical outcomes together with their recorded costs, consistent with Q2(a).

R1-Table5: FRR computation at the three high-budget ceilings; task-level outcomes agree with S26.

Fit ceilingMatched tasksMean fitsMaximum fitsMean verifier calls
647685.5911575.6654
1287685.5911575.6654
1927685.5911575.6654

The fit ceiling is an upper limit, not a target number of fits. At the 64-fit ceiling, no task used more than 57 fits. Raising this limit to 128 or 192 therefore adds unused capacity: each task follows the same repair trace and stops at the same point. The maximum is 57 in all three settings, and both the outcomes and the per-task computation counts remain unchanged.

Manuscript update.

  • Appendix M.2 — p. 44, lines 2353–2358: matched outcomes, mean fit and verifier-call counts, and the maximum of 57 fits.
  • Appendix M.3 — p. 45, lines 2378–2382: explanation of the compound budget pattern.
  • Table S29 — p. 45, lines 2383–2402: the complete budget grid.
AuthorsOpenReview ↗

Response to Reviewer oPda — Part 3: W3 / Q3

W3 / Q3: Are “Tests” and verifier calls counted consistently? Why were zero Tests reported, and was the full verification/replay procedure executed?

A3: Verification and replay were executed in the compound FRR runs. We now report their recorded calls separately from numerical fits, using the same accounting rule throughout the comparison and budget tables.

(a) Consistent definitions and accounting.

A verifier call is an executable check during repair: diagnosing a failure, testing a refitted candidate, or replaying a previously passed check. Numerical refits are counted separately. Initialization and the final evaluation on the sealed audit split are excluded from the repair-time verifier-call count.

(b) Zero entries and execution of verification/replay.

The compound FRR runs executed verification and replay where applicable. Their recorded mean is 5.67 verifier calls, compared with 4.81 in the one-edit comparison. The revised “Verifier calls” column reports these recorded FRR counts consistently across the comparison and budget tables.

R1-Table6: Mean fitting and verification costs in the method-identification comparison.

SettingMethodTasksMean fitsMean verifier calls
One editLocal search18063.980.00
One editResidual-guided18063.020.00
One editFRR1808.244.81
CompoundLocal search768174.980.00
CompoundResidual-guided768156.450.00
CompoundFRR7685.595.67

All methods use the same task cohort and fit ceiling within each setting: 64 for one edit and 192 for compound tasks. The compound rows use the same records as Q2(a).

FRR uses approximately 87% fewer fits on the one-edit cohort and over 96% fewer on the compound cohort than either search control. These percentages quantify reductions in numerical fitting count, with repair-time verifier calls reported separately. In the same one-edit comparison (Supplementary Table S25), FRR achieves 39.4% exact recovery with 8.24 fits, versus 1.1% with 63.98 fits for Local search and 2.8% with 63.02 fits for Residual-guided search. Thus the fitting reduction accompanies higher exact recovery in that setting.

The search baselines fit and score candidates using validation error instead of invoking the FRR verifier during repair. They still undergo the final audit. Fits and verifier calls capture their different computational costs; both methods receive the independent final audit.

Manuscript update.

  • Appendix M.1 — p. 42, lines 2234–2242: repair-time cost accounting, including the exclusion of initialization and the final audit.
  • Appendix M.1 — p. 42, lines 2254–2258: fitting reductions.
  • Table S25 — p. 42, lines 2259–2265; Table S26 — p. 43, lines 2268–2275: mean fits and verifier calls.

Thank you again for the careful examination of our results. The controlled comparisons distinguish the contributions of typed editing and numerical acceptance, while the fit-budget experiments quantify their computational cost. We hope these results clarify FRR's contribution to local structural repair. We welcome further questions about the comparisons and cost accounting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of local structural repair as a valuable problem and of our modular design and controlled experiments. For clarity, we refer to Reviewers oPda, FZk4, and 5Pn7 as R1, R2, and R3, respectively.

We have responded to each comment individually. Following the expanded cross-model evaluation, R3 confirmed that our responses had resolved most of their concerns. The revised manuscript incorporates additional experiments, clearer methodological explanations, and consistent result and cost reporting. Below, we summarize the contributions and principal updates.


Core Contributions of Our Work

  1. Local repair as constrained statistical synthesis: FRR links typed witnesses to admissible edits through a proposal-agnostic interface, combining numerical acceptance, replay, and abstention to make structural changes traceable to evidence.

  2. Formal characterization: We connect repair-basin reachability, witness localization, grammar coverage, and validation margins to conditional recovery, and characterize replay preservation, budgeted termination, and false-repair control under conditionally valid tests and error spending.

  3. Controlled experimental attribution: A complete 242^4 factorial design isolates component effects: typed editing yields +12.5 semantic and +25.6 exact-recovery percentage points. Separately, fixed-candidate acceptance recovers 20 additional exact structures on 768 paired tasks.

  4. Transfer and practical evaluation: Seven held-out ODEBench systems, discovery outputs, OOD and null controls, selective deployment, and independent rollouts assess behavioral improvement, structural recovery, and preservation of existing models.


Experimental Updates during Rebuttal

  • Appendix M.1–M.3, Tables S25–S29: Added local-search and residual-guided comparisons, paired-effect estimates, and fit-budget analyses to evaluate structural recovery and numerical fitting efficiency (R1).

  • Appendix M.3, Table S30: Added the fixed-candidate comparison to isolate the acceptance-rule contribution (R1).

  • Appendix F and M.4, Tables S16 and S32: Added held-out calibration and model-update experiments to evaluate selective intervention and preservation of correct structures (R2).

  • Appendix E.1, Table S12: Added independent free-rollout results; all six evaluated system–horizon comparisons favor FRR in mean failure-aware loss (R3).

  • Appendix N, Tables S33–S35: Added six-model, four-family evaluation on 180 controlled and 200 held-out compound tasks, reporting both action interfaces and numerical costs (R3).


Discussion and Presentation Updates during Rebuttal

  • Appendix M.1–M.3, Tables S25–S29: Unified task-level results and clarified fit budgets, verifier calls, and replay accounting (R1).

  • Sections 3.2–3.3 and Appendix B.3–B.4: Clarified grammar coverage, recovery conditions, and the relationship between formal guarantees and empirical calibration (R2).

  • Sections 3–5 and Appendix B: Added notation and proof pointers, distinguished grammar actions from initial AST distance, and clarified experimental organization (R3).

These additions strengthen the evidence for typed local repair and make the method and evaluation easier to follow. The corresponding revisions are highlighted in blue in the main text and appendix.


We sincerely thank the Area Chair and all reviewers for their time, careful consideration, and contributions to the review process. The reviewers' constructive questions have helped us strengthen the paper. We hope the revised manuscript and detailed responses address the concerns raised, and we welcome any remaining feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.