- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FASim, an LLM-agent pipeline that builds an executable agent-based simulator from a task brief and data, calibrates it, and iteratively refines its structure. Issues are routed to one of two paths. “Directly verifiable” implementation issues go straight to repair. “Structurally ambiguous” residuals go through a falsification loop: a Hypothesis Agent proposes 2–4 competing explanations, and a Falsification Agent designs one counterfactual probe with predicted signatures. The plan is frozen and executed by a deterministic ProbeExecutor, usually with matched seeds. Each hypothesis is labeled FALSIFIED, SURVIVES or INCONCLUSIVE, and a structural edit is allowed only if exactly one hypothesis survives and all others are falsified.
Critical strengths
- The framework looks sound and well motivated. The freeze-then-evaluate protocol (plan hash, evidence audit, controller-owned thresholds, execution failure never counted as falsification) is careful and have some potential of being reused by other agentic refinement systems.
- The empirical analysis is thorough. The injected-defect study (App. D.2) is the right kind of experiment for testing structural diagnosis, and Table 9 honestly shows cases where lower validation loss favors a structurally wrong simulator. The paper also reports unfavorable process statistics.
Thank you for the response. Some of my concerns have been addressed and I've raised my rating accordingly.
Critical weaknesses
W1. The central claim, i.e., “falsify before repair”, is not isolated by the evidence. The paper attributes its gains to the falsification gate, but the pipeline has many other components: blueprint generation, a direct-repair route that bypasses the gate, recalibration, and a retain-if-better rule. The experiments do not separate these contributions. In some settings, such as User modeling, FASim’s advantage is largest even though the process analysis shows the gate never admitted a diagnosis, so the gain there must come from somewhere else. The paper also never tests the gate itself: it reports that most hypotheses are “blocked”, but not whether blocking was the right call, i.e., whether the blocked repairs would actually have hurt. Compute-matched comparisons are also missing, so it is unclear whether the extra probing and LLM calls could achieve the same by simply trying more candidate repairs. This matters because the paper’s contribution is a design principle, not a leaderboard entry. If the benefit cannot be attributed to the principle, the main conclusion is unsupported.
W2. The notion of “trust” delivered by falsification is weaker than the framing suggests. Every step of the falsification loop is authored by the same LLM: the competing hypotheses, the probe, and the predicted signatures. The evidence comes from the simulator’s own behavior under intervention. A hypothesis that “survives” is therefore only consistent with the current program and with the explanations the LLM happened to propose. It is not tested against the real system, and the true cause may never be in the hypothesis set. The method is also restricted to defects in pathways that already exist in the simulator, and the authors acknowledge that surviving does not establish the real mechanism. The only setting with known ground truth is a small set of synthetic defects injected into one simulator, which is a favorable setting for this approach. So the paper shows that the procedure can recover known, designed faults, but not that it makes diagnoses on natural tasks more reliable. More broadly, the idea of testing hypotheses before acting on them is an application of existing ideas (automated falsification, metamorphic testing) to simulator refinement. The conceptual novelty is modest, which raises the bar for the empirical evidence in W1.
Questions for authors
- On User, where no gated repair was ever admitted, what produces MAE 0.10? Please give the exact test protocol and the fraction of retained edits from the direct vs gated route for every task and ablation variant.
3 comments
Response to W2.
W2. Reliability, Scope, and Conceptual Novelty
W2.1 — Can falsification support correct structural repairs beyond the original Mask defects?
Experiment E5 — Cross-Task Injected-Defect Recovery (Official Comment [E5]). We extended the original seven Mask Adoption defects to 15 injected defects across Mask Adoption, User Modeling, and Personal Mobility, achieving 15/15 verified program-level recoveries.
For eight new controlled cases, we compared three diagnosis strategies:
- Forced One-shot: Selects a diagnosis from competing hypotheses using static evidence without executing probes.
- Static-Abstention: May withhold diagnosis when static evidence cannot discriminate competing hypotheses.
- FASim: Uses frozen executable counterfactual probes to authorize structural repair.
Both Forced One-shot and FASim correctly identified 8/8 injected causes, while Static-Abstention returned UNRESOLVED in all eight cases.
Why is falsification still valuable? Forced One-shot selected the correct diagnoses without executable verification. FASim reached the same diagnoses through pre-specified counterfactual tests and observable execution evidence, providing auditable justification before repair. Unlike Static-Abstention, FASim could also authorize all eight repairs, with recovery verified against hidden program truth.
Therefore, E5 demonstrates evidence-supported actionability and verified structural recovery, rather than superior diagnosis accuracy over one-shot reasoning.
Full experimental details are provided in Official Comment [E5].
W2.2 — Can surviving hypotheses establish the true structural cause?
We agree that falsification is conditional on the proposed hypothesis set and cannot guarantee that the true cause is included.
A unique surviving hypothesis is supported relative to the tested alternatives and predefined observable signatures; survival does not establish real-world causal truth. When execution evidence is inconclusive or insufficiently discriminating, FASim withholds structural repair. However, an incomplete hypothesis set may still lead to an incorrect diagnosis.
Our intended contribution is therefore executable, auditable evidential support for structural-edit decisions, not guaranteed identification of real-world causal mechanisms.
The hidden-oracle evaluation in E5 verifies recovery of known program mechanisms rather than real-world causal identification. We will clarify this distinction in the paper.
W2.3 — What is the conceptual contribution beyond existing falsification techniques?
We acknowledge that falsification and metamorphic testing are established principles. FASim's contribution lies in their operational integration into autonomous simulator refinement:
- Separating directly verifiable implementation issues from ambiguous structural failures.
- Representing structural ambiguity through competing hypotheses.
- Freezing counterfactual predictions and decision criteria before execution.
- Using executable evidence and controller-owned rules to authorize persistent structural edits.
- Connecting supported diagnoses to targeted code repair and retention.
Unlike standalone testing, this protocol governs whether an LLM-generated structural explanation is permitted to modify an executable simulator.
Our additional experiments demonstrate measurable consequences of this integration: incremental fidelity gains (E1), prevention of harmful edits (E2), and evidence-supported structural recovery (E5).
Response to Questions
Questions for Authors: User Modeling and Retained-Edit Attribution
Q1 — What produces User Modeling MAE 0.10?
As established in E4, FASim autonomously constructed an LLM-assisted, review-conditioned rating mechanism using a precomputed cache. Its broader construction, direct-repair, calibration, and refinement pipeline explains the strong performance. E1 independently confirms that no gate-approved structural correction is needed to achieve MAE 0.10.
Q2 — What is the exact evaluation protocol?
In E1, FASim and Direct-Only share the same fixed blueprint, GPT-5.2 backbone, direct-repair mechanism, calibration, retention, and stopping procedures. Each independently executes five refinement runs, with performance reported as mean ± standard deviation. Holding the blueprint fixed isolates downstream structural-refinement effects.
Q3 — How many repairs are retained from each route, and what do they contribute?
Experiment E1 — Repair Process Audit (Official Comment [E1]). In addition to five-run performance evaluation, we recorded diagnosis, gate-approval, and repair-retention decisions from one complete FASim refinement trajectory per task:
| Task | Direct diagnoses | Gated approvals | Direct retained | Gated retained |
|---|---|---|---|---|
| User | 14 | 0 | 3 | 0 |
| Mask | 13 | 5 | 5 | 2 |
| N→N | 12 | 6 | 5 | 2 |
| A→A | 8 | 9 | 1 | 2 |
| N→A | 9 | 6 | 5 | 2 |
| Total | 56 | 26 | 19 | 8 |
Retained repairs are modifications accepted after evaluation and preserved in the simulator. These process counts represent one illustrative trajectory per task, not five-run aggregates.
Three observations clarify the contributions of the repair routes:
First, direct repair is an independently effective component. On User Modeling, three direct repairs were retained while no gated structural repair was admitted. Combined with the identical FASim/Direct-Only MAE in E1, this confirms that the task's performance does not depend on successful gated structural correction.
Second, falsification produces persistent structural modifications, not merely diagnostic verdicts. Across the other four tasks, 26 gated diagnoses were approved and eight gated repairs were ultimately retained. These tasks also show the incremental fidelity gains reported in E1, demonstrating that falsification-supported refinement contributes beyond simply withholding uncertain edits.
Third, direct and gated routes have complementary roles. Of the 27 retained repairs in these illustrative trajectories, 19 (70.4%) came from the direct route and eight (29.6%) from the gated route. These are proportions of retained edits, not proportions of performance improvement: individual structural repairs can affect fidelity differently.
For the Direct-Only ablation, ambiguous structural repairs are disabled by design, so all retained repairs necessarily come from the direct route. Detailed task-level process statistics are provided in Official Comment [E1].
Together, these observations clarify the attribution: FASim's overall effectiveness reflects multiple components, while falsification contributes selectively by authorizing evidence-supported structural corrections in tasks where they are needed.
Closing
We appreciate the reviewer's emphasis on empirical attribution and diagnostic reliability. Our five additional experiments systematically investigate:
- E1: Incremental benefits beyond direct-only refinement.
- E2: Counterfactual consequences of rejected repairs.
- E3: Additional repair attempts under aligned compute budgets.
- E4: Root causes of strong User Modeling performance.
- E5: Evidence-supported diagnosis and verified recovery across simulator families.
Together, these results provide stronger empirical support for falsification before repair as a conservative, executable, and auditable structural-edit decision principle, while explicitly recognizing the limits of hypothesis completeness and real-world causal identification.
Response to Reviewer o8NG - W1
We agree that the central contribution should be evaluated through the effectiveness of falsification-guided structural refinement, rather than benchmark performance alone. To directly address these concerns, we conducted five complementary experiments, documented in separate Official Comments [E1]–[E5]. We summarize the key evidence below and invite the reviewer to examine the corresponding comments for detailed protocols and results.
W1. Isolating the Contribution of Falsification
W1.1 — What explains FASim's performance, particularly on User Modeling?
On User Modeling, Direct-Only achieves 0.100 ± 0.003 MAE, matching FASim's 0.100 ± 0.004. This confirms the reviewer's observation: User Modeling gains do not originate from gate-approved structural corrections. Nevertheless, the gate still performs conservative admission control by withholding ambiguous structural edits lacking sufficient evidence.
Importantly, on the other four tasks where gated structural repairs were admitted, FASim improves six of eight fidelity metrics relative to Direct-Only, including Mask RMSE/TFE, N→N JSD, A→A WD, and N→A WD. Two mobility metrics favor Direct-Only, reflecting trade-offs across fidelity dimensions.
Thus, E1 demonstrates task-dependent incremental value from falsification-supported structural refinement beyond direct repairs and unconditional abstention.
We then reran five baselines with explicitly equal review-information availability and the same GPT-5.2 backbone, while preserving each agent's autonomy in selecting its implementation. All baselines generated review-conditioned predictors, yet the strongest, SRR, achieved 0.150 ± 0.005 MAE, compared with FASim's 0.100 ± 0.004 over five runs.
Together, E1 and E4 clarify the attribution: FASim's User Modeling advantage reflects its broader autonomous simulator-construction, direct-repair, calibration, and refinement capabilities, rather than successful gated structural correction or unequal awareness of available review information.
W1.2 — Does the falsification gate actually prevent harmful repairs?
Among 26 contract-valid comparisons in the standard post-gate audit:
Thus, 80.8% of rejected repairs were non-beneficial, including 50.0% that were actively harmful.
A complementary nine-case audit, selected before observing counterfactual outcomes, yielded five harmful, two neutral, and zero beneficial repairs among seven valid cases.
E2 directly demonstrates the preventive value of falsification, rather than merely reporting rejection counts. We also acknowledge that some beneficial repairs were blocked, revealing the strict gate's conservative trade-off between structural-edit risk and repair opportunities.
Please see Official Comment [E2] for the full counterfactual protocol and results.
W1.3 — Could additional computation and repair attempts replace falsification?
Among seven cases allowing valid numerical comparison, FASim achieved better fidelity in five. In three additional N→A cases, Extra-Repair failed the required OOD evaluation contract.
Although actual resource consumption differed, both strategies operated under controlled budget limits. E3 shows that allocating bounded computation to additional repair attempts does not consistently reproduce falsification-guided outcomes.
Detailed results and computational accounting are provided in Official Comment [E3].