acceptodds
Under review as a conference paper at ICLR 2027

Agentic Simulator Construction with Falsification-Guided Structural Diagnosis

Abstract

Large language models can turn natural-language briefs into runnable simulation code, yet automated simulator refinement remains vulnerable to unreliable struc- tural diagnosis: the same macro-level mismatch may admit multiple plausible explanations, while an LLM-generated diagnosis can be acted upon before it is tested. We present FASim, an agentic framework for automated simulator con- struction with falsification-guided structural diagnosis. FASim constructs and calibrates an executable simulator, routes directly verifiable implementation is- sues to repair, and treats structurally ambiguous failures as competing falsifiable hypotheses. A Hypothesis Agent proposes alternative explanations, while a sepa- rate Falsification Agent derives their observable counterfactual signatures, freezes the test plan before execution, and challenges them through controlled simulator probes. Directly observable signatures are evaluated from execution traces, while stochastic or borderline responses use repeated matched-seed executions. Only sufficiently discriminated diagnoses are allowed to guide structural repair; falsified or inconclusive explanations are blocked from triggering persistent edits. Across user modeling, intervention-driven mask adoption, and mobility under ID and OOD shifts, FASim achieves strong multi-metric simulator fidelity. Ablation and process analyses further show that pre-repair falsification improves refinement quality, pro- duces actionable structural diagnoses, and prevents unsupported explanations from being converted into edits. These results support falsify before repair as a practical principle for more reliable and auditable agentic simulator construction. Code and data: https://anonymous.4open.science/status/FASim-8D87.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
automated simulator construction, agentic systems, structural diagnosis, falsification, counterfactual probing, large language models, multi-agent systems, simulation calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_o8NGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FASim, an LLM-agent pipeline that builds an executable agent-based simulator from a task brief and data, calibrates it, and iteratively refines its structure. Issues are routed to one of two paths. “Directly verifiable” implementation issues go straight to repair. “Structurally ambiguous” residuals go through a falsification loop: a Hypothesis Agent proposes 2–4 competing explanations, and a Falsification Agent designs one counterfactual probe with predicted signatures. The plan is frozen and executed by a deterministic ProbeExecutor, usually with matched seeds. Each hypothesis is labeled FALSIFIED, SURVIVES or INCONCLUSIVE, and a structural edit is allowed only if exactly one hypothesis survives and all others are falsified.

Critical strengths
  1. The framework looks sound and well motivated. The freeze-then-evaluate protocol (plan hash, evidence audit, controller-owned thresholds, execution failure never counted as falsification) is careful and have some potential of being reused by other agentic refinement systems.
  2. The empirical analysis is thorough. The injected-defect study (App. D.2) is the right kind of experiment for testing structural diagnosis, and Table 9 honestly shows cases where lower validation loss favors a structurally wrong simulator. The paper also reports unfavorable process statistics.

Thank you for the response. Some of my concerns have been addressed and I've raised my rating accordingly.

Critical weaknesses

W1. The central claim, i.e., “falsify before repair”, is not isolated by the evidence. The paper attributes its gains to the falsification gate, but the pipeline has many other components: blueprint generation, a direct-repair route that bypasses the gate, recalibration, and a retain-if-better rule. The experiments do not separate these contributions. In some settings, such as User modeling, FASim’s advantage is largest even though the process analysis shows the gate never admitted a diagnosis, so the gain there must come from somewhere else. The paper also never tests the gate itself: it reports that most hypotheses are “blocked”, but not whether blocking was the right call, i.e., whether the blocked repairs would actually have hurt. Compute-matched comparisons are also missing, so it is unclear whether the extra probing and LLM calls could achieve the same by simply trying more candidate repairs. This matters because the paper’s contribution is a design principle, not a leaderboard entry. If the benefit cannot be attributed to the principle, the main conclusion is unsupported.

W2. The notion of “trust” delivered by falsification is weaker than the framing suggests. Every step of the falsification loop is authored by the same LLM: the competing hypotheses, the probe, and the predicted signatures. The evidence comes from the simulator’s own behavior under intervention. A hypothesis that “survives” is therefore only consistent with the current program and with the explanations the LLM happened to propose. It is not tested against the real system, and the true cause may never be in the hypothesis set. The method is also restricted to defects in pathways that already exist in the simulator, and the authors acknowledge that surviving does not establish the real mechanism. The only setting with known ground truth is a small set of synthetic defects injected into one simulator, which is a favorable setting for this approach. So the paper shows that the procedure can recover known, designed faults, but not that it makes diagnoses on natural tasks more reliable. More broadly, the idea of testing hypotheses before acting on them is an application of existing ideas (automated falsification, metamorphic testing) to simulator refinement. The conceptual novelty is modest, which raises the bar for the empirical evidence in W1.

Questions for authors
  1. On User, where no gated repair was ever admitted, what produces MAE 0.10? Please give the exact test protocol and the fraction of retained edits from the direct vs gated route for every task and ablation variant.
3 comments
AuthorsOpenReview ↗

Response to Reviewer o8NG - W1

We agree that the central contribution should be evaluated through the effectiveness of falsification-guided structural refinement, rather than benchmark performance alone. To directly address these concerns, we conducted five complementary experiments, documented in separate Official Comments [E1]–[E5]. We summarize the key evidence below and invite the reviewer to examine the corresponding comments for detailed protocols and results.


W1. Isolating the Contribution of Falsification

W1.1 — What explains FASim's performance, particularly on User Modeling?

Experiment E1 — Direct-Only Ablation (Official Comment [E1]). We introduced a Direct-Only variant that preserves FASim's blueprint, direct repairs, calibration, retention, and stopping protocol but disables falsification-guided structural repairs. Both methods were evaluated over five independent refinement runs using GPT-5.2.

On User Modeling, Direct-Only achieves 0.100 ± 0.003 MAE, matching FASim's 0.100 ± 0.004. This confirms the reviewer's observation: User Modeling gains do not originate from gate-approved structural corrections. Nevertheless, the gate still performs conservative admission control by withholding ambiguous structural edits lacking sufficient evidence.

Importantly, on the other four tasks where gated structural repairs were admitted, FASim improves six of eight fidelity metrics relative to Direct-Only, including Mask RMSE/TFE, N→N JSD, A→A WD, and N→A WD. Two mobility metrics favor Direct-Only, reflecting trade-offs across fidelity dimensions.

Thus, E1 demonstrates task-dependent incremental value from falsification-supported structural refinement beyond direct repairs and unconditional abstention.

Experiment E4 — User Modeling Root-Cause Analysis (Official Comment [E4]). To investigate why User Modeling performs so well without gated structural corrections, we inspected FASim's generated simulator and identified an autonomously constructed LLM-assisted, review-conditioned rating estimation mechanism using a precomputed cache.

We then reran five baselines with explicitly equal review-information availability and the same GPT-5.2 backbone, while preserving each agent's autonomy in selecting its implementation. All baselines generated review-conditioned predictors, yet the strongest, SRR, achieved 0.150 ± 0.005 MAE, compared with FASim's 0.100 ± 0.004 over five runs.

Together, E1 and E4 clarify the attribution: FASim's User Modeling advantage reflects its broader autonomous simulator-construction, direct-repair, calibration, and refinement capabilities, rather than successful gated structural correction or unequal awareness of available review information.


W1.2 — Does the falsification gate actually prevent harmful repairs?

Experiment E2 — Blocked-Repair Counterfactual Audit (Official Comment [E2]). We directly tested the consequences of gate decisions by forcibly implementing rejected structural diagnoses and comparing them with the original simulator under matched evaluation conditions.

Among 26 contract-valid comparisons in the standard post-gate audit:

  • 13 (50.0%) forced repairs degraded fidelity.
  • 8 (30.8%) produced no improvement.
  • 5 (19.2%) improved fidelity.

Thus, 80.8% of rejected repairs were non-beneficial, including 50.0% that were actively harmful.

A complementary nine-case audit, selected before observing counterfactual outcomes, yielded five harmful, two neutral, and zero beneficial repairs among seven valid cases.

E2 directly demonstrates the preventive value of falsification, rather than merely reporting rejection counts. We also acknowledge that some beneficial repairs were blocked, revealing the strict gate's conservative trade-off between structural-edit risk and repair opportunities.

Please see Official Comment [E2] for the full counterfactual protocol and results.


W1.3 — Could additional computation and repair attempts replace falsification?

Experiment E3 — Compute-Budget Reallocation (Official Comment [E3]). We compared FASim with an Extra-Repair strategy at ten structural-issue checkpoints under aligned computational budget ceilings. Instead of executing falsification probes, Extra-Repair generated up to two independent repairs, evaluated and recalibrated each, and selected the best valid candidate.

Among seven cases allowing valid numerical comparison, FASim achieved better fidelity in five. In three additional N→A cases, Extra-Repair failed the required OOD evaluation contract.

Although actual resource consumption differed, both strategies operated under controlled budget limits. E3 shows that allocating bounded computation to additional repair attempts does not consistently reproduce falsification-guided outcomes.

Detailed results and computational accounting are provided in Official Comment [E3].

AuthorsOpenReview ↗

Response to W2.


W2. Reliability, Scope, and Conceptual Novelty

W2.1 — Can falsification support correct structural repairs beyond the original Mask defects?

Experiment E5 — Cross-Task Injected-Defect Recovery (Official Comment [E5]). We extended the original seven Mask Adoption defects to 15 injected defects across Mask Adoption, User Modeling, and Personal Mobility, achieving 15/15 verified program-level recoveries.

For eight new controlled cases, we compared three diagnosis strategies:

  • Forced One-shot: Selects a diagnosis from competing hypotheses using static evidence without executing probes.
  • Static-Abstention: May withhold diagnosis when static evidence cannot discriminate competing hypotheses.
  • FASim: Uses frozen executable counterfactual probes to authorize structural repair.

Both Forced One-shot and FASim correctly identified 8/8 injected causes, while Static-Abstention returned UNRESOLVED in all eight cases.

Why is falsification still valuable? Forced One-shot selected the correct diagnoses without executable verification. FASim reached the same diagnoses through pre-specified counterfactual tests and observable execution evidence, providing auditable justification before repair. Unlike Static-Abstention, FASim could also authorize all eight repairs, with recovery verified against hidden program truth.

Therefore, E5 demonstrates evidence-supported actionability and verified structural recovery, rather than superior diagnosis accuracy over one-shot reasoning.

Full experimental details are provided in Official Comment [E5].


W2.2 — Can surviving hypotheses establish the true structural cause?

We agree that falsification is conditional on the proposed hypothesis set and cannot guarantee that the true cause is included.

A unique surviving hypothesis is supported relative to the tested alternatives and predefined observable signatures; survival does not establish real-world causal truth. When execution evidence is inconclusive or insufficiently discriminating, FASim withholds structural repair. However, an incomplete hypothesis set may still lead to an incorrect diagnosis.

Our intended contribution is therefore executable, auditable evidential support for structural-edit decisions, not guaranteed identification of real-world causal mechanisms.

The hidden-oracle evaluation in E5 verifies recovery of known program mechanisms rather than real-world causal identification. We will clarify this distinction in the paper.


W2.3 — What is the conceptual contribution beyond existing falsification techniques?

We acknowledge that falsification and metamorphic testing are established principles. FASim's contribution lies in their operational integration into autonomous simulator refinement:

  1. Separating directly verifiable implementation issues from ambiguous structural failures.
  2. Representing structural ambiguity through competing hypotheses.
  3. Freezing counterfactual predictions and decision criteria before execution.
  4. Using executable evidence and controller-owned rules to authorize persistent structural edits.
  5. Connecting supported diagnoses to targeted code repair and retention.

Unlike standalone testing, this protocol governs whether an LLM-generated structural explanation is permitted to modify an executable simulator.

Our additional experiments demonstrate measurable consequences of this integration: incremental fidelity gains (E1), prevention of harmful edits (E2), and evidence-supported structural recovery (E5).

AuthorsOpenReview ↗

Response to Questions


Questions for Authors: User Modeling and Retained-Edit Attribution


Q1 — What produces User Modeling MAE 0.10?

As established in E4, FASim autonomously constructed an LLM-assisted, review-conditioned rating mechanism using a precomputed cache. Its broader construction, direct-repair, calibration, and refinement pipeline explains the strong performance. E1 independently confirms that no gate-approved structural correction is needed to achieve MAE 0.10.


Q2 — What is the exact evaluation protocol?

In E1, FASim and Direct-Only share the same fixed blueprint, GPT-5.2 backbone, direct-repair mechanism, calibration, retention, and stopping procedures. Each independently executes five refinement runs, with performance reported as mean ± standard deviation. Holding the blueprint fixed isolates downstream structural-refinement effects.


Q3 — How many repairs are retained from each route, and what do they contribute?

Experiment E1 — Repair Process Audit (Official Comment [E1]). In addition to five-run performance evaluation, we recorded diagnosis, gate-approval, and repair-retention decisions from one complete FASim refinement trajectory per task:

TaskDirect diagnosesGated approvalsDirect retainedGated retained
User14030
Mask13552
N→N12652
A→A8912
N→A9652
Total5626198

Retained repairs are modifications accepted after evaluation and preserved in the simulator. These process counts represent one illustrative trajectory per task, not five-run aggregates.

Three observations clarify the contributions of the repair routes:

First, direct repair is an independently effective component. On User Modeling, three direct repairs were retained while no gated structural repair was admitted. Combined with the identical FASim/Direct-Only MAE in E1, this confirms that the task's performance does not depend on successful gated structural correction.

Second, falsification produces persistent structural modifications, not merely diagnostic verdicts. Across the other four tasks, 26 gated diagnoses were approved and eight gated repairs were ultimately retained. These tasks also show the incremental fidelity gains reported in E1, demonstrating that falsification-supported refinement contributes beyond simply withholding uncertain edits.

Third, direct and gated routes have complementary roles. Of the 27 retained repairs in these illustrative trajectories, 19 (70.4%) came from the direct route and eight (29.6%) from the gated route. These are proportions of retained edits, not proportions of performance improvement: individual structural repairs can affect fidelity differently.

For the Direct-Only ablation, ambiguous structural repairs are disabled by design, so all retained repairs necessarily come from the direct route. Detailed task-level process statistics are provided in Official Comment [E1].

Together, these observations clarify the attribution: FASim's overall effectiveness reflects multiple components, while falsification contributes selectively by authorizing evidence-supported structural corrections in tasks where they are needed.


Closing

We appreciate the reviewer's emphasis on empirical attribution and diagnostic reliability. Our five additional experiments systematically investigate:

  • E1: Incremental benefits beyond direct-only refinement.
  • E2: Counterfactual consequences of rejected repairs.
  • E3: Additional repair attempts under aligned compute budgets.
  • E4: Root causes of strong User Modeling performance.
  • E5: Evidence-supported diagnosis and verified recovery across simulator families.

Together, these results provide stronger empirical support for falsification before repair as a conservative, executable, and auditable structural-edit decision principle, while explicitly recognizing the limits of hypothesis completeness and real-world causal identification.

Reviewer_6AkUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FASim, which uses LLM agents to construct and refine simulators. It repairs clear implementation bugs directly, and testing competing explanations for uncertain structural problems before allowing code changes. Experiments on user ratings, mask adoption, and mobility show improvements over a reflection-based baseline in general. The benefit of identifying useful explanations, rather than simply avoiding uncertain edits, and the rule for retaining repairs need clearer evidence.

Critical strengths
  • FASim requires the expected test outcomes to be specified before the tests run. This makes it harder for the model to change its explanation to fit the result and turn an unsupported guess into a code edit.

  • The evaluation covers different simulation tasks and includes a reflection-based baseline with access to the same code and execution evidence. This is a useful comparison for examining whether testing a diagnosis helps beyond simply reasoning about it.

Critical weaknesses
  • The paper is difficult to follow, and the text in the figures is too small to read comfortably. Clearer figures and a more direct description of the main workflow would help communicate the contribution.

  • Table 1 shows a strong improvement on user modeling, but Table 2 shows that none of the uncertain diagnoses was approved for repair. The gain may therefore come from avoiding questionable edits rather than identifying useful fixes. Comparing with a version that fixes clear bugs but always skips uncertain changes would help show what the hypothesis tests add.

  • Section 3.3 says a repair is kept only if it improves the overall score. However, some controlled experiments keep repairs with worse validation error. This could be valid if the repair also reduces penalties for violating constraints, but the paper does not show this.

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6AkU - W1

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of FASim's pre-specified counterfactual predictions and its comparison with reflection-based refinement.

We particularly appreciate the suggestion to compare FASim against a variant that repairs clear implementation issues but always skips ambiguous structural changes. We implemented this comparison and conducted additional experiments to clarify falsification's corrective and preventive contributions.

The five supplementary experiments are documented in separate Official Comments [E1]–[E5]. We summarize the relevant evidence below and invite the reviewer to examine these comments for full results.


W1. Presentation and Workflow Clarity

We thank the reviewer for highlighting the readability issues. We have redesigned Figure 1 and revised Section 3, with all textual changes highlighted in blue in the revised manuscript for easy inspection.

1. Redesigned Figure 1. We increased text size, reduced visual complexity, and reorganized the framework into three clearly connected stages:

  • Construction and Calibration: Task/data analysis, mechanistic blueprint generation, simulator construction, and initial calibration.
  • Issue Routing and Falsification: Execution issues are separated into two explicit paths. Directly verifiable implementation issues bypass falsification, while structurally ambiguous diagnoses require competing hypotheses, frozen executable probes, and the strict unique-survivor gate.
  • Repair, Recalibration, and Retention: Both routes converge into a shared actionable repair set and Code Generation Agent. The repaired candidate is recalibrated, then either promoted or rejected according to the overall fidelity objective.

The revised figure also explicitly illustrates the Improvement / No Improvement branches and their return to the next refinement iteration, clarifying that falsification approval and checkpoint retention are separate decisions.

2. Revised Section 3 (blue-highlighted changes). We made corresponding improvements to the method description:

  • Section 3 opening: Added a concise three-stage workflow overview aligned with Figure 1.
  • Section 3.1: Clarified how the calibrated simulator and execution evidence initiate the refinement loop.
  • Section 3.2: Added a concrete Personal Mobility example showing how competing explanations of excessive trip distance are distinguished through a frozen counterfactual probe. We also clarified direct versus falsification-guided issue routing.
  • Section 3.2.4: Explicitly explained the gate's outcomes: structural repair requires exactly one surviving hypothesis with all alternatives falsified; inconclusive or insufficient evidence does not authorize repair.
  • Section 3.3: Clarified how both diagnosis routes form one actionable repair set, and distinguished repair authorization from fidelity-based candidate retention under Eq. (10). We also explained how accepted and rejected candidates lead into the next iteration.

We invite the reviewer to inspect the blue-highlighted revisions in Section 3 and the redesigned Figure 1. Together, these changes provide a more direct explanation of FASim's end-to-end workflow while preserving the technical details of its falsification mechanism.

AuthorsOpenReview ↗

Response to Reviewer 6AkU - W2

W2. Does Falsification Identify Useful Repairs or Merely Avoid Uncertain Edits?


Experiment E1 — Direct-Only Ablation (Official Comment [E1]).

Following your specific suggestion, we implemented Direct-Only, which retains FASim's fixed blueprint, GPT-5.2 backbone, direct repairs, calibration, best-checkpoint retention, and stopping protocol, but always withholds structurally ambiguous repairs. Both methods were evaluated over five independent refinement runs.

On User Modeling, Direct-Only achieves 0.100 ± 0.003 MAE, matching FASim's 0.100 ± 0.004. We therefore agree that User Modeling's strong performance does not originate from gate-approved structural corrections.

However, in the four remaining tasks where FASim admits falsification-supported structural diagnoses, FASim improves six of eight fidelity metrics relative to Direct-Only:

MetricDirect-OnlyFASim
Mask RMSE0.2148 ± 0.0100.170 ± 0.007
Mask TFE0.1703 ± 0.0050.140 ± 0.006
N→N JSD0.1242 ± 0.0070.050 ± 0.003
A→A WD1.2068 ± 0.0080.710 ± 0.003
N→A WD3.1596 ± 0.0110.860 ± 0.005

The remaining improvement is smaller (N→A JSD), while Direct-Only performs better on N→N WD and A→A JSD. These results show that always skipping uncertain structural changes cannot reproduce FASim's gains across all fidelity dimensions, although structural refinement involves metric-specific trade-offs.

The repair-process analysis in [E1] further shows that FASim retained eight gated structural repairs across the four non-User tasks in illustrative refinement trajectories. Thus, falsification is not merely rejecting diagnoses; it also enables persistent structural corrections.


Experiment E5 — Cross-Task Injected-Defect Recovery (Official Comment [E5]).

To establish whether falsification-supported diagnoses can produce correct structural repairs, we extended the original Mask Adoption injected-defect study to User Modeling and Personal Mobility, achieving 15/15 verified program-level recoveries across three simulator families.

In eight new controlled cases, we compared FASim with two alternatives:

  • Forced One-shot: Must choose a structural diagnosis using static evidence without executing probes.
  • Static-Abstention: May return UNRESOLVED when static evidence cannot distinguish competing hypotheses.

Forced One-shot and FASim both identified all eight injected causes, whereas Static-Abstention withheld all eight diagnoses. Crucially, FASim reached its diagnoses through frozen executable counterfactual probes, enabling eight repairs verified against hidden program truth.

Thus, [E5] demonstrates evidence-supported corrective action beyond unconditional abstention, without claiming higher diagnosis accuracy than Forced One-shot.


Experiment E2 — Blocked-Repair Counterfactual Audit (Official Comment [E2]).

We also examined falsification's preventive role by forcibly implementing rejected structural repairs and measuring the consequences.

Among 26 contract-valid comparisons under the standard post-gate protocol, 13 (50.0%) forced repairs degraded fidelity, eight (30.8%) were neutral, and five (19.2%) improved fidelity.

Therefore, falsification does not simply reject every uncertain edit: it selectively authorizes evidence-supported repairs while preventing many modifications that would otherwise harm simulator fidelity. We also acknowledge the conservative gate's trade-off, as some beneficial repairs are blocked.

Together, E1, E2, and E5 distinguish falsification from both unconditional abstention and unconditional repair. E1 demonstrates incremental fidelity benefits, E5 demonstrates verified corrective action, and E2 demonstrates preventive value.


Why does User Modeling perform well without gated structural repairs? (Official Comment [E4])

We additionally inspected FASim's generated User Modeling simulator and identified an autonomously constructed LLM-assisted, review-conditioned rating estimation mechanism using a precomputed cache.

In Experiment E4, we reran five baselines under the same GPT-5.2 backbone and explicit review-information availability, allowing each to choose its own implementation. All generated review-conditioned predictors, but the strongest, SRR, achieved 0.150 ± 0.005 MAE, compared with FASim's 0.100 ± 0.004 over five runs.

This clarifies that User Modeling's advantage reflects FASim's broader autonomous simulator-construction, direct-repair, calibration, and refinement capabilities, rather than successful gated structural correction or unequal awareness of available review information.

AuthorsOpenReview ↗

Response to Reviewer 6AkU - W3

W3. Clarification of Repair Retention (Eq. 10 and Table 9)

We thank the reviewer for identifying this ambiguity. The distinction is between structural recovery and fidelity-based checkpoint retention, which serve different purposes.

1. Experimental objective: isolating structural recovery

The controlled D1–D7 study (Appendix D.2) asks:

Given known injected structural defects, can FASim identify them through executable falsification and generate repairs that restore the correct mechanisms?

It does not evaluate whether those candidates should replace the incumbent under Eq. (10).

We separate these objectives because a defective simulator may achieve lower validation loss through parameter compensation. Using fidelity-based retention as the recovery criterion would conflate whether diagnosis and repair recover the correct mechanism with whether the fidelity objective favors promoting that repair.

Evaluating candidates against hidden program truth independently of checkpoint promotion isolates structural recovery.

2. Why Eq. (10) is not applied in D1–D7

The study preserves the same falsification mechanism as standard FASim: competing hypotheses, frozen executable probes, pre-specified signatures, and the strict unique-survivor gate. Injected defects and clean references remain hidden from diagnosis and repair agents.

However, D1–D7 did not apply Eq. (10)'s checkpoint-promotion rule. Instead, the experiment ran fixed repair iterations, preserved candidates, and evaluated recovery against hidden program truth.

In standard FASim, two decisions occur sequentially:

  • Falsification gate: Authorizes repair when executable evidence supports its diagnosis.
  • Eq. (10): After recalibration, compares the overall objective defined in Eq. (1) and promotes the candidate only upon sufficient improvement.

Thus, Eq. (10) is a fidelity-based retention rule, not a structural-correctness test. A correct repair may not qualify for promotion if the task-level objective does not improve. Standard FASim retains the incumbent in such cases, while candidates can still be preserved for audit.

This addresses the reviewer's suggestion regarding constraint penalties: Table 9 does not report candidates promoted through Eq. (10). Its recovery outcomes therefore do not rely on reduced constraint penalties.

3. What Table 9 demonstrates

Table 9 shows cases where a simulator containing an injected defect achieves lower validation loss than its repaired candidate:

DefectCorrupted loss ↓Repaired loss ↓Mechanism restored
D10.10470.1449Yes
D30.09220.0942Yes
D40.09240.1242Yes
D60.09280.1004Yes

FASim generated candidates restoring the campaign pathway (D1), social-influence mechanisms (D3–D4), and individual-level persistence (D6), despite higher validation losses.

These cases demonstrate structural recovery, not promotion under Eq. (10). Observational fit alone may fail to distinguish correct from defective mechanisms.

4. Why observational fidelity remains essential

This does not undermine the metrics used in our main benchmarks. In User Modeling, Mask Adoption, and natural Mobility tasks, complete ground-truth simulator structures are generally unavailable.

We use MAE, RMSE/TFE, and JSD/WD to evaluate how well simulators reproduce observable task behavior under common criteria.

These metrics assess behavioral fidelity, whereas executable falsification provides additional evidence for structural diagnoses. The mechanisms are complementary: falsification supports repair authorization, while fidelity determines candidate retention.

5. Revisions incorporated into the manuscript

We acknowledge that Table 9's original "Repair retained" label misleadingly suggested checkpoint promotion under Eq. (10).

We have addressed this in the revised manuscript:

  • Section 4.3: Clarifies that the injected-defect study evaluates structural recovery independently of standard checkpoint promotion.
  • Appendix D.2: Explains why D1–D7 evaluates structural recovery without applying Eq. (10), while preserving the same falsification mechanism.
  • Tables 8–9: Clarify stable recovery and replace "Repair retained" with "Reference mechanism restored", distinguishing repaired candidates from promoted incumbents.

All modifications are highlighted in blue in Section 4.3 and Appendix D.2 of the revised manuscript.

These revisions distinguish structurally repaired candidates from incumbents promoted through fidelity-based retention, without changing the falsification mechanism or experimental findings.

Reviewer_U9rvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a problem in LLM-based simulator construction: a plausible explanation for a mismatch may lead to an incorrect structural repair. To address this, FASim builds and calibrates a simulator from a natural-language brief, then tests competing explanations before making structural changes. A Hypothesis Agent proposes explanations, and a separate Falsification Agent tests them through executable probes. Repair proceeds only when one hypothesis survives and all alternatives are falsified; directly verifiable implementation issues are handled separately.

The authors evaluate FASim on user modeling, mask adoption, and personal mobility, reporting the best average rank among the compared methods. Overall it is a nicd read!

Critical strengths
  • The gate is concrete. Plans carry a hash, thresholds come from baseline variability, and a failed run never counts as falsification.

  • The injected-defect study tests structural recovery against known ground truth. FASim recovers all seven defects, including cases where the corrupted simulator has lower validation loss than the repaired one (Tables 8-9).

Critical weaknesses
  • FASim's largest improvement over the baselines occurs on User Modeling, where no diagnosis passes the gate. The variant without structural refinement already outperforms the baselines. Weakening the gate does hurt performance, but the paper does not separate the benefit of withholding unsupported edits from that of direct-route repairs, which skip the gate (Tables 2-3). This leaves the source of the gains unclear.

  • The Mask RMSE results for GPT-5.2 are inconsistent: Table 11 reports 0.07, while Table 1 reports 0.17. This discrepancy changes the comparison with the strongest baseline and needs clarification. I might be wrong here.

Questions for authors
  • Which Mask RMSE for GPT-5.2 is correct, Table 11 or Table 1, and does each seed rerun the LLM construction and refinement loop?

  • How many direct-route repairs, and which ones, account for the User Modeling improvement over fix /lambda?

  • Given that Table 9 retains some repairs despite higher loss than the corrupted one, which loss does Eq. 10 compare?

5 comments
AuthorsOpenReview ↗

Response to Reviewer U9rv - W1 & W2

We thank the reviewer for the positive assessment of FASim's methodological soundness. We also appreciate the careful identification of ambiguities in our performance attribution and experimental reporting.

To address these concerns, we conducted five supplementary experiments, documented in separate Official Comments [E1]–[E5]. Below, we summarize the relevant evidence and directly answer each question.


W1. Attribution of FASim's Performance Gains

Experiment E1 — Direct-Only Ablation (Official Comment [E1]).

To separate the contribution of direct repairs from falsification-guided structural refinement, we introduced a Direct-Only variant that preserves FASim's blueprint, direct repairs, calibration, retention, and stopping protocol while withholding all structurally ambiguous repairs. Both methods were evaluated over five independent refinement runs using GPT-5.2.

On User Modeling, Direct-Only achieves 0.100 ± 0.003 MAE, matching FASim's 0.100 ± 0.004. We therefore agree with the reviewer that User Modeling gains do not originate from gate-approved structural corrections. The falsification gate nevertheless enforces conservative admission control by withholding ambiguous structural changes lacking sufficient evidence.

On the four remaining tasks where falsification-supported repairs were admitted, FASim improves six of eight fidelity metrics relative to Direct-Only, including Mask RMSE/TFE, N→N JSD, A→A WD, and N→A WD. Two mobility metrics favor Direct-Only, reflecting task-dependent trade-offs rather than universal dominance.

Thus, E1 distinguishes the contribution of the direct-repair pipeline from the incremental benefits of falsification-guided refinement.

Experiment E2 — Blocked-Repair Counterfactual Audit (Official Comment [E2]).

To determine whether withholding unsupported edits is actually beneficial, we forcibly implemented diagnoses rejected by the falsification gate and evaluated their consequences under matched conditions.

Among 26 contract-valid standard post-gate comparisons, 13 (50.0%) forced repairs degraded fidelity, eight (30.8%) were neutral, and five (19.2%) improved it. Thus, 80.8% of rejected repairs provided no fidelity benefit, including 50.0% that were actively harmful.

These results demonstrate the gate's preventive role while acknowledging that conservative blocking can also reject useful repairs. The complete counterfactual protocol and results are provided in Official Comment [E2].

Experiment E4 — User Modeling Root-Cause Analysis (Official Comment [E4]).

To investigate the source of User Modeling's strong performance, we inspected FASim's generated simulator and identified an autonomously constructed LLM-assisted, review-conditioned rating estimation mechanism using a precomputed cache.

We then reran five baselines under explicitly equal review-information availability, the same GPT-5.2 backbone, and five independent runs per method, while allowing each agent to choose its own implementation.

All baselines generated review-conditioned predictors, yet the strongest, SRR, achieved 0.150 ± 0.005 MAE, compared with FASim's 0.100 ± 0.004.

Importantly, the final evaluation was strictly leakage-free: ground-truth test ratings were inaccessible to simulator construction, user/item profiling, calibration, and prediction, and were used exclusively by the final evaluator.

Together, E1 and E4 clarify that User Modeling's advantage reflects FASim's broader autonomous simulator-construction, direct-repair, calibration, and refinement capabilities, rather than gated structural correction or unequal access to review information.


W2. Clarification of Mask RMSE in Tables 1 and 11

We thank the reviewer for identifying this discrepancy. Both values are correct but use different semi-synthetic target datasets.

  • Table 1: Main Mask Adoption benchmark with GPT-5.2: RMSE 0.170 ± 0.007.
  • Table 11: Backbone-sensitivity experiment on an independently synthesized dataset: GPT-5.2 RMSE 0.070 ± 0.009.

Within Table 11, all backbones share the same target dataset, blueprint, calibration, refinement budget, and evaluation protocol; only the backbone varies.

Each backbone underwent five independent construction-and-refinement runs, not repeated evaluations of a fixed simulator. Updated results (mean ± 95% CI):

BackboneRMSE ↓
GPT-5.20.070 ± 0.009
Qwen3-Next-80B-A3B-Instruct0.049 ± 0.003
Llama-3.3-70B-Instruct0.734 ± 0.007

These findings support cross-backbone portability, not backbone invariance. The RMSE difference between Tables 1 and 11 reflects distinct evaluation targets.

We have updated Appendix D.4 and Table 11 in the revised manuscript, including dataset distinctions and five-run protocols. All revisions are highlighted in blue in our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer U9rv - Q1 & Q2

Q1 — Does each seed rerun the LLM construction and refinement loop?

Yes. Each seed corresponds to an independent downstream LLM-based workflow run, including simulator construction, diagnosis, code repair, calibration, and evaluation, rather than repeated evaluations of a fixed final simulator.

The initial mechanistic blueprint, evaluation targets, and data splits are held fixed across runs to control for differences in initial design and evaluation conditions. Thus, the reported five-run means and 95% confidence intervals reflect independent downstream workflow executions.

We have explicitly clarified this protocol in the Evaluation metrics paragraph of Section 4.1 (Experiment Settings) in the revised manuscript. The corresponding modifications are highlighted in blue for ease of inspection.


Q2 — How many direct-route repairs, and which ones, account for User Modeling's improvement?

The process audit in Official Comment [E1] records 14 direct-route diagnoses, three retained direct repairs, and zero gate-approved structural repairs in an illustrative complete User Modeling refinement run.

The three retained repairs addressed:

  1. Dataset loading and schema compatibility: Corrected loading failures that otherwise triggered synthetic placeholder-data fallback.
  2. Test-label leakage prevention: Corrected user/item profiling logic that could incorporate held-out test ratings into predictive features.
  3. Evaluation-contract compliance: Corrected simulator outputs and evaluation procedures to match the task's specified prediction requirements.

These are directly verifiable implementation and evaluation issues, not falsification-gated structural diagnoses.

Crucially, the final reported User Modeling results were obtained under a verified leakage-free evaluation protocol. The leakage-related issue was identified and corrected before final evaluation; test ratings were used only by the evaluator and never as predictive features.

As shown in [E4], the simulator's autonomous review-conditioned rating estimation mechanism also contributes to the broader construction pipeline. We do not attribute the entire MAE improvement to any single direct repair.

AuthorsOpenReview ↗

Response to Reviewer U9rv - Q3

Q3 — Which loss does Eq. (10) compare, and why does Table 9 retain higher-loss repairs?

We thank the reviewer for identifying this ambiguity. The key distinction is between structural recovery and fidelity-based checkpoint retention, which serve different evaluation purposes.

1. Experimental objective: isolating structural recovery

The controlled D1–D7 study (Appendix D.2) asks:

Given known injected structural defects, can FASim identify them through executable falsification and generate repairs that restore the correct mechanisms?

It does not ask whether those candidates should replace the incumbent under Eq. (10).

We separate these objectives because a corrupted simulator may achieve lower validation loss through parameter compensation despite containing a known structural defect. Using fidelity-based retention as the recovery-success criterion would conflate the ability to generate structurally correct repairs with whether the current fidelity objective favors their promotion.

Evaluating candidates independently against hidden program truth allows us to isolate structural recovery.

2. Why Eq. (10) is not applied in D1–D7

D1–D7 preserve the same falsification mechanism as standard FASim: competing hypotheses, frozen executable probes, pre-specified signatures, and the strict unique-survivor gate. Injected defects and clean references remain hidden from diagnosis and repair agents.

However, the original D1–D7 experiment did not apply Eq. (10)'s checkpoint-promotion rule. Instead, it ran fixed repair iterations, preserved candidates, and verified mechanism recovery against hidden program truth.

In standard FASim, two decisions occur sequentially:

  • Falsification gate: Authorizes a structural repair when executable evidence supports its diagnosis.
  • Eq. (10): After recalibration, compares the overall task-level objective defined in Eq. (1) and promotes the candidate only upon sufficient improvement.

Thus, Eq. (10) is a fidelity-based retention rule, not a structural-correctness test. A candidate may restore the correct mechanism without qualifying for promotion. Standard FASim conservatively retains the incumbent when the task-level objective does not improve, while rejected candidates can still be preserved for audit.

3. What Table 9 demonstrates

Table 9 illustrates cases where a simulator with an injected structural defect achieves lower validation loss than its structurally repaired candidate:

DefectCorrupted loss ↓Repaired loss ↓Mechanism restored
D10.10470.1449Yes
D30.09220.0942Yes
D40.09240.1242Yes
D60.09280.1004Yes

FASim generated repairs restoring the campaign pathway (D1), social-influence mechanisms (D3–D4), and individual-level persistence (D6), despite higher validation losses.

These cases demonstrate successful structural recovery, not promotion under Eq. (10). They show that observational fit alone may fail to distinguish correct from defective mechanisms.

4. Why observational fidelity remains essential

This does not undermine the metrics used in our main benchmarks. For User Modeling, Mask Adoption, and natural Mobility tasks, complete ground-truth simulator structures are generally unavailable.

We therefore use MAE, RMSE/TFE, and JSD/WD to measure how well simulators reproduce observable task behavior and to compare methods under common evaluation criteria.

These metrics assess predictive or behavioral fidelity, not structural correctness. FASim combines this evaluation with executable falsification, which supplies additional evidence for structural edits.

The two mechanisms are complementary: falsification supports repair authorization; fidelity determines candidate retention. Neither alone guarantees the true real-world mechanism.

5. Revisions incorporated into the manuscript

We acknowledge that Table 9's original "Repair retained" label misleadingly suggested checkpoint promotion under Eq. (10).

We have now addressed this ambiguity in the revised manuscript:

  • Section 4.3: Clarifies that the injected-defect study evaluates controlled structural recovery independently of standard checkpoint retention.
  • Appendix D.2: Explicitly explains the distinct experimental objective, why Eq. (10) is not applied, and how structural recovery differs from observational fidelity.
  • Tables 8–9: Clarify the stable-recovery criterion and replace "Repair retained" with "Reference mechanism restored", distinguishing repaired candidates from promoted incumbents.

All corresponding revisions are highlighted in blue in Section 4.3 and Appendix D.2 of the revised manuscript for ease of inspection.

These changes make the distinction between structural recovery and fidelity-based checkpoint retention explicit without altering the experimental results or falsification mechanism.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Author Response Overview: Five Additional Experiments (E1–E5) Addressing Reviewer Concerns

We sincerely thank all reviewers for their constructive feedback and insightful suggestions. In response to the concerns regarding the attribution of FASim's performance gains, the effectiveness of falsification-guided structural diagnosis, and the reliability of structural repairs, we have conducted five additional experiments.

To facilitate discussion and avoid repeating extensive experimental details across individual responses, we present the experimental setups, results, analyses, and findings in five separate Official Comments:

  • [E1] Direct-Only Ablation: Isolating the incremental contribution of falsification-guided structural refinement.
  • [E2] Blocked-Repair Counterfactual Audit: Evaluating whether rejected structural repairs would degrade simulator fidelity.
  • [E3] Compute-Budget Reallocation: Comparing falsification-guided diagnosis with additional repair attempts under controlled computational budgets.
  • [E4] User Modeling Root-Cause Analysis: Explaining FASim's strong User Modeling performance through simulator inspection and controlled baseline comparisons.
  • [E5] Cross-Task Injected-Defect Recovery: Evaluating evidence-supported diagnosis and verified structural recovery across three simulator families.

We will respond to each reviewer's specific concerns and questions separately under their respective review threads, explicitly referencing the relevant experiments above.

We hope this organization makes the additional evidence easier to examine and facilitates a focused discussion of FASim's contributions, empirical findings, and limitations.

We thank the reviewers and Area Chairs again for their time and consideration.

AuthorsOpenReview ↗

E1: Direct-Only Ablation — Isolating the Contribution of Falsification-Guided Refinement

Motivation. Reviewers U9rv, 6AkU, and o8NG questioned whether FASim's improvements originate from falsification-guided structural refinement or from other components, particularly direct repairs, recalibration, and retention. To isolate the contribution of falsification, we introduce a Direct-Only ablation that preserves the existing refinement pipeline but disables falsification-guided structural repairs.

1. Experimental Setup

Both methods share the same fixed blueprint, task specification, LLM (GPT-5.2), Feedback Agent, direct-repair route, Code Agent, calibration, best-checkpoint retention, iteration budget, and stopping protocol. Their only intended methodological difference is how they handle structurally ambiguous issues:

  • FASim: Generates competing hypotheses, evaluates frozen counterfactual probes, and permits structural repair only when the unique-survivor gate is satisfied.
  • Direct-Only: Withholds all structurally ambiguous diagnoses from repair while retaining directly verifiable repairs.

We conduct five independent refinement runs per method, following the same evaluation protocol and reporting mean ± standard deviation. The blueprint is fixed across runs to control for initial design differences, while downstream refinement is independently re-executed. The runs are not paired by identical random-seed IDs.

2. Fidelity Results

All metrics are lower-is-better.

MetricDirect-OnlyFASim
User MAE0.100 ± 0.0030.100 ± 0.004
Mask RMSE0.2148 ± 0.0100.170 ± 0.007
Mask TFE0.1703 ± 0.0050.140 ± 0.006
N→N JSD0.1242 ± 0.0070.050 ± 0.003
N→N WD0.2400 ± 0.0090.500 ± 0.003
A→A JSD0.00435 ± 0.0010.040 ± 0.003
A→A WD1.2068 ± 0.0080.710 ± 0.003
N→A JSD0.0437 ± 0.0030.040 ± 0.005
N→A WD3.1596 ± 0.0110.860 ± 0.005

FASim improves six of eight non-User metrics, including Mask RMSE (20.9%), Mask TFE (17.8%), N→N JSD (59.7%), A→A WD (41.2%), and N→A WD (72.8%). Direct-Only performs better on N→N WD and A→A JSD, revealing trade-offs across fidelity dimensions.

3. Diagnosis and Repair Process Analysis

To clarify how the two repair routes operate, we additionally audit one illustrative complete FASim refinement run per task. These process counts are separate from the five-run fidelity statistics.

TaskDirect diagnosesGated approvalsDirect retainedGated retained
User14030
Mask13552
N→N12652
A→A8912
N→A9652
Total5626198

These are issue-level process counts from the illustrated trajectories, not five-run aggregates or independent defect counts. Across these runs, 19 direct and eight gated repairs were retained (i.e., accepted after evaluation and preserved in the simulator), confirming that both routes produce persistent modifications.

4. Key Findings

F1 — User Modeling gains are not attributable to gated structural correction. Direct-Only and FASim achieve identical mean MAE (0.100). The User process trace contains three retained direct repairs but no gate-approved or retained structural repairs. We therefore agree with the reviewers: User Modeling performance reflects the shared construction, direct-repair, calibration, and evaluation pipeline, rather than successful falsification-gated structural correction.

F2 — Falsification provides incremental value beyond conservative abstention. In the other four tasks, FASim admits and retains falsification-supported structural repairs, while Direct-Only withholds all ambiguous modifications. FASim improves six of eight corresponding metrics, with particularly substantial gains in Mask fidelity and mobility distribution matching. Thus, always skipping uncertain structural edits does not reproduce the full pipeline's performance.

F3 — Direct and gated repairs serve complementary roles. The process audit records 19 retained direct repairs and eight retained gated repairs. Direct repair is an important independent component, while falsification enables additional evidence-supported structural modifications. Neither route alone should be credited with all end-to-end improvements.

5. Conclusion

This controlled ablation clarifies the source of FASim's gains. Direct repairs explain an important part of the overall performance, particularly on User Modeling, while falsification-guided structural refinement provides selective, task-dependent improvements beyond direct repair and unconditional abstention. The five-run results also reveal metric-specific trade-offs rather than universal dominance. Complementary experiment [E2] (Blocked-Repair Counterfactual Audit) further evaluates whether withholding unsupported structural edits prevents harmful modifications.

AuthorsOpenReview ↗

E2: Blocked-Repair Counterfactual Audit — Evaluating Rejected Structural Edits

Motivation. Reviewer o8NG questioned whether FASim's falsification gate actually prevents harmful structural repairs rather than merely blocking plausible diagnoses. We directly evaluate this through a counterfactual audit: what happens if a rejected structural diagnosis is forcibly implemented?

1. Experimental Setup

Each case represents one structural diagnosis rejected by FASim's strict unique-survivor gate. We compare:

  • Gate arm: Follow FASim's decision and withhold the blocked repair.
  • Forced-repair arm: From the same checkpoint, use the Code Agent to implement exactly one additional blocked diagnosis as a minimal structural edit.

Both arms share other approved repairs, task specifications, calibration, evaluation procedures, and simulation seed (1729). We evaluate one repair step without further refinement.

Case selection and protocols. We report two complementary audit sets:

  • Pre-specified targeted audit (9 cases): Nine blocked diagnoses were selected and frozen before generating forced repairs or observing outcomes. They cover all five tasks and both FALSIFIED and UNRESOLVED/INCONCLUSIVE diagnoses. Selection was based on coverage, not known harmfulness.

  • Standard post-gate audit (29 cases): We expanded the audit to 37 eligible blocked diagnoses. Of these, 29 had historical post-gate checkpoints, enabling uniform comparisons between the original simulator after approved repairs and the same simulator with one additional blocked edit. The remaining eight occurred at terminal iterations and required a different protocol.

Thus, the nine-case set is defined by pre-outcome selection, while the 29-case set is defined by a common comparison protocol. They may overlap and must not be added together.

Forced repairs undergo syntax, runtime, and evaluation checks. Invalid candidates are contract failures, not harmful repairs.

2. Counterfactual Results

We define:

ΔJ=Jforced−Jgate\Delta J = J_{\mathrm{forced}} - J_{\mathrm{gate}}

Lower JJ indicates better fidelity. Positive, zero, and negative differences indicate harmful, neutral, and beneficial repairs.

AuditTotalHarmfulNeutralBeneficialContract failuresNon-beneficial (valid)
Pre-specified targeted952027/7 (100%)
Standard post-gate291385321/26 (80.8%)

Targeted audit: Of seven valid cases selected before observing outcomes, five were harmful, two neutral, and none beneficial, providing evidence without outcome-based selection.

Standard audit: Of 26 contract-valid cases, 13 (50.0%) degraded fidelity, eight (30.8%) were neutral, and five (19.2%) improved fidelity. Thus, 80.8% of rejected edits provided no benefit.

3. Key Findings

F1 — Blocking has demonstrable preventive value. Half of the valid rejected repairs in the standard audit would have degraded fidelity. This directly supports the gate's preventive role beyond rejection counts.

F2 — The gate balances safety against repair opportunities. Although 21/26 rejected repairs were non-beneficial, five were beneficial. The strict gate reduces harmful-edit exposure but can reject useful repairs. It is a conservative admission controller, not a perfect oracle.

F3 — Outcome classification is conservative. Neutral repairs do not establish incorrect diagnoses; they simply provide no observed fidelity benefit. Contract failures are excluded rather than counted as successful rejections.

4. Scope and Limitations

Each case evaluates one concrete code modification from a blocked diagnosis. A harmful outcome does not prove that every possible implementation would fail. The audit tests immediate effects, not subsequent refinement.

Multiple diagnoses may share checkpoints, so cases are not necessarily independent. Matched-seed comparisons control simulation randomness between arms but do not establish multi-seed robustness.

The eight terminal-local cases, evaluated separately under a different protocol, yielded three harmful and five beneficial repairs, suggesting possible over-blocking. They are not pooled with the standard audit.

5. Conclusion

The audit provides direct evidence for falsification's preventive value: under the standard protocol, 50.0% of valid rejected repairs degraded fidelity, while another 30.8% provided no improvement. Although some beneficial repairs were blocked, the gate reduces exposure to harmful edits.

Together with [E1] (Direct-Only Ablation), these findings support FASim's complementary mechanisms: selectively admitting evidence-supported structural repairs while withholding insufficiently supported modifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.