- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical audit of validator reliability when agents revise executable world models and patch code repositories. The authors compare repairing versus rebuilding programmatic world models in a controlled Crafter-OO slice and audit test-passing patches on SWE-bench Verified. Their findings indicate that validator-accepted revisions still suffer from high failure rates during execution (primarily due to collateral regressions) and that consensus-based checks fail to catch shared defects. Consequently, the authors advocate gating revisions on sealed held-out checks and regression checks anchored on the unpatched code.
Critical strengths
-
Rigorous experimental protocol and reproducibility effort. The authors pre-registered their experimental designs, prompts, and statistical tests, and transparently reported negative or inconclusive results.
-
Investigating validator blind spots by directly executing confident plans and counterexample tests is a practical perspective that exposes meaningful evaluation gaps in current agent benchmarks.
-
Clear categorization and empirical breakdown of failure modes (missing requirements vs. collateral regressions) across both testbeds.
Critical weaknesses
-
Despite the thorough empirical audit, the methodological novelty is somewhat limited. The core contributions are primarily empirical observations, while the theoretical claims (e.g., Observation 1 on consensus limitations) are relatively elementary. Moreover, the proposed remedies—using sealed held-out sets and regression checks against unpatched code—closely mirror standard software testing and ML evaluation practices rather than introducing novel learning mechanisms.
-
The experimental setup for the world model testbed is quite constrained (evaluating only one learner and two hand-picked rule changes on a simplified Crafter-OO slice). It is unclear whether the findings on repair versus rebuild can generalize to more complex or stochastic RL environments.
-
While the proposed regression detector in Section 6 reduces false flags, it incurs a noticeable drop in recall across backbones alongside higher API costs.
Questions for authors
-
Given that the Crafter-OO evaluation is limited to two simple deterministic rule changes and the per-arm failure rates are confounded with acceptance leniency (Section 4.2), it is unclear how well the "repair over rebuild" finding extrapolates to general world-modeling tasks. Would these conclusions still hold if all arms were gated at a matched development accuracy threshold or tested in more complex, stochastic environments?
-
In the code testbed, the pre-registered comparison of certified rates in Section 5.1 fails to separate resolved patches from coder-cleared controls (and is noted as compatible with zero true defects at the controls' specificity bound). Meanwhile, the frozen detector in Section 6 cuts false flags only by sacrificing more than half of the recall on witness-confirmed positives (e.g., finding only 6/17 on DeepSeek and Qwen) at a much higher cost per detection. To solidify the practical utility of this auditing pipeline, how can the authors improve the discriminative power over controls without losing the majority of true positives?
4 comments
Details for 8D98: each check at the changed rule; matched thresholds by arm; operating points
Terms as in the general response; probes are test transitions built from a change by a fixed rule; the comparison flags plans where a revision departs from the program it replaces, outside actions the post-change evidence shows that program mispredicting; the per-action rule distrusts any action a program mispredicted on its development transitions.
W1, check by check, on one pool (two rule changes picked by a fixed rule, never run before; 24 runs, two reviser families, 970 programs; pre-registered; rows marked * frozen before computing; App. L.2):
| Check | Failing / admitted plans, any rule | At the changed rule | Task cells covered |
|---|---|---|---|
| Best development score (as used) | 32.3% (1,426 / 4,413) | 505 | 91.5% |
| Perfect on the 27-transition held-out panel | 10.6% (104 / 984) | 104 | 38.7% |
| The per-action rule on the gate's choices* | 8.6% (180 / 2,093) | 179 | 64.8% |
| Boundary-value tests from the same change, then the comparison* | 9.2% (488 / 5,329) | 197 | 57.6% |
| Mutation-adequate tests given the true edit, then the comparison* | no program passes | none | 0% |
| Probes with the changed rule's held-out transitions | 46.8% (4,887 / 10,434) | 0 | 71.0% |
| Probes with the changed rule's held-out transitions, then the comparison | 7.0% (273 / 3,897) | 0 | 37.9% |
The probes remove the changed-rule failures, which the per-action rule alone leaves (179 against 0); the comparison removes most of the rest, all at untouched rules (46.8% to 7.0%). At the first change, where the compared program is the correct source, probes then the comparison admit no failing plan (0 of 2,361, against 8.5% for the panel). All 273 that remain sit at the second change, where the compared program is the gate's own first-change pick; 84.2% (230) repeat its errors, and anchoring the comparison on the source program cuts the rate from 7.0% to 4.9% (177 plans; frozen before computing). Against the panel's 10.6%, the end-to-end rate missed our pre-set bar of half.
Why the panel protected before, and where it stops. On the paper's two changes, of 2,339 rules within two grammar edits of the replaced one, only the true rule fits the reviser's evidence (frozen before computing; App. L.3), so the panel had nothing left to separate: programs perfect on it fail none of 1,008 confident plans sealed and none of 902 visible. On new changes its 27 transitions separate none of the 114 plausible wrong rules we enumerated (frozen before computing; App. L, separation of plausible wrong rules). Probes add transitions at the new threshold, where the wrong rules revisers learned sit. Kept out of the reviser's evidence, the probe check admits 0 of 992 plans failing at the changed rule; 21.1% of programs perfect on probes the reviser had seen (12 of 57) still failed there (post hoc; App. L.8). On 12 more runs (pre-registered), 0.4% of plans kept by the probes fail (16 of 3,722), from one run whose programs dropped a condition no edit or probe touches; the full panel rejects all four programs (App. L.7).
Against existing gates, on 24 fresh runs (pre-registered). Probes, the comparison and the per-action rule together leave 2.4% of covered task cells failing at any rule (12 of 509), against 5.8% (52 of 904) for the best of 180 frozen gates built from development scores, held-out panels and plan filters; the check deploys in fewer cells and declines the plans it cannot trust (App. L.10). At the changed rule they leave a failure in 1.4% of covered cells (7 of 509), against 4.9% (27 of 550) for the best of 88 standard test-adequacy gates covering as many; on the 388 cells both admit, each has 3 failures, so the gap reflects which cells each admits (App. L.12; the pre-set comparison is inconclusive).
Q1, by arm at matched thresholds (exploratory; nine runs on the paper's two changes; the best admissible program re-selected per cell). From 19 of 26 up, repair's admitted programs fail least, and at 25-26 of 26 none of repair's 200 confident plans fails. At 24 of 26:
| Arm, development ≥ 24/26 | Changed cells admitted (of 18) | Failing / confident plans (each distinct program once) |
|---|---|---|
| Repair | 7 | 1.7% (4/240) |
| Rebuild, post-change evidence | 13 | 5.5% (33/600) |
| Rebuild, same archive | 2 | 15.0% (12/80) |
With 5 admitted repair programs at this threshold, repair against the post-change rebuild is not separable (permutation p = 0.54).
Q2 and W3, operating points (three backbones pooled):
| Operating point | Positives (of 51) | Reference false flags (of 51) | Control false flags (of 60) | Cost per detection |
|---|---|---|---|---|
| Auditor | 37 | 34 | 27 | USD 1.14 |
| Anchored filter (exploratory) | 28 | 15 | 18 | USD 1.51 |
| Detector or anchored filter (rule fixed before computing) | 30 | 16 | 21 | USD 3.88 |
Thank you for the detailed response. My concerns about the paper’s methodological novelty and the clarity of its core contribution have been only partially addressed, so I will maintain my original score.
Thank you for clarifying the remaining concerns.
The core contribution. The contribution is empirical: in our experiments, where an accepted revision fails depends on how it was made. On the same 1,391 task cells, rebuilds break rules the change left valid in 38.4% of cells, against 0.3% for repairs, whose failures concentrate at the changed rule.
What follows for checking. The tested checks have complementary blind spots, motivating separate checks of the requested change and preserved behaviour. On new changes, a sealed held-out panel admits plans of which 10.6% fail at the changed rule. Probes built from the change, combined with held-out transitions at that rule, admit no failures there (0 of 10,434). Failures at unchanged rules remain; comparison with the replaced program filters many of them, but can retain errors inherited from that program.
Does this clarify the contribution? If it changes your overall assessment, we would be grateful if you could update your review to reflect it.
Reply to Reviewer 8D98
Thank you for a careful review. We ran the matched-threshold test you proposed (Q1) and tested the findings on rule changes we did not choose (W2).
Q1, matched thresholds. Held to 24 of 26 on development, with the best admissible program re-selected per cell over nine runs, accepted programs still fail 4.2% of confident plans (59 of 1,409; frozen before computing), so a matched threshold does not remove the failure. By arm, repair's admitted programs fail least at every matched threshold from 19 of 26 up, and none of repair's 200 confident plans fails at 25-26 of 26; at 24 of 26, repair keeps a program in 38.9% of the changed cells (7 of 18; exploratory; table in the details below).
W1: novelty; remedies mirror standard practice. A sealed held-out set fails on new changes, so we now build that check from the change. We measured those practices against checks built from the change on 24 runs of two rule changes picked by a fixed rule and never run before (pre-registered; the boundary-value and mutation readings frozen before computing). Failing plans at the changed rule, of plans admitted: the gate as used, 11.4% (505 of 4,413); a sealed held-out panel, the standard ML practice, 10.6% (104 of 984); boundary-value tests built from the same change, then the comparison, 3.7% (197 of 5,329); mutation tests given the true edit keep no program; tests built from the change, none of 10,434. A check can reject a wrong revision only on a transition that separates it from the right one, the condition oracle-guided synthesis uses to choose among programs (Jha et al., 2010); on new changes the sealed panel separates none of the wrong rules that fit the reviser's evidence, while the change-built tests sit at the new threshold, where the revisers' wrong rules sit. Given to the reviser as training evidence, the same tests leave failure where it was (10.4% against 10.8%; pre-registered), so they work as an acceptance check, with no change to how the reviser learns. Observation 1 needs no independence assumption, and its condition occurs where agents are ranked (all 8 top systems ship one crash; a majority vote fixed in advance flags none of 16 surviving defects). On 24 fresh runs of the same changes, the change-built tests with the comparison and a rule that distrusts mispredicted actions leave a changed-rule failure in 1.4% of covered task cells, against 4.9% for the best of 88 standard test-adequacy gates covering as many (pre-registered; App. L.12; the pre-set comparison is inconclusive).
W2 and Q1: one learner, two hand-picked changes, a deterministic world. On the paper's two changes the reviser's evidence already ruled out every wrong rule within two grammar edits (of 2,339, only the true rule fits; frozen before computing), which is why the sealed panel reached zero there; on changes we did not pick it does not (10.6%, above). The failure itself persists: on the paper's changes a second reviser family fails on 51.0%; four changes drawn from a grammar give 34.0%; under sticky actions with replanning at every surprise, failures persist in 36.9% of covered task cells. On the same 1,391 task cells of fresh runs, rebuilds break a rule the change left valid in 38.4% of cells against 0.3% for repairs, and repairs' failures sit at the changed rule (15.6% of cells; all pre-registered). The revised paper's claim is this split; which strategy solves more tasks is a measured side result (Sec. 4.1).
Q2 and W3: discrimination, recall and cost. Running the detector alongside a filter that keeps an auditor counterexample only when the unpatched code passes it keeps 58.8%, 64.7% and 52.9% of the 17 positives per backbone (10, 11 and 9; detector alone: 6, 7 and 6), and removes 52.9% of the auditor's 34 false flags on reference patches, adding none (exact McNemar p = 0.000008; rule fixed before computing). The filter alone adds no model call to the auditor's, so no added API spend (its cost per detection rises from USD 1.14 to USD 1.51, as it keeps fewer positives), and removes 55.9% of those false flags (19 of 34) while keeping 54.9% of positives (28 of 51; exploratory). Against "zero true defects", five patches carry an executed regression and none of the 13 controls does; the maintainers' later releases keep the behaviour each of the 11 certified patches breaks.
What changed since your review. (1) Matched development thresholds by arm, and the same-cell split (Q1; Sec. 4.2, 4.3). (2) Rule changes drawn or picked by a fixed rule, a second reviser family, and stochastic execution with replanning (W2, Q1; App. L). (3) The held-out check is now built from the change and kept out of the reviser's evidence; the regression check against the replaced version stands (W1; Sec. 4.4, 8). (4) An operating point that keeps 58.8% of positives and removes 52.9% of reference false flags (Q2, W3; Sec. 6).
If these answers resolve your concerns, we would be grateful if you would reconsider your rating.