- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies iterative artifact correction under partial verification. Across eight models and three domains—audio generation, SVG repair, and API workflows—it evaluates eight deterministic checks while exposing only a subset to the model. Coverage conditions share the same initial draft, and the main loop stops when all exposed checks pass or after five revisions. Additional experiments examine fixed revision horizons, generic failure notices, full diagnostic feedback, reasoning, and model-selected checks. The main finding is that repairs concentrate on reported defects, particularly in audio and SVG, and that additional revisions or generic failure notices do not fully recover the benefit of detailed feedback in the tested workflow.
Critical strengths
-
The experimental controls provide credible evidence for repair locality within the stated setting: Shared initial drafts and evaluation of all checks strengthen the comparisons. Fixed-horizon controls address premature stopping, while GS-versus-neutral and GS-versus-full-feedback comparisons distinguish continuation, knowledge of remaining errors, and detailed diagnosis. Directive-removal, report-length, mask-redraw, and within-check analyses address further alternatives. The findings therefore go beyond simply observing that a partial verifier can miss errors.
-
The study reports useful domain differences and makes its boundaries explicit: Audio and SVG extend the investigation beyond conventional code repair. The authors separately analyze API completeness and provide probe diagnostics rather than treating API results as identical corroboration. They also disclose the round-0 contribution to false completion, define K within each domain's check set, and explicitly restrict the conclusion to settings where models cannot run the checks themselves.
Critical weaknesses
-
Conceptual novelty is limited: The importance of external feedback and the risks of incomplete verification are established. The paper’s value lies in its controlled measurements; the discussion could better explain which design decisions they inform, particularly when generic feedback suffices versus when detailed diagnosis matters, in relation to Stechly et al.’s findings.
-
Some interpretations need qualification: Generic-feedback gains vary substantially, notably for Sonnet-5/SVG. API analyses support poor error localization as a plausible explanation, but do not isolate it from task complexity or observation access; these limits do not invalidate the main comparisons.
Questions for authors
-
Contribution: Which finding most materially changes an existing expectation or design decision beyond prior external-feedback and test-overfitting results? In particular, what should a reader learn from the contrast with Stechly et al. about when detailed diagnosis is valuable?
-
Heterogeneity: How should the substantial Sonnet-5/SVG response in the matched-continuation control be reconciled with the summary that generic feedback helps only a little? Can the discussion distinguish the pooled result from practically important model/domain exceptions, without treating exploratory cell estimates as definitive subgroup effects?
-
API interpretation: Which of the existing check-specific, probe, replay, or documentation analyses most directly supports error localization as the explanation for API repair difficulty, rather than other differences in task complexity or observation access? If these factors remain unresolved, would the authors frame localization as a plausible hypothesis? A targeted comparison could be useful future work, but I am not requiring a new experimental suite to answer this clarification.
3 comments
Thank you for the response and additional experiments. The matched naming comparison and revised discussion address my main concerns about the design implications, model/domain heterogeneity, and API interpretation. I maintain my weak accept recommendation.
Thank you for the follow-up and for your support of the paper. We are glad that the matched naming comparison and the revised discussion address your concerns.
Thank you for the careful reading and the three questions.
When is detailed diagnosis valuable? (Question 1, Weakness 1)
Our data show that naming the failing checks is worth far more than knowing that an error remains. At a false-clear state, the exposed checks pass but a hidden check fails. In the paper, the "not fully correct" (GS) message repairs 18.1% of hidden defects by the end of the run, against 9.1% for a neutral offer of another round. At 790 of these states we added a third message, GS plus the failing checks with one counterexample each, using seven models since Google retired Gemini-3.5-Flash on 8 October 2026. By the end of the run, naming raises repair from 19.9% to 61.6% (+41.7 points [37.7, 45.6]) and runs ending fully correct from 10.5% to 48.6%, with gains in every domain and for every model (Sec. 6.3, Appendix C).
This holds even under a sound acceptance gate, the setting in which Stechly et al. (2025) found critique detail to matter little. The GS and named branches above both accept only when all eight checks pass. A candidate explanation is that our artifacts have eight separately checked properties, so a revision must first find the failing one. For design, our data argue for checking broadly before accepting, since 668 of 888 false completions at K=2 happen at round 0, and for reporting failures to the model instead of only gating on them.
How to reconcile the Sonnet-5/SVG response? (Question 2)
The pooled effect is small, and a few model-domain pairs are large exceptions. Over all 904 states in the paper, the GS message adds 9.0 points [7.0, 11.2] of hidden-defect repair by the end of the run; for Sonnet-5 on SVG it adds 55.6 [41.4, 69.5], the largest of the 24 pairs, and for Gemini-3.5 on SVG 29.4. Sonnet-5's fully correct SVG rate is 20% at K=2, 76% under the GS policy and 96% at K=8. Naming still adds even where the generic message works best: in the matched naming comparison, Sonnet-5's one-step repair of hidden SVG defects rises from 44% with GS to 70% with the failing checks named (Appendix C). Sec. 6.2 now states the pooled effect and these exceptions together, presenting the exceptions as observed differences between model-domain pairs.
What supports error localization in API workflows? (Question 3)
We agree, and the revised Sec. 5.3 frames poor error localization as a plausible hypothesis. The most direct support is the check-specific comparison. The outcome check reports only that a result is wrong, and when exposed it is repaired in 5% of trajectories, against 42–100% for checks that point to a specific API request (Sec. 5.3). Naming the failing checks also helps much less in API workflows than in audio or SVG (+4.8 versus +26.9 and +33.9 points). Observation matters as well. In a separate five-round experiment with all checks reported, probes raised Qwen3.8-Max's fully correct runs on ten hard tasks from 1 of 20 to 10 of 20 (Appendix D). In the probe experiment of Sec. 7.3, GPT-5.4 and DeepSeek-V4-Pro revised all 50 API-workflow tasks for ten rounds with only two checks reported. There, probes fixed mostly argument-format and call-order errors, and in drafts that contain a program only 2 of 75 outcome-check failures. Separating localization from task complexity needs a targeted comparison, as Sec. 5.3 now states.
Brackets are 95% bootstrap intervals over tasks.