acceptodds
Under review as a conference paper at ICLR 2027

What You Don't Check, Models Don't Fix: False Completion under Partial Verification

Abstract

Iterative correction loops revise LLM outputs until their checks pass, and self-improving pipelines favor changes that score better on theirs. However, the checks rarely cover every way an output can be wrong, so an output accepted as correct may well contain defects that the checks missed. We ask what happens to these defects. We study a correction loop on three task domains (music-as-code audio, SVG repair, and API workflows) with eight proprietary and open-weight models. Each task has eight deterministic checks. All eight checks are scored in every round, but the model sees only some of them (the exposed checks; the rest are hidden), and the loop stops when the exposed checks pass. Because the hidden checks are also scored, each run shows which defects the loop repairs and which it leaves behind when it stops. First, models fix what they are shown: exposed defects are repaired far more often than hidden ones, and at low coverage most of the loop's stops leave a hidden defect, usually before any revision. Second, stopping too early is not the main reason these defects remain: extra rounds recover little, and telling the model that its output is still wrong helps, but far less than naming the failing checks. Third, models can compensate only partly: reasoning reduces the effect of coverage without removing it, and letting models choose which checks they see does not detectably reduce the remaining defects. What you don't check, models don't fix: a caution for any correction loop trusting partial verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
partial verification, false completion, iterative correction, self-correction, recursive self-improvement, LLM evaluation, execution feedback, verification coverage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UhFkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies iterative artifact correction under partial verification. Across eight models and three domains—audio generation, SVG repair, and API workflows—it evaluates eight deterministic checks while exposing only a subset to the model. Coverage conditions share the same initial draft, and the main loop stops when all exposed checks pass or after five revisions. Additional experiments examine fixed revision horizons, generic failure notices, full diagnostic feedback, reasoning, and model-selected checks. The main finding is that repairs concentrate on reported defects, particularly in audio and SVG, and that additional revisions or generic failure notices do not fully recover the benefit of detailed feedback in the tested workflow.

Critical strengths
  1. The experimental controls provide credible evidence for repair locality within the stated setting: Shared initial drafts and evaluation of all checks strengthen the comparisons. Fixed-horizon controls address premature stopping, while GS-versus-neutral and GS-versus-full-feedback comparisons distinguish continuation, knowledge of remaining errors, and detailed diagnosis. Directive-removal, report-length, mask-redraw, and within-check analyses address further alternatives. The findings therefore go beyond simply observing that a partial verifier can miss errors.

  2. The study reports useful domain differences and makes its boundaries explicit: Audio and SVG extend the investigation beyond conventional code repair. The authors separately analyze API completeness and provide probe diagnostics rather than treating API results as identical corroboration. They also disclose the round-0 contribution to false completion, define K within each domain's check set, and explicitly restrict the conclusion to settings where models cannot run the checks themselves.

Critical weaknesses
  • Conceptual novelty is limited: The importance of external feedback and the risks of incomplete verification are established. The paper’s value lies in its controlled measurements; the discussion could better explain which design decisions they inform, particularly when generic feedback suffices versus when detailed diagnosis matters, in relation to Stechly et al.’s findings.

  • Some interpretations need qualification: Generic-feedback gains vary substantially, notably for Sonnet-5/SVG. API analyses support poor error localization as a plausible explanation, but do not isolate it from task complexity or observation access; these limits do not invalidate the main comparisons.

Questions for authors
  1. Contribution: Which finding most materially changes an existing expectation or design decision beyond prior external-feedback and test-overfitting results? In particular, what should a reader learn from the contrast with Stechly et al. about when detailed diagnosis is valuable?

  2. Heterogeneity: How should the substantial Sonnet-5/SVG response in the matched-continuation control be reconciled with the summary that generic feedback helps only a little? Can the discussion distinguish the pooled result from practically important model/domain exceptions, without treating exploratory cell estimates as definitive subgroup effects?

  3. API interpretation: Which of the existing check-specific, probe, replay, or documentation analyses most directly supports error localization as the explanation for API repair difficulty, rather than other differences in task complexity or observation access? If these factors remain unresolved, would the authors frame localization as a plausible hypothesis? A targeted comparison could be useful future work, but I am not requiring a new experimental suite to answer this clarification.

3 comments

Thank you for the careful reading and the three questions.

When is detailed diagnosis valuable? (Question 1, Weakness 1)

Our data show that naming the failing checks is worth far more than knowing that an error remains. At a false-clear state, the exposed checks pass but a hidden check fails. In the paper, the "not fully correct" (GS) message repairs 18.1% of hidden defects by the end of the run, against 9.1% for a neutral offer of another round. At 790 of these states we added a third message, GS plus the failing checks with one counterexample each, using seven models since Google retired Gemini-3.5-Flash on 8 October 2026. By the end of the run, naming raises repair from 19.9% to 61.6% (+41.7 points [37.7, 45.6]) and runs ending fully correct from 10.5% to 48.6%, with gains in every domain and for every model (Sec. 6.3, Appendix C).

This holds even under a sound acceptance gate, the setting in which Stechly et al. (2025) found critique detail to matter little. The GS and named branches above both accept only when all eight checks pass. A candidate explanation is that our artifacts have eight separately checked properties, so a revision must first find the failing one. For design, our data argue for checking broadly before accepting, since 668 of 888 false completions at K=2 happen at round 0, and for reporting failures to the model instead of only gating on them.

How to reconcile the Sonnet-5/SVG response? (Question 2)

The pooled effect is small, and a few model-domain pairs are large exceptions. Over all 904 states in the paper, the GS message adds 9.0 points [7.0, 11.2] of hidden-defect repair by the end of the run; for Sonnet-5 on SVG it adds 55.6 [41.4, 69.5], the largest of the 24 pairs, and for Gemini-3.5 on SVG 29.4. Sonnet-5's fully correct SVG rate is 20% at K=2, 76% under the GS policy and 96% at K=8. Naming still adds even where the generic message works best: in the matched naming comparison, Sonnet-5's one-step repair of hidden SVG defects rises from 44% with GS to 70% with the failing checks named (Appendix C). Sec. 6.2 now states the pooled effect and these exceptions together, presenting the exceptions as observed differences between model-domain pairs.

What supports error localization in API workflows? (Question 3)

We agree, and the revised Sec. 5.3 frames poor error localization as a plausible hypothesis. The most direct support is the check-specific comparison. The outcome check reports only that a result is wrong, and when exposed it is repaired in 5% of trajectories, against 42–100% for checks that point to a specific API request (Sec. 5.3). Naming the failing checks also helps much less in API workflows than in audio or SVG (+4.8 versus +26.9 and +33.9 points). Observation matters as well. In a separate five-round experiment with all checks reported, probes raised Qwen3.8-Max's fully correct runs on ten hard tasks from 1 of 20 to 10 of 20 (Appendix D). In the probe experiment of Sec. 7.3, GPT-5.4 and DeepSeek-V4-Pro revised all 50 API-workflow tasks for ten rounds with only two checks reported. There, probes fixed mostly argument-format and call-order errors, and in drafts that contain a program only 2 of 75 outcome-check failures. Separating localization from task complexity needs a targeted comparison, as Sec. 5.3 now states.

Brackets are 95% bootstrap intervals over tasks.

Thank you for the response and additional experiments. The matched naming comparison and revised discussion address my main concerns about the design implications, model/domain heterogeneity, and API interpretation. I maintain my weak accept recommendation.

Thank you for the follow-up and for your support of the paper. We are glad that the matched naming comparison and the revised discussion address your concerns.

Reviewer_kvfCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies iterative correction under partial verification across eight language models and three task domains. It varies the checks shown to the model and examines hidden defects, stopping rules, feedback content, reasoning, and model-selected checks.

Critical strengths
  • Shared initial drafts and controls for check selection, feedback wording, and report length support the finding that repair concentrates on reported failures. These controls strengthen the credibility of the main comparisons.
  • Matched continuation and fixed-draft reasoning experiments measure the contributions of failure notices and reasoning during repair. They provide useful evidence about which interventions help recover hidden defects.
Critical weaknesses
  • The main results largely quantify expected effects of incomplete verification and feedback dependence. The additional studies provide limited new explanation of repair behavior or guidance for designing better correction loops.
  • The stopping analysis compares exactly three revisions with runs that can continue for five. This mixes stopping rules with revision budgets and weakens the estimate of the effect of early stopping.
Questions for authors

Could you recompute the stopping effect using a common revision horizon and report changes in hidden-defect repair and residual defects?

3 comments

Thank you for the careful review. The revised PDF contains every change described below.

1. Stopping early versus revising longer (Weakness 2, Question)

At a common horizon, stopping at the first all-clear lowers hidden-defect repair but leaves about as many remaining defects as revising on. We recomputed the stopping effect from the check results stored for every round, varying one factor at a time. For the stopping rule, we stop each three-round run at its first all-clear, as the main loop does; the prompts are identical up to that point. For the round limit, we compare the main-grid runs after three and after five rounds. The table covers 8 models, 30 tasks and the 303 hidden defects in their first drafts (Table 15).

Change at K=2Hidden defects repairedRemaining defects per task
Stopping rule: stop at all-clear → revise on (3 rounds)11.6% → 19.5% (+7.9 points [4.2, 11.7])+0.00 [−0.07, +0.07]
Round limit: 3 → 5 rounds (stop at all-clear)11.6% → 11.6%, every model−0.02 [−0.04, −0.004]

At the same three-round limit, revising on after an all-clear repairs 24 hidden defects but breaks 24 passing checks, so the number of remaining defects does not change (Sec. 6.1). At K=7, the missing rounds 4 and 5 account for +0.121 [0.067, 0.179] of the +0.204 [0.100, 0.329] extra remaining defects per task in the original comparison, and the stopping rule for +0.037 [0.004, 0.079].

A new ten-round control shows the same pattern at longer horizons (Sec. 6.1, Appendix C). Seven models revised the same 30 tasks from their first drafts for ten rounds with no early stop, and we apply the all-clear stop within these runs at horizons 3, 5 and 10. Google retired Gemini-3.5-Flash on 8 October 2026, so this control uses the other seven models.

K=2, revise on minus stop at the first all-clearHorizon 3Horizon 5Horizon 10
Hidden defects repaired (points)+10.1 [6.6, 13.6]+10.5 [7.1, 13.7]+14.1 [10.3, 17.7]
Remaining defects per task−0.02 [−0.10, +0.04]−0.02 [−0.11, +0.09]−0.05 [−0.13, +0.01]

2. What the paper shows about repair (Weakness 1)

Reports increase repair mainly for the defects they name. Because every check is scored while the model sees only some, we can see which defects a report helps and which it leaves behind. Starting from the same drafts and revising for three rounds, we compare the K=2 report with no report at all (Sec. 5.1, Table 15):

From the same drafts, 3 roundsK=2 reportNo report (K=0)Difference
Defects shown in the K=2 report (115)57.4%15.7%+41.7 points [26.0, 57.1]
Defects the report left out (303)19.5%20.5%−1.0 points [−5.7, +4.3]

At matched states, naming the failing checks adds far more than the "not fully correct" message alone (Sec. 6.3, Fig. 3d). From 790 first false-clear states, where the exposed checks pass but a hidden check fails, paired branches received "The artifact is not fully correct yet. Please continue revising."; one branch also received the failing checks with one counterexample each. Both branches accept only when all eight checks pass. Naming raises one-step repair of the 1,258 hidden defects from 10.7% to 34.9% (+24.2 points [21.3, 27.2]) and runs ending fully correct from 10.5% to 48.6%.

Sec. 8 draws three practices for correction loops from these results:

  1. Check broadly before accepting. At K=2, 668 of the 888 false completions happen at round 0, because the first draft already passes the two shown checks; checking all eight properties at that point would reject these drafts.
  2. Report the failing checks to the model instead of only using them to decide when to stop. Under the same acceptance rule, K=8 tells the model which checks fail and why, and repairs more hidden defects than the global-stop policy, with intervals above zero in 20 of 24 model–domain comparisons. Stechly et al. (2025) found that critique detail mattered little in several settings, with large differences across domains; in ours, report content matters in most model–domain comparisons.
  3. Judge the cost of coverage per fully correct artifact rather than per run. K=8 uses 2.6 times the model tokens of K=2 per run but 35% fewer per fully correct artifact. Our harness runs all eight checks every round at every K, so the number of check runs depends only on the number of rounds.

Brackets are 95% confidence intervals from resampling tasks.

Thank you for the detailed response and additional experiments. The common-horizon analysis resolves my concern about confounding stopping rules with revision budgets, and the matched naming comparison strengthens the empirical contribution. While the conceptual novelty remains limited, these additions address my main concerns, and I am updating my recommendation to weak accept.

Thank you for the follow-up and for updating your recommendation to weak accept. We are glad that our additional analyses and experiments address your main concerns. Your question about a common revision horizon also improved the paper: Sec. 6.1 and Appendix C now measure the effect of early stopping at common horizons of 3, 5 and 10 rounds.

Reviewer_FVJ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates LLM correction loops under partial verification by scoring models on all possible defects while exposing only a subset. Across audio, SVG repair, and API workflows, eight language models exhibit "repair locality," fixing the defects they are explicitly shown while largely ignoring hidden ones. At low verification coverage, loops frequently trigger "false completion," stopping early because exposed checks pass despite lingering hidden failures. The study demonstrates that providing guaranteed extra rounds or generic warnings yields minimal improvement compared to explicitly naming failing checks, and while reasoning helps, it cannot fully bridge the coverage gap.

Critical strengths

Isolates Information Over Opportunity: By forcing a fixed three-round horizon without early stopping, the authors prove that models leave bugs unfixed primarily because they lack specific failure information, not just because the loop cuts them off.

Refutes Prior "Blind Repair" Claims: The findings specifically push back against recent literature suggesting that content-free failure notices or generic "try again" prompts are nearly as effective as detailed critiques.

Controls for Defect Obscurity: Using randomized masks on small, fixed-length artifacts ensures that failures stem from overfitting to visible tests, rather than the model losing a tiny bug within a massive context window.

Critical weaknesses

No Interactive Execution Environment: The evaluation restricts models from actively running code, rendering outputs, or writing their own test suites. An agent with access to an interactive terminal and a larger round budget might successfully recover far more hidden defects on its own. (The authors also state this in the paper, but i still think it is a weakness)

Questions for authors

Interactive Environments and Round Budgets (Question on method, details)

While the fixed-horizon control isolates the effect of missing information within a strict three-round limit, does the overall five-round horizon artificially cap the potential of blind repair? If models were given an interactive execution environment to actively run their own tests and a significantly larger budget, would they successfully recover hidden defects without explicit hints?

Architectural Constraints in Self-Selection (Question on method, details)

The self-selection study isolates test-picking into a stateless, single-line subagent call with reasoning disabled. Does preventing the model from deliberating in a multi-turn, reasoning-enabled context artificially handicap its ability to formulate a cohesive, long-term testing strategy?

Scientific Contribution vs. Triviality (Question on significance, not so important, just curious)

Given that explicit feedback intuitively outperforms generic hints, how do these findings on short, controlled artifacts scale to massive, real-world repositories? Does the repair locality effect hold when the sheer volume of context makes locating the defect computationally expensive, even with explicit feedback?

1 comment

Thank you for the review and the three questions.

Does the round budget cap blind repair? (Question 1)

A larger budget raises blind repair only slowly. We reran the 30 tasks of the three-domain fixed-horizon control (Sec. 6.1) from the same first drafts for ten new rounds with no early stop, so that all rounds come from the same model in the same period. Google retired Gemini-3.5-Flash on 8 October 2026, so the new runs use seven models. At K=2, hidden-defect repair rises from 22.0% at round 3 to 27.8% at round 10 (+5.8 points [1.8, 10.7]), and by less than one point per round after round 8, while rounds 6 to 10 use about two thirds of the model tokens. With no report at all, the same defects reach 28.5% by round 10, so the K=2 report shows no advantage for them. Naming the failing checks helps much more: at 790 states where both exposed checks pass but a hidden check fails, adding the failing checks and one counterexample for each to the "not fully correct" message raises repair one revision later from 10.7% to 34.9%.

Would running code recover hidden defects? (Weakness)

Running code recovers some hidden defects, but most remain. We tested this on all 50 API-workflow tasks at K=2: GPT-5.4 and DeepSeek-V4-Pro revised for ten rounds, with or without up to two probe programs run in a sandbox copy of the task world. Hidden-defect repair rose from 6.6% to 21.3% (+14.8 points [8.9, 21.6]), but only 3 of 100 runs with probes became fully correct. The repairs were mostly argument-format and call-order errors, and in drafts that contain a program the completeness check was repaired in 2 of 75 cases. In a separate five-round experiment with all eight checks reported, probes raised Qwen3.8-Max's fully correct runs on ten hard tasks from 1 of 20 to 10 of 20 (Appendix D). API workflows are where the artifact is itself an executable program, and where repair is hardest even when every check is reported (Sec. 5.3). Appendix G lists rendering access for SVG and audio as future work.

Does the stateless, reasoning-off selection call handicap testing strategy? (Question 2)

In audio and API workflows, our experiments point to repair rather than check selection as the limit. There, models picked failing checks more often than random choice did, yet repair of the draft's defects stayed within about three points of the random arm (Sec. 7.2). In SVG, models picked render-required checks less often than random choice did (30% versus 54% of picks), although these checks hold 86% of SVG round-0 defects. The selection call sees the task, the current artifact, the exposed checks with their results and the list of checks not yet exposed (Appendix A); Sec. 7.2 now states this context explicitly. What it lacks is the earlier revisions and reasoning. Reasoning narrows the coverage gap without closing it: when the same drafts are revised for three rounds with reasoning on in every revision call, residual defects fall in every domain and for every model, and the gap between K=2 and K=8 narrows from 0.56 to 0.31 defects per task (Sec. 7.1).

Does repair locality hold in large repositories? (Question 3)

Beyond showing that explicit feedback outperforms generic hints, the paper shows which defects a report helps. From the same drafts, defects shown in the K=2 report are repaired 57.4% of the time against 15.7% with no report (+41.7 points [26.0, 57.1]), while defects it leaves out are repaired about 20% of the time either way (−1.0 points [−5.7, +4.3]; Table 15). API workflows are our closest tested analogy: there, hidden defects stay largely unrepaired, but the gap to exposed defects shrinks (Sec. 5.3). Their main check reports which outcome is wrong but not which step of a long program caused it, and even when exposed it is repaired in only 5% of trajectories. Naming the failing checks with counterexamples adds 4.8 points of one-step repair there, against 26.9 in audio and 33.9 in SVG. On SWE-bench, patches that pass generated tests often fail hidden reference tests (Ahmed et al., 2026). Appendix G now states the scale of the artifacts we tested.

The revised paper adds the ten-round control (Sec. 6.1, Appendix C), the matched naming comparison (Sec. 6.3, Fig. 3d) and the probe experiment (Sec. 7.3, Appendix D). Brackets are 95% bootstrap intervals over tasks, or over API task families for the probe experiment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for their careful reviews. The revised PDF addresses their questions in the sections listed below.

New results

  • Ten-round control (Sec. 6.1; Appendix C, Table 16, Fig. 7): later rounds raise hidden-defect repair only slowly, and stopping at the first all-clear forgoes some hidden repairs but leaves about as many defects at horizons of 3, 5 and 10 rounds (kvfC, FVJ5).
  • Matched naming comparison (Sec. 6.3, Fig. 3d; Appendix C, Tables 21 and 22): at the same false-clear states and under the same acceptance rule, naming the failing checks with one counterexample each raises one-step repair of hidden defects from 11% to 35% (kvfC, UhFk, FVJ5).
  • Probe experiment (Sec. 7.3; Appendix D, Table 29): with two checks reported, running probe programs on all 50 API-workflow tasks recovers some hidden defects but leaves most unrepaired (FVJ5).

Analyses and text

  • Table 15 separates the stopping rule from the round limit and adds K=7, and Sec. 6.1 states the trade-off between repairs and new failures (kvfC).
  • Sec. 5.1 compares the K=2 report with no report on the same drafts (kvfC).
  • Sec. 6.2 states the pooled effect of the "not fully correct" message together with the Sonnet-5 and Gemini-3.5 exceptions in SVG (UhFk).
  • Sec. 5.3 presents poor error localization in API workflows as a plausible explanation (UhFk).
  • Sec. 7.2 states what the selection call sees (FVJ5).
  • Sec. 8 adds design practices for correction loops (kvfC, UhFk), and Appendix G states the scale of the tested artifacts (FVJ5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.