acceptodds
Under review as a conference paper at ICLR 2027

What Net Accuracy Hides: Auditing Repair and Breakage in Verify–Refine Workflows

Abstract

A small accuracy change can hide substantial changes in which problems an LLM workflow solves—and regressions may be attributed to refinement that never executed. We present an execution-path audit for verify–refine workflows that pairs baseline and workflow outcomes with the initial answer's source, verifier decisions, and calls admitted or refused by the budget. Across program synthesis and mathematics, these records distinguish answer regressions from the execution paths on which they occur. In a GPT-4o study, a code workflow changes accuracy by −3.3 percentage points (95% CI: [−8.7,+2.0]), while repairing 24.4% of baseline failures and breaking 13.8% of baseline successes. Under tighter budgets, 96 of 450 executions stop before any refinement runs: a wrong final answer in these runs cannot be attributed to an executed refinement. With equal refinement allowances and no blocked suffix calls, reusing a stored baseline output yields 5/347 breakages versus 27/347 for same-policy regeneration on one mathematics reference set. A second frozen reference set on the same tasks repeats the breakage ordering, but not the net-accuracy advantage. A serving control changes 8.2% of paired workflow outcomes under batching while net accuracy moves by only −0.22 percentage points. These findings support reporting repairs, breakages, and actual execution paths together, and evaluating preservation against the repairs it forgoes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, verify-refine workflows, evaluation methodology, regression and negative flips, verifier signal, inference budget, path-level audit, program synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JZAHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  1. The writing quality of the paper is very poor, making it difficult to read, and it has not reached the level required for acceptance at ICLR. 2. Because the paper is difficult to read and the method section is too brief, I find it difficult to identify what the task of this paper is and what its contributions are.
Critical strengths

None

Critical weaknesses
  1. The main structure of the paper is missing, and the overall quality does not meet the ICLR standard: the paper does not even include a Method section.
  2. The paper's writing logic is very poor and difficult to follow.
Questions for authors

See Weeknesses.

Reviewer_M4BCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies repairs and breakages in verify–refine workflows using paired outcomes, initial-answer provenance, verifier decisions, and budget records. Experiments compare retained and regenerated answers and examine serving effects. The contribution is an evaluation procedure and empirical findings, not a uniformly better refinement policy.

Critical strengths
  1. Table 30 identifies 45 breakages terminating before the first refinement executes, avoiding a misleading attribution to refinement. The analysis also correctly clusters repeated seeds by task.

  2. The separately funded drafting comparison addresses budget asymmetry. The second mathematics cohort shows that fewer breakages need not improve accuracy, and the paper candidly reports unsuccessful transfer and certification results.

Critical weaknesses
  1. A matched comparison with QualityFlow's protective acceptance [Q25] would clarify the practical lesson. Appendix W's replay does not cover the equal-suffix-budget study supporting the strongest provenance result. Please also explain what execution-path attribution adds to SCoRe's correction/preservation analysis [SC25], and compare whole-answer regeneration here with the semantic drift studied in ORLoopBench [O26]. The latter uses solver feedback and a different correctness criterion, so it is a comparison of mechanisms rather than raw scores.

  2. Reproducibility limits are scattered across the paper. Overwritten drafts are unavailable, and some regrading and ledger replays require omitted inputs. Table 19 also shows sensitivity to ledger overruns in mathematics. The authors acknowledge these issues; a concise mapping from main results to available records would make independent checking easier.

I lean toward acceptance for the execution-path findings and careful qualifications, rather than a general advantage of stored assignment.

Questions for authors
  1. Can the reversion rule be replayed on Table 3's budget-parity records? Relative to [Q25, SC25, O26], please distinguish incumbent preservation, verifier information, feedback, and budget refusal. Existing analyses and discussion would suffice.

  2. Which retained files reproduce Tables 1–3 and Table 30, and which checks require omitted text or other inputs?

[Q25] Hu, Y., Zhou, Q., Chen, Q., Li, X., Liu, L., Zhang, D., Kachroo, A., Oz, T., and Tripp, O. (2025). QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks. arXiv:2501.17167. https://arxiv.org/abs/2501.17167.

[SC25] Kumar, A., Zhuang, V., Agarwal, R., et al. (2025). Training Language Models to Self-Correct via Reinforcement Learning. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/871ac99fdc5282d0301934d23945ebaa-Abstract-Conference.html.

[O26] Ao, R., Simchi-Levi, D., and Wang, X. (2026). ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research. arXiv:2601.21008. https://arxiv.org/abs/2601.21008.

Reviewer_FL2iOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how net accuracy can hide both successful repairs and regressions in LLM verify–refine workflows. The authors propose an execution-path audit that tracks repair and breakage rates, verifier decisions, budget constraints, and the sources of initial answers. Experiments on code generation and mathematical reasoning tasks reveal that workflows can repair incorrect answers while breaking previously correct ones, and some failures occur without refinement being executed. The results further show that reusing stored answers can reduce regressions compared with regenerating them, but does not consistently improve overall accuracy.

Critical strengths
  1. The paper addresses a meaningful problem in verify–refine workflows: improvements in overall accuracy can hide regressions on previously correct tasks. Understanding this trade-off is important for evaluating the reliability of LLM-based workflows.

  2. The paper goes beyond repair and breakage rates by examining verifier decisions, budget constraints, and actual execution paths. This helps distinguish failures caused by refinement from those occurring before refinement is executed.

Critical weaknesses
  1. Previous studies have already investigated repair–breakage trade-offs, negative flips, and regressions in self-correction workflows. This paper mainly extends these analyses by incorporating execution paths and budget records. While useful, the conceptual contribution remains largely incremental.

  2. Many findings, such as budget constraints preventing refinement and reusing stored answers reducing certain regressions, are relatively intuitive or follow directly from the workflow design. The paper does not establish sufficiently new insights or demonstrate how its analysis can systematically improve workflow reliability.

Questions for authors
  1. Previous studies, such as SCoRe [1] and QualityFlow [2], have already investigated self-correction, repair–breakage trade-offs, and verifier-guided refinement. I would like the authors to further clarify the main contributions of this work. What new insights does the proposed execution-path audit provide beyond existing analyses of these phenomena?

  2. While the paper identifies different sources of breakage, it remains unclear how these findings can guide the design of more reliable verify–refine workflows. Could the authors demonstrate whether using execution-path diagnostics leads to measurable improvements in workflow performance or reliability?

  3. The experiments primarily focus on short-form code generation and mathematical reasoning. How well do the identified failure patterns generalize to more complex agent tasks involving long-horizon execution, multiple tool interactions, and changing environment states?

References

[1] Kumar, A., et al. Training Language Models to Self-Correct via Reinforcement Learning (SCoRe). ICLR 2025. https://arxiv.org/pdf/2409.12917

[2] Hu, Y., et al. QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks. arXiv:2501.17167, 2025. https://arxiv.org/abs/2501.17167

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.