acceptodds
Under review as a conference paper at ICLR 2027

WHERE DOES THE FAULT LIE? TRACEABLE REPAIR FOR CODING AGENTS BEYOND PASS/FAIL

Abstract

Passing tests does not reveal whether a coding agent found the fault, kept edits focused, or avoided repair loops. Across eight models on three external CLI benchmarks, native outcome scores correlate weakly with process proxies from trajectory logs. We introduce TracebenchCLI, a benchmark of 126 CLI repair tasks derived from Codeforces problems. Staged tests and known fault spans support direct measurement of fault attribution, edit locality, progress, wasted turns, and diagnosis. We formalize what the terminal label leaves unresolved through a repair state comprising stage, fault, and accumulated unnecessary work. An information decomposition motivates intervention through action selection. Our controller, **ARC** (Accountable Repair Control), reweights a frozen model's candidates using visible trajectory evidence and requires no training. With fixed settings across five base models, **ARC** raises the Qwen3-32B solve rate from to , reduces wasted turns by up to , and increases explicit diagnosis by up to . Under matched sampling budgets on Qwen2.5-Coder-14B, it solves 10 of 126 tasks against 3 for the base. On three external CLI benchmarks without oracle repair spans, the same controller improves native outcomes and process proxies. In an independent evaluation of 30 pairs matched on terminal outcome, non-author raters prefer **ARC** on 19 pairs and the base on 4, with 7 ties. Code: https://anonymous.4open.science/r/Tracebench-cli/https://anonymous.4open.science/r/Tracebench-cli/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Coding agent, Accountablity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5Ej5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARC (Accountable Repair Control), a train-free technique to improve model's solve rate across CLI tasks. The extensive experimental results show that ARC lifts the Qwen3-32B solve rate from 21.4% to 34.1%.

Critical strengths
  1. The problem solved in this paper is very important
  2. This paper tries to solve the problem with train-free techniques
  3. The experimental results shows that ARC improves the model's solve rate from 21.4% to 34.1%
Critical weaknesses
  1. ARC improves the Qwen3-32B solve rate from 21.4% to 34.1%. However, ARC cannot lifts the GPT-5.4-nano solve rate as shown in Figure 5. Could you explain it?
  2. The models used in this paper are not up to date. It looks like this paper has been rejected for several times, maybe I am wrong
  3. I cannot open the code link: https://anonymous.4open.science/r/Tracebench-cli/ Could you check it?
Questions for authors

I do not have any questions for the authors, since I also have a similar idea and I think the solution proposed in this paper is workable. However, the limitation for this paper is that if the model is not powerful enough, the solution proposed in this paper does not have marginal improvement.

Reviewer_uLe6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the quality of coding-agent repair trajectories beyond whether the final program passes its tests. It introduces TraceBench-CLI, a 126-task benchmark with staged command-line interactions and annotated fault regions, and measures fault attribution, edit locality, progress, and unproductive actions. ARC is a training-free controller that reweights candidate actions using diagnostic information, verification, edit spread, and repetition penalties. The evaluation combines full-trajectory outcomes with equal-candidate-budget baselines, execution-based patch audits, same-state branch replay, external CLI benchmarks, and blinded preferences from non-author raters. The central contribution is therefore both an evaluation framework for repair processes and a controller intended to improve those processes.

Critical strengths
  1. The evaluation target is important, and the paper does more than introduce another final-success benchmark. Two agents can reach the same terminal outcome while differing substantially in how accurately they identify the fault and how much unnecessary work they perform. The annotated environment makes these differences measurable. The necessary-edit audit in Appendix C.1 is especially valuable: it explicitly checks whether correct repairs require changes outside the original fault region, rather than treating the initial annotation as unquestionable ground truth. This makes the benchmark a useful vehicle for studying what process metrics capture and where they fail.

  2. There is substantive evidence that ARC can improve outcomes, not just its own process scores. In the matched-candidate comparison on 126 tasks, ARC solves 10 tasks, compared with 3 for the base selector, 3 for best-of-N, and 2 for heuristic reranking. Matching the proposal and auxiliary-generation budget addresses an important confound in controller evaluations. The external CLI results also include native task outcomes. Across three non-author raters, majority preferences favor ARC in 22 of 30 outcome-matched trajectory pairs, versus 3 for base and 5 ties, with reported Fleiss kappa of .54. This independent preference evidence is a meaningful complement to the constructed metrics.

Critical weaknesses
  1. The independent checks leave the interpretation of the traceability improvements unresolved. Within the ten-patch audit in Table C.5, four patches require edits outside the original fault spans, and adjusted diff precision is .900 for both conditions. This limits what original-span locality alone establishes about those audited successful patches. The manuscript should identify which paired model/condition runs contribute this audit before its result is generalized to the other successful repairs. It does not negate the separate task-outcome or blinded-preference evidence. The same-state branch replay in Table D.10 raises a related issue about the controller: ARC has higher regret than heuristic reranking (133/400 versus 102/400), higher waste (34.2% versus 31.8%), and no stage-advancing selection in the 11 eligible pools, compared with one for the heuristic. The stage-advance comparison has only 11 eligible pools; regret and waste summarize the 400-state replay. These one-step comparisons do not overturn the separate full-trajectory gains. However, the results do make the mechanism less clear: improvements in aggregate process scores and trajectory preferences coexist with local decision measures that do not consistently favor ARC. For the paper's main claim, it matters which dimensions represent independently useful repair behavior and how the utility function produces that behavior.

  2. The magnitude and cost of the benefit on natural repair tasks remain weakly characterized. The paper already includes external transfer and a check on 12 naturally occurring multi-file bugs, so the concern is the strength of that evidence, not its absence. The natural-bug success change is 3/12 to 4/12, which is too small a sample to establish the distribution of gains across repository issues. The API-cost pilot in Table D.8 reports completed-run averages and excludes failed attempts. Appendix D.13 already gives completion counts: GPT-5.4 with heuristic reranking completes 7 of 9 attempts, while the other rows complete 10 of 10. The missing accounting is therefore the cost of failed attempts, rather than the completion counts themselves. Its favorable cost comparison with other multi-candidate selectors does not establish a cost advantage over a single-proposal base; for the GPT-5.4 pilot, the reported relative costs imply that ARC is substantially more expensive than that base. The equal-candidate-budget experiment is useful, but the deployment question requires task success and total attempted-run cost to be considered together.

My main reservation is whether the measured traceability gains identify a consistent improvement in repair quality at a clearly characterized cost. The current audits make that conclusion less settled than the aggregate improvements suggest.

Questions for authors
  1. Table D.10 measures one-step outcomes on a fixed replay distribution. Which existing evidence explains ARC's higher replay regret and waste than heuristic reranking, alongside its full-trajectory gains? Is the difference attributable to the replay-state distribution, longer-horizon effects, or a mismatch between the local oracle ranking and ARC's objective?

  2. Which paired model/condition runs contribute the ten passing patches in Table C.5, and how does that audit population relate to the 126-task comparisons consolidated in Table D.11? This mapping would determine how broadly the equal adjusted precision can be interpreted.

  3. Using the attempted/completed counts already reported in Appendix D.13, can you add total API cost including failed attempts, keeping single-proposal and matched-candidate baselines explicit? Paired outcomes for the 12 natural bugs would also help characterize the practical benefit beyond the reported aggregate change from 3/12 to 4/12.

Reviewer_diXTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a final pass/fail result does not show how a coding agent repaired the code. It introduces TRACEBENCH-CLI, a benchmark of 126 Codeforces-derived repair tasks in the TerminalBench format. The tasks have staged tests and recorded fault regions for evaluation. The benchmark tracks fault-location claims, edit locality, stage progress, and wasted turns. The authors also propose ARC, a training-free controller that scores candidate actions and uses fixed rules to guide diagnosis, editing, and testing. With the diagnostic tool enabled, ARC raises solve rates from 24.6% to 37.3% for Qwen3-32B and from 12.7% to 20.6% for GPT-5.4. Gains are smaller for gpt-4o-mini (5.6% to 6.3%), while GPT-5.4-nano remains at 1.6%.

Critical strengths
  • The benchmark is well designed. Known fault regions, staged tests, and explicit fault-location records make it possible to study repair behavior that final pass/fail scores miss. This could be a useful resource for evaluating other coding agents.

  • ARC is a promising idea. Guiding a frozen model toward diagnosis, testing after edits, and avoiding repeated actions is a simple and practical way to improve an agent without training another model.

  • The Qwen3-32B solve-rate gains are encouraging: 21.4% to 34.1% in DIRECT and 24.6% to 37.3% in HARNESS.

  • The independent review of 30 outcome-matched pairs is a valuable addition.

Critical weaknesses
  • The main-model comparisons (figure 5) do not fully separate the controller from additional sampling. The ordinary baseline uses one generation per turn, while API ARC uses three proposals and three auxiliary generations, with different temperatures. The full 126-task matched-sampling comparison helps, but uses a different smaller model from the main Qwen3/GPT comparisons. For example, from GPT 5.4 runs, ARC costs about 3.7 times the one-call GPT-5.4 baseline (Table D.8 in appendix). This should made explicit along with the main results in the paper i.e. the performance improvement is coming at a higher cost.

  • The paper is hard to read despite the fairly simple benchmark and controller ideas. The results section could be better organized.

Questions for authors
  • How does the Figure 4 look for Qwen3-32B which has a higher pass rate on the benchmark?

  • Add cost of baseline vs ARC for the experiments in figure 5.

Comments on the paper

2 comments

Dear Reviewers,

The authors have now submitted their responses. Please review them and indicate whether your concerns have been addressed. Please also update your scores if the responses have changed your assessment.

Thank you.

AC

AuthorsOpenReview ↗

A General Response by the Authors

Dear Area Chairs and Reviewers,

Thank you for handling our submission.

To support your evaluation, we briefly summarize below: (1) an overview of our work, (2) the strengths highlighted by the reviewers, and (3) the responses we provided to address their concerns during the rebuttal period.

(1) Overview of our work. The community is moving beyond pass/fail evaluation of coding agents, as recent trajectory benchmarks show, because a passing patch can still be broad, poorly localized, or wasteful. We study what the pass/fail label leaves unresolved in a repair. To this end we introduce TraceBench-CLI, a benchmark of 126 CLI repair tasks, and ARC, a training-free controller for frozen models. Our key contributions are:

  • Traceability gap. Show that outcome scores correlate only weakly with process proxies across eight models on three external CLI benchmarks.
  • Direct process measurement. Measure fault attribution, edit locality, wasted turns, and diagnosis through staged tests and known fault spans.
  • Training-free control. Keep the base model's preferred action unless the trajectory loops, scatters edits, or skips diagnosis.

(2) Strengths. We are glad that all three reviewers (diXT, uLe6, 5Ej5) highlighted the value of our work. In particular:

  • Important problem. The problem is very important, and the paper does more than introduce another final success benchmark (uLe6, 5Ej5).
  • Well-designed benchmark. Known fault regions and staged tests expose repair behavior that pass/fail scores miss (diXT, uLe6).
  • Simple and practical controller. ARC is a promising and workable way to improve a frozen agent without training (diXT, 5Ej5).
  • Substantive and independent evidence. The matched-budget comparison addresses an important confound, and the blinded ratings complement the metrics (diXT, uLe6).

(3) Responses to address the reviewers' concerns. We provided detailed clarifications and added new experiments along several axes:

  • Consistent improvement in repair quality. Reviewer uLe6's "main reservation is whether the measured traceability gains identify a consistent improvement in repair quality at a clearly characterized cost." We scoped the ten-patch audit to the gpt-4o-mini pair and showed that the heuristic's lower one-step regret comes from picking low-probability proposals, which solves 2/126 tasks over full runs against 10/126 for ARC. Blinded non-author raters preferred ARC on 22 of 30 matched pairs against 3 for the base (uLe6 W1, Q1-Q2).
  • Clearly characterized cost. We reported total API cost over all attempts, including failed ones, and added paired outcomes for the 12 natural bugs. With the point above, this resolves both parts of the main reservation (uLe6 W2, Q3; diXT Q2).
  • Separating the controller from additional sampling. We addressed the concern that the main-model comparisons "do not fully separate the controller from additional sampling" with matched-budget controls. With the same six generations per turn, ARC solves 10/126 tasks on Qwen2.5-Coder-14B against 3/126 for both Matched Base and Best-of-N (diXT W1).
  • Clarifying what the reported cost measures. The reported cost comes from a controlled experiment that gives every selector six generations per turn. Under this protocol, ARC costs 2.0× to 6.0× a one-generation base and 0.74× to 1.02× matched Best-of-N. ARC is adaptive: in actual runs it keeps the base model's first choice on 78% to 88% of turns and needs extra samples only when a trigger fires, so its cost falls to near 1× (diXT W1, Q2; uLe6 W2).
  • Clarifying the claim on recent models. Our claim is that ARC keeps reliable models from getting worse and makes weaker models trace-aware, which cuts wasted turns and improves repair quality. On four recent models, ARC reaches the most total solves (37, against 35 for Best-of-N and 33 for Single Base) and never falls below Single Base. Base solve rates of 7.1% to 35.7% show that the benchmark is not saturated (diXT; 5Ej5 W2).
  • Testing weaker models. On GPT-5.4-nano, ARC cuts wasted turns from 9.64 to 1.27. Its solve rate stays flat because the model returns identical proposals on 42% of turns (5Ej5 W1).
  • Improving presentation. We reorganized Section 6 into four subsections, added the Qwen3-32B counterpart of Figure 4, reported sampling budgets and cost ratios with the main results, and verified the code link (diXT W2, Q1; 5Ej5 W3).

In addition, we have incorporated the above responses in Section 6 and Appendices C and D, and highlighted the revisions in blue.

We believe that the detailed point-by-point responses and the updated submission can address reviewers' concerns. The revised submission presents a clearer framework for traceable repair, with demonstrated generalization across models and benchmarks and robust behavior under matched budgets.

Thank you again for your time and consideration.

Sincerely,

Authors of #23323

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.