acceptodds
Under review as a conference paper at ICLR 2027

ExecCritic: Learn to Test, Test to Improve for Coding Agents

Abstract

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test–verify–revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agents, Test Generation, Execution Feedback, Reinforcement Learning, Repository-Level Code Repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v1g5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ExecCritic, a framework for test generation and feedback optimization for code repair agents. Its core idea is to decouple test construction from code repair, highlighting that execution feedback does not necessarily improve code repair performance, and that the quality of the feedback tests themselves is a key factor. The proposed framework is well structured, and the experimental results provide a reasonable degree of empirical support.

Critical strengths
  1. The research problem holds practical relevance. In repository-level code repair tasks, an agent must not only generate a correct patch but also determine whether the patch actually satisfies the behavioral requirements described in the issue. The paper shifts the focus from simply enhancing code generation capability to improving the reliability of execution feedback.
  2. ExecCritic divides the process into two stages, Learn to Test and Test to Improve, connected through a Harness. This separation provides a clear and coherent framework.
Critical weaknesses
  1. The main technical components include training the test generation agent using Base-to-Gold and candidate patch discrimination capabilities, fixing the generated tests through a Harness, and training the repair agent based on execution feedback. These designs are reasonable, but the contribution currently relies primarily on combining existing framework components and differentiating training objectives. The paper does not sufficiently establish which designs constitute irreplaceable technical innovations or identify the specific novel mechanism responsible for the main performance gains.
  2. The paper primarily uses Qwen3.5-35B-A3B as the trainable backbone and SWE-bench Verified as main evaluation benchmark. Although it also reports experiments on test feedback with GPT-5.6 on SWE-bench Pro and presents some cross-language results in the appendix, these results are not yet sufficient to demonstrate that the entire learning framework can generalize reliably across other models, programming languages, and task types.
  3. ExecCritic requires maintaining and training two separate models: the Test Agent and the Repair Agent. During inference, test generation and multiple rounds of repair revision introduce additional token and latency consumption.
Questions for authors
  1. Compared to existing issue-conditioned test generation and fixed-test-driven repair methods, what is ExecCritic's most critical technical innovation? Can more fine-grained ablation experiments be provided to quantify the individual contributions of test generation RL, the candidate patch discrimination objective, the fixed test mechanism, and Repair Agent training?
  2. The cross-language test-generation results reported in the appendix exhibit substantial variation. Could the authors further explain these differences and report the complete combinatorial system results across alternative programming languages, model backbones, and benchmarks?
Reviewer_Fx7HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ExecCritic, a framework that separately trains test generation agent and repair agent for better self-validation in code repair tasks. The test generation agent is trained through SFT and RL to generate tests that fail on buggy code and pass after applying the reference patch, and should be able to distinguish a pool of correct and incorrect patches. The repair agent is trained to revise patches with the execution feedback from hidden tests. At inference time, the authors use one-shot generated tests from the test agent to guide the iterative repair process of the repair agent. Experiments on SWE-bench verified show improvements in test quality and repair success.

Critical strengths
  1. The paper is well-motivated. The experiments clearly demonstrate the necessity of deployable LLMs and even proprietary LLMs to improve test generation.
  2. The experiments can generally support the claims, except for missing details on the labeled candidate repair patches used in training.
Critical weaknesses
  1. The paper did not establish a clear position in the literature. There are two main arguments in this paper. When coding agents write tests to test its implementation, it is likely to make the same mistake as in generation due to the shared context. Therefore, the authors separate the test generation process from the repairing process and keep generated tests unchanged throughout the iterative repairing process. However, this practice is relatively not novel and is adopted in recent works such as TDFlow (which is also mentioned by the authors). The second argument is poor quality in generated tests decreases performance in the self-validation process. Thus, the authors train the test agent following an SFT+RL paradigm. While the authors emphasize that the learning goal is distinct, I found several prior works [1,2,3,4,5] that also train test agents with the same goal, some of which are also briefly mentioned by the authors. Overall, these make the contribution of this paper is a bit incremental.
  2. The presentation of this paper is hard to follow due to the jargony wording and extensive usage of unnecessary notations. I don't think the notation helps understanding, but rather increases the burden. Many key ideas could have been explained more simply. And there are many terms and notations that are used before being defined or without being defined.

[1] Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents

[2] SWE-Swiss: A Multi-Task Fine-Tuning and RL Recipe for High-Performance Issue Resolution

[3] Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning

[4] Learning to Generate Unit Test via Adversarial Reinforcement Learning

[5] Learning to Generate Unit Tests for Automated Debugging

Questions for authors
  1. Could the authors elaborate more on the distinct contribution compared to the existing works mentioned in the weaknesses section, as well as any unique insights behind the key designs in this paper?
  2. Could the authors explain the procedure of obtaining the labeled candidate repair patches?
  3. Could the authors provide explanation for not providing the artifacts?
Reviewer_NqXYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ExecCritic separately trains a Test agent to generate repository-native regression tests and a Repair agent to revise source patches using fixed-test execution feedback. Test training combines teacher-trajectory SFT with RL rewards for Base-to-Gold success and candidate-patch discrimination. Repair training uses Oracle feedback and rewards both direct solving and subsequent correction. Composing the trained agents achieves 72.6% on SWE-bench Verified, compared with 68.3% for the trained Repair agent without the additional feedback loop.

Critical strengths
  1. The paper addresses a practically important problem: generated tests can mislead repair when they encode incorrect requirements. The separation of test construction, source revision, and official evaluation provides a concrete framework for studying this problem.
  2. Table 2 evaluates both trained and untrained versions of each component, providing evidence that both training procedures improve the composed system. Table 7 also includes a standard Repair-training baseline and an ablation of the direct-solve bonus.
Critical weaknesses
  1. The benefit of candidate-discrimination training is not sufficiently established. The paper does not isolate its contribution from Base-to-Gold-only RL or demonstrate improved discrimination on unseen candidate patches. This leaves the value of a central part of the Test-training objective unclear.
  2. The evidence for the system’s additional gains remains incomplete. Without a compute-matched self-revision baseline, it is difficult to separate the benefit of test feedback from additional inference computation. The gain over standard Repair training is also modest, without uncertainty estimates establishing its reliability.
Questions for authors
  1. Appendix E discusses shared-weight training as future work, but does separate training offer an advantage under matched total training and inference budgets? How does it compare with a shared model trained on both tasks while keeping test generation and repair in separate contexts? The related tasks may benefit from shared training.
  2. Does candidate-discrimination training improve validation of unseen patches beyond Base-to-Gold-only RL? Could the authors compare the two objectives under the same training budget and report false acceptance and false rejection on held-out candidate patches?
  3. How does the trained Repair agent perform when the same inference budget is allocated to self-revision without the independent Test agent? Please also report a paired issue-level confidence interval for the gain over standard Repair training in Table 7.
  4. The values in Figure 3 are confusing. The Qwen/GPT-test result is 66.2% in Figure 3b but 65.3% in Table 2 and the text; the GPT-5.6 Pro baseline is 61.2% in Figure 3a but 61.6% in Section 5.2.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.