- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ExecCritic, a framework for test generation and feedback optimization for code repair agents. Its core idea is to decouple test construction from code repair, highlighting that execution feedback does not necessarily improve code repair performance, and that the quality of the feedback tests themselves is a key factor. The proposed framework is well structured, and the experimental results provide a reasonable degree of empirical support.
Critical strengths
- The research problem holds practical relevance. In repository-level code repair tasks, an agent must not only generate a correct patch but also determine whether the patch actually satisfies the behavioral requirements described in the issue. The paper shifts the focus from simply enhancing code generation capability to improving the reliability of execution feedback.
- ExecCritic divides the process into two stages, Learn to Test and Test to Improve, connected through a Harness. This separation provides a clear and coherent framework.
Critical weaknesses
- The main technical components include training the test generation agent using Base-to-Gold and candidate patch discrimination capabilities, fixing the generated tests through a Harness, and training the repair agent based on execution feedback. These designs are reasonable, but the contribution currently relies primarily on combining existing framework components and differentiating training objectives. The paper does not sufficiently establish which designs constitute irreplaceable technical innovations or identify the specific novel mechanism responsible for the main performance gains.
- The paper primarily uses Qwen3.5-35B-A3B as the trainable backbone and SWE-bench Verified as main evaluation benchmark. Although it also reports experiments on test feedback with GPT-5.6 on SWE-bench Pro and presents some cross-language results in the appendix, these results are not yet sufficient to demonstrate that the entire learning framework can generalize reliably across other models, programming languages, and task types.
- ExecCritic requires maintaining and training two separate models: the Test Agent and the Repair Agent. During inference, test generation and multiple rounds of repair revision introduce additional token and latency consumption.
Questions for authors
- Compared to existing issue-conditioned test generation and fixed-test-driven repair methods, what is ExecCritic's most critical technical innovation? Can more fine-grained ablation experiments be provided to quantify the individual contributions of test generation RL, the candidate patch discrimination objective, the fixed test mechanism, and Repair Agent training?
- The cross-language test-generation results reported in the appendix exhibit substantial variation. Could the authors further explain these differences and report the complete combinatorial system results across alternative programming languages, model backbones, and benchmarks?