- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies data synthesis for training code-repair agents and proposes TRACEGEN, which mines defect patterns from existing bugs, transfers them to new code locations, and creates easy, medium, and hard variants by controlling the amount of localization information provided. Experiments show that, combined with curriculum learning, it improves baseline model performance from 3.2% to 24.7% on SWE-Bench verified.
Critical strengths
- The paper proposes a simple, intuitive framework to synthesize problems from the original bugs and provide different levels of localization information.
Critical weaknesses
-
I found the comparison with baselines under-specified and potentially unfair, which is elaborated on further in my Questions 1 & 2 below.
-
Ordering trajectories from easy to hard and doing curriculum training with SFT is not a new training pipeline, even in the automatic program repair field, similar ideas already exist [1,2]. The main contribution is therefore its combination with DefectChain, which itself combines several known methods from LocAgent, ChangeDistiller, and GumTree.
[1] Armin Zirak and Hadi Hemmati. 2024. Improving Automated Program Repair with Domain Adaptation. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 65 (March 2024), 43 pages. https://doi.org/10.1145/3631972
[2] Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. 2022. Bridging pre-trained models and downstream tasks for source code understanding. In Proceedings of the 44th International Conference on Software Engineering (ICSE '22). Association for Computing Machinery, New York, NY, USA, 287–298. https://doi.org/10.1145/3510003.3510062
Questions for authors
-
Can the authors clarify how other methods in Figure 2 were implemented? For example, native LM rewrite enumerates functions and classes and asks an LM to reimplement the selected entity, without asking the model to introduce a defect. Similarly, FeatAdd is designed to add functionality, so without newly generated tests, its changes would not normally cause existing passing tests to fail. Its low PASS to FAIL validation rate therefore is largely expected from the method’s objective. Were these baselines modified to explicitly generate defects at externally supplied locations? The implementation seems to be missing from the submitted code in the Supplementary Material.
-
Can the authors clarify whether the results in Table 2 are reproduced using a common evaluation protocol or taken directly from the respective papers? Although the methods share the same base model, they may use different agent frameworks, decoding settings, trajectory counts, and inference budget. Without more details, it's hard to make a fair comparison or draw conclusions.
-
I found the manuscript's description of several prior works misleading. For instance, the manuscript claims that "existing synthesis methods depend on the known patch location", while SWE-Smith provides 4 different generation strategies, and only one of them (PR Mirror is based on the files and changes from a real PR. Similarly, the claim that "no current method produces bugs with structured difficulty labels" ignores that SWE-Smith trains a classifier from SWE-bench annotations to label the difficulty level of synthetic instances, although it is not the same difficulty (how much localization information is available). Can the authors provide more evidence behind these claims? (There are still other valid motivations for the method: varying the amount of localization information or task difficulty can affect performance, and curriculum-based training schedules can also improve performance. These are not articulated in the current manuscript.)