acceptodds
Under review as a conference paper at ICLR 2027

TraceGen: Synthesizing Difficulty-Controllable Bugs for Training and Evaluating Code Repair Agents

Abstract

Training and evaluating code repair agents both demand large volumes of quality bug data, but existing synthesis methods depend on known patch locations and produce bugs without difficulty labels, leaving training data flat and evaluation unable to distinguish agents at different capability levels. We present TraceGen, a framework that mines structured causal paths, termed DefectChains, from repair traces and uses them to synthesize bugs at independently discovered code locations without access to the ground-truth patch. The causal depth of each DefectChain yields three difficulty tiers that serve a dual purpose: ordering trajectories for curriculum fine-tuning and producing calibrated evaluation gradients. On SWE-bench Verified seeds, TraceGen validates 194 of 318 attempts at a 61.0% rate, while baselines deprived of the original patch location suffer 63-92% validation drops. An agent trained via curriculum SFT on TraceGen trajectories achieves 24.7% on SWE-bench Verified at the 7B scale, surpassing SWE-Dev at 23.4%, R2E-Gym at 19.0%, and Lingma SWE-GPT at 18.2%; curriculum ordering alone contributes 3.3 percentage points over random-order SFT on identical data. All six evaluated frontier models exhibit monotonic resolve-rate increases across the three tiers, confirming the stability of the difficulty gradient.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
bug synthesis, code repair agents, difficulty-controllable data, curriculum fine-tuning, SWE-bench, software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wc4jOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies data synthesis for training code-repair agents and proposes TRACEGEN, which mines defect patterns from existing bugs, transfers them to new code locations, and creates easy, medium, and hard variants by controlling the amount of localization information provided. Experiments show that, combined with curriculum learning, it improves baseline model performance from 3.2% to 24.7% on SWE-Bench verified.

Critical strengths
  1. The paper proposes a simple, intuitive framework to synthesize problems from the original bugs and provide different levels of localization information.
Critical weaknesses
  1. I found the comparison with baselines under-specified and potentially unfair, which is elaborated on further in my Questions 1 & 2 below.

  2. Ordering trajectories from easy to hard and doing curriculum training with SFT is not a new training pipeline, even in the automatic program repair field, similar ideas already exist [1,2]. The main contribution is therefore its combination with DefectChain, which itself combines several known methods from LocAgent, ChangeDistiller, and GumTree.

[1] Armin Zirak and Hadi Hemmati. 2024. Improving Automated Program Repair with Domain Adaptation. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 65 (March 2024), 43 pages. https://doi.org/10.1145/3631972

[2] Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. 2022. Bridging pre-trained models and downstream tasks for source code understanding. In Proceedings of the 44th International Conference on Software Engineering (ICSE '22). Association for Computing Machinery, New York, NY, USA, 287–298. https://doi.org/10.1145/3510003.3510062

Questions for authors
  1. Can the authors clarify how other methods in Figure 2 were implemented? For example, native LM rewrite enumerates functions and classes and asks an LM to reimplement the selected entity, without asking the model to introduce a defect. Similarly, FeatAdd is designed to add functionality, so without newly generated tests, its changes would not normally cause existing passing tests to fail. Its low PASS to FAIL validation rate therefore is largely expected from the method’s objective. Were these baselines modified to explicitly generate defects at externally supplied locations? The implementation seems to be missing from the submitted code in the Supplementary Material.

  2. Can the authors clarify whether the results in Table 2 are reproduced using a common evaluation protocol or taken directly from the respective papers? Although the methods share the same base model, they may use different agent frameworks, decoding settings, trajectory counts, and inference budget. Without more details, it's hard to make a fair comparison or draw conclusions.

  3. I found the manuscript's description of several prior works misleading. For instance, the manuscript claims that "existing synthesis methods depend on the known patch location", while SWE-Smith provides 4 different generation strategies, and only one of them (PR Mirror is based on the files and changes from a real PR. Similarly, the claim that "no current method produces bugs with structured difficulty labels" ignores that SWE-Smith trains a classifier from SWE-bench annotations to label the difficulty level of synthetic instances, although it is not the same difficulty (how much localization information is available). Can the authors provide more evidence behind these claims? (There are still other valid motivations for the method: varying the amount of localization information or task difficulty can affect performance, and curriculum-based training schedules can also improve performance. These are not articulated in the current manuscript.)

Reviewer_LV8UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The premise of the paper is that existing bug synthesis strategies rely on using known patch locations and can plausibly identify where bugs reside, therefore guiding the agent to the code that it should perturb to generate a bug. I am dubious of this claim, since prior techniques (BugPilot) did not provide information to the agent that a bug was located in a particular location. However, the claim that they have successfully implemented a technique that allows for scaling up and down of the difficulty level seems to be more directly supported by their results. Moreover, while BugPilot doesn't require exact locations to inject bugs, it is a limitation of this technique that it requires many attempts to successfully generate a test case.

Critical strengths
  1. The paper presents a method that seems to have a higher yield of interesting bugs and a more granular approach to curriculum learning than past Bug synthesis techniques. Their training pipeline on Qwen 2.5 Coder seems to improve on the SWE-Smith and R2E-Gym bug datasets.
Critical weaknesses
  1. The claims made in the introduction and throughout the results section that ablating location information from BugInstruct / FeatAdd in the BugPilot paper would make any difference are unsupported. Please see Appendix A in the BugPilot information for the prompts that require no localization information.
  2. While performance gains in Table 2 are clear, I wonder if the authors would be able to ablate on a more recent model besides Qwen2.5-Coder and report error bars, perhaps SFT on Qwen3.5-4B?
Questions for authors

This is the prompt for FeatAdd (BugPilot's instructions). It doesn't have any location-specific information in it; neither does BugInstruct's pipeline. I am very confused about where we see the location-specific information and why ablating it would make a difference.

< uploaded_files > {{ working_dir }} </ uploaded_files > I ’ ve uploaded a python code repository in the directory {{ working_dir }}. Your task is to implement a new feature in this codebase . First go through the codebase and identify a suitable new feature to add . Come up with a plan to implement it and then make the necessary changes to the codebase . You can use the tools provided to edit files , run tests , and submit your changes . The feature you introduce should not break any existing functionality . Make sure the edit you make is complex - you should introduce at least two related changes in the codebase in different files .

Reviewer_8tJEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents TraceGen, a pipeline for synthesizing SWE-bench-style bugs from seed instances. For each seed, an LLM traces a “localization chain” through a LocAgent-style code graph, running from the symptom to the root-cause entity the developer modified. The gold diff is abstracted into a “repair chain” of AST-level repair intents drawn from a 10-category taxonomy. Candidate injection sites are retrieved by embedding similarity to the root-cause node and filtered by graph depth and intent compatibility. A ReAct agent then injects an analogous bug at each site and writes tests for it. An LLM writes three problem statements per bug (hard, medium, easy) that expose progressively more chain nodes. Trajectories from a frontier model are weighted by outcome tier (gold, silver, bronze) and by repository, then ordered easy to medium to hard within each epoch for LoRA SFT of Qwen2.5-Coder-7B.

Critical strengths
  • Producing matched variants of the same bug that differ only in how much localization evidence the problem statement reveals could be useful because it turns “issue-report specificity” into a controlled variable. That could help both diagnostic evaluation and data design, and it is cleaner than difficulty ratings.
  • They also stress-test prior synthesizers by moving them away from the original patch location. The finding that SWE-Smith-style rewriting and BugInstruct depend heavily on that location is interesting.
Critical weaknesses
  • The training data comes from the test set, and the numbers don’t add up. The seeds are SWE-bench Verified instances. Each DefectChain is built from the issue and the gold diff, and ends at the entity the developer modified. Retrieval then searches for similar code in the same repositories. So the training bugs are anchored on test-set fix locations and patterns, and the agent is then evaluated on those same 500 instances. This also undercuts the “location-independent” claim, since the gold-patch location seeds the search. The comparison to SWE-Dev and the other baselines is not fair, because several of them exclude Verified data entirely.
  • The curriculum and difficulty claims are not well supported. Because tiers are sorted within each epoch, training always ends on hard trajectories. Their minimal-hint problem statements likely resemble real SWE-bench issues most closely. The 3.3-point gain (about 16 instances, single run) could come from seeing that data last rather than from easy-to-hard ordering. A hard-to-easy control would test this. The baselines are not controlled: the paper says all baselines use random-order SFT, but its own related work says Lingma SWE-GPT uses a curriculum. The baseline numbers also come from different papers with different teachers and setups. The difficulty gradient is by construction because tiers are defined by how many hints the problem statement gives, so higher resolve rates with more hints is expected.
Questions for authors
  • How are 2,847 trajectories obtained from 194 validated bugs? Were additional bugs synthesized beyond the 318-attempt experiment? If so, from which seeds and repositories, and what was their validation rate?
  • Were any of the synthesized training bugs located in files or functions touched by SWE-bench Verified gold patches?
  • Can you evaluate the trained agent on a benchmark disjoint from the seeds, such as SWE-bench-Live, SWE-rebench, or Verified instances from repositories with no seeds?
  • Can you add an anti-curriculum (hard to easy) control and a random-order-with-hard-last control, and report mean and std over at least three training seeds for all ordering variants?
  • What happens to performance if silver and bronze trajectories are dropped? Which frontier model produced the trajectories, and what fraction of trajectories falls in each tier?
  • For Fig 3, which bugs were evaluated, and how many? Were any of them used in training? Can you report the actual numbers with CIs, along with resolve rates broken down by chain length |L| within each tier?
  • How do you make sure easy-tier problem statements, which by design give function-level cues, don’t reveal the fix? This seems to conflict with the patch-sanitization rule that statements must not reveal the fix location.
  • In Fig 2, how many attempts did each baseline receive? Were baselines given the same retry budget and the same differential-testing criteria? Since DefectChain sites are selected for TraceGen’s intent compatibility, could you also run TraceGen at the original patch locations, which the paper marks as “n/a”?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.