acceptodds
Under review as a conference paper at ICLR 2027

SWE-Craft: Graph-Guided Task Synthesis for Weak Spots in Coding Agents

Abstract

Executable software engineering (SWE) tasks provide controlled environments for studying and improving coding agents. Existing perturbation-based synthesis scales task generation but offers limited control over the engineering practice represented in the resulting data. We study target-conditioned SWE task synthesis: a supplied capability target specifies desired model behavior or reasoning and guides task design before construction. In our experiments, model-assisted analysis and human synthesis of base-model trajectories motivate these targets. We introduce SWE- Craft, which uses a source-linked code property graph to explore functional modules across functions and files. A task design agent matches repository behavior to a capability target and specifies required and preserved behaviors. The task specification then guides separate construction and validation of executable bug- fixing, feature-addition, and feature-enhancement tasks. Quality-screened teacher trajectories provide supervision for fine-tuning. In a source-blinded LLM audit, SWE-Craft scores 93.16 out of 100 for alignment with a shared task profile, compared with 53.36 for SWE-smith and 42.67 for BugPilot. Fine-tuning Ling-3.0- flash on 1,500 trajectories achieves 46.88% and 68.00% Pass@1 on SWE-bench Pro and SWE-bench Multilingual, outperforming SWE-smith by 2.74 and 3.55 percentage points, and BugPilot by 2.05 and 3.00 points, respectively

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data synthesis, code agent, SWE

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2zeZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes SWE Craft, where the authors study target-conditioned SWE task synthesis: it uses a capability target that specifies desired behavior to guide task design before construction.

To facilitate this, SWE Craft leverages a source-linked code property graph, which allows the agent to better understand the code structure and design task specifications that align with the target capability.

Using SWE Craft to finetune Ling-3.0-flash, it outperforms SWE-smith by 2.74 and 3.55 percentage points, and BugPilot by 2.05 and 3.00 points on SWE-bench Pro and SWE-bench Multilingual.

Critical strengths
  • The studied problem, target-conditioned SWE task synthesis, seems interesting.
  • The proposed source-linked code property graph seems to be an intuitive improvement over baselines such as SWE Smith.
Critical weaknesses
  • The improvement over baseline, such as SWE Smith, is surprisingly small, as such baselines are not conditioned on target capability? This makes me question the usefulness of the identified target capability.
  • The scope of experiments is somewhat limited: (1) Only one backbone model is finetuned (Ling-3.0-flash). (2) Only one task design agent setup is used. One might argue that a stronger task design agent does not need the source-linked code property graph. (3) Only a fixed number of trajectories (1500) is experimented with. It might be true that the proposed approach might have similar performance to baselines once we scale up the number of trajectories.
Questions for authors
  • I don't fully understand how target capability is specified/derived. Does it mean that we need to specify these target capability for different base models we want to improve? Does not mean that the target-conditioned SWE task synthesis procedure need to be done differently for different base models?
2 comments
AuthorsOpenReview ↗

Thank you for your valuable feedbacks.

We sincerely thank you for your insightful comments on evaluating the usefulness of capability targets and expanding our experimental scope. We respectfully address each of your questions and concerns below.


Q1: How are capability targets derived, and does each base model require a different synthesis procedure?

A1: We would like to clarify that different base models do not require different synthesis procedures, as SWE-Craft decouples model-specific weakness diagnosis from task synthesis (Section 3.1 and Appendix B). Only the capability targets are updated, while the CPG-guided synthesis pipeline remains unchanged.

Specifically, we derive capability targets by analyzing base-model evaluation trajectories through LLM-assisted inspection and human synthesis, translating recurring failure patterns into four standardized fields: task type, task subtype, behavioral focus, and structural reasoning scope (e.g., cross-module error reporting in Pyflakes, Appendix C.1). For a new base model, we simply update these targets from its observed failures without altering the pipeline. Moreover, existing synthesized tasks also transfer effectively to other models, as shown on Qwen3-14B in W3.


W2: Are the improvements over untargeted baselines sufficient to demonstrate the value of capability targeting?

A2: We would like to emphasize that our results clearly demonstrate the value of capability targeting, even under a setup that also benefits the untargeted baselines. As stated in Section 4.1, all methods share the same 500 repositories selected using our capability targets, giving the baselines access to target-relevant repository contexts (which already boosts them by +5.06 to +8.33 points over the base model in Table 2).

Even on top of this shared repository pool, SWE-Craft outperforms both baselines by +2.05 to +3.55 percentage points across SWE-bench Pro and Multilingual (Table 2) and achieves substantially higher profile alignment (93.16 vs. 53.36 and 42.67, Figure 3). Moreover, on benchmark subsets specifically testing our targeted capabilities (Configuration Consistency and Result/Error Propagation), SWE-Craft's lead over SWE-smith reaches +6.00 to +6.89 points on Multilingual, exceeding the overall gain of +3.55 points. This result confirms the direct benefit of incorporating capability targets into task design.


W3: Limited experimental scope in terms of backbone models, task-design agents, and training data scale.

A3: We would like to clarify that our experimental setup aligns with standard controlled evaluation protocols, and that extending our evaluation across backbone models, stronger task-design agents, and larger trajectory scales consistently confirms our main findings.

First, to evaluate cross-backbone generalization, we generate tasks and fine-tuned Qwen3-14B, a dense model from a different model family than Ling-3.0-flash, under the same 1,500-trajectory setting. SWE-Craft outperforms both baselines by +1.96 to +2.67 percentage points across both benchmarks, showing that the training benefits generalize beyond Ling-3.0-flash:

Training Data (1,500 Trajectories)SWE-bench Pro (%)SWE-bench Multilingual (%)
SWE-smith14.2315.78
BugPilot (BugInstruct)14.4615.22
SWE-Craft (Ours)16.4217.89

Second, to test whether CPG remains beneficial with a stronger task-design agent, we compared 11 matched same-repository task pairs synthesized with and without CPG using Kimi-K3 (released in July 2026 with 2.8T parameters and a 1M-token context window) with Claude Code. Even with this stronger agent, CPG exposes cross-module dependencies missed by unguided exploration (e.g., connecting the single-run driver, sweep launcher, and shared execution function across modules in Hydra), consistent with our No-Graph ablation where CPG improves Pass@1 by +1.33 and +1.89 points (Section 4.5).

Third, while 1,500 trajectories already match controlled SFT scales in prior work (500 to 2,136 trajectories),we scaled the training set from 1,500 to 4,500 trajectories per method on the same 1,500 underlying tasks:

Training Data1,500 Trajectories Pro / ML (%)4,500 Trajectories Pro / ML (%)
SWE-smith44.14 / 64.4545.14 / 64.89
BugPilot (BugInstruct)44.83 / 65.0045.60 / 65.33
SWE-Craft (Ours)46.88 / 68.0048.25 / 69.33

Because the 4,500 trajectories are sampled from the same 1,500 tasks, the additional gains are relatively modest. Nevertheless, the baselines do not catch up: SWE-Craft's lead widens to +2.65–4.44 percentage points across both benchmarks. This demonstrates that collecting more trajectories from the same untargeted tasks does not close the performance gap in our evaluated setting.

Dear Reviewer,

Thank you again for your valuable suggestions. We have supplemented our rebuttal with additional experiments directly addressing the limited experimental scope and the usefulness of capability targeting.

Specifically, we have extended our evaluation to Qwen3-14B and increased the number of training trajectories from 1,500 to 4,500. SWE-Craft consistently outperforms both synthesis baselines in these experiments, with a widening performance gap at the larger training scale. We have also evaluated CPG-guided task synthesis using Kimi-K3 and clarified how capability targets are derived and adapted to different base models.

We would greatly appreciate it if you could review these additional results and let us know whether they address your concerns. We would be glad to clarify any remaining questions during the discussion period.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_Fq8kOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SWE-Craft, an agentic pipeline for generating synthetic SWE tasks. Traditional approaches for task synthesis mainly rely on either rule-based or agentic approach to perturb a given code repository in order to create tasks. However, the type of skills those tasks happen to exercise is often limited and may not cover the full spectrum of capabilities required for real-world software engineering. Instead, the proposed pipeline leverages a mix of classical code synthesis tools (like CodeQL and Joern) and agentic system to generate tasks to target particular competencies for a model to acquire. The authors empirically show that SWE-Craft can produce a diverse set of tasks that better reflect the range of skills needed in real-world software engineering scenarios.

Critical strengths
  • Target-conditioned SWE task synthesis. The authors formalize a synthesis setting in which the desired behavior/reasoning capability precedes construction. Their capability targets specify combinations of task type, task subtype, behavioral focus, and structural reasoning scope. This is a meaningful difference from mutation-oriented synthesis.
  • Graph-guided task design. SWE-Craft uses code graph relationships as navigation and evidence for identifying functional modules spanning functions/files. So, it uses explicit program-structure evidence to guide an LLM in deciding where and how a target capability can be instantiated.
Critical weaknesses

The evaluation is somewhat self-referential. The evaluation demonstrates that SWE-Craft is effective in generating tasks that align with the desired task profile and structural/behavioral requirements. That said, the synthesis model, the teacher and the judge all uses the same LLM model (GLM-5.2). Using another strong judge model could provide a more objective assessment of the generated tasks' quality and alignment.

Questions for authors
  • In the paper, it is mentioned that all BugPilot results use the BugInstruct variant, which explicitly requests bug injection. How does this affect the profile alignment results? It seems to me that BugPilot's FeatAdd would fair better given the tasks are created from bugs introduced while working on a codebase to implement new features.

  • Given that SWE-Craft identifies model-specific weak spots, could the fine-tuned Ling-3.0-flash model be used to start another iteration of the pipeline? If so, have the authors considered evaluating SWE-Craft as an iterative, closed-loop training procedure?

2 comments
AuthorsOpenReview ↗

Thank you for your valuable feedbacks.

We sincerely thank you for your strong support of our work and for highlighting the value of our weakness-targeted synthesis paradigm, our use of Code Property Graphs (CPG) to ground realistic multi-component tasks, and our consistent empirical gains across benchmarks. We respectfully address each of your questions and suggestions below.


Q1: Why use BugPilot's BugInstruct variant rather than FeatAdd, and how does this affect profile alignment?

A1: We would like to clarify that FeatAdd was not excluded without empirical evaluation. As stated in Section 4.1, FeatAdd has a low task construction success rate in our setting. Specifically, in our preliminary experiment with 36 FeatAdd attempts across 8 repositories, 20 introduced no regression bugs, while 14 failed due to validation errors, compilation issues, or timeouts, yielding only 2 valid tasks (5.6%). This is consistent with BugPilot's observation in their paper[1] that stronger models tend to introduce fewer bugs during feature development. Therefore, we adopted BugInstruct for a scalable comparison. The reported profile-alignment score (42.67) thus applies only to BugInstruct, not FeatAdd.


Q2: Can the fine-tuned Ling-3.0-flash model be used to start another iteration of SWE-Craft in a closed-loop training procedure?

A2: We would like to clarify that SWE-Craft can naturally support closed-loop iterative training without modifying the synthesis pipeline. As described in Section 3.1, weakness diagnosis is decoupled from task synthesis, allowing the fine-tuned model to be re-evaluated to identify remaining weaknesses and update capability targets for the next synthesis round. While this paper validates single-round effectiveness (+7.80/+11.33 percentage-point gains on SWE-bench Pro/Multilingual, Table 2), we leave systematic multi-round evaluation to future work, as discussed in Section 6.


W3: Does using GLM-5.2 as the synthesis model, teacher, and judge introduce self-referential bias?

A3: We would like to clarify that SWE-Craft's advantages do not depend on GLM-5.2 serving as the judge. To address potential self-referential bias, we conducted an additional evaluation using an independent LLM judge and verified downstream performance through executable benchmark tests.

First, we re-evaluated 200 tasks per method using Kimi-K3 from an independent model family under the same source-blinded protocol. SWE-Craft achieves a profile-alignment score of 89.66, compared with 50.75/42.38 for SWE-smith/BugPilot (BugInstruct). Similarly, its 86.0% statement-to-test alignment exceeds the baselines (74.5%/50.5%), confirming that the task-quality advantages persist under an independent judge.

Second, as shown in Table 2, downstream performance is measured entirely through executable unit tests, without LLM judgments. SWE-Craft improves Pass@1 by +7.80/+11.33 percentage points over the base model and +2.05 to +3.55 points over the baselines on SWE-bench Pro/Multilingual.


References

[1] Sonwane et al., BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills. arXiv preprint arXiv:2510.19898v2, 2025.

Thank you for the responses to my questions. The additional experiments using a different LLM judge addresses my main concern.

Reviewer_3Na4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Craft proposes a targeted task generation method. It introduces capability target into task synthesis and specifically search for code repositories that matches required capability target and code behaviors, via code property graph.

Critical strengths
  1. The problem it tries to tackle is valuable: how to control the task distribution of training data generation to precisely match what the model weakness is. It tries to address the data efficiency problem.
Critical weaknesses
  1. It does not fully demonstrate that targeted synthesis remedies actual model weaknesses. The task profile alignment shows the generated tasks matches the predefined capability target, and overall improved model performance on SWE-Bench Pro and SWE-Bench Multilingual, but it doesn’t show the specific model weakness is addressed.
  2. Missing ablation on how code property graph compare to a simple code search tool, in terms of discovering cross-function and cross-file dependency.
Questions for authors
  1. Does training on tasks targeting a specific weak spot produce a larger improvement on that weak spot than equally difficult untargeted tasks?
  2. What is the incremental benefit of CPG over simpler repository exploration?
7 comments
AuthorsOpenReview ↗

Thank you for your valuable feedbacks.

We sincerely thank you for your constructive comments and for recognizing the value of controlling training task distributions to address model weaknesses. We respectfully address each of your questions and concerns below.


Q1: Does training on tasks targeting a specific weak spot produce a larger improvement on that weak spot than equally difficult untargeted tasks?

A1: We would like to clarify that SWE-Craft's improvements cannot be explained simply by generating harder tasks. To examine this, we conducted an additional SFT experiment using untargeted tasks with comparable difficulty to SWE-Craft.

First, we selected 1,500 training samples from SWE-smith and BugPilot with a teacher solve rate of 56.47%, close to SWE-Craft's 52.47%. Despite the comparable difficulty, this baseline achieves only 44.92% Pass@1 on SWE-bench Pro, compared with 46.88% for SWE-Craft (+1.96 percentage points). This shows that increasing the difficulty of untargeted tasks does not close the performance gap.

Second, our additional analysis shows that SWE-Craft's gains are particularly pronounced on the targeted weaknesses (detailed in W3). Compared with SWE-smith, SWE-Craft improves Configuration Consistency (CFG) and Result/Error Propagation (RES) on SWE-bench Multilingual by +6.89/+6.00 percentage points, exceeding the overall gain of +3.55 points. On SWE-bench Pro, the CFG gain (+3.70 points) also exceeds the overall gain (+2.74 points).

Importantly, all methods share the same target-informed repository pool, ensuring that the comparison controls for repository selection. These results show that SWE-Craft provides additional training value beyond using more difficult tasks or selecting suitable repositories.


Q2: How does the CPG-based approach compare to a simple code search tool in discovering cross-function and cross-file dependencies, and what is its incremental benefit?

A2: We would like to clarify that CPG explicitly exposes cross-function and cross-file dependency relations that standard code search tools cannot directly surface. As described in Section 4.5 and Appendix E.4, our No-Graph baseline already operates as a code-search agent equipped with Read, Grep, Glob, LSP, and Bash, which retrieve text or symbol matches but require manual dependency reconstruction across files. We evaluated CPG's incremental benefit through both controlled ablations and paired task analysis.

First, under the same capability targets, and construction pipeline (Section 4.5), SWE-Craft (Full) outperforms 71.4% of blinded pairwise task-quality comparisons against No-Graph (Figure 4) and improves downstream SFT Pass@1 by +1.33 and +1.89 percentage points on SWE-bench Pro and Multilingual (Table 2):

Synthesis MethodExploration Tools AvailableSWE-bench Pro (%)SWE-bench Multilingual (%)
SWE-Craft (No-Graph)Standard Code Search (Read, Grep, Glob, LSP, Bash)45.55±0.4145.55 \pm 0.4166.11±0.5166.11 \pm 0.51
SWE-Craft (Full)Standard Code Search + CPG46.88±0.5546.88 \pm 0.5568.00±0.5868.00 \pm 0.58

Second, across 11 matched same-repository task pairs synthesized with and without CPG, graph guidance helps capture cross-module requirements. For example, in Hydra, CPG connected the single-run driver, sweep launcher, and shared execution function. This enabled the agent to design more comprehensive tests for environment setup, working-directory restoration, and configuration saving, which the No-Graph agent overlooked.

W3: Does targeted synthesis improve the model’s actual weaknesses?

A3: We would like to clarify that SWE-Craft not only generates tasks aligned with identified weaknesses (93.16/100 profile alignment and 92.0% test alignment in Figure 3), but also directly improves the model's corresponding capabilities after SFT. We verified this through weakness-specific subset evaluations and post-SFT trajectory analysis.

First, we evaluated two recurring weaknesses identified from base-model trajectories: Configuration Consistency (CFG) and Result/Error Propagation (RES). Across all four benchmark subsets grouped by public issue descriptions, SWE-Craft improves Pass@1 by +7.32 to +15.52 percentage points over the base model and outperforms SWE-smith by +2.54 to +6.89 points:

Targeted Weakness SubsetBenchmark# TasksBase Model (%)SWE-smith SFT (%)SWE-Craft SFT (%)Gain vs. Base ModelGain vs. SWE-smith
Configuration Consistency (CFG)SWE-bench Pro38734.4538.0741.77+7.32+3.70
Configuration Consistency (CFG)SWE-bench Multilingual5860.3468.9775.86+15.52+6.89
Result/Error Propagation (RES)SWE-bench Pro34134.0239.2041.74+7.72+2.54
Result/Error Propagation (RES)SWE-bench Multilingual5061.3368.0074.00+12.67+6.00

Pass@1 (%) averaged over three runs; subsets grouped by public issue descriptions in our additional analysis. Base Model denotes the base model before SFT.

Second, across 30 shared CFG tasks, configuration-consistency violations decrease from 6 (Base Model) and 4 (SWE-smith) to 2 (SWE-Craft). For instance, on a qutebrowser issue, SWE-Craft correctly traces a configuration option to its downstream consumer, whereas both Base Model and SWE-smith fail to maintain cross-module consistency.

Dear Reviewer,

Thank you again for your constructive feedback. We have provided detailed responses and additional experiments addressing your main concerns.

In particular, we have added a difficulty-comparable untargeted baseline, which achieves only 44.92% Pass@1 on SWE-bench Pro, trailing SWE-Craft by 1.96 percentage points. We have also analyzed performance and trajectories on problems requiring the targeted capabilities and clarified our No-Graph ablation against standard code-search tools.

We would greatly appreciate it if you could review these additional results and let us know whether they address your concerns. We would be glad to clarify any remaining questions during the discussion period.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response from Authors

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their insightful feedback and constructive suggestions, and the Area Chairs for their time and support during the review process. We are encouraged by the recognition of the valuable and interesting problem of controlling training task distributions to address model weaknesses (Reviewers 3Na4 and 2zeZ). We also appreciate the acknowledgment of our target-conditioned synthesis as a meaningful departure from mutation-oriented methods (Reviewer Fq8k), and our graph-guided task design that uses explicit program-structure evidence to construct cross-component tasks (Reviewers Fq8k and 2zeZ).

In response to the reviewers' feedback, we have further strengthened our experiments and clarified the key contributions of our work. Our key efforts are summarized below:

  • Effectiveness of weakness-targeted synthesis (Reviewers 3Na4 and 2zeZ). A difficulty-comparable untargeted baseline achieves only 44.92% Pass@1, compared with SWE-Craft's 46.88% on SWE-bench Pro. Additional analyses further confirm improvements on targeted capabilities and corresponding model behaviors.

  • Benefits of graph-guided task synthesis (Reviewers 3Na4 and 2zeZ). Our No-Graph ablation demonstrates +1.33/+1.89 percentage-point gains from CPG on Pro/Multilingual. Additional task analyses illustrate how explicit structural relations help construct cross-module requirements.

  • Generalization and reliability (Reviewers 2zeZ and Fq8k). SWE-Craft retains its advantages on Qwen3-14B and with 4,500 training trajectories, while independent evaluation using Kimi-K3 confirms its task-alignment advantages.

We believe these additional experiments and clarifications substantially address the reviewers' main concerns and strengthen our paper.

Finally, we extend our sincere gratitude to the Area Chairs, all reviewers, and the entire ICLR community for their time, dedication, and valuable contributions to the review process.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.