- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes SWE Craft, where the authors study target-conditioned SWE task synthesis: it uses a capability target that specifies desired behavior to guide task design before construction.
To facilitate this, SWE Craft leverages a source-linked code property graph, which allows the agent to better understand the code structure and design task specifications that align with the target capability.
Using SWE Craft to finetune Ling-3.0-flash, it outperforms SWE-smith by 2.74 and 3.55 percentage points, and BugPilot by 2.05 and 3.00 points on SWE-bench Pro and SWE-bench Multilingual.
Critical strengths
- The studied problem, target-conditioned SWE task synthesis, seems interesting.
- The proposed source-linked code property graph seems to be an intuitive improvement over baselines such as SWE Smith.
Critical weaknesses
- The improvement over baseline, such as SWE Smith, is surprisingly small, as such baselines are not conditioned on target capability? This makes me question the usefulness of the identified target capability.
- The scope of experiments is somewhat limited: (1) Only one backbone model is finetuned (Ling-3.0-flash). (2) Only one task design agent setup is used. One might argue that a stronger task design agent does not need the source-linked code property graph. (3) Only a fixed number of trajectories (1500) is experimented with. It might be true that the proposed approach might have similar performance to baselines once we scale up the number of trajectories.
Questions for authors
- I don't fully understand how target capability is specified/derived. Does it mean that we need to specify these target capability for different base models we want to improve? Does not mean that the target-conditioned SWE task synthesis procedure need to be done differently for different base models?
2 comments
Dear Reviewer,
Thank you again for your valuable suggestions. We have supplemented our rebuttal with additional experiments directly addressing the limited experimental scope and the usefulness of capability targeting.
Specifically, we have extended our evaluation to Qwen3-14B and increased the number of training trajectories from 1,500 to 4,500. SWE-Craft consistently outperforms both synthesis baselines in these experiments, with a widening performance gap at the larger training scale. We have also evaluated CPG-guided task synthesis using Kimi-K3 and clarified how capability targets are derived and adapted to different base models.
We would greatly appreciate it if you could review these additional results and let us know whether they address your concerns. We would be glad to clarify any remaining questions during the discussion period.
Thank you for your time and consideration.
Best regards,
The Authors
Thank you for your valuable feedbacks.
We sincerely thank you for your insightful comments on evaluating the usefulness of capability targets and expanding our experimental scope. We respectfully address each of your questions and concerns below.
Q1: How are capability targets derived, and does each base model require a different synthesis procedure?
A1: We would like to clarify that different base models do not require different synthesis procedures, as SWE-Craft decouples model-specific weakness diagnosis from task synthesis (Section 3.1 and Appendix B). Only the capability targets are updated, while the CPG-guided synthesis pipeline remains unchanged.
Specifically, we derive capability targets by analyzing base-model evaluation trajectories through LLM-assisted inspection and human synthesis, translating recurring failure patterns into four standardized fields: task type, task subtype, behavioral focus, and structural reasoning scope (e.g., cross-module error reporting in Pyflakes, Appendix C.1). For a new base model, we simply update these targets from its observed failures without altering the pipeline. Moreover, existing synthesized tasks also transfer effectively to other models, as shown on Qwen3-14B in W3.
W2: Are the improvements over untargeted baselines sufficient to demonstrate the value of capability targeting?
A2: We would like to emphasize that our results clearly demonstrate the value of capability targeting, even under a setup that also benefits the untargeted baselines. As stated in Section 4.1, all methods share the same 500 repositories selected using our capability targets, giving the baselines access to target-relevant repository contexts (which already boosts them by +5.06 to +8.33 points over the base model in Table 2).
Even on top of this shared repository pool, SWE-Craft outperforms both baselines by +2.05 to +3.55 percentage points across SWE-bench Pro and Multilingual (Table 2) and achieves substantially higher profile alignment (93.16 vs. 53.36 and 42.67, Figure 3). Moreover, on benchmark subsets specifically testing our targeted capabilities (Configuration Consistency and Result/Error Propagation), SWE-Craft's lead over SWE-smith reaches +6.00 to +6.89 points on Multilingual, exceeding the overall gain of +3.55 points. This result confirms the direct benefit of incorporating capability targets into task design.
W3: Limited experimental scope in terms of backbone models, task-design agents, and training data scale.
A3: We would like to clarify that our experimental setup aligns with standard controlled evaluation protocols, and that extending our evaluation across backbone models, stronger task-design agents, and larger trajectory scales consistently confirms our main findings.
First, to evaluate cross-backbone generalization, we generate tasks and fine-tuned Qwen3-14B, a dense model from a different model family than Ling-3.0-flash, under the same 1,500-trajectory setting. SWE-Craft outperforms both baselines by +1.96 to +2.67 percentage points across both benchmarks, showing that the training benefits generalize beyond Ling-3.0-flash:
Second, to test whether CPG remains beneficial with a stronger task-design agent, we compared 11 matched same-repository task pairs synthesized with and without CPG using Kimi-K3 (released in July 2026 with 2.8T parameters and a 1M-token context window) with Claude Code. Even with this stronger agent, CPG exposes cross-module dependencies missed by unguided exploration (e.g., connecting the single-run driver, sweep launcher, and shared execution function across modules in Hydra), consistent with our No-Graph ablation where CPG improves Pass@1 by +1.33 and +1.89 points (Section 4.5).
Third, while 1,500 trajectories already match controlled SFT scales in prior work (500 to 2,136 trajectories),we scaled the training set from 1,500 to 4,500 trajectories per method on the same 1,500 underlying tasks:
Because the 4,500 trajectories are sampled from the same 1,500 tasks, the additional gains are relatively modest. Nevertheless, the baselines do not catch up: SWE-Craft's lead widens to +2.65–4.44 percentage points across both benchmarks. This demonstrates that collecting more trajectories from the same untargeted tasks does not close the performance gap in our evaluated setting.