- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes training models to synthesize software engineering (SWE) tasks from a given code context by learning from reversed real-world bug fixes. SWE-at-Scale is trained to directly generate bug-inducing patches, while SWE-at-Scale+ first generates a natural-language bug pattern, which an executor model then realizes as code changes.
Critical strengths
- The proposed approach offers a scalable way to generate SWE tasks, and bug diversity continues to grow with the sampling budget.
- Learning from reversed real-world fixes yields a more diverse and balanced bug distribution than prompting-based baselines.
- The paper is well-written and easy to follow.
Critical weaknesses
- It is unclear whether the difficulty of the generated tasks comes from the bugs themselves or from “hidden contracts” in the test cases that the problem statement does not convey.
- It would be helpful to verify that the tasks are actually solvable, for instance by checking whether a strong reasoning model achieves a reasonable pass@N.
- Alternatively, the authors could show that the generated dataset helps train more effective coding agents.
- The usability metric could be strengthened. Although HPR and the manual non-triviality check are informative, the paper does not show whether the generated bugs are useful for evaluation or training.
- Generalization beyond the current setting remains to be shown.
- The evaluation covers 12 Python repositories, so it would be valuable to see results on more diverse codebases and other programming languages.
Questions for authors
- Could the authors provide concrete examples of the bug patterns?
- Since the issue generator sees the P2F test and its output, how do the authors ensure that the problem statement contains enough information to solve the task without leaking the solution?
2 comments
Dear Reviewer KM7L,
We would like to share an additional update regarding repository generalization and the practical utility of our synthesized tasks.
We have publicly released 2,563 SWE-at-Scale instances covering 52 Python repositories, extending beyond the 12 repositories used in our original evaluation. These instances were synthesized using SWE-smith's repository environments, with patches, test information, container image references, and reproduction instructions provided.
The dataset is available at:
https://huggingface.co/datasets/SWE4ICLR2027/SWE-at-Scale
We hope this release provides further evidence of our approach's applicability across diverse repositories and supports reproducibility. We would be happy to address any remaining concerns.
Best regards,
The Authors
We thank the reviewer for the encourging comments.
Task Solvability and Issue Description Quality
In our experimentsm, we strictly followed the issue-generation procedures of SWE-smith, with all methods using the same pipeline to ensure a fair comparison. Our issue-generation pipeline explicitly instructs the LLM to include the relevant function or class names and expected behaviors required by the P2F tests, ensuring that the problem statements convey the necessary behavioral requirements without revealing the specific code modifications or solutions. We further manually inspected the generated problem statements to verify that they accurately describe the introduced bugs and provide sufficient information for debugging. Our inspection found no cases where the P2F tests imposed hidden behavioral requirements that were not conveyed by the problem statements.
Utility of Synthesized SWE Tasks and Generalization Across Repositories
To further validate the practical utility and generalizability of our approach, we extend SWE-at-Scale beyond the 12 repositories used in our original evaluation by synthesizing tasks across a broader collection of repository environments provided by SWE-smith. We additionally conduct a downstream SFT experiment to examine whether these tasks improve coding agents on real-world SWE issues. Specifically, we compare two training configurations: (1) 3k SWE-smith instances and (2) 1.5k SWE-smith instances combined with 1.5k SWE-at-Scale instances. SWE-at-Scale instances are generated using the same repository environments and context-selection procedure as SWE-smith, with bug injection being the primary difference. We use DeepSeek-V4-Flash-0731 as the teacher model to collect successful repair trajectories. Both configurations follow SWE-smith's training setup, using identical student models and training budgets, with 4 NVIDIA A100 GPUs for two epochs. We evaluate the resulting models on SWE-bench Verified.
Training on SWE-smith improves Pass@1 from 44.4% to 48.2%, while incorporating SWE-at-Scale tasks further increases performance to 52.0%. This represents a 7.6-percentage-point improvement over the base model and a 3.8-point improvement over SWE-smith-only training under the same data and training budgets. These results demonstrate that our synthesized tasks are not merely challenging for existing agents but also provide useful training signals that improve generalization to held-out real-world SWE issues.
Concrete Examples of Learned Bug Patterns
We have added four concrete examples of generated bug patterns and their corresponding injection diffs in Appendix A.5. For example, the SymPy pattern exchanges the
Piecewisecode-generation strategies of the Python and NumPy backends. The resulting NumPy implementation uses scalarif/elseexpressions instead ofnumpy.select, causing array-valued conditions to fail. Another Django pattern replaces proxy-model identities with concrete-model identities across deletion eligibility checks and signal dispatch, causing proxy-specific signal receivers to be overlooked. Each example includes the complete six-field natural-language bug pattern and the corresponding executor-produced diff, making explicit what modification the pattern specifies and how it is realized in code.