acceptodds
Under review as a conference paper at ICLR 2027

SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes

Abstract

As coding agents become increasingly capable, robust evaluation requires a broader and more diverse supply of verifiable software engineering (SWE) tasks. However, real-world tasks are constrained by the natural pace at which issues arise and the substantial cost of manual curation. Furthermore, existing synthetic approaches rely primarily on the parametric knowledge of large models combined with heuristic prompting, often failing to capture the authentic and complex mechanisms underlying software failures. To scale SWE task production, we introduce SWE-at-Scale, a framework that automatically synthesizes realistic issues by reverse-engineering real-world software fixes. Our key insight is that historical fixes encode mechanistic knowledge of bug formation: reversing a fix transforms repair supervision into bug-induction supervision grounded in actual development scenarios. SWE-at-Scale transfers these learned bug patterns to new code contexts and repeatedly samples diverse bug types to construct executable SWE tasks. Extensive experiments show that SWE-at-Scale generates more diverse and realistic SWE tasks than existing synthesis baselines while posing greater challenges to state-of-the-art coding agents. Notably, the model can generate distinct bug types from the same code context, with bug diversity continuing to increase even after 200 generations per file. Overall, SWE-at-Scale turns finite real-world fixes into reusable supervision for scalable synthesis of realistic, diverse, and challenging SWE tasks.

Primary area:
generative models
Keywords:
agents, coding agents, synthetic data, swe agents, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KM7LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes training models to synthesize software engineering (SWE) tasks from a given code context by learning from reversed real-world bug fixes. SWE-at-Scale is trained to directly generate bug-inducing patches, while SWE-at-Scale+ first generates a natural-language bug pattern, which an executor model then realizes as code changes.

Critical strengths
  • The proposed approach offers a scalable way to generate SWE tasks, and bug diversity continues to grow with the sampling budget.
  • Learning from reversed real-world fixes yields a more diverse and balanced bug distribution than prompting-based baselines.
  • The paper is well-written and easy to follow.
Critical weaknesses
  • It is unclear whether the difficulty of the generated tasks comes from the bugs themselves or from “hidden contracts” in the test cases that the problem statement does not convey.
    • It would be helpful to verify that the tasks are actually solvable, for instance by checking whether a strong reasoning model achieves a reasonable pass@N.
    • Alternatively, the authors could show that the generated dataset helps train more effective coding agents.
  • The usability metric could be strengthened. Although HPR and the manual non-triviality check are informative, the paper does not show whether the generated bugs are useful for evaluation or training.
  • Generalization beyond the current setting remains to be shown.
    • The evaluation covers 12 Python repositories, so it would be valuable to see results on more diverse codebases and other programming languages.
Questions for authors
  • Could the authors provide concrete examples of the bug patterns?
  • Since the issue generator sees the P2F test and its output, how do the authors ensure that the problem statement contains enough information to solve the task without leaking the solution?
2 comments

We thank the reviewer for the encourging comments.

Task Solvability and Issue Description Quality

In our experimentsm, we strictly followed the issue-generation procedures of SWE-smith, with all methods using the same pipeline to ensure a fair comparison. Our issue-generation pipeline explicitly instructs the LLM to include the relevant function or class names and expected behaviors required by the P2F tests, ensuring that the problem statements convey the necessary behavioral requirements without revealing the specific code modifications or solutions. We further manually inspected the generated problem statements to verify that they accurately describe the introduced bugs and provide sufficient information for debugging. Our inspection found no cases where the P2F tests imposed hidden behavioral requirements that were not conveyed by the problem statements.

Utility of Synthesized SWE Tasks and Generalization Across Repositories

To further validate the practical utility and generalizability of our approach, we extend SWE-at-Scale beyond the 12 repositories used in our original evaluation by synthesizing tasks across a broader collection of repository environments provided by SWE-smith. We additionally conduct a downstream SFT experiment to examine whether these tasks improve coding agents on real-world SWE issues. Specifically, we compare two training configurations: (1) 3k SWE-smith instances and (2) 1.5k SWE-smith instances combined with 1.5k SWE-at-Scale instances. SWE-at-Scale instances are generated using the same repository environments and context-selection procedure as SWE-smith, with bug injection being the primary difference. We use DeepSeek-V4-Flash-0731 as the teacher model to collect successful repair trajectories. Both configurations follow SWE-smith's training setup, using identical student models and training budgets, with 4 NVIDIA A100 GPUs for two epochs. We evaluate the resulting models on SWE-bench Verified.

Training DataSWE-bench Verified Pass@1 (%) ↑
Qwen 3.5 4b (No SFT)44.4
SWE-smith (3k)48.2
SWE-smith (1.5k) + SWE-at-Scale (1.5k)52.0

Training on SWE-smith improves Pass@1 from 44.4% to 48.2%, while incorporating SWE-at-Scale tasks further increases performance to 52.0%. This represents a 7.6-percentage-point improvement over the base model and a 3.8-point improvement over SWE-smith-only training under the same data and training budgets. These results demonstrate that our synthesized tasks are not merely challenging for existing agents but also provide useful training signals that improve generalization to held-out real-world SWE issues.

Concrete Examples of Learned Bug Patterns

We have added four concrete examples of generated bug patterns and their corresponding injection diffs in Appendix A.5. For example, the SymPy pattern exchanges the Piecewise code-generation strategies of the Python and NumPy backends. The resulting NumPy implementation uses scalar if/else expressions instead of numpy.select, causing array-valued conditions to fail. Another Django pattern replaces proxy-model identities with concrete-model identities across deletion eligibility checks and signal dispatch, causing proxy-specific signal receivers to be overlooked. Each example includes the complete six-field natural-language bug pattern and the corresponding executor-produced diff, making explicit what modification the pattern specifies and how it is realized in code.

Dear Reviewer KM7L,

We would like to share an additional update regarding repository generalization and the practical utility of our synthesized tasks.

We have publicly released 2,563 SWE-at-Scale instances covering 52 Python repositories, extending beyond the 12 repositories used in our original evaluation. These instances were synthesized using SWE-smith's repository environments, with patches, test information, container image references, and reproduction instructions provided.

The dataset is available at:

https://huggingface.co/datasets/SWE4ICLR2027/SWE-at-Scale

We hope this release provides further evidence of our approach's applicability across diverse repositories and supports reproducibility. We would be happy to address any remaining concerns.

Best regards,

The Authors

Reviewer_ykcsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SWE-at-Scale, a framework that reverses historical bug-fix commits into supervision pairs and uses this corpus to fine-tune a dedicated "bug-inducing" model that can be repeatedly sampled to synthesize new, executable bugs in arbitrary code contexts. The authors report that human evaluators prefer SWE-at-Scale and SWE-at-Scale+ generated bugs as more realistic than baseline synthetic bugs. The synthesized tasks remain challenging for four different coding-agents and that bug-type diversity scales favorably with training repository coverage, sampling temperature, and sampling budget.

Critical strengths
  • Addresses a practically important and timely problem, as scalable benchmarks of realistic, diverse, and challenging SWE tasks, is directly relevant to the software-engineering and coding-agent evaluation (also agent construction).
Critical weaknesses
  • Multiple soundness issues in evaluation:

  • (1) Although the paper involves human evaluation to show the realism of generated bugs, it is unclear the criteria for human labelling the so-called “Human Preference Rate”. In particular, it is not intuitive even if human would have consistent criteria on which bug is preferred. The current paper just mentions that “judge which one they would be more likely to encounter in real-world software development”, and such a vague criteria here make the main human evaluation results not convincing to support the benefits of the proposed approach.

  • (2) Metrics concern. The current metrics to indicate the synthesized bugs are indeed better (realistic or usable) are not that convincing. For example, the proposed Usability Rate primarily measures whether a generated code modification triggers at least one previously passing test failure and passes a non-triviality filter. Triggering a test failure does not necessarily imply that the resulting task is meaningful, realistic, or suitable for training and evaluating coding agents. How does this metric establish that the resulting issue is meaningful, realistic, and suitable for coding-agent training or evaluation.

  • (3) Downstream training utility. The paper demonstrates that the synthesized tasks are challenging for existing coding agents, but does not establish whether these tasks are useful for improving coding agents. Can the authors provide downstream training experiments comparing agents trained on SWE-at-Scale tasks against those trained on tasks generated by SWE-smith and BugPilot, under matched training budgets, and evaluate their generalization to held-out real-world issues?

  • Lack of independent validation of issue descriptions. The issue descriptions are generated by an LLM with access to the bug-inducing patches and failing tests. However, the paper does not independently evaluate whether these descriptions accurately capture the bugs, provide sufficient information for debugging, or inadvertently reveal implementation details. This raises concerns about the quality and realism of the synthesized tasks.

  • Missing ablations to establish the effectiveness of reverse-fix training. The paper does not sufficiently isolate the contribution of learning from reversed real-world fixes. It remains unclear whether the improvements come from reverse-fix supervision itself, general supervised fine-tuning, or other components such as the explicit bug-pattern representation and executor in SWE-at-Scale+. More controlled ablations would strengthen the central technical claim.

  • Potentially unfair baseline comparisons. The experimental setup restricts SWE-smith's LLM-Modify and LLM-Rewrite to the smallest enclosing functions or classes containing the original fix hunks, while SWE-at-Scale operates on complete source files. These differences in permitted modification scope may influence synthesis diversity, usability, and task difficulty. Comparisons under consistent constraints would be more convincing.

Questions for authors

Justify the soundness of the human preference rate metric. E.g., Could the authors clarify the specific criteria and instructions provided to human evaluators when assessing the Human Preference Rate? Given the subjective nature of judging which bugs are more likely to occur in real-world development, how reliable are the reported preference results?

Justify the soundness of the usability metric. E.g., How do the authors justify using Pass-to-Fail tests and non-triviality checks as evidence of task usability? Have the authors conducted additional validation to establish that the generated tasks are meaningful, realistic, and suitable for coding-agent training or evaluation?

Can the authors demonstrate that training coding agents on SWE-at-Scale-generated tasks improves their performance on held-out real-world issues? In particular, how does the training utility compare with synthetic tasks from SWE-smith and BugPilot under matched training budgets?

Have the authors independently evaluated the accuracy, completeness, and clarity of the generated issue descriptions? Given that the description generator has access to bug-inducing patches and failing tests, how do the authors ensure that the descriptions do not inadvertently reveal implementation details or introduce unrealistic debugging hints?

11 comments
AuthorsOpenReview ↗

Concise response to the reviewer's concerns

We thank the reviewer for the constructive feedback. For ease of review, we first provide concise responses to the reviewer‘s concerns.

Reliability of HPR

HPR measures bug realism rather than repair difficulty, considering Bug Plausibility, Contextual Consistency, and Behavioral Realism (full instructions in Appendix A.4). Three professional engineers independently assessed blinded bug pairs using majority vote, achieving substantial agreement (Fleiss' κ = 0.78). Against real SWE-bench Verified bugs, SWE-at-Scale achieved 55.9% HPR (95% CI: [50.8%, 60.9%]) and SWE-at-Scale+ achieved 61.2% ([56.7%, 65.5%]).

Soundness of the Usability Rate

Our usability validation strictly follows SWE-smith's P2F and non-triviality checks. Additional manual inspection confirmed that all examined injected bugs represent meaningful behavioral regressions rather than arbitrary test failures.

Issue Description Quality

All methods follow SWE-smith's issue-generation pipeline. Manual inspection identified 8 descriptions mentioning relevant file paths, but none revealed specific fixes. Similar characteristics, including location hints and occasional ambiguity, also appear in real-world issue demonstrations.

Downstream Training Utility

We compare SFT on 3k SWE-smith instances against 1.5k SWE-smith + 1.5k SWE-at-Scale instances. Both use matched repository environments and context selection, differing primarily in bug injection. DeepSeek-V4-Flash-0731 serves as the teacher model. Training follows SWE-smith's setup with identical student models and budgets (four NVIDIA A100 GPUs), evaluated on SWE-bench Verified.

Training DataSWE-bench Verified Pass@1(%)↑
Qwen 3.5 4b(base)44.4
SWE-smith(3k)48.2
SWE-smith(1.5k)+SWE-at-Scale(1.5k)52.0

Replacing half the SWE-smith data improves Pass@1 by 3.8 percentage points over SWE-smith-only training (+7.6 over the base model), demonstrating downstream training utility.

Comparison with BugPilot. Its original SWE-instances haven't been officially released. Our reproduction using DeepSeek-V4-Flash-0731 yielded only 29 validated bugs from 500 candidates (~2,500 attempts), costing USD150 (USE 5.17 per success). At this yield, generating 1.5k usable instances would cost ~$7,760, excluding trajectory collection and training. We therefore focus our matched-budget comparison on SWE-smith.

Effectiveness of Reverse-Fix Supervision

SWE-at-Scale

We compare base Qwen3.5-9B, SFT on official SWE-smith LLM-Modify/Rewrite instances, and our reverse-fix SFT. All share the same backbone, input/output interface, and validation; SFT configurations are identical. All metrics use 200 generations/file at temperature 0.6. Behavioral uniqueness deduplicates usable bugs by P2F tests and failure behaviors.

Training DataUsability Rate(%)↑Behavioral Uniqueness(%)↑MBTE↑
None(Qwen3.5-9B)10.238.00.67
LLM-Modify SFT41.340.90.92
LLM-Rewrite SFT35.069.31.19
SWE-at-Scale(Reverse-Fix SFT)47.983.742.063

LLM-Modify SFT achieves relatively high usability but limited diversity due to localized edits. LLM-Rewrite SFT improves diversity but has lower usability, as rewriting primarily teaches alternative implementations rather than targeted bug injection. Reverse-fix SFT outperforms both across all metrics, demonstrating benefits beyond general fine-tuning. Notably, the three fine-tuned models generate largely non-overlapping bugs, suggesting that our approach of reversing bug-fix pairs into bug-injection supervision is applicable across different data sources, with real-world fixes yielding the strongest overall performance.

SWE-at-Scale+

We compare base and reverse-fix-trained Qwen3.5-9B pattern generators using the same executor.

VariantUsability Rate(%)↑Behavioral Uniqueness(%)↑MBTE↑
Qwen3.5-9B+Executor38.664.21.45
SWE-at-Scale+(Reverse-Fix SFT)+Executor62.482.72.08

With the executor fixed, reverse-fix pattern learning improves usability by 23.8 percentage and substantially increases diversity, confirming benefits beyond the executor alone.

Fairness of Baseline Comparisons

We introduce LLM-Modify (Full-File), allowing modifications anywhere within the target file. Both variants use identical models, prompts, sampling budgets, and validation, differing only in edit scope.

MethodUsability Rate(%)↑Behavioral Uniqueness(%)↑MBTE↑
LLM-Modify44.146.31.22
LLM-Modify(full-file)32.671.01.89
SWE-at-Scale47.983.742.06

Expanding edit scope improves behavioral uniqueness (46.3%→71.0%) and MBTE (1.22→1.89), but reduces usability (44.1%→32.6%), often causing widespread failures or non-executable repositories. Extending LLM-Rewrite to multiple masked code blocks yielded only 13/500 successful injections (2.6%) at higher inference costs. SWE-at-Scale remains stronger under the same file-level edit scope.

Hi Reviewer ykcs,

We appreciate your recognition of the practical importance of our work. We would love to know whether our response has addressed your concerns about evaluation reliability, downstream training utility, and baseline fairness. We are happy to continue the discussion and answer any further questions you may have.

Dear Reviewer ykcs,

We would like to share an additional update regarding the reproducibility of our downstream training experiments.

We have publicly released 2,563 SWE-at-Scale instances synthesized using SWE-smith's repository environments, including problem statements, patches, test information, and references to the original container images.

The dataset and reproduction instructions are available at:

https://huggingface.co/datasets/SWE4ICLR2027/SWE-at-Scale

We hope this release further addresses your concerns regarding the practical utility and reproducibility of our synthesized tasks. We would be happy to answer any additional questions.

Best regards, The Authors

Reviewer_sR9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWE-at-Scale, a a method for synthesizing repository-level SWE tasks by learning bug injection from reversed real-world fixes. The authors collect about 134K bug fix instances from SWE-Fixer, SWE-rebench, SWE-bench-Live, and SWE-Gym. They exclude repositories that appear in SWE-bench Verified, filter the data, and rebalance it into 13k (fixed file + bug-inducing diff) pairs from ~700 repositories. They then LoRA-fine-tune Qwen3.5-9B to produce bug-inducing diffs from a full, line-numbered golden file. A variant, SWE-at-Scale+, instead predicts a natural-language “bug pattern,” using supervision distilled from DeepSeek-V4-Flash, and an executor model then turns that pattern into code. Candidates are kept only if they cause at least one pass-to-fail test, and an LLM writes the issue text. The method is evaluated against SWE-smith and BugPilot using the same target files as SWE-bench Verified instances. The paper reports usability (94.0% / 72.6%), human preference over real issues (61.2% / 55.9%), lower Pass@1 across four solver models, and a within-file bug-type entropy (MBTE) that increases with the number of training repositories and with the sampling budget, up to K=200.

Critical strengths
  • The idea is simple and well motivated: reversed real fixes give grounded supervision for where and how bugs get introduced, so the method doesn’t have to rely on an LLM’s prior for “what a bug looks like.” Training a small 9B model on one GPU that beats prompted frontier-model baselines on usability is nice. If the diversity and difficulty claims are true, this could be a cheap source of evaluation tasks
  • Careful experimental design. They use training data that's disjoint across repos, the same target files across all methods, model-disjoint issue generation and resolution, 3 runs per configuration, and an ablation of repository coverage
Critical weaknesses
  • Greater difficulty doesn't mean better SWE tasks. An additional human evaluation of issue quality and solvability with qualitative examples of tasks that coding agents can't solve would support the claim that it's generating high quality tasks.
  • The diversity and realism claims are not supported. The paper reports increasing Mean Bug Type Entropy and an 88.27% unique-patch rate at 200 generations/file. However, distinct patches are not necessarily distinct failures because small changes can produce different diffs while introducing the same bug. The diversity curve in Figure 8 appears to approach saturation by approximately 100 generations.
  • The paper demonstrates that its trained models outperform prompt-based synthesis baselines. However, several factors differ between methods, such as dedicated fine-tuning versus prompting, the training distribution, the permissible edit locations, and direct patch generation versus natural-language pattern generation with an executor. It is unclear how much of the improvement comes from reverse-fix supervision, the intermediate representation, or the executor.
Questions for authors
  • Have the authors evaluated whether the generated issue descriptions contain sufficient information for a competent developer to resolve the problem? Can they distinguish meaningful task difficulty from ambiguity or artificial regressions?
  • Can the authors provide an ablation that separates the effect of training on reversed fixes from the effect of the pattern-generation and execution pipeline?
  • What are the sample sizes, inter-annotator agreement, and confidence intervals for the human preference results?
  • How are distinct bug semantics differentiated from syntactically different patches, and what explains nonzero within-file diversity at temp 0?
7 comments
AuthorsOpenReview ↗

Issue Description Quality and Semantic Bug Diversity and Scalability

We thank the reviewer for the constructive comments.

Issue Description Quality

We first clarity that in our experiments, we strictly followed the issue-generation procedures of SWE-smith, with all methods using the same pipeline to ensure a fair comparison. In response to the reviewer's concern, we further manually inspected the generated problem statements, focusing on whether they accurately describe the introduced bugs and provide sufficient information to understand the expected behavior. Among the tasks generated by SWE-at-Scale, we identified 8 problem statements that explicitly mention the file paths associated with the bugs and 5 instances whose bug descriptions are relatively ambiguous. However, these ambiguous descriptions still provide sufficient information to understand the expected behavior. Upon examining the corresponding real-world issue demonstrations, we found similar characteristics, including references to potentially affected files and relatively vague descriptions of the intended behavior. These observations suggest that such characteristics are not unique to SWE-at-Scale but also reflect how issues are described in real-world SWE instances.

Semantic Bug Diversity and Scalability

Semantic Bug Diversity. To further address the reviewer's concern, we additionally deduplicated the generated bugs based on their Pass-to-Fail (P2F) tests and corresponding failure behaviors. The results show that 83.74% of usable patches exhibit distinct failure behaviors, compared to the originally reported 88.27% patch uniqueness rate. This provides additional execution-based evidence that the generated bugs exhibit behavioral diversity beyond syntactic differences. Nonzero Diversity at Temperature 0. We identified an error in the data preprocessing for the original figure, where unified-diff metadata differences (varying index lines) were not properly normalized. We have corrected this issue, manually verified the bug-subtype assignments, and recalculated MBTE across all temperature settings. The revised results are presented below. After correction, the MBTE at temperature 0 decreases from 0.527 to 0.131, with only 8 out of 50 target files exhibiting multiple bug subtypes across 10 repeated generations. Prior work has shown that greedy decoding does not guarantee deterministic outputs in practice, as variations in batching and floating-point computations can lead to different generations even at temperature 0 (Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference, NIPS '25: Proceedings of the 39th International Conference on Neural Information Processing Systems). This provides a plausible explanation for the remaining nonzero diversity. Importantly, temperature 0.6 still achieves the highest MBTE (1.109), preserving our main finding regarding the effect of sampling temperature on bug-type diversity.

TemperatureRevised MBTE (bits)
0.00.131
0.20.753
0.41.013
0.61.109
0.80.866
1.00.644

Additional experiments and detailed analyses regarding Contributions of Reverse-Fix Supervision and Individual Components and Statistical Analysis of Human Preference Evaluation are provided in our subsequent response.

Contributions of Reverse-Fix Supervision and Individual Components

SWE-at-Scale

We conduct a controlled ablation to isolate the contribution of reverse-fix supervision from general supervised fine-tuning. To investigate this, we compare four variants: (1) the base Qwen3.5-9B model without fine-tuning, (2) the same model fine-tuned on bug-inducing patches generated by SWE-smith's LLM-Modify strategy, (3) the same model fine-tuned on patches generated by LLM-Rewrite, and (4) SWE-at-Scale trained on reversed real-world fixes. All three models are fine-tuned under identical training configurations. Specifically, the LLM-Modify and LLM-Rewrite variants are trained on instances from the official SWE-smith dataset, while SWE-at-Scale is trained on reversed real-world fixes. All variants share the same Qwen3.5-9B backbone, input format, and patch-generation interface, and are evaluated on identical target files using the same sampling budgets and execution-based validation pipeline.

Training DataUsability Rate (%) ↑Behavioral Uniqueness (%) ↑MBTE ↑
None (Base Qwen3.5-9B)10.238.00.67
LLM-Modify SFT41.340.90.92
LLM-Rewrite SFT35.069.31.19
SWE-at-Scale (Reverse-Fix SFT)47.983.742.063

All three metrics—Usability Rate, Behavioral Uniqueness, and MB2TE—are evaluated under the same repeated-sampling setting, with 200 generations per target file at a temperature of 0.6. Behavioral Uniqueness is measured by deduplicating usable bugs based on their Pass-to-Fail (P2F) tests and corresponding failure behaviors.

The results reveal a trade-off between bug usability and diversity across different training data sources. LLM-Modify SFT achieves high usability but limited diversity, as its training examples primarily involve localized and relatively simple modifications. In contrast, LLM-Rewrite SFT exhibits slightly higher diversity but lower usability. Since LLM-Rewrite introduces bugs as a byproduct of code rewriting rather than through intentional bug injection, the trained model tends to learn how to generate alternative implementations with similar functionality instead of deliberately introducing targeted bugs. SWE-at-Scale achieves a better performance, maintaining high usability while substantially improving behavioral uniqueness and bug-subtype diversity. Notably, the three fine-tuned models generate largely non-overlapping bugs, suggesting that our approach of reversing bug-fix pairs into bug-injection supervision is applicable across different data sources, with real-world fixes yielding the strongest overall performance.

Fairness of Edit Scope.

To address the concern regarding differences in permissible edit locations, we additionally introduce LLM-Modify (full file), which extends the original LLM-Modify baseline by allowing bug injection anywhere within the target file rather than restricting modifications to selected functions or classes. Both variants use the same underlying LLM, bug-injection instructions, sampling budgets, and validation pipeline, with the permissible edit scope being the primary experimental variable.

Training DataUsability Rate (%) ↑Behavioral Uniqueness (%) ↑MBTE ↑
LLM-Modify44.146.31.22
LLM-Modify (full file)32.671.01.89
SWE-at-Scale47.983.742.06

Expanding LLM-Modify's edit scope improves behavioral uniqueness (40.9% → 71.0%) and MBTE (1.22 → 1.89), but reduces usability (44.1% → 32.6%). We also explored extending LLM-Rewrite by masking and simultaneously rewriting multiple code blocks. However, this frequently caused widespread test failures and substantially increased runtime and inference costs, yielding only 13 successful bug injections out of 500 attempts (2.6%). Our inspection reveals that unrestricted modifications often cause widespread test failures or render repositories non-executable, rather than introducing controlled bugs.

SWE-at-Scale+

To further isolate the contribution of bug-pattern learning in SWE-at-Scale+, we compare two variants: (1) the base Qwen3.5-9B model prompted to generate bug patterns, and (2) our reverse-fix-trained Qwen3.5-9B model. Both variants use the same input format, bug-pattern representation, agent executor, and execution-based validation pipeline. Considering the computational cost of agent-based execution, all three metrics are evaluated under the same repeated-sampling setting, with 50 generations per target file at temperature 0.6.

VariantUsability Rate (%) ↑Behavioral Uniqueness (%) ↑MBTE ↑
Base Qwen3.5-9B + Executor38.664.21.45
SWE-at-Scale+ (Reverse-Fix SFT + Executor)62.482.72.08

Under the same agent executor, SWE-at-Scale+ consistentlu achieves higher usability, behavioral uniqueness, and bug-subtype diversity than the base model. This suggests that explicitly learning bug patterns from reversed real-world fixes provides more effective and diverse bug-injection instructions, rather than relying solely on the capabilities of the executor.

Statically Analysis of Human Preference Evaluation

We conducted a blind pairwise evaluation of all generated bug pairs, with each pair independently assessed by three professional software engineers and the final preference determined by majority vote. The annotations showed substantial inter-annotator agreement (Fleiss' κ = 0.78). Specifically, SWE-at-Scale achieved a Human Preference Rate of 55.9% (95% CI: [51.8%, 60.0%]), while SWE-at-Scale+ achieved 61.2% (95% CI: [57.1%, 65.3%]). Both confidence intervals lie above the 50% reference level, supporting the reliability of our human preference results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Additional Experiments

We thank all reviewers for their constructive comments and valuable suggestions. Based on the reviewers' feedback, we have substantially revised our paper and conducted additional experiments to strengthen our claims. The major changes are summarized as follows:

  • Semantic Bug Diversity and Scalability. We added a behavioral diversity analysis by deduplicating generated bugs based on their Pass-to-Fail (P2F) tests and corresponding failure behaviors, showing that 83.74% of usable patches exhibit distinct behavioral regressions. We also corrected the preprocessing error affecting the original temperature analysis, manually verified bug-subtype assignments, and recalculated MBTE across all temperature settings. Additionally, we reported the diversity growth from K=100 to K=200.

  • Human Evaluation and Task Quality. We clarified the criteria and instructions for Human Preference Rate (HPR), covering Bug Plausibility, Contextual Consistency, and Behavioral Realism. We further reported the agreement among three independent professional software engineers (Fleiss' κ = 0.78) and the corresponding 95% confidence intervals. We also manually inspected the generated problem statements and injected bugs to assess their meaningfulness and potential solution leakage.

  • Effectiveness of Reverse-Fix Supervision. We introduced controlled SFT ablations comparing the base Qwen3.5-9B model with models trained on SWE-smith's LLM-Modify, LLM-Rewrite, and our reverse-fix supervision. The results show that SWE-at-Scale achieves the highest usability (47.9%), behavioral uniqueness (83.74%), and MBTE (2.063), demonstrating the benefits of learning bug-inducing transformations from reversed real-world fixes beyond general supervised fine-tuning.

  • Fairness of Baseline Comparisons and Component Ablations. We introduced LLM-Modify (Full-File) to control for differences in permissible edit scopes. We also compared prompted and reverse-fix-trained bug-pattern generators under the same SWE-at-Scale+ executor. These experiments further isolate the contributions of edit scope and reverse-fix-trained bug-pattern learning.

  • Downstream Training Utility. We conducted controlled SFT experiments comparing 3k SWE-smith instances against a combination of 1.5k SWE-smith and 1.5k SWE-at-Scale instances. Using identical training configurations and repository environments, the latter improves SWE-bench Verified Pass@1 from 48.2% to 52.0%, demonstrating the practical training utility of our synthesized instances. We also investigated the reproduction cost of BugPilot as an additional reference for synthesis efficiency.

  • Dataset Release and Reproducibility. We have publicly released 2,672 SWE-at-Scale instances covering 52 Python repositories on Hugging Face, including problem statements, patches, test information, and references to the corresponding container images. This release enables independent evaluation and facilitates future research.

We believe these additional experiments, analyses, and clarifications substantially strengthen the empirical evidence and address the reviewers' concerns. We will incorporate these additional experiments, analyses, and clarifications into the revised version of the manuscript. We welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.