acceptodds
Under review as a conference paper at ICLR 2027

DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch

Abstract

As the capabilities of LLM-based code agents continue to advance, their expected role is expanding beyond localized bug fixing in existing codebases toward architecting and implementing complete software repositories from high-level specifications. However, training agents for such long-horizon software engineering tasks remains difficult due to the scarcity of large-scale, verifiable whole-repository generation data. In this paper, we introduce DeNovoSWE, a large-scale dataset for whole-repository generation. DeNovoSWE comprises 4,818 high-quality instances, where each instance requires generating a complete repository from documentation. Our dataset is automatically constructed through a carefully designed sandboxed agentic workflow, enabling scalable curation without human annotation. DeNovoSWE is constructed with "divide and conquer" and critic-repair philosophy. To balance data quality and diversity, we further introduce a difficulty-aware trajectory filtering strategy. Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
SWE, Code Agent, Agent, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RwDXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeNovoSWE, a dataset and construction pipeline for training code agents to generate complete repositories from documentation. The pipeline selects executable, well-tested open-source repositories; decomposes each repository into functional capabilities; generates capability-level documentation with a draft-critic-repair workflow grounded in source code and tests; removes the reference implementation and potential leakage channels; and collects execution trajectories from DeepSeek-V4-Pro. The final dataset contains 4,818 tasks and approximately 11k filtered trajectories. The paper also proposes difficulty-aware trajectory filtering, in which a difficulty score derived from code size and two LLM judgments determines a task-specific minimum test-pass threshold. Fine-tuning Qwen3-30B-A3B improves BeyondSWE-Doc2Repo from 5.8% to 47.2% and NL2RepoBench from 4.3% to 23.0%; fine-tuning Qwen3.5-35B-A3B improves the same benchmarks from 43.8% to 50.0% and from 23.5% to 27.1%. An ablation compares the adaptive filter with three fixed thresholds.

Critical strengths

1.The paper addresses an important and under-served training regime at meaningful scale.

Most existing SWE post-training data focus on issue resolution, whereas DeNovoSWE targets full repository reconstruction. The 4,818 executable tasks and roughly 11k trajectories are substantially larger than the cited repository-generation evaluation sets. This is consequential because lack of scalable, verifiable data is a genuine bottleneck for long-horizon code agents.

2.The end-to-end transfer results are strong and consistent across two backbones and two external benchmarks.

The gain from 5.8% to 47.2% on BeyondSWE-Doc2Repo for Qwen3-30B-A3B is especially large, while the stronger Qwen3.5-35B-A3B also improves on both benchmarks. Exact repository overlap with the evaluation sets is excluded. These results provide credible evidence that the overall data-generation and post-training recipe produces useful repository-level training signal rather than merely fitting one backbone or benchmark.

Critical weaknesses

1.The quality of the final dataset is not adequately validated.

The paper describes source-repository filters and an automated critic-repair pipeline, but provides no independent instance-level audit of the resulting 4,818 tasks. It does not report human checks of documentation fidelity, quantitative API/behavioral coverage, ambiguity or leakage rates, or how many instances were rejected or revised. Although DeepSeek-V4-Pro rollouts could provide useful indirect evidence, the paper also omits basic solvability and coverage statistics, such as the rollout-score distribution, Pass@1/3/6, and how many unique tasks are represented in the final 11k trajectories. Consequently, the reported repository coverage and downstream gains do not establish that the individual task documents are consistently accurate, complete, non-leaking, and solvable. This is central because the dataset itself is the paper’s primary contribution.

2.The difficulty-aware signal may conflate intrinsic difficulty with data defects, and its validation is not independent.

Low rollout pass rates can result not only from genuine task complexity, but also from incomplete documentation, environment problems, test-suite differences, or teacher-model limitations. Without a quality audit, malformed tasks may be labeled as difficult and admitted under lower trajectory thresholds, thereby retaining noisy data. Moreover, the difficulty weights are optimized for correlation with rollout scores, and apparently the same relationship is then used to validate the difficulty estimate. The paper reports neither held-out validation nor the learned weights and threshold-selection procedure. Since the adaptive filter improves over the best fixed threshold by only 1.2 and 0.7 points without uncertainty estimates or clearly matched data budgets, its specific benefit remains uncertain.

Questions for authors

1.Final dataset quality: What independent verification was performed on the final 4,818 tasks? Please report the manually reviewed fraction, criteria and error rates for documentation correctness, completeness, ambiguity, and test leakage, together with the numbers of rejected or revised instances. Please also report rollout-score statistics, Pass@1/3/6, and how many unique tasks are covered by the final 11k trajectories. Were failed reconstructions inspected to distinguish model failures from specification or environment defects?

2.Difficulty signal and threshold selection: How does the method distinguish genuinely difficult tasks from low-scoring tasks caused by incomplete documentation or environment defects? Please report the learned weights, calibration-set size, held-out correlation with rollout success, and how the threshold schedule was selected. Did either downstream benchmark influence this selection?

Reviewer_AAi5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DeNovoSWE, designed to train intelligent agents capable of generating complete code repositories from documentation. The authors constructed 4,818 tasks through functional decomposition and documentation inspection revisions, and selected approximately 11,000 execution traces based on task difficulty for fine-tuning. Two Qwen models achieved performance improvements on both BeyondSWE Doc2Repo and NL2RepoBench benchmarks.

Critical strengths
  1. Scalable Data Pipeline: The paper effectively converts real-world code repositories into executable generation tasks equipped with test suites and environments, providing a substantial dedicated training dataset for full-repository code generation.
  2. Consistent Performance Improvements: As shown in Table 3, both evaluated models demonstrate notable performance gains across two benchmark datasets. Specifically, the stronger Qwen3.5 model achieves improvements of 6.2 and 3.6 percentage points, validating the utility and quality of the proposed dataset.
Critical weaknesses
  1. Incomplete Ablation Analysis: The current ablation study only compares different trajectory filtering thresholds. It does not independently verify the individual contributions of function decomposition or documentation checking and revision to overall task quality or downstream performance.
  2. Unclear Experimental Controls: Table 4 lacks details regarding the volume of retained data and the training budget (e.g., token count) for each group. Consequently, it remains ambiguous whether the minor performance gains stem directly from difficulty-based filtering or simply from variations in training budget.
Questions for authors
  1. Documentation-Test Consistency Audit: Could the authors conduct an independent manual audit to evaluate the alignment between generated documentation and test requirements? This would help distinguish whether execution failures are caused by documentation omissions/discrepancies or by agent implementation errors.
  2. Controlled Comparison of Filtering Strategies: To properly evaluate the difficulty-based filtering strategy, could the authors ensure the total number of training tokens is controlled across comparisons? Additionally, reporting the difficulty distribution of retained tasks along with variance across multiple runs would help clarify whether retaining harder tasks yields consistent performance gains.
  3. Granular Task Performance & Full-Pass Rate: Could the authors provide the proportion of repositories where all tests pass completely? It would also be insightful to see a performance breakdown categorized by repository scale/complexity to confirm whether gains hold for larger, more complex repository generation tasks.
Reviewer_5dxrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DeNovoSWE, a dataset of 4,818 tasks designed to train AI coding agents to build entire software repositories from documentation, addressing the shortage of training data for work beyond individual bug fixes. Its automated pipeline breaks existing repositories into functional capabilities, writes documentation grounded in source code and test behavior, and uses a draft, critique, and repair process to improve completeness and accuracy. Agents then reconstruct the software in isolated environments that prevent access to the original implementation, with unit tests checking their results. A key idea is to filter training examples according to task difficulty, retaining useful partial solutions to harder problems instead of requiring uniformly near-perfect results. Fine-tuning Qwen3-30B-A3B on this data raises its BeyondSWE-Doc2Repo score from 5.8% to 47.2%, suggesting that targeted, verifiable training data can substantially improve agents’ ability to complete complex coding projects over many step

Critical strengths

One major strength is that DeNovoSWE provides 4,818 tasks for training AI agents to build complete software projects. These tasks require agents to make different files and components work together, which goes beyond fixing a single bug. The authors select projects with reliable tests, so they can check whether the generated software works. They also block access to the original code during task execution, helping ensure that agents solve the problems themselves.

Another strength is how carefully the authors prepare the instructions for each task. They divide a project into its main features, examine the code and tests to identify required behavior, and repeatedly review and improve the documentation. This matters because an agent can fail simply because its instructions leave out something essential. The approach aims to provide enough information to build working software while still leaving meaningful design and coding decisions to the agent.

The paper also makes a useful choice when selecting examples for training: it keeps some partially successful attempts at difficult tasks while requiring better results on easier ones. This helps prevent the training data from becoming dominated by easy problems. Experiments show that this approach performs slightly better than applying the same standard to every task. Training on the full dataset produces much larger gains, including an increase from 5.8% to 47.2% on Doc2Repo for one model, with improvements also seen on another model and a second benchmark

Critical weaknesses

The biggest weakness is that the paper misses several closely related approaches, making its contribution harder to judge. The most important omission is SWE-Playground, which already generates synthetic software projects and trains agents to implement libraries from scratch. This directly challenges the paper’s broad framing that earlier training data mainly covers bug fixing. SWE-smith, R2E-Gym, SWE-Factory, MEnvAgent, and SWE-rebench V2 also address automated construction and verification of software environments. DeNovoSWE cites an earlier SWE-rebench paper, but does not discuss V2. These methods have different goals, so their benchmark scores should not be compared directly. However, the authors should explain what their approach adds in task complexity, documentation quality, verification, and construction efficiency.

The missing terminal-agent literature also leaves important parts of the method insufficiently compared. Endless Terminals already builds verified environments automatically and uses them for RL. TermiGen uses iterative multi-agent refinement and a generator-critic process, making it relevant to DeNovoSWE’s critique-and-repair design. Nemotron-Terminal studies filtering, curriculum learning, and data scaling, while Terminal-World jointly generates task instructions, environments, and training trajectories. These comparisons would help establish whether DeNovoSWE’s design choices improve on existing approaches. Reporting construction costs, rejection rates, and performance at different dataset sizes would also provide stronger evidence for scalability. Its supervised fine-tuning results are useful, but its value as an RL environment collection remains untested.

The experiments also leave uncertainty about which proposed components actually matter. The paper does not separately test capability decomposition, test-based profiling, or the critic-repair loop. Difficulty-aware filtering improves over the strongest fixed threshold by only 1.2 percentage points on Doc2Repo and 0.7 on NL2Repo, without reported uncertainty or a comparison holding the amount of retained training data constant. Recent approaches such as SETA, Recursive Synthesis, and Envs-FORGE offer additional comparisons for difficulty control and environment expansion, while MindForge is particularly relevant to building software from scratch. These newer papers appeared after DeNovoSWE’s June 2026 public release, so they should be treated as comparisons for an updated evaluation, rather than omissions from the original release.

  1. Training Versatile Coding Agents in Synthetic Environments (SWE-Playground). December 2025. https://arxiv.org/abs/2512.12216
  2. SWE-smith: Scaling Data for Software Engineering Agents. April 2025. https://arxiv.org/abs/2504.21798
  3. R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. April 2025. https://arxiv.org/abs/2504.07164
  4. SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks. June 2025. https://arxiv.org/abs/2506.10954
  5. MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering. January 2026. https://arxiv.org/abs/2601.22859
  6. SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale. February 2026. https://arxiv.org/abs/2602.23866
  7. Endless Terminals: Scaling RL Environments for Terminal Agents. January 2026. https://arxiv.org/abs/2601.16443
  8. TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents. February 2026. https://arxiv.org/abs/2602.07274
  9. On Data Engineering for Scaling LLM Terminal Capabilities (Nemotron-Terminal). February 2026. https://arxiv.org/abs/2602.21193
  10. Terminal-World: Scaling Terminal-Agent Environments via Agent Skills. May 2026. https://arxiv.org/abs/2605.20876
  11. SETA: Scaling Environments for Terminal Agents. July 2026. https://arxiv.org/abs/2607.10891
  12. Recursive Synthesis for Long-Horizon Terminal Tasks. August 2026. https://arxiv.org/abs/2608.05466
  13. Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL. August 2026. https://arxiv.org/abs/2608.14312
  14. MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis. July 2026. https://arxiv.org/abs/2607.27146
Questions for authors

How does DeNovoSWE advance beyond SWE-Playground, which already trains agents to implement libraries from scratch? What specific limitation of that approach does your method address?

Would DeNovoSWE still outperform existing software training datasets when using the same base model, teacher model, training-token budget, and evaluation setup? This would help separate the benefit of repository-generation tasks from differences in data quantity or teacher quality.

How much does each pipeline component contribute? Could you remove capability decomposition, test-based profiling, and critic-repair separately, and compare difficulty-aware filtering against fixed filtering while keeping the amount of retained training data equal?

What does it cost to produce a valid task, including rejected candidates, documentation revisions, and solution attempts? How does performance change as the number of training environments grows, and how does this compare with other automated environment pipelines?

Since documentation is constructed using the original code and tests, how do you establish that tasks teach broader software engineering skills? Would improvements persist with independently written requirements and additional tests, and would the environments also support effective RL training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.