- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DeNovoSWE, a dataset and construction pipeline for training code agents to generate complete repositories from documentation. The pipeline selects executable, well-tested open-source repositories; decomposes each repository into functional capabilities; generates capability-level documentation with a draft-critic-repair workflow grounded in source code and tests; removes the reference implementation and potential leakage channels; and collects execution trajectories from DeepSeek-V4-Pro. The final dataset contains 4,818 tasks and approximately 11k filtered trajectories. The paper also proposes difficulty-aware trajectory filtering, in which a difficulty score derived from code size and two LLM judgments determines a task-specific minimum test-pass threshold. Fine-tuning Qwen3-30B-A3B improves BeyondSWE-Doc2Repo from 5.8% to 47.2% and NL2RepoBench from 4.3% to 23.0%; fine-tuning Qwen3.5-35B-A3B improves the same benchmarks from 43.8% to 50.0% and from 23.5% to 27.1%. An ablation compares the adaptive filter with three fixed thresholds.
Critical strengths
1.The paper addresses an important and under-served training regime at meaningful scale.
Most existing SWE post-training data focus on issue resolution, whereas DeNovoSWE targets full repository reconstruction. The 4,818 executable tasks and roughly 11k trajectories are substantially larger than the cited repository-generation evaluation sets. This is consequential because lack of scalable, verifiable data is a genuine bottleneck for long-horizon code agents.
2.The end-to-end transfer results are strong and consistent across two backbones and two external benchmarks.
The gain from 5.8% to 47.2% on BeyondSWE-Doc2Repo for Qwen3-30B-A3B is especially large, while the stronger Qwen3.5-35B-A3B also improves on both benchmarks. Exact repository overlap with the evaluation sets is excluded. These results provide credible evidence that the overall data-generation and post-training recipe produces useful repository-level training signal rather than merely fitting one backbone or benchmark.
Critical weaknesses
1.The quality of the final dataset is not adequately validated.
The paper describes source-repository filters and an automated critic-repair pipeline, but provides no independent instance-level audit of the resulting 4,818 tasks. It does not report human checks of documentation fidelity, quantitative API/behavioral coverage, ambiguity or leakage rates, or how many instances were rejected or revised. Although DeepSeek-V4-Pro rollouts could provide useful indirect evidence, the paper also omits basic solvability and coverage statistics, such as the rollout-score distribution, Pass@1/3/6, and how many unique tasks are represented in the final 11k trajectories. Consequently, the reported repository coverage and downstream gains do not establish that the individual task documents are consistently accurate, complete, non-leaking, and solvable. This is central because the dataset itself is the paper’s primary contribution.
2.The difficulty-aware signal may conflate intrinsic difficulty with data defects, and its validation is not independent.
Low rollout pass rates can result not only from genuine task complexity, but also from incomplete documentation, environment problems, test-suite differences, or teacher-model limitations. Without a quality audit, malformed tasks may be labeled as difficult and admitted under lower trajectory thresholds, thereby retaining noisy data. Moreover, the difficulty weights are optimized for correlation with rollout scores, and apparently the same relationship is then used to validate the difficulty estimate. The paper reports neither held-out validation nor the learned weights and threshold-selection procedure. Since the adaptive filter improves over the best fixed threshold by only 1.2 and 0.7 points without uncertainty estimates or clearly matched data budgets, its specific benefit remains uncertain.
Questions for authors
1.Final dataset quality: What independent verification was performed on the final 4,818 tasks? Please report the manually reviewed fraction, criteria and error rates for documentation correctness, completeness, ambiguity, and test leakage, together with the numbers of rejected or revised instances. Please also report rollout-score statistics, Pass@1/3/6, and how many unique tasks are covered by the final 11k trajectories. Were failed reconstructions inspected to distinguish model failures from specification or environment defects?
2.Difficulty signal and threshold selection: How does the method distinguish genuinely difficult tasks from low-scoring tasks caused by incomplete documentation or environment defects? Please report the learned weights, calibration-set size, held-out correlation with rollout success, and how the threshold schedule was selected. Did either downstream benchmark influence this selection?