- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SWE-Next, a pipeline for constructing executable software engineering tasks from real-world repositories. The pipeline collects base and merged commit pairs from merged pull requests and retains those where at least one previously non-passing test becomes passing, with no regressions among the compared tests. Task descriptions can optionally be generated using an LLM. To reduce environment storage and construction costs, the authors introduce repo-quarter profiles, which share dependency environments across commits within the same repository and calendar quarter. The pipeline also collects agent trajectories without exposing ground-truth test lists and requires code changes and test execution before submission. Using this pipeline, the authors construct 2,308 tasks and collect trajectories for supervised fine-tuning. They report that the resulting 7B and 14B models outperform the compared baselines of the same model size on SWE-bench Lite and Verified.
Critical strengths
- The paper addresses an important and well-established problem: constructing executable, verifiable training tasks from real-world GitHub repositories while reducing environment storage and setup costs.
- The authors present an end-to-end pipeline spanning task construction, environment management, and trajectory collection. They use the collected trajectories to fine-tune student models and report improvements on SWE-bench Lite and Verified, providing empirical evidence of the resulting data’s utility for training SWE agents.
Critical weaknesses
- Limited differentiation from prior task-construction pipelines. My main concern is that the methodological novelty is not sufficiently established. SWE-Gym and SWE-rebench already construct executable tasks from real-world repositories, and Table 1 reports substantially more tasks for SWE-rebench than for SWE-Next. While SWE-Next combines execution-grounded filtering with specific trajectory-collection mechanisms, the paper does not clearly establish which components provide a substantive advance over existing pipelines or isolate their individual contributions.
- The incremental storage benefit of repo-quarter profiles remains insufficiently established. SWE-smith already reduces storage overhead by sharing an execution environment across multiple tasks generated from the same repository version. The paper should more clearly explain their additional contribution beyond existing environment-sharing mechanisms. The storage comparison in Section 6 relies on an estimated per-candidate environment baseline rather than a measured comparison against an alternative reuse strategy on the same candidate commits. A controlled comparison of actual storage usage after accounting for shared layers, build time, and execution reliability would help establish the additional value of repo-quarter profiles.
- The downstream improvements cannot be cleanly attributed to data quality. Table 2 does not consistently control the training setup across methods. SWE-Next uses 3.7k training trajectories versus 3.3k for R2E-Gym, and the trajectory-generating models also differ. Although both use OpenHands, the comparison with SWE-smith additionally involves a different agent scaffold. These differences limit attribution of the gains specifically to the task-construction pipeline or data quality. The controlled experiment in Table 3 partially addresses this concern by matching the teacher model, trajectory count, and training hyperparameters. However, the absolute gains are modest: from 5.0% to 6.7% on Lite and from 7.8% to 8.2% on Verified. Without repeated runs or uncertainty estimates, it remains unclear whether these gains demonstrate a robust advantage.
Questions for authors
- Task-construction novelty: Could you clarify the methodological differences between SWE-Next and existing real-world executable task pipelines such as SWE-Gym and SWE-rebench? Which components are primarily responsible for the reported gains, and can their individual contributions be isolated?
- Environment storage efficiency: SWE-smith reduces storage overhead by sharing an execution environment across tasks generated from the same repository version. While SWE-Next addresses a different setting involving historical commits, could you clarify the additional benefits of repo-quarter profiles over existing environment-sharing mechanisms? Can you provide a controlled comparison against an alternative reuse strategy on the same candidate commits, measuring actual storage usage after accounting for shared layers, build time, and execution reliability?
- Controlled evaluation: Do the downstream gains persist when the teacher model, training data budget, and evaluation scaffold are matched? For the controlled experiment in Table 3, could you report results across multiple training seeds or uncertainty estimates to assess the robustness of the relatively small improvements, particularly on Verified?