acceptodds
Withdrawn from ICLR 2027

SWE-Next: Scalable Real-World Software Engineering Tasks for Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Executable software engineering data is valuable for training SWE agents, but scaling it remains difficult for two reasons: only a small fraction of real repository changes yield verifiable, high-signal task instances, and naively building repository-specific environments quickly becomes the dominant systems cost. We present **SWE-Next**, an execution-grounded framework for scalable SWE task and trajectory collection. On the data side, SWE-Next mines real merged pull requests, executes candidate base/merged commit pairs, and retains only those that produce strict test improvements without regressions, yielding self-verifying instances. It also applies strict submission gating so that collected trajectories remain evidence-driven rather than speculative. On the systems side, SWE-Next introduces **reusable repo-quarter profiles**, which reuse the same environment across nearby commits in time while keeping each task run separate and reproducible. Using only **30 hours** and **639GB** of environment storage, SWE-Next processes 3,971 seed repositories and 102,582 candidate commit pairs mined from real merged PRs to construct a dataset of 2,308 self-verifying instances. Experiments show that SWE-Next improves downstream pass@1 with fewer or comparable training trajectories, indicating that its gains come not from a stronger trajectory generator, but from higher-signal execution-grounded supervision and more efficient data collection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
software engineering agents, executable task synthesis, coding agent, trajectory-based post-training, environment reuse

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ycC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SWE-Next, a pipeline for constructing executable software engineering tasks from real-world repositories. The pipeline collects base and merged commit pairs from merged pull requests and retains those where at least one previously non-passing test becomes passing, with no regressions among the compared tests. Task descriptions can optionally be generated using an LLM. To reduce environment storage and construction costs, the authors introduce repo-quarter profiles, which share dependency environments across commits within the same repository and calendar quarter. The pipeline also collects agent trajectories without exposing ground-truth test lists and requires code changes and test execution before submission. Using this pipeline, the authors construct 2,308 tasks and collect trajectories for supervised fine-tuning. They report that the resulting 7B and 14B models outperform the compared baselines of the same model size on SWE-bench Lite and Verified.

Critical strengths
  1. The paper addresses an important and well-established problem: constructing executable, verifiable training tasks from real-world GitHub repositories while reducing environment storage and setup costs.
  2. The authors present an end-to-end pipeline spanning task construction, environment management, and trajectory collection. They use the collected trajectories to fine-tune student models and report improvements on SWE-bench Lite and Verified, providing empirical evidence of the resulting data’s utility for training SWE agents.
Critical weaknesses
  1. Limited differentiation from prior task-construction pipelines. My main concern is that the methodological novelty is not sufficiently established. SWE-Gym and SWE-rebench already construct executable tasks from real-world repositories, and Table 1 reports substantially more tasks for SWE-rebench than for SWE-Next. While SWE-Next combines execution-grounded filtering with specific trajectory-collection mechanisms, the paper does not clearly establish which components provide a substantive advance over existing pipelines or isolate their individual contributions.
  2. The incremental storage benefit of repo-quarter profiles remains insufficiently established. SWE-smith already reduces storage overhead by sharing an execution environment across multiple tasks generated from the same repository version. The paper should more clearly explain their additional contribution beyond existing environment-sharing mechanisms. The storage comparison in Section 6 relies on an estimated per-candidate environment baseline rather than a measured comparison against an alternative reuse strategy on the same candidate commits. A controlled comparison of actual storage usage after accounting for shared layers, build time, and execution reliability would help establish the additional value of repo-quarter profiles.
  3. The downstream improvements cannot be cleanly attributed to data quality. Table 2 does not consistently control the training setup across methods. SWE-Next uses 3.7k training trajectories versus 3.3k for R2E-Gym, and the trajectory-generating models also differ. Although both use OpenHands, the comparison with SWE-smith additionally involves a different agent scaffold. These differences limit attribution of the gains specifically to the task-construction pipeline or data quality. The controlled experiment in Table 3 partially addresses this concern by matching the teacher model, trajectory count, and training hyperparameters. However, the absolute gains are modest: from 5.0% to 6.7% on Lite and from 7.8% to 8.2% on Verified. Without repeated runs or uncertainty estimates, it remains unclear whether these gains demonstrate a robust advantage.
Questions for authors
  1. Task-construction novelty: Could you clarify the methodological differences between SWE-Next and existing real-world executable task pipelines such as SWE-Gym and SWE-rebench? Which components are primarily responsible for the reported gains, and can their individual contributions be isolated?
  2. Environment storage efficiency: SWE-smith reduces storage overhead by sharing an execution environment across tasks generated from the same repository version. While SWE-Next addresses a different setting involving historical commits, could you clarify the additional benefits of repo-quarter profiles over existing environment-sharing mechanisms? Can you provide a controlled comparison against an alternative reuse strategy on the same candidate commits, measuring actual storage usage after accounting for shared layers, build time, and execution reliability?
  3. Controlled evaluation: Do the downstream gains persist when the teacher model, training data budget, and evaluation scaffold are matched? For the controlled experiment in Table 3, could you report results across multiple training seeds or uncertainty estimates to assess the robustness of the relatively small improvements, particularly on Verified?
Reviewer_CyJ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SWE-Next is a pipeline for collecting executable SWE training tasks and trajectories from real merged pull requests. For each PR it runs the repository's tests on the pre- and post-merge commits and keeps only pairs where at least one test turns from non-passing to passing with no regressions ("NewCommitBetter"). To control environment cost, it builds one reusable dependency image per (repository, calendar quarter), with no source code baked in, and mounts the commit-specific snapshot at runtime with copy-on-start isolation, falling back to per-commit builds when needed. From 3,971 seed repositories and 102,582 executed candidate pairs, the run yields 2,308 instances from 311 repositories (2.2% yield) in 30 hours, using 639 GB of environment storage. Trajectories are collected with an R2E-Gym/OpenHands-style scaffold that hides the FAIL_TO_PASS lists and requires a non-empty diff plus at least one test run before submission. SFT of Qwen2.5-Coder-Instruct 7B/14B on 3.7k trajectories from Claude 4.5 Sonnet and GPT-5-mini gives 17.4/30.0 on SWE-bench Verified and 13.7/23.3 on Lite, above the R2E-Gym and SWE-smith numbers at the same sizes. A teacher-controlled comparison against R2E-Gym (400 Claude-3.5-Sonnet trajectories each) gives 8.2 vs. 7.8 (Verified) and 6.7 vs. 5.0 (Lite).

Critical strengths
  1. A practical, real-PR pipeline with effective environment reuse. Grounding every task in a real merged PR with a strict paired fail-to-pass/no-regression check gives reliable labels without synthetic bug injection. The repo-quarter profile is simple, and the appendix shows it works in practice: 1,273 profiles collapse to 173 environment signatures, only 1 of 2,308 retained instances needed the per-commit fallback, and residual environment/harness failures make up just 2.4% of failed teacher trajectories. The final set covers 311 repositories, far more diverse than SWE-Gym or R2E-Gym, at a storage cost an academic lab can afford.

  2. Consistent downstream gains and transparent data accounting. At both 7B and 14B, SFT on SWE-Next trajectories outperforms the R2E-Gym (and, at 7B, SWE-smith) numbers. The appendix gives a full execution-outcome audit (Table 7), a complexity profile (Table 8), failure attribution (B.2), and a dataset schema, which makes the release easy to use and audit.

Critical weaknesses
  1. The central causal claim is not supported by the controlled evidence, and the novelty is incremental. The abstract claims the gains "come not from a stronger trajectory generator, but from higher-signal execution-grounded supervision." The only teacher-controlled experiment (Table 3) shows differences of 0.4 points on Verified (~2 of 500 tasks) and 1.7 points on Lite (~5 of 300 tasks), from single runs with no variance. Meanwhile, the gap between the controlled setting (8.2) and the main result (17.4) at the same 7B size suggests that the stronger teachers and larger trajectory count drive most of the gain. The Table 2 baselines also differ in teacher, scaffold (SWE-smith uses SWE-agent), and step budget. None of the claimed "high-signal" ingredients (strict submit gating, non-leaky prompting, clean vs. recovery buckets) is ablated. Finally, paired fail-to-pass execution filtering is standard in SWE-bench, SWE-Gym, and SWE-rebench, and per-repository environment reuse already exists in SWE-smith (128 images, 295 GB for 50k instances) and SWE-rebench. Table 1 omits SWE-rebench's environment footprint, and the 30.8 TB no-reuse figure is an assumption (300 MB × candidates) rather than a measurement.
Questions for authors
  1. (Clarification) Please reconcile the "no overlap with SWE-Bench" statement with Table 9, which includes pytest-dev/pytest, sphinx-doc/sphinx, and psf/requests. How many training instances and trajectories come from these repositories? Do any of them correspond to (or post-date and touch the same functions as) SWE-bench Verified/Lite instances? Please report results after removing all instances from SWE-bench repositories.

  2. (Clarification) Can you provide a data-matched comparison with the same teacher (Claude 4.5 Sonnet / GPT-5-mini), the same scaffold and step budget, and the same trajectory count on R2E-Gym or SWE-Gym tasks, with multiple seeds or bootstrap confidence intervals? In Table 3, are the R2E-Gym trajectories the released ones (on R2E-Gym tasks) and the SWE-Next ones newly collected on 800 sampled SWE-Next tasks? Were the step budgets and filtering identical?

Reviewer_vMktOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SWE-Next, a framework for collecting execution-grounded software engineering tasks and trajectories from real-world GitHub repositories. The proposed approach identifies self-verifying tasks by executing candidate base and merged commits, and introduces reusable repository-quarter environments to reduce the computational and storage costs of environment construction. The authors demonstrate that the framework can process thousands of repositories efficiently and report improvements in downstream model performance using the collected data.

Critical strengths
  1. The paper is generally well-written and easy to follow.
  2. A potentially valuable contribution to SWE data.
Critical weaknesses
  1. In my view, the primary drawback of this paper is that it is already outdated. Cutting-edge models have effectively saturated the SWE-bench-verified benchmark and are now exploring more complex coding domains, such as those involving entire repositories (e.g., nl2repo).
  2. Furthermore, I find it confusing that, according to Table 1, SWE-next utilizes a larger number of repositories than SWE-Rebench yet results in fewer synthesized environments. And authors forgot to add latest related work, such as SWE-Factory, SWE-Universe, etc.
  3. Additionally, the paper highlights that it can process 3,971 repositories within 30 hours using 639 GB of environment space; frankly, I see little value in this. The current priority in the field is scaling—we need a greater volume of higher-quality data, rather than faster synthesis within resource-constrained environments.
  4. Finally, the absence of a comparison with SWE-Rebench in Table 2 means that the authors fail to demonstrate whether the data synthesized by SWE-next offers greater value than that provided by SWE-Rebench.

Reference:

  1. SWE-Factory: https://arxiv.org/html/2506.10954v1
  2. SWE-Universe: https://arxiv.org/abs/2602.02361
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.