- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies scaling interactive trajectories for code-agent mid-training. To reduce the reliance of existing data generation methods on executable sandbox environments, it proposes the World Rollout framework, which constructs agent trajectories from GitHub open issues. The proposed pipeline is evaluated through mid-training experiments, real-sandbox patch verification, and agent behavior analysis.
Critical strengths
-
This work addresses the challenge of scaling interaction trajectory generation for code-agent mid-training by constructing training data from open GitHub issues, providing a practical and scalable solution with strong practical relevance.
-
A systematic evaluation, including mid-training experiments, real-environment validation, and agent behavior analysis, demonstrates the effectiveness of the proposed data generation framework from multiple complementary perspectives.
Critical weaknesses
1.The claimed four to sixfold token throughput improvement is based on resource modeling rather than end-to-end measurements. The abstract and main text do not clearly distinguish theoretical estimates from empirical results, leaving the actual acceleration gain unverified.
2.Simulated runtime feedback may cause distribution shift. Only final patches are validated, leaving intermediate observations unverified by real execution. Moreover, the experiments cannot distinguish gains from increased task diversity from those due to simulated trajectories.
Questions for authors
1.The standalone benefit of World Rollout-generated interaction trajectories remains unclear. Could the authors compare SFT performance using only issue text, real execution trajectories, and simulated execution trajectories under a fixed data budget?
2.The advantages of World Rollout over SWE-World remain unclear. Could the authors provide comparative experiments under matched task settings and training configurations, or a more detailed qualitative analysis?
3.Please analyze the modest performance gains on Terminal-Bench-2.0 and clarify whether they stem primarily from limited simulation fidelity or the inherent characteristics of the benchmark tasks.
4.Performance degrades when all trajectories are replaced with simulated ones in the WR100 setting. Please discuss the practical deployment risks when fully relying on simulated trajectories for training.
5.Please clarify which Terminal-Bench version (2.0 or 2.1) is used in the experiments, as the manuscript refers to both. The version number should be consistent throughout the paper and explicitly specified in the experimental setup.
4 comments
The authors have responded to the first two weaknesses. However, they did not address the five follow-up questions I raised. These important questions remain unresolved. I therefore revise my score from 3 to 2.
Response to Reviewer cit1 Part II
Sorry for the delayed part 2 &3 as we are revising the responses to the five questions:
Q1. The standalone benefit of World Rollout-generated interaction trajectories remains unclear. Could the authors compare SFT performance using only issue text, real execution trajectories, and simulated execution trajectories under a fixed data budget?
Thank you for this suggestion. The comparison between real execution and simulated trajectories under a fixed data budget is exactly the Original vs. WR100 setting in §3.2 (Figure 2). Both use the same base model, the same SFT budget. Original uses 100% execution-based SWE traces, and WR100 replaces all of them with World Rollout traces (pass@1 / pass@k, %):
| Setting | SWE-bench Verified | SWE-bench Multilingual | SWE-bench Pro | Terminal-Bench 2.1 | Avg. |
|---|---|---|---|---|---|
| Original (real execution) | 56.2 / 68.8 | 50.2 / 67.3 | 30.4 / 41.2 | 20.5 / 41.6 | 39.3 / 54.7 |
| WR100 (simulated) | 53.4 / 66.4 | 50.1 / 68.7 | 29.2 / 41.0 | 22.5 / 42.7 | 38.8 / 54.7 |
Trained on simulated trajectories alone, the model reaches 38.8% average pass@1 (similar to execution-based's 39.3%) and the same average pass@k (54.7%).
Regarding issue text only: our SFT loss is applied only to assistant turns (reasoning and tool calls), while system/user messages and tool observations are masked (Appendix C.1.1). The issue text is the user message of every trajectory, so it is already present as context in both settings above. A corpus of issue text alone contains no actions or observations, so it provides no supervision for the multi-turn tool use these benchmarks evaluate; a model trained on it would not learn to solve the agentic problems.
Q2. The advantages of World Rollout over SWE-World remain unclear. Could the authors provide comparative experiments under matched task settings and training configurations, or a more detailed qualitative analysis?
Thank you for the question. World Rollout and SWE-World [2] target different training stages and data regimes, which makes a matched-setting comparison difficult to define:
- SWE-World targets post-training on a curated set of 16.6K high-quality merged issue–PR instances (≈5.7K filtered SFT trajectories). Its transition model conditions on the ground-truth patch, and its reward model on F2P/P2P tests.
- World Rollout targets mid-training scale: 3M+ unique tasks and 100B+ training tokens, drawn largely from open issues and development-branch states (Table 7) that have no reference fix or tests.
Matching on our task pool is not possible, since SWE-World's simulators require a gold patch and F2P/P2P tests that these tasks lack. Matching on SWE-World's curated tasks would compare two datasets that differ by orders of magnitude in volume and in curation level, and would leave out the setting we target. We therefore compare simulated traces against the strongest available reference, real execution on the same task source (§3.2 and §4.1).
For a qualitative comparison, Table 1, Table 6 (Appendix B) and Table 8 (Appendix C.2) contrast the two systems. The main differences are:
- Oracle-free simulation. Our world model sees only the public task, the current real diff and the interaction history; gold patches and tests are excluded by construction. This is what admits unresolved issues, open PRs, and tasks without PR test changes.
- Multilingual routing. A three-way capability router covers runtimes beyond Python (Go, C/C++, Java, CUDA, Julia, Elixir, OCaml), routes compound commands atomically, and explicitly refuses unsupported actions. SWE-World's released dispatch primarily detects Python and
pytest, and other commands run locally. - State ownership. Our simulator is state-inert: only controller-owned edits can change files, every observation records its backend and routing decision, and a hash-chained turn journal supports auditable recovery.
World Rollout builds directly on SWE-World's separation of real repository state and simulated runtime feedback (§1, §2.1) with enhanced functionalities on a much larger scale and real world setting. We will move a condensed version of this comparison into the main text.
[2] Sun et al. (2026). SWE-World: Building Software Engineering Agents in Docker-Free Environments.
Response to Reviewer cit1 Part III
Q3. Please analyze the modest performance gains on Terminal-Bench-2.0 and clarify whether they stem primarily from limited simulation fidelity or the inherent characteristics of the benchmark tasks.
Thank you for this question. We believe the modest Terminal-Bench gains mainly reflect the characteristics of the benchmark relative to our training data rather than limited simulation fidelity, for following reasons:
- Task OOD Both the World Rollout traces and the execution-based traces they are compared with are repository-level GitHub issue-resolution traces. Terminal-Bench 2.1 covers a broad range of command-line tasks (e.g., system administration, environment setup, data processing, security) and is evaluated with a different agent, Terminus-2 through Harbor (Appendix C.1.2). Gains on it measure out-of-domain transfer, which we expect to be smaller than in-domain gains on the SWE benchmarks.
- Terminal-Bench does not degrade as simulated data increases. If simulation fidelity were the bottleneck, replacing real traces with simulated ones should hurt Terminal-Bench. Instead, pass@1 goes 20.5 → 20.4 → 21.6 → 22.5 from Original to WR25/50/100 (Figure 2), so real and simulated SWE traces transfer to it to a similar degree. While for mid-training, pass@k improves more clearly (41.6% → 46.1%, Table 3).
Our interpretation is that the modest gains on Terminal-Bench 2.1 mainly reflect limited coverage of terminal tasks in our training data, rather than the fidelity of individual simulated responses.We can broaden the coverage of World Rollout by adding terminal-native task sources. Qwen-AgentWorld already models terminal and OS environments, so this extension could expose the agent to a wider range of interactions relevant to Terminal-Bench 2.1. We will discuss this direction in the revision.
Q4. Performance degrades when all trajectories are replaced with simulated ones in the WR100 setting. Please discuss the practical deployment risks when fully relying on simulated trajectories for training.
Thank you for this question. The average drop is moderate: WR100 reaches 38.8% pass@1 vs. 39.3% for Original, with identical average pass@k (54.7%); the largest drop is on SWE-bench Verified (56.2 → 53.4), while Terminal-Bench improves (20.5 → 22.5).
We agree that relying entirely on simulated trajectories carries a real risk: distribution shift between training feedback and deployment feedback. A world model leaves systematic patterns in the feedback it generates; Table 4b shows, for example, fewer failed calls and shorter long-tail outputs than real execution. A model trained only on simulated traces sees only this feedback distribution, while evaluation and deployment take place in real execution environments. Behaviors that depend on real feedback, such as recovering from verbose or unexpected errors, may then be under-trained, and occasional hallucinated feedback is never corrected by a real signal.
For this reason, World Rollout is designed to complement execution-based collection, for two purposes: (1) scaling data volume and task coverage on top of existing execution-based traces, and (2) providing trajectories for tasks that execution-based collection cannot reach, such as open issues, development branches, and tasks with broken images or non-reproducing gold solutions, where simulated traces are the only interaction data available. Our main results follow this usage: in mid-training, World Rollout traces are added alongside execution-based traces (Table 3, 44.4% → 46.4% average pass@1), and in the additional 200B-token setting reported in our response to Reviewer epPN, real-data exposure is kept unchanged and only simulated traces are added, reaching 46.8%. We use WR100 as a stress test of simulated-trace utility rather than as a recommended configuration, and will add this guidance and the associated risks to the Limitations section.
Q5. Please clarify which Terminal-Bench version (2.0 or 2.1) is used in the experiments.
Thank you for catching this, and sorry for the confusion. All experiments use Terminal-Bench 2.1, evaluated with the Terminus-2 agent through Harbor (Appendix C.1.2). The mentions of "Terminal-Bench 2.0" in the abstract and Introduction are typos. We will use "Terminal-Bench 2.1" consistently throughout the paper, including Table 3, and state the version explicitly in the experimental setup (§3.1).
Thank you again for your constructive feedback. We will incorporate these clarifications and fixes into the final version and acknowledge your helpful feedback. If the additional evidence addresses your concerns, we would appreciate you can keep your positive score.
Response to Reviewer cit1 Part I
Thank you so much for your manual review and for the time and effort. We appreciate your recognition that World Rollout offers a practical and scalable way to build code-agent mid-training data from open GitHub issues, and that our evaluation examines the framework from complementary perspectives.
We will address your concerns and questions one by one:
The throughput gain was first observed in our end-to-end production data collection. Using the 80-node GB300 setup described in Appendix D, with the same budget of 320 GPUs for each method, we measured training tokens collected per day. Simulated-runtime collection produced more than 4× as many training tokens per day as execution-based collection.
Then we formalizes this observation with a throughput model in Appendix D.2. The model uses measured decode times for policy and world-model requests (13.4 s and 88.9 s per request), the average number of calls per trajectory (35.2 and 5.65, respectively), and historical wall-clock durations for execution-based rollouts (717.9 s mean, 519.3 s median). Under the same GPU budget, it estimates a 4.1× gain using the median execution time and a 5.7× gain using the mean. We will clarify this further in the paper: the speedup was indeed the actual acceleration, and the theoretical framework in Appendix D.2 provides a formal account of that empirical result.
Thank you for raising these points. We address them in turn.
Intermediate observation quality. We chose Qwen-AgentWorld-35B-A3B[1] to balance fidelity and collection speed. On the SWE subset of AgentWorldBench, it scores 65.6/100 on a rubric that compares intermediate simulated responses with real observations, providing direct evidence of per-turn fidelity. If intermediate fidelity is the priority, a frontier model such as Claude Opus is a plausible alternative, at the cost of slower collection. While at the end-to-end level, patches from simulated and execution-based rollouts passed real tests at nearly identical rates (22.29% vs. 22.39%). This provides indirect evidence that the simulated intermediate responses were informative enough to guide the agent toward correct fixes.
Distribution shift. We agree that simulated runtime feedback can shift the distribution of observations the agent sees during training. Empirically, however, adding simulated trajectories to execution-based data made the agent more robust on the tasks we evaluated, as shown by the behavioral analysis in Section 4.3. Training on both sources exposes the agent to a broader range of runtime responses: simulated feedback complements the observations available from execution-based trajectories.
Task diversity vs. simulated trajectories. We would like to clarify that broader task coverage is precisely the benefit World Rollout is designed to enable, and the world model is just the mechanism that makes these tasks usable. Our fixed-budget SFT replacement experiment (Section 3.2, Figure 2) directly compares simulated and execution-based trajectories without expanding the task pool. Replacing all execution-based traces with simulated ones yields comparable performance (38.8% vs. 39.3% average pass@1 and identical average pass@k). We therefore do not attribute the gains in our experiments to simulated trajectories being intrinsically better training data. Rather, simulation lets us collect trajectories for a much more diverse set of tasks that execution-based collection cannot readily cover. We interpret the gains primarily as a benefit of that added task diversity.
[1] Zuo et al. (2026). Qwen-AgentWorld: Language World Models for General Agents.