- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Terminal-Universe, a pipeline that transforms recorded terminal-agent trajectories into reusable, task-sufficient execution environments. Deterministic replay restores observed files to their pre-solution states, an agent reconstructs missing project context, and a read-only judge filters out incomplete or insufficient workspaces. The resulting environments support re-solving tasks with the original intent, generating new tasks within or across workspaces, and conducting multi-round interactions with evolving requirements and verifier-driven user feedback. Using this pipeline, the authors construct 37k task-sufficient environments. SFT on Qwen3.5-27B yields improvements of over 10 percentage points on Terminal-Bench 2.1 and EvoCode-Bench v2. Ablation studies covering re-solving, context completion, verification, breadth/depth expansion, environment allocation, and other components, further validating the effectiveness of the proposed approach.
Critical strengths
-
A useful and well-motivated approach to scaling executable agent environments. The paper repurposes existing demonstrations into reusable workspaces that support renewed interaction and the creation of new tasks, addressing a practical bottleneck in terminal-agent post-training. The overall pipeline is clearly presented and easy to follow.
-
Strong overall improvements with multiple evaluations of generality. Using the same Qwen3.5-27B base model, the full training mixture delivers consistent gains on both single-round and persistent multi-round evaluations. Additional experiments demonstrate that the approach remains effective across different agent scaffolds, additional benchmarks, and an alternative teacher model.
Critical weaknesses
-
The experiments do not sufficiently disentangle the benefits of the proposed interaction structure from those of increased training volume. This issue affects both the main results and ablations on breadth/depth expansion and related components, making it difficult to isolate the contribution of the method itself. In addition, all experiments are conducted on a single model at a single scale, Qwen3.5-27B, which limits the evidence that the resulting data generalizes across different model families or capacities.
-
Benchmark independence is not documented clearly enough for the multi-round and transfer evaluations. Section 3.3 states that demonstrations are decontaminated against the evaluation benchmarks, but Appendix A explicitly describes only a 13-gram check against Terminal-Bench and the exclusion of Terminal-Bench-derived sources. Appendix C.1 further discusses removing SWE-bench Verified repositories, but this does not establish whether similar screening was performed for EvoCode-Bench v2, SWE-bench Pro, or SWE-bench Multilingual.
Questions for authors
For Tables 6 and 7, what are the total training tokens and optimizer updates for each variant? Do you have a control that spends the added budget on Single-WS supervision, matching training tokens and updates, and does the breadth/depth advantage remain?
3 comments
Thanks for the responses! They address most of my concerns. I will keep my already positive score.
Thank you for your time and constructive suggestions! we will improve documentation and include the additional experiments.
Best regards,
Authors of Submission
Authors' Response
Thank you for the positive remarks and constructive suggestions regarding the experimental controls, model generalization, and benchmark decontamination of our work. Below, we address each concern in detail.
W1 (Part 1) & Q: Disentangling Interaction Structure from Training Volume.
(1) Training tokens and optimizer updates in Tables 6 and 7.
(2) Token- and update-matched SFT control (Qwen3.5-27B, DeepSeek-V4.1-Flash teacher). At a matched budget of about 107M tokens per epoch, replacing half of the Single-WS tokens with Cross-WS data improves TB2.1 by 1.9 points, and replacing half with Multi-Round data improves TB2.1 by 1.1 points and EvoCode by 4.1 points.
W1 (Part 2): Generalization to an Additional Base Model.
We additionally evaluate generalization on Qwen3.6-27B: fine-tuning on the same Terminal Universe (Full Mixture) data improves TB2.1 by 6.2 points (51.7 to 57.9) and EvoCode Case Score by 3.0 points (82.9 to 85.9). We will include these cross-base results in the revised paper.
W2: Benchmark Independence Across All Four Benchmarks.
Models evaluated on EvoCode-Bench v2, SWE-bench Pro, and SWE-bench Multilingual are trained on the terminal pool, and our further audit across text (13-gram), query (TF-IDF cosine similarity at most 0.17), and repository levels confirms zero leakage. Thank you for pointing this out; we will improve the documentation for decontamination.
We appreciate your constructive feedback, which has helped us strengthen the experimental controls and decontamination documentation of the paper. We hope the clarifications and additional results above address your concerns, and we welcome any further questions.
Best regards,
Authors of Submission 970