- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OpenTerminal, a pipeline that generates executable tasks for training terminal agents. It consists of three steps:
- Task synthesis from real data. The authors collect about 307K problem patterns from 36 public sources. They cluster these patterns and sample from the clusters in a way that increases the probability of rare topics. For each task, the pipeline samples a group of patterns from several clusters, and a teacher model selects the compatible ones and combines them into a single task description.
- Contract-based construction. For each task, the pipeline first writes a contract that specifies the task details, the initial environment, and the verification criteria. Both the environment and the verifier are generated from this contract, so that they remain consistent with each other.
- Validation and repair. Each task must build successfully, and its verifier must give zero reward when the agent takes no action. If a check fails, a model predicts which step caused the failure, and the pipeline regenerates that step and all subsequent ones.
In total, the pipeline produces 11,713 tasks and 7,365 training trajectories, which are collected with DeepSeek-V4-Flash and kept if they score at least 0.5. The authors fine-tune Qwen3.5-4B and Qwen3.5-9B on these trajectories, and show improved performance on Terminal-Bench 2.1 and OpenThoughts-TBLite.
Critical strengths
-
Task concepts are derived from real technical problems rather than generated from the model’s own prior or from a hand-designed taxonomy, which gives broad coverage.
-
The shared contract is a simple way to keep the environment and verifier consistent, and its effect is measured directly: removing it raises the mismatch rate from 4.5% to 49.5%.
-
The paper reports the construction process in detail, including the size of each stage of the pipeline, the types of verifier defects caught by the no-op check, and the routing statistics of the repair step.
Critical weaknesses
-
Neither task solvability nor verifier correctness is checked, since the pipeline does not generate a reference solution. The build and no-op checks only ensure that the environment runs and that taking no action yields zero reward. As a result, it is unclear whether the 21.6% of tasks that receive zero reward in all three rollouts are hard or simply unsolvable. There is no human audit of task quality.
-
The comparison with other datasets is not controlled. The SFT datasets used by the baselines and by this method differ in the number of trajectories, and OpenTerminal uses more trajectories than three of the four baselines. The paper also does not state whether the baseline trajectories are generated by the same model. It is therefore unclear whether the gains come from better task quality, or simply from more training data or a stronger trajectory-generating model. A comparison under the same number of trajectories or tokens would make this clearer. Moreover, the gains over the strongest baseline (2.4 to 3.2 points) are within the reported standard deviations.
-
Only the task concepts come from real data. The environments and files are generated by an LLM, and specialized services are replaced with mocks, which may behave differently from the real systems. The paper does not analyze how realistic these environments are, or whether skills learned on mocked services transfer to real ones.
-
The tasks generated are relatively short (median of 12 turns, with 75% finishing within 15), which limits their relevance to long-horizon terminal work.
Questions for authors
Please refer to the weaknesses part.
4 comments
W3. Environment realism and mock-to-real transfer
Across our 11,713 executable environments, only 12.6% use a shim or mock, while the remaining 87.4% run in real containers without mocks. Mocks are used mainly for operations restricted inside unprivileged, network-isolated containers such as ssh, sudo, systemctl, external databases, and cloud CLIs, preserving sysadmin and DevOps coverage rather than dropping those domains. Empirically, OpenTerminal improves pass@1 by 56.8% at 4B and 63.1% at 9B on Terminal-Bench 2.1, whose 89 human-built containers run real software stacks and human-written tests, confirming strong transfer to real environments. We will include this breakdown and note transfer to live external services in Limitations.
W4. Trajectory horizon
We agree that OpenTerminal is not specifically designed for long-horizon workflows and will clarify this scope in Limitations. However, our primary goal is to establish a general, scalable framework for synthesizing diverse, high-quality executable terminal tasks, whose effectiveness is consistently validated across both model scales on Terminal-Bench 2.1 and OpenThoughts-TBLite. Extending OpenTerminal to construct dedicated long-horizon tasks is a promising direction that we plan to explore in future work.
Response to rebuttal
Thank you for the detailed response and additional experiments. The response addresses my first concern.
However, my second concern (W2) is only partially addressed.
First, the trajectory generator is still not controlled. The baselines are trained on their officially released trajectories, which may come from different models, while OpenTerminal uses DeepSeek-V4-Flash. The new same-generator experiment compares the full pipeline with first-pass-only tasks, which is an ablation within OpenTerminal rather than a comparison with other datasets. A cleaner comparison would regenerate trajectories on the baseline tasks with the same model.
Second, the gains over the strongest baseline remain within the reported standard deviations, as the authors also acknowledge. For example, in the new 6,000-trajectory comparison, OpenTerminal improves over CLI-Universe by 1.8 points on TB 2.1, while the standard deviations are 2.2 to 2.4 points. Therefore, it remains unclear whether the gains come from better task quality. I will maintain my original score.
We thank the reviewer for the thoughtful follow-up and are glad that our response resolved your first concern. Below we address your two remaining points on W2.
- Trajectory generator across datasets
OpenTerminal does not benefit from a stronger teacher model than the baselines. Specifically, OpenTerminal collects rollouts with a lightweight model, DeepSeek-V4-Flash. By contrast, our closest baseline CLI-Universe collects its 6,000 trajectories with 1T-scale Kimi-K2.6, and its own experimental results show that Kimi-K2.6 already outperforms the flagship model DeepSeek-V4-Pro by 2.2 points on the same tasks. The other baselines likewise use large or frontier teachers, including MiniMax-M2 and M2.1 in LiteCoder-Terminal, MiniMax-M2.7 in SkillSynth, and o3 in Endless Terminals.
Meanwhile, regenerating baseline SFT datasets with DeepSeek-V4-Flash is not currently feasible because the leading SFT baselines release static trajectory logs for training rather than their underlying runnable task containers and verifiers. Specifically, CLI-Universe and SkillSynth have not released their task Docker environments, and LiteCoder-Terminal released containers for only 602 RL tasks rather than its 11,255 SFT tasks.
Conversely, collecting trajectories on OpenTerminal tasks with the baseline teacher models is a valid same-teacher comparison, though it cannot be completed within the remaining discussion period. We will document each baseline's trajectory generator in Section 4.1 and include this cross-teacher comparison in the revised paper.
- Margins relative to reported standard deviations
-
Single-trial SD vs. standard error of the 5-trial mean. In our tables,
±reports the standard deviation across five independent Harbor runs, which measures single-trial spread. Because each reported score is the mean over trials, the uncertainty of that mean is the standard error , so a single-trial SD of2.2to2.4corresponds to an SE of only0.98to1.07points on the 5-trial mean. -
Statistical significance across benchmarks and model scales. Moreover, no single baseline ranks second on both benchmarks. Against our 6,000-trajectory model, CLI-Universe narrows the gap to 1.8 points on TB 2.1 yet falls behind by 3.7 points on TBLite with , whereas LiteCoder-Terminal narrows the gap to 1.7 points on TBLite yet falls behind by 3.8 points on TB 2.1 despite nearly twice as many trajectories. Evaluating the two-benchmark average across the independent five-trial Harbor runs via two-sided Welch -tests confirms statistically significant gains in every 4B and 9B comparison:
| Backbone | Comparison | TB 2.1 | TBLite | Two-Benchmark Mean ± SE | Mean | Two-Sided | |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | OpenTerminal, 6,000 vs. CLI-Universe, 6,000 | +1.8 | +3.7 | 31.50 ± 0.68 vs. 28.75 ± 0.61 | +2.75 | 3.00 | 0.009 |
| Qwen3.5-4B | OpenTerminal, 6,000 vs. LiteCoder, 11,255 | +3.8 | +1.7 | 31.50 ± 0.68 vs. 28.75 ± 0.93 | +2.75 | 2.37 | 0.032 |
| Qwen3.5-4B | OpenTerminal, 7,365 vs. CLI-Universe, 6,000 | +2.7 | +5.2 | 32.70 ± 0.90 vs. 28.75 ± 0.61 | +3.95 | 3.63 | 0.003 |
| Qwen3.5-4B | OpenTerminal, 7,365 vs. LiteCoder, 11,255 | +4.7 | +3.2 | 32.70 ± 0.90 vs. 28.75 ± 0.93 | +3.95 | 3.04 | 0.008 |
| Qwen3.5-9B | OpenTerminal, 7,365 vs. CLI-Universe, 6,000 | +2.7 | +4.4 | 39.20 ± 0.65 vs. 35.65 ± 1.13 | +3.55 | 2.72 | 0.020 |
| Qwen3.5-9B | OpenTerminal, 7,365 vs. LiteCoder, 11,255 | +6.0 | +2.4 | 39.20 ± 0.65 vs. 35.00 ± 0.73 | +4.20 | 4.29 | < 0.001 |
Consistent and statistically significant gains across all SFT models at 4B and 9B, achieved with equal or fewer trajectories and a lightweight teacher, confirm that OpenTerminal's improvements are driven by task quality. We hope this resolves your remaining concern and would greatly appreciate it if you could reconsider your score.
We thank the reviewer for the insightful and constructive comments. Below we address each concern and will incorporate the clarifications and additional results into the revision. If our response resolves your concerns, we would greatly appreciate it if you could raise your score (2: Weak rejection). We are happy to clarify any remaining issues.
OpenTerminal does not claim to guarantee that every generated task is perfectly solvable or free of verifier flaws, as build and no-op gates are necessary rather than sufficient conditions. Instead, our pipeline combines contract alignment, execution gates, and rollout filtering to screen out low-quality instances so the retained data is predominantly high quality. Even when minor verifier imperfections remain, coherent teacher trajectories executed in runnable containers still provide valuable supervision on tool use and environment interaction, as confirmed by our downstream gains and 120-task human audit below.
Zero-reward exclusion. Because we require r≥0.5 across three rollouts, none of the 21.6% zero-reward tasks enter our 7,365 SFT trajectories, where 39.5% achieve full success at r=1 and 62.3% reach r≥0.8.
Stratified 120-task human audit. Two annotators inspected 120 live container tasks across four cohorts of 30 tasks each: one authored a reference solution and a wrong final state, while the other verified solvability and tested the verifier on solvable tasks:
Across the accepted training cohorts, 93.3% of audited tasks are solvable, reaching 98.3% when weighted by training cohort shares, with 92.9% reference acceptance and 96.4% wrong-state rejection. In the excluded zero-reward pool, 56.7% are genuinely hard solvable tasks missed in three rollouts while 43.3% have residual defects, confirming that rollout filtering removes most broken tasks before training.
OpenTerminal outperforms all competing datasets at matched or lower trajectory and supervised-token counts.
We trained Qwen3.5-4B on a random 6,000-trajectory subset of OpenTerminal using the same SFT recipe: