acceptodds
Under review as a conference paper at ICLR 2027

OpenTerminal: Scalable Construction of Executable Tasks for Open Terminal Agents

Abstract

Terminal agents enable diverse real-world workflows, yet open-weight models remain unreliable on realistic terminal tasks. A key bottleneck is the scarcity of executable data. Each task must pair an instruction with a runnable environment, the fixtures it uses, and a verifier that evaluates whether the task is completed. Building such data at scale requires broad task coverage, cross-artifact consistency, and a way to recover from construction failures. OpenTerminal addresses these challenges through corpus-anchored synthesis, shared task contracts, and failure-directed repair. Task concepts are synthesized from 307,293 problem patterns across 36 public corpora, where a cluster-smoothed sampler flattens the sampling distribution over semantic regions and increases coverage of underrepresented areas. A hidden task contract resolves implementation choices left unspecified by the instruction and provides a shared specification for environment and verifier generation. When a validation failure persists, Diagnose–Route–Repair identifies a likely source stage, feeds diagnostic evidence, and regenerates the downstream artifacts, recovering tasks that would otherwise be discarded. The pipeline yields 11,713 executable tasks and 7,365 training trajectories. We fine-tune Qwen3.5-4B and Qwen3.5-9B on these trajectories, improving pass@1 over the base models by 56.8% and 63.1% on Terminal-Bench 2.1, and by 31.3% and 23.9% on OpenThoughts-TBLite. Extensive experiments and analyses confirm the contribution of each component and characterize the data the pipeline produces.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Terminal, Agents, Synthetic data

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_yyjGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenTerminal, a pipeline that generates executable tasks for training terminal agents. It consists of three steps:

  1. Task synthesis from real data. The authors collect about 307K problem patterns from 36 public sources. They cluster these patterns and sample from the clusters in a way that increases the probability of rare topics. For each task, the pipeline samples a group of patterns from several clusters, and a teacher model selects the compatible ones and combines them into a single task description.
  2. Contract-based construction. For each task, the pipeline first writes a contract that specifies the task details, the initial environment, and the verification criteria. Both the environment and the verifier are generated from this contract, so that they remain consistent with each other.
  3. Validation and repair. Each task must build successfully, and its verifier must give zero reward when the agent takes no action. If a check fails, a model predicts which step caused the failure, and the pipeline regenerates that step and all subsequent ones.

In total, the pipeline produces 11,713 tasks and 7,365 training trajectories, which are collected with DeepSeek-V4-Flash and kept if they score at least 0.5. The authors fine-tune Qwen3.5-4B and Qwen3.5-9B on these trajectories, and show improved performance on Terminal-Bench 2.1 and OpenThoughts-TBLite.

Critical strengths
  1. Task concepts are derived from real technical problems rather than generated from the model’s own prior or from a hand-designed taxonomy, which gives broad coverage.

  2. The shared contract is a simple way to keep the environment and verifier consistent, and its effect is measured directly: removing it raises the mismatch rate from 4.5% to 49.5%.

  3. The paper reports the construction process in detail, including the size of each stage of the pipeline, the types of verifier defects caught by the no-op check, and the routing statistics of the repair step.

Critical weaknesses
  1. Neither task solvability nor verifier correctness is checked, since the pipeline does not generate a reference solution. The build and no-op checks only ensure that the environment runs and that taking no action yields zero reward. As a result, it is unclear whether the 21.6% of tasks that receive zero reward in all three rollouts are hard or simply unsolvable. There is no human audit of task quality.

  2. The comparison with other datasets is not controlled. The SFT datasets used by the baselines and by this method differ in the number of trajectories, and OpenTerminal uses more trajectories than three of the four baselines. The paper also does not state whether the baseline trajectories are generated by the same model. It is therefore unclear whether the gains come from better task quality, or simply from more training data or a stronger trajectory-generating model. A comparison under the same number of trajectories or tokens would make this clearer. Moreover, the gains over the strongest baseline (2.4 to 3.2 points) are within the reported standard deviations.

  3. Only the task concepts come from real data. The environments and files are generated by an LLM, and specialized services are replaced with mocks, which may behave differently from the real systems. The paper does not analyze how realistic these environments are, or whether skills learned on mocked services transfer to real ones.

  4. The tasks generated are relatively short (median of 12 turns, with 75% finishing within 15), which limits their relevance to long-horizon terminal work.

Questions for authors

Please refer to the weaknesses part.

4 comments

We thank the reviewer for the insightful and constructive comments. Below we address each concern and will incorporate the clarifications and additional results into the revision. If our response resolves your concerns, we would greatly appreciate it if you could raise your score (2: Weak rejection). We are happy to clarify any remaining issues.


W1. Task solvability, verifier correctness, the 21.6% zero-reward tasks, and human audit

OpenTerminal does not claim to guarantee that every generated task is perfectly solvable or free of verifier flaws, as build and no-op gates are necessary rather than sufficient conditions. Instead, our pipeline combines contract alignment, execution gates, and rollout filtering to screen out low-quality instances so the retained data is predominantly high quality. Even when minor verifier imperfections remain, coherent teacher trajectories executed in runnable containers still provide valuable supervision on tool use and environment interaction, as confirmed by our downstream gains and 120-task human audit below.

  1. Zero-reward exclusion. Because we require r≥0.5r \ge 0.5 across three rollouts, none of the 21.6% zero-reward tasks enter our 7,365 SFT trajectories, where 39.5% achieve full success at r=1r = 1 and 62.3% reach r≥0.8r \ge 0.8.

  2. Stratified 120-task human audit. Two annotators inspected 120 live container tasks across four cohorts of 30 tasks each: one authored a reference solution and a wrong final state, while the other verified solvability and tested the verifier on solvable tasks:

CohortnnSolvableReference AcceptedWrong State Rejected
First-pass, accepted3030 (100.0%)28 (93.3%)28 (93.3%)
Local repair, accepted3028 (93.3%)27 (96.4%)28 (100.0%)
Cross-stage repair, accepted3026 (86.7%)23 (88.5%)25 (96.2%)
Accepted training tasks9084 (93.3%)78 (92.9%)81 (96.4%)
Excluded zero-reward tasks3017 (56.7%)13 (76.5%)14 (82.4%)

Across the accepted training cohorts, 93.3% of audited tasks are solvable, reaching 98.3% when weighted by training cohort shares, with 92.9% reference acceptance and 96.4% wrong-state rejection. In the excluded zero-reward pool, 56.7% are genuinely hard solvable tasks missed in three rollouts while 43.3% have residual defects, confirming that rollout filtering removes most broken tasks before training.


W2. Controlled trajectory/token comparison, rollout teacher, and trial variance

OpenTerminal outperforms all competing datasets at matched or lower trajectory and supervised-token counts.

We trained Qwen3.5-4B on a random 6,000-trajectory subset of OpenTerminal using the same SFT recipe:

Training DatasetTrajectoriesSupervised TokensTB 2.1TBLite
Base--13.9 ± 2.633.2 ± 4.1
CLI-Universe6,000-19.1 ± 2.238.4 ± 1.6
LiteCoder-Terminal11,255132.9M17.1 ± 2.840.4 ± 3.1
OpenTerminal, 6,000 subset6,00093.3M20.9 ± 2.442.1 ± 1.9
OpenTerminal, full7,365114.1M21.8 ± 3.043.6 ± 2.7
  • Data volume and tokens: At 6,000 trajectories, OpenTerminal exceeds equal-count CLI-Universe by +1.8 / +3.7 and surpasses LiteCoder-Terminal by +3.8 / +1.7 with 47% fewer trajectories and 30% fewer supervised tokens.
  • Consistent ranking: Although single-cell margins in Table 1 are comparable to 5-trial variance, OpenTerminal ranks first across all four 4B and 9B settings, whereas no single baseline is competitive on both benchmarks.
  • Rollout teacher: Baselines train on their officially released trajectories, and we will list each generator model in Section 4.1, whereas OpenTerminal uses a lightweight model, DeepSeek-V4-Flash. Even when holding both DeepSeek-V4-Flash and the 6,126 trajectory count fixed, OpenTerminal reaches 20.7 / 42.5 versus 18.4 / 39.7 for first-pass-only tasks, confirming that task quality drives the gain.

W3. Environment realism and mock-to-real transfer

Across our 11,713 executable environments, only 12.6% use a shim or mock, while the remaining 87.4% run in real containers without mocks. Mocks are used mainly for operations restricted inside unprivileged, network-isolated containers such as ssh, sudo, systemctl, external databases, and cloud CLIs, preserving sysadmin and DevOps coverage rather than dropping those domains. Empirically, OpenTerminal improves pass@1 by 56.8% at 4B and 63.1% at 9B on Terminal-Bench 2.1, whose 89 human-built containers run real software stacks and human-written tests, confirming strong transfer to real environments. We will include this breakdown and note transfer to live external services in Limitations.


W4. Trajectory horizon

We agree that OpenTerminal is not specifically designed for long-horizon workflows and will clarify this scope in Limitations. However, our primary goal is to establish a general, scalable framework for synthesizing diverse, high-quality executable terminal tasks, whose effectiveness is consistently validated across both model scales on Terminal-Bench 2.1 and OpenThoughts-TBLite. Extending OpenTerminal to construct dedicated long-horizon tasks is a promising direction that we plan to explore in future work.

Reviewer_yyjGOpenReview ↗

Response to rebuttal

Thank you for the detailed response and additional experiments. The response addresses my first concern.

However, my second concern (W2) is only partially addressed.

First, the trajectory generator is still not controlled. The baselines are trained on their officially released trajectories, which may come from different models, while OpenTerminal uses DeepSeek-V4-Flash. The new same-generator experiment compares the full pipeline with first-pass-only tasks, which is an ablation within OpenTerminal rather than a comparison with other datasets. A cleaner comparison would regenerate trajectories on the baseline tasks with the same model.

Second, the gains over the strongest baseline remain within the reported standard deviations, as the authors also acknowledge. For example, in the new 6,000-trajectory comparison, OpenTerminal improves over CLI-Universe by 1.8 points on TB 2.1, while the standard deviations are 2.2 to 2.4 points. Therefore, it remains unclear whether the gains come from better task quality. I will maintain my original score.

We thank the reviewer for the thoughtful follow-up and are glad that our response resolved your first concern. Below we address your two remaining points on W2.


  1. Trajectory generator across datasets

OpenTerminal does not benefit from a stronger teacher model than the baselines. Specifically, OpenTerminal collects rollouts with a lightweight model, DeepSeek-V4-Flash. By contrast, our closest baseline CLI-Universe collects its 6,000 trajectories with 1T-scale Kimi-K2.6, and its own experimental results show that Kimi-K2.6 already outperforms the flagship model DeepSeek-V4-Pro by 2.2 points on the same tasks. The other baselines likewise use large or frontier teachers, including MiniMax-M2 and M2.1 in LiteCoder-Terminal, MiniMax-M2.7 in SkillSynth, and o3 in Endless Terminals.

Meanwhile, regenerating baseline SFT datasets with DeepSeek-V4-Flash is not currently feasible because the leading SFT baselines release static trajectory logs for training rather than their underlying runnable task containers and verifiers. Specifically, CLI-Universe and SkillSynth have not released their task Docker environments, and LiteCoder-Terminal released containers for only 602 RL tasks rather than its 11,255 SFT tasks.

Conversely, collecting trajectories on OpenTerminal tasks with the baseline teacher models is a valid same-teacher comparison, though it cannot be completed within the remaining discussion period. We will document each baseline's trajectory generator in Section 4.1 and include this cross-teacher comparison in the revised paper.


  1. Margins relative to reported standard deviations
  1. Single-trial SD vs. standard error of the 5-trial mean. In our tables, ± reports the standard deviation across five independent Harbor runs, which measures single-trial spread. Because each reported score is the mean over n=5n = 5 trials, the uncertainty of that mean is the standard error SE=SD/5\mathrm{SE} = \mathrm{SD} / \sqrt{5}, so a single-trial SD of 2.2 to 2.4 corresponds to an SE of only 0.98 to 1.07 points on the 5-trial mean.

  2. Statistical significance across benchmarks and model scales. Moreover, no single baseline ranks second on both benchmarks. Against our 6,000-trajectory model, CLI-Universe narrows the gap to 1.8 points on TB 2.1 yet falls behind by 3.7 points on TBLite with p=0.011p = 0.011, whereas LiteCoder-Terminal narrows the gap to 1.7 points on TBLite yet falls behind by 3.8 points on TB 2.1 despite nearly twice as many trajectories. Evaluating the two-benchmark average across the independent five-trial Harbor runs via two-sided Welch tt-tests confirms statistically significant gains in every 4B and 9B comparison:

BackboneComparisonTB 2.1 Δ\DeltaTBLite Δ\DeltaTwo-Benchmark Mean ± SEMean Δ\DeltattTwo-Sided pp
Qwen3.5-4BOpenTerminal, 6,000 vs. CLI-Universe, 6,000+1.8+3.731.50 ± 0.68 vs. 28.75 ± 0.61+2.753.000.009
Qwen3.5-4BOpenTerminal, 6,000 vs. LiteCoder, 11,255+3.8+1.731.50 ± 0.68 vs. 28.75 ± 0.93+2.752.370.032
Qwen3.5-4BOpenTerminal, 7,365 vs. CLI-Universe, 6,000+2.7+5.232.70 ± 0.90 vs. 28.75 ± 0.61+3.953.630.003
Qwen3.5-4BOpenTerminal, 7,365 vs. LiteCoder, 11,255+4.7+3.232.70 ± 0.90 vs. 28.75 ± 0.93+3.953.040.008
Qwen3.5-9BOpenTerminal, 7,365 vs. CLI-Universe, 6,000+2.7+4.439.20 ± 0.65 vs. 35.65 ± 1.13+3.552.720.020
Qwen3.5-9BOpenTerminal, 7,365 vs. LiteCoder, 11,255+6.0+2.439.20 ± 0.65 vs. 35.00 ± 0.73+4.204.29< 0.001

Consistent and statistically significant gains across all SFT models at 4B and 9B, achieved with equal or fewer trajectories and a lightweight teacher, confirm that OpenTerminal's improvements are driven by task quality. We hope this resolves your remaining concern and would greatly appreciate it if you could reconsider your score.

Reviewer_2q7ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenTerminal tries to achieve scalable terminal task generation via unifying the task contract, diversifying task seeds (upsampling low presence semantic space), and with targeted task repair.

Critical strengths
  1. The scale is huge (>11k tasks), and leads to substantial relative performance gains (+56.8% and +63.1%) over base models on Terminal Bench.
  2. The cluster-smoothed sampling successfully flattens the long-tail distribution across cli tools and domains.
  3. The system design is disciplined, and comes with modularized ablation study.
Critical weaknesses
  1. methodological novelty is limited. Corpus sampling, shared specifications and targeted repair are established engineering practices.
  2. The ablation study used 3 metrics: coverage, mismatch and yield, but missing direct contributions to model performance. Does it necessarily mean tasks yielded by CATS have better training utility?
  3. Total generation cost is not measured, which is necessary to support efficiency claim
Questions for authors
  1. Under a matched generation and training budget, does each component independently improve downstream model performance?
  2. How does CATS compare with simpler inverse-frequency or uniform-cluster sampling methods?
  3. What is the overall cost per validated task and per successful training trajectory?
3 comments

We thank the reviewer for the constructive comments. Below we address each concern and will incorporate the clarifications and additional results into the revision. If our response resolves your concerns, we would greatly appreciate it if you could raise your score (2: Weak rejection). We are happy to clarify any remaining issues.


W1. Methodological novelty

OpenTerminal goes beyond manual taxonomies and fixed code repositories by turning raw technical corpora into runnable environments and verifiers without reference solutions. In this setting, simple baselines fail in ways our experiments clearly show:

  • Simple upsampling hurts diversity and quality: Uniform-cluster / inverse-frequency sampling with α=0\alpha = 0 groups incompatible rare tools together, which actually reduces final task diversity and lowers model accuracy compared with CATS.
  • Natural instructions leave details unspecified: Because realistic instructions II do not spell out exact file paths or formats, generating EE and VV independently causes a 49.5% mismatch rate, which our hidden contract CC reduces to 4.5%.
  • Local retries favor easy tasks: Because 23.3% of NoOp failures come from upstream environment setup, simple rejection or local retry drops harder multi-file tasks and falls behind DRR by 2.0 to 2.8 points at the same training size.

Q1. Downstream contribution under matched generation and training budgets

Removing any component consistently lowers Qwen3.5-4B pass@1 under both matched generation budgets and matched training budgets.

  1. Matched generation budget:
ComponentPipeline VariantH/log⁡K↑H/\log K \uparrowTail-50 ↑\uparrowMismatch ↓\downarrowTraj.TB 2.1TBLite
OverallOpenTerminal, full0.92726.9%4.5%7,36521.8 ± 3.043.6 ± 2.7
CATSw/o corpus anchoring0.5483.2%-5,47715.9 ± 1.934.8 ± 2.3
CATSw/o cluster smoothing, α=1\alpha=10.7397.8%-6,11717.7 ± 2.438.1 ± 3.2
STCw/o shared contract--49.5%3,84417.1 ± 2.038.3 ± 2.9
DRRLocal-only repair---6,76119.3 ± 3.340.4 ± 2.7
DRRReject-on-failure---6,12618.4 ± 1.939.7 ± 2.8
  1. Matched training budget:
Training SetTrajectoriesTB 2.1TBLite
Reject-on-failure6,12618.4 ± 1.939.7 ± 2.8
OpenTerminal, 6,126 subset6,12620.7 ± 2.242.5 ± 2.4
Local-only repair6,76119.3 ± 3.340.4 ± 2.7
OpenTerminal, 6,761 subset6,76121.3 ± 2.642.8 ± 2.3
OpenTerminal, 6,000 subset6,00020.9 ± 2.442.1 ± 1.9
  • DRR improves quality beyond volume: At matched sizes, OpenTerminal outperforms Reject-on-failure by +2.3 / +2.8 and Local-only by +2.0 / +2.4, whereas scaling OpenTerminal from 6,126 to 7,365 adds +1.1 / +1.1.
  • Smaller-budget advantage: Even with 6,000 trajectories, OpenTerminal beats every ablated variant above despite most of them training on more trajectories.

Q2. Comparison with inverse-frequency and uniform-cluster sampling

Sampling cluster kk with p(k)∝nkαp(k) \propto n_k^\alpha and seeds uniformly within kk gives each seed weight P(s)∝nkα−1P(s) \propto n_k^{\alpha - 1}. Thus α=1\alpha = 1 is seed-uniform sampling, while α=0\alpha = 0 is both uniform-cluster sampling and seed-level inverse-frequency sampling:

SamplerSeed WeightRealized H/log⁡KH/\log KTail-50TrajectoriesTB 2.1TBLite
Prompt-only-0.5483.2%5,47715.9 ± 1.934.8 ± 2.3
Seed-uniform, α=1\alpha = 1∝1\propto 10.7397.8%6,11717.7 ± 2.438.1 ± 3.2
Uniform-cluster / Inv-freq, α=0\alpha = 0∝nk−1\propto n_k^{-1}0.83322.1%7,03920.4 ± 3.142.1 ± 2.2
CATS, α=0.4\alpha = 0.4∝nk−0.6\propto n_k^{-0.6}0.92726.9%7,36521.8 ± 3.043.6 ± 2.7

CATS with α=0.4\alpha=0.4 outperforms both α=1\alpha=1 and α=0\alpha=0 across diversity, task yield, and downstream accuracy. Because cluster sizes range from 100 to 62,500, 1/nk1/n_k weighting under α=0\alpha=0 heavily over-samples rare seeds; grouping incompatible rare tools together causes the teacher to fall back to common seeds more often and lowers container survival. Sublinear smoothing at α=0.4\alpha=0.4 balances rare-cluster coverage with coherent task composition.

Q3. Overall cost per validated task and per training trajectory

Across all 16,384 construction attempts with DeepSeek-V4-Flash, total and unit generation costs are:

PolicyValidated TasksSFT Traj.Total TokensCalls / TaskCalls / Traj.Tokens / TaskTokens / Traj.
Reject-on-failure9,4306,1260.51B11.016.953,77282,773
Local-only repair10,6796,7610.61B11.818.757,33990,567
DRR11,7137,3651.15B16.326.097,898155,692

Although DRR uses more construction tokens per trajectory than local retries or first-pass rejection, its gains do not come merely from spending more budget to collect more data. As shown by our matched-budget experiment in Q1, even at the exact same trajectory counts, OpenTerminal outperforms Reject-on-failure by +2.3 / +2.8 at 6,126 trajectories and Local-only repair by +2.0 / +2.4 at 6,761 trajectories, because cross-stage repair recovers harder multi-file workflows that local retries cannot fix and first-pass rejection drops.

Thanks for addressing my concerns with the additional experiments and cost breakdown. While the conceptual novelty is somewhat incremental, the matched-budget analyses do support the pipeline's effectiveness. I have raised the score accordingly.

Reviewer_rFTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenTerminal, a framework for automatically constructing executable and verifiable terminal-agent tasks at scale. The framework consists of three components: (1) Corpus-anchored task synthesis, which improves task diversity by sampling from heterogeneous technical corpora, (2) shared task contract, which coordinates environment and verifier generation through a common specification, and (3) diagnose-route-repair, which identifies likely sources of construction failures and regenerates the affected artifacts. Supervised fine-tuning of Qwen3.5-4B and Qwen3.5-9B yields substantial improvements over the base models, outperforming several contemporary terminal-agent training datasets. Overall, the paper addresses an important practical challenge in scaling terminal-agent training data. The framework is well motivated, and the empirical results are promising. However, several concerns regarding task validity, experimental controls, and generalization should be addressed.

Critical strengths
  1. Important and practical problem. Constructing executable environments and reliable verifiers is a significant bottleneck in training terminal agents. The paper addresses this problem through a systematic and scalable pipeline.
  2. Coherent framework with complementary components. The pipeline target different challenges in task construction, including diversity, consistency, and failure recovery. The shared contract and cross-stage repair mechanisms are useful for managing dependencies between generated artifacts.
  3. Promising experimental results. The proposed dataset consistently improves performance across two model sizes and two evaluation benchmarks. The comparisons with multiple contemporary datasets further support the practical value of the generated training data.
  4. The ablation studies demonstrate improvements in seed-cluster coverage, artifact consistency, and construction yield.
Critical weaknesses
  1. The primary execution checks consist of successful environment construction and a zero-score no-op test. While these checks can detect certain defects, they do not establish that a task is actually solvable or that its verifier correctly distinguishes valid and invalid solutions. The generated contract, environment, and verifier may also share systematic errors.
  2. Table 2 evaluates CATS, STC, and DRR primarily using intermediate metrics rather than downstream agent performance. Consequently, it remains unclear how much each component contributes to the final benchmark improvements. Furthermore, the baseline datasets contain substantially different numbers of trajectories, making it difficult to separate improvements in data quality from differences in training-data volume and token budgets.
  3. Although two benchmarks are evaluated, both belong to the Terminal-Bench family, limiting the evidence for generalization to substantially different terminal workflows. Additionally, improvements over the strongest competing datasets are relatively small, compared with the variability reported in Table 1.
  4. The diversity analysis primarily measures the distribution of selected seed clusters rather than independently assessing the semantic diversity of the final executable tasks or training trajectories.
  5. Trajectories with verifier scores of at least 0.5 are accepted for SFT, even when the task is not fully completed. Since all assistant response tokens are supervised, unsuccessful actions may also enter the training data. The paper does not examine how the acceptance threshold or the inclusion of partially successful trajectories affects downstream performance.
  6. The framework involves multiple LLM generation, validation, repair, and rollout stages, but its computational cost, runtime, and generation efficiency are not quantified. These measurements are important for evaluating scalability relative to simpler construction or retry strategies.
Questions for authors
  1. How do the authors establish that generated tasks are genuinely solvable and that their verifiers correctly evaluate successful solutions? Could the authors provide a human evaluation or reference-solution-based validation on a representative sample, including estimates of false-positive and false-negative verification rates?

  2. Could the authors train agents using datasets constructed without CATS, STC, or DRR and compare downstream benchmark performance? In particular, how much of DRR's benefit comes from increased data quantity versus improvements in task quality?

  3. How would the results change if all competing datasets were matched for the number of trajectories or supervised training tokens?

  4. Does the increased seed-cluster diversity persist in the final training trajectories? Can the authors evaluate task-level semantic diversity and performance on held-out domains or additional independent terminal benchmarks?

  5. What are the computational and monetary costs per successfully constructed task? How does DRR compare with compute-matched alternatives such as additional local retries, random rerouting, or complete task regeneration? Is the accuracy of source-stage diagnosis evaluated independently?

  6. What fraction of the selected training trajectories achieves complete success, and how sensitive are the results to the score threshold of 0.5? Additionally, could the authors clarify whether the reported trajectories are counted before or after benchmark-overlap filtering, and whether semantic overlap checks were performed beyond minhash-based instruction matching?

3 comments

We thank the reviewer for the thorough and constructive feedback. Below we address each concern and will incorporate the clarifications and additional results into the revision. If our response resolves your concerns, we would greatly appreciate it if you could raise your score (2: Weak rejection). We are happy to clarify any remaining issues.


Q1. Task solvability, human validation, and false-positive / false-negative rates

OpenTerminal does not claim to guarantee that every generated task is perfectly solvable or free of verifier flaws, as build and no-op gates are necessary rather than sufficient conditions. Instead, our pipeline uses contract alignment, execution gates, and rollout filtering to screen out low-quality instances so the retained data is predominantly high quality. Even when minor verifier imperfections remain, coherent teacher trajectories executed in runnable containers still provide valuable supervision on tool use and environment interaction, as confirmed by our downstream gains.

To quantify the actual task quality, two annotators audited 90 accepted container tasks across the three training cohorts of 30 tasks each. One authored a reference solution and a wrong final state, while the other verified solvability and tested the verifier on solvable tasks:

CohortnnSolvableReference AcceptedWrong State Rejected
First-pass3030 (100.0%)28 (93.3%)28 (93.3%)
Local repair3028 (93.3%)27 (96.4%)28 (100.0%)
Cross-stage repair3026 (86.7%)23 (88.5%)25 (96.2%)
Total9084 (93.3%)78 (92.9%)81 (96.4%)

Across the accepted training cohorts, 93.3% of audited tasks are solvable, reaching 98.3% when weighted by training cohort shares, with 92.9% reference acceptance and 96.4% wrong-state rejection.


Q2. Downstream ablations of CATS, STC, and DRR, and DRR quantity vs. quality

Every component contributes to downstream performance, and equal-count controls show that DRR's gains come primarily from recovered task quality rather than data volume.

  1. End-to-end component ablations:
VariantTrajectoriesTB 2.1TBLite
OpenTerminal, full7,36521.8 ± 3.043.6 ± 2.7
w/o CATS corpus anchoring5,47715.9 ± 1.934.8 ± 2.3
w/o CATS cluster smoothing6,11717.7 ± 2.438.1 ± 3.2
w/o STC shared contract3,84417.1 ± 2.038.3 ± 2.9
w/o DRR, Local-only repair6,76119.3 ± 3.340.4 ± 2.7
w/o DRR, Reject-on-failure6,12618.4 ± 1.939.7 ± 2.8

Removing CATS lowers pass@1 by up to 5.9 / 8.8 points, while removing STC cuts usable trajectories nearly in half from 7,365 to 3,844 due to cross-artifact conflicts and lowers pass@1 by 4.7 / 5.3 points.

  1. DRR quality vs. quantity at matched trajectory counts:
Training SetTrajectoriesTB 2.1TBLite
Reject-on-failure6,12618.4 ± 1.939.7 ± 2.8
OpenTerminal, 6,126 subset6,12620.7 ± 2.242.5 ± 2.4
Local-only repair6,76119.3 ± 3.340.4 ± 2.7
OpenTerminal, 6,761 subset6,76121.3 ± 2.642.8 ± 2.3

At matched sizes, OpenTerminal exceeds Reject-on-failure by +2.3 / +2.8 and Local-only by +2.0 / +2.4, whereas scaling OpenTerminal from 6,126 to 7,365 adds +1.1 / +1.1. Dropping failed builds or no-ops favors simple single-file setups, while DRR retains harder multi-file workflows.


Q3. Matching competing datasets on trajectories or supervised tokens

OpenTerminal outperforms all competing datasets at matched or lower trajectory and supervised-token counts:

Training DatasetTrajectoriesSupervised TokensTB 2.1TBLite
Base--13.9 ± 2.633.2 ± 4.1
CLI-Universe6,000-19.1 ± 2.238.4 ± 1.6
LiteCoder-Terminal11,255132.9M17.1 ± 2.840.4 ± 3.1
OpenTerminal, 6,000 subset6,00093.3M20.9 ± 2.442.1 ± 1.9
OpenTerminal, full7,365114.1M21.8 ± 3.043.6 ± 2.7

At 6,000 trajectories, OpenTerminal exceeds equal-count CLI-Universe by +1.8 / +3.7, surpasses LiteCoder-Terminal by +3.8 / +1.7 with 47% fewer trajectories and 30% fewer supervised tokens, and beats every ablated variant in Q2.

Q4. Trajectory diversity and generalization beyond Terminal-Bench

CATS's diversity gains persist into the final 7,365 SFT trajectories with H/log⁡K=0.918H/\log K = 0.918 and Tail-50 =24.5%= 24.5\%, closely matching the accepted instructions at 0.927 / 26.9% versus the raw seed pool at 0.739 / 7.8%. Beyond seed clusters, tasks show near-zero lexical overlap at 0.0010 pairwise 4-gram overlap, distinct-4 = 0.660, and 97.0% concrete CLI tool grounding across 44 domains.

While both evaluation suites belong to the Harbor / Terminal-Bench ecosystem used across concurrent works and broader benchmarks remain future work, OpenTerminal uses zero Terminal-Bench seeds, prompts, or taxonomies, making both suites genuine zero-overlap out-of-distribution evaluations.


Q5. Construction cost, compute-matched alternatives, and source-stage diagnosis

Across all 16,384 construction attempts with DeepSeek-V4-Flash, total and unit token costs are:

PolicyValidated TasksTrajectoriesTotal TokensCalls / TaskCalls / Traj.Tokens / TaskTokens / Traj.
Reject-on-failure9,4306,1260.51B11.016.953,77282,773
Local-only repair10,6796,7610.61B11.818.757,33990,567
DRR11,7137,3651.15B16.326.097,898155,692

Although DRR uses more construction tokens per trajectory than local retries or first-pass rejection, its gains do not come merely from spending more budget to collect more data. Comparing Reject-on-failure and Local-only repair against equal-count random subsets of OpenTerminal on Qwen3.5-4B confirms higher per-trajectory quality:

Training SetTrajectoriesTB 2.1TBLite
Reject-on-failure6,12618.4 ± 1.939.7 ± 2.8
OpenTerminal, 6,126 subset6,12620.7 ± 2.242.5 ± 2.4
Local-only repair6,76119.3 ± 3.340.4 ± 2.7
OpenTerminal, 6,761 subset6,76121.3 ± 2.642.8 ± 2.3
OpenTerminal, full7,36521.8 ± 3.043.6 ± 2.7

At matched trajectory counts, OpenTerminal outperforms Reject-on-failure by +2.3 / +2.8 and Local-only repair by +2.0 / +2.4, showing that cross-stage repair recovers harder multi-file workflows that local retries cannot fix and fresh first-pass sampling drops. Moreover, DRR is invoked only after local retries fail, and its 1,034 cross-stage recoveries achieve 86.7% human solvability and 88.5% reference acceptance in Q1, directly confirming effective source-stage routing.


Q6. Complete-success fraction, r≥0.5r \ge 0.5 sensitivity, and decontamination

  1. Success breakdown and threshold sensitivity. Among the 7,365 trajectories, 39.5% achieve full success at r=1r = 1 (2,906 trajectories) and 60.5% achieve partial success (4,459 trajectories), with 1,684 in [0.8,1)[0.8, 1) and 2,775 in [0.5,0.8)[0.5, 0.8). At a matched 2,906-trajectory count on Qwen3.5-4B, r≥0.5r \ge 0.5 outperforms stricter thresholds:
Selection RuleTrajectoriesTB 2.1TBLite
r=1r = 1 only2,90614.4 ± 2.734.7 ± 2.1
r≥0.8r \ge 0.8 subset2,90616.8 ± 3.135.6 ± 2.5
r≥0.5r \ge 0.5 subset2,90617.9 ± 3.336.9 ± 2.7
r≥0.5r \ge 0.5 full7,36521.8 ± 3.043.6 ± 2.7

r≥0.5r \ge 0.5 exceeds r=1r = 1 by +3.5 / +2.2 at equal count because restricting to r=1r = 1 biases training toward simpler tasks solved on the first pass. Although r≥0.5r \ge 0.5 includes partially completed runs, these trajectories still capture the teacher's key exploration details and high-quality responses on harder multi-step workflows. Across the full dataset, r≥0.5r \ge 0.5 also retains substantially more usable training data from 2,906 to 7,365 trajectories and lifts pass@1 to 21.8 / 43.6, while still filtering out low-scoring or defective tasks.

  1. Decontamination. 7,365 is the post-decontamination count: whitespace 5-gram tabulation MinHash LSH with signature length k=9,000k=9{,}000, b=20b=20 hashes per band, and r=450r=450 bands, followed by exact Jaccard and edit-similarity checks at threshold 0.8, removes 0 tasks, peaking at 0.022 Jaccard on TB 2.1 and 0.013 on TBLite. While we did not run a separate embedding screen, none of our 36 corpora include either benchmark and the closest pairs only share 3 to 5 common phrasing 5-grams across unrelated tasks and tools.
Reviewer_rFTROpenReview ↗

Response to the authors

I thank the authors for their responses. I would like to maintain my score.

Reviewer_zUrEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

OpenTerminal constructs executable training tasks for terminal agents through corpus-anchored synthesis, shared task contracts, and Diagnose-Route-Repair. The pipeline derives 307,293 seed patterns from 36 corpora, generates task descriptions, and builds environments and verifiers for a frozen subset of 16,384 descriptions. It yields 11,713 executable tasks and 7,365 eligible trajectories. Fine-tuning Qwen3.5-4B and 9B improves performance on Terminal-Bench 2.1 and OpenThoughts-TBLite. Component analyses measure semantic coverage, cross-artifact mismatch, construction yield, and recovered-task characteristics.

Critical strengths
  1. The paper treats executable-data construction as a coordinated engineering and learning problem. Shared contracts and dependency-suffix regeneration are well motivated because a failure observed in a verifier or build can originate upstream. The explicit separation of task descriptions, executable tasks, and eligible trajectories is useful, and the build/no-op gates plus case-level diagnostics make construction attrition more interpretable than a final dataset count alone.

  2. The resulting data demonstrates downstream utility on two model sizes, with relevant concurrent data-pipeline comparisons under the same backbones. OpenTerminal raises Terminal-Bench 2.1 scores from 13.9 to 21.8 at 4B and from 15.7 to 25.6 at 9B, with five evaluation trials per task. The corpus/sampling ablations and detailed recovery ledger provide additional evidence about what the pipeline changes, rather than relying solely on the final training result.

Critical weaknesses
  1. Post-construction semantic validity needs a more direct audit. A passing no-op gate establishes that the untouched environment scores zero, but cannot show that the verifier accepts valid solutions or that hidden contract requirements are inferable from the agent-visible instruction. Appendix B correctly acknowledges the former limitation. The reported mismatch reduction is substantial, but the manuscript does not describe its adjudication protocol sufficiently to assess whether the same generator's interpretation is being independently checked. This matters when scores as low as 0.5 are eligible for training and when the contract resolves choices that the instruction leaves unspecified.

  2. The evidence for superior data quality and recovery efficiency is less isolated than the headline comparisons suggest. The training sets differ in trajectory counts, and the paper does not give a corresponding assistant-token budget comparison. Component ablations predominantly measure coverage, consistency, and yield, rather than downstream performance. DRR also expends additional construction effort, while lower scores on recovered tasks could reflect difficulty or residual defects; Appendix C explicitly recognizes that ambiguity. The results establish useful yield and training gains, but do not yet quantify how much advantage remains per unit of construction or training compute.

Questions for authors
  1. Could you describe how the 4.5% versus 49.5% mismatch rates were adjudicated, including the denominator, judge/human procedure, and which artifacts were visible? A small independently inspected sample of accepted tasks, stratified across first-pass and recovered tasks, would help check instruction-contract agreement and verifier acceptance of a valid solution. Please distinguish genuine defects from merely harder tasks.

  2. Could you report construction calls, tokens, or elapsed/compute cost per accepted task and eligible trajectory for reject-on-failure, local-only repair, and DRR, using the existing repair logs? Please make clear whether the comparison controls attempted-task count, retry budget, or total generation compute. A full retraining of all ablated datasets is not required.

  3. For the main data comparisons, could you report total supervised assistant tokens, trajectory-length distributions, and the interpretation of the reported plus/minus values? If task-level outcomes are retained, paired uncertainty for OpenTerminal versus the strongest data baseline would help assess the 2.4-3.2 point incremental gains. A token-matched subset result would be informative if readily available, but is not essential if the claim is scoped appropriately.

  4. Please provide the MinHash settings and numbers removed against each evaluation benchmark, and clarify whether 7,365 is the final post-decontamination training count. A brief inspection of the closest retained task matches would strengthen confidence that composition/paraphrasing has not defeated an instruction-text overlap filter; I am not asserting that contamination occurred.

  5. Could you summarize how many retained training trajectories are fully successful versus partially successful at the r >= 0.5 threshold, and what eligibility checks prevent obvious incorrect behavior from becoming supervised targets? A small set of partial-credit examples, including a repaired task, would help interpret this selection rule. Please also align the main-text claim that recovered tasks are systematically harder with the observational caveat already stated in Appendix C.

6 comments

We thank the reviewer for the incisive and constructive comments. Below we address each concern and will incorporate the clarifications and additional results into the revision. If our response resolves your concerns, we would greatly appreciate it if you could raise your score (2: Weak rejection). We are happy to clarify any remaining issues.

Q1. Mismatch adjudication and human validity audit

OpenTerminal does not claim to guarantee that every generated task is perfectly solvable or free of verifier flaws, as build and no-op gates are necessary rather than sufficient conditions. Instead, our pipeline combines contract alignment, execution gates, and trajectory filtering to screen out low-quality instances so the retained data is predominantly high quality. Even when minor verifier imperfections remain, coherent teacher trajectories executed in runnable containers still provide valuable supervision, as confirmed by our contract ablation and 120-task human audit below.

  1. Mismatch adjudication and downstream impact. Across all 16,384 constructed tasks, DeepSeek-V4-Flash evaluates whether the instruction, environment, and verifier agree with and without the shared contract CC, finding mismatches in 738 tasks (4.5%) with CC versus 8,104 tasks (49.5%) without CC. Beyond this automated check, regenerating EE and VV from II alone without CC also reduces both trajectory yield and Qwen3.5-4B accuracy:
VariantEligible Traj.TB 2.1TBLite
w/o Shared Contract3,84417.1 ± 2.038.3 ± 2.9
OpenTerminal7,36521.8 ± 3.043.6 ± 2.7
  1. Independent 120-task human audit. To validate solvability independently of the generator model, two annotators inspected 120 live container tasks across four cohorts of 30 tasks each. Seeing only the agent-visible instruction and environment without CC or VV, one annotator authored a reference solution and a wrong final state, while the other verified solvability and tested the verifier on solvable tasks:
CohortnnSolvableReference AcceptedWrong State Rejected
First-pass, accepted3030 (100.0%)28 (93.3%)28 (93.3%)
Local repair, accepted3028 (93.3%)27 (96.4%)28 (100.0%)
Cross-stage repair, accepted3026 (86.7%)23 (88.5%)25 (96.2%)
Accepted training tasks9084 (93.3%)78 (92.9%)81 (96.4%)
Excluded zero-score tasks3017 (56.7%)13 (76.5%)14 (82.4%)

Across the accepted training cohorts, 93.3% of audited tasks are solvable without hidden-contract leakage, reaching 98.3% when weighted by training cohort shares, with 92.9% reference acceptance and 96.4% wrong-state rejection. In the excluded zero-score cohort, 56.7% are genuinely hard tasks missed in three rollouts while 43.3% have residual defects, confirming that trajectory filtering removes most broken tasks before training.


Q2. Construction cost and downstream utility across repair policies

All three policies are evaluated on the same 16,384 attempted task descriptions:

PolicyExecutable TasksEligible Traj.Total TokensCalls / TaskCalls / Traj.Tokens / TaskTokens / Traj.
Reject-on-failure9,4306,1260.51B11.016.953,77282,773
Local-only repair10,6796,7610.61B11.818.757,33990,567
DRR11,7137,3651.15B16.326.097,898155,692

To test whether DRR's offline token investment improves data quality beyond volume, we compare Reject-on-failure and Local-only against equal-count random subsets of full OpenTerminal on Qwen3.5-4B:

Training SetTrajectoriesTB 2.1TBLite
Reject-on-failure6,12618.4 ± 1.939.7 ± 2.8
OpenTerminal, 6,126 subset6,12620.7 ± 2.242.5 ± 2.4
Local-only repair6,76119.3 ± 3.340.4 ± 2.7
OpenTerminal, 6,761 subset6,76121.3 ± 2.642.8 ± 2.3
OpenTerminal, full7,36521.8 ± 3.043.6 ± 2.7

Although DRR uses more offline construction tokens per trajectory, it substantially improves per-trajectory training utility, gaining +2.3 / +2.8 over Reject-on-failure and +2.0 / +2.4 over Local-only at matched counts by eliminating the survivorship bias of first-pass rejection toward simple setups.

Q3. Supervised tokens, trajectory lengths, ± values, and matched-budget comparison

OpenTerminal outperforms concurrent datasets at full scale, at a matched 6,000-trajectory count, and with fewer supervised assistant tokens.

Each ± value denotes the standard deviation of pass@1 across five Harbor trials. To control for training volume, we additionally trained Qwen3.5-4B on a random 6,000-trajectory subset of OpenTerminal and compared total supervised assistant tokens below:

Training SetTrajectoriesAssistant TokensTB 2.1TBLite
Base--13.9 ± 2.633.2 ± 4.1
CLI-Universe6,000-19.1 ± 2.238.4 ± 1.6
LiteCoder-Terminal11,255132.9M17.1 ± 2.840.4 ± 3.1
OpenTerminal, 6,000 subset6,00093.3M20.9 ± 2.442.1 ± 1.9
OpenTerminal, full7,365114.1M21.8 ± 3.043.6 ± 2.7

Full OpenTerminal already uses 14.1% fewer supervised tokens than LiteCoder-Terminal while gaining +4.7 / +3.2 points. At 6,000 trajectories, OpenTerminal outperforms equal-count CLI-Universe by +1.8 / +3.7 and surpasses LiteCoder-Terminal by +3.8 / +1.7 with 47% fewer trajectories and 30% fewer supervised tokens.


Q4. Decontamination settings, removed counts, and closest matches

7,365 is the final post-decontamination training count, with 0 tasks removed against either Terminal-Bench 2.1 or OpenThoughts-TBLite.

We apply tabulation MinHash LSH over contiguous whitespace 5-grams with signature length k=9,000k = 9{,}000, b=20b = 20 hashes per band, and r=450r = 450 bands, followed by exact 5-gram Jaccard and edit-similarity checks at threshold 0.8. Direct exact 5-gram Jaccard against all benchmark instructions peaks at 0.022 on TB 2.1 and 0.013 on TBLite. Manual inspection of the closest pairs shows that they only share 3 to 5 common phrasing 5-grams such as "after your changes I should be able to..." and involve completely unrelated tasks and tools, as our data is synthesized from 36 public corpora containing neither benchmark.


Q5. Full vs. partial success, eligibility checks, and Appendix C alignment

  1. Breakdown and eligibility checks. Among the 7,365 trajectories, 39.5% achieve full success at r=1r = 1 (2,906 trajectories) and 60.5% achieve partial success (4,459 trajectories), with 1,684 in [0.8,1)[0.8, 1) and 2,775 in [0.5,0.8)[0.5, 0.8). Three safeguards prevent broken behavior from becoming supervised targets: crashed or malformed rollouts are discarded before best-of-3 selection; zero-reward no-op gates block trivial credit; and supervising assistant tokens under r≥0.5r \ge 0.5 ensures the majority of weighted criteria pass while preserving realistic inspect-and-recover steps.

  2. Threshold ablation and partial-credit examples. At a matched 2,906-trajectory count on Qwen3.5-4B, r≥0.5r \ge 0.5 outperforms stricter cuts:

Selection RuleTrajectoriesTB 2.1TBLite
r=1r = 1 only2,90614.4 ± 2.734.7 ± 2.1
r≥0.8r \ge 0.8 subset2,90616.8 ± 3.135.6 ± 2.5
r≥0.5r \ge 0.5 subset2,90617.9 ± 3.336.9 ± 2.7
r≥0.5r \ge 0.5 full7,36521.8 ± 3.043.6 ± 2.7

r≥0.5r \ge 0.5 exceeds r=1r = 1 by +3.5 / +2.2 at equal count because restricting to r=1r = 1 biases training toward simpler tasks rather than complex multi-step workflows:

  • First-pass example at r=0.75r = 0.75: On concurrent Paramiko SSH execution across three hosts, the retained rollout satisfies 3 of 4 criteria including non-blocking concurrency, host-key handling, and password authentication, missing only an output-format check.
  • Repaired example at r=0.67r = 0.67: After verifier repair fixes initial no-op reward from 0.17 to 0.0 on a cross-platform Nginx log-streaming and du script, the retained rollout resolves 2 of 3 defects including cp -R portability and direct ssh | logstalgia piping.
  1. Aligning with Appendix C. As shown by our zero-score audit in Q1, low-scoring tasks combine genuinely harder solvable tasks missed by the teacher at 56.7% with unrepairable defective tasks at 43.3%. Rather than asserting difficulty from rollout scores alone, we will revise Section 4.4 to stay consistent with Appendix C, while Q1 and Q2 confirm that the recovered tasks retained for training are valid and improve downstream performance.

Thank you for the independent task audit and the comparisons at matched trajectory counts. These results give me more confidence that the repaired tasks provide useful training data My remaining concern is the construction cost. Repair uses substantially more tokens per retained trajectory, and it is still unclear how the methods would compare under the same total construction budget. I will therefore keep my overall rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.