acceptodds
Under review as a conference paper at ICLR 2027

CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion

Abstract

Command-line-interface (CLI) agents must diagnose and repair dependencies, permissions, system files, runtime configurations, and source code, yet executable training data for these interactions remain limited. We introduce CLI-Gym, an execution-feedback-guided agentic environment inversion pipeline. Starting from healthy, unit-test-verified repository environments, an inversion agent constructs counterfactual degraded states, summarizes the changes as reproducible Dockerfiles, and pairs observed failures with natural-language issues and executable verifiers. From 4,066 inversion prompts, CLI-Gym produces 1,655 verified CLI task instances; 81.0% contain an environment modification. A repair model solves 417 tasks, from which shortcut and length filtering retains 291 trajectories for supervised fine-tuning. Under the same OpenHands scaffold, these trajectories improve Qwen3-235B-A22B-Instruct by 21.1 points to 46.1% pass@1 on Terminal-Bench 1.0 and by 12.9 points to 31.0% on Terminal-Bench 2.0, with consistent gains at the 32B scale. A three-expert stratified audit separately characterizes the realism, solvability, issue–test consistency, and shortcut resistance of the full task collection. These results establish the training value of the filtered teacher-solvable subset while making the quality boundary of the complete collection explicit.

Primary area:
datasets and benchmarks
Keywords:
Agentic Coding, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CTCGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The author presents CLI-Gym, a pipeline to synthesize executable environments for Terminal Bench from GitHub. The pipeline produces 1,655 TB-like environments from 29 Python repositories, though environment inversion (letting LLMs freely break a known-good environment such that a random subset of unit tests fail and the break direction differs from previous breaks). Rejection sampling from Deekseep-V3.2 yields 291 trajectories. Fine-tuning Qwen3-32B and Qwen3-235B-A22B-Instruct with the teacher trajectories and open source SWE trajectories yields competitive Terminal Bench 1.0/2.0 performance among open weights models.

Critical strengths
  • Data efficiency: with only SFT 291 teacher trajectories from Deepseek V3.2, the model's Terminal Bench 1.0 performance raised significantly, across 2 model sizes and with and without SWE trajectory data training before the teacher trajectory SFT.
Critical weaknesses
  • Lack of fair comparison with previous TB-focused data curation work: The authors only compared the benchmark scores of their models against other open or closed weight models, that may use different agentic harnesses and inference budgets. There is no controlled comparison (training on same backbone model and same token budget, or comparison on yield rate and quality) with other open TB-focused datasets such as TerminalTraj or TerminalGen. The authors therefore frame their data synthesis approach as complementary with other works rather than superior.
Questions for authors
  1. Can you specify details about the source and composition of the 48k SWE initialization trajectories used in Sec 4. ?
  2. Do the LiberCoder models hack the reward or use shortcuts in evaluation, as found in teacher trajectories?
  3. What is the performance of the teacher model Deepseek V3.2 on Terminal Bench in the evaluation environment and harness of this work?
5 comments

Thank you for thoughtfully reviewing our work and recognizing the data efficiency of the generated supervision. We appreciate your specific questions about fair comparisons, the SWE initialization data, and evaluation behavior, which have helped us identify where the experimental setup and its limitations need clearer explanation.


Q1. How should readers interpret the comparisons with TerminalTraj and TermiGen?

A1. Thank you for raising comparability. TermiGen-Qwen3-32B provides the closest published backbone comparison. The following TB1.0 pass@1 results use the same Qwen3-32B backbone; our CLI-Gym-only setting excludes SWE initialization.

SettingBefore trainingAfter training
TermiGen10.4%27.5%
CLI-Gym-only10.3%32.4%

TermiGen uses BashAgent and averages three runs at temperature 0.6, while we use OpenHands and greedy decoding. The similar starting scores and shared backbone provide context for the observed gains, although scaffold, sampling, and training differences limit causal attribution. TermiGen, Sections 4.1-4.2

TerminalTraj-32B uses Qwen2.5-Coder-32B-Instruct, trajectories generated with Qwen3-Coder-480B, and the Terminus-2 scaffold. Our full LiberCoder-32B setting uses Qwen3-32B with SWE initialization followed by training on CLI-Gym trajectories.

ModelTB1.0 pass@1TB2.0 pass@1
TerminalTraj-32B35.3%22.0%
LiberCoder-32B38.9%19.5%

This gives mixed cross-benchmark rankings and illustrates the distinct strengths of the resulting systems. TerminalTraj, Sections 2.3.2 and 4, Table 2

Our central empirical conclusion is that the filtered trajectories produced by CLI-Gym provide useful supervision across two student scales and both benchmark versions under a fixed OpenHands scaffold. The within-study ablations support that conclusion, while a matched training comparison would further establish the relative value of different CLI data sources. We will make these comparison conditions explicit and retain the complementary positioning of the construction methods.


Q2. What is the source and composition of the 48K SWE initialization trajectories?

A2. The 48K trajectories come from solving tasks drawn from SWE-smith, SWE-Gym, and R2E-Gym, all based on open-source repositories. We use them to initialize repository navigation, tool use, and code repair. These training tasks and their source repositories are disjoint from Terminal-Bench. We will name these sources in Section 4.1.


Q3. Did LiberCoder use shortcuts during Terminal-Bench evaluation?

A3. We used the standard Terminal-Bench 1.0 verifier, which was hidden from the agent during task execution. We inspected one complete evaluation run for each of LiberCoder-235B-A22B and LiberCoder-32B, covering 80 trajectories per model and 160 in total. We observed no apparent shortcut behavior in the inspected trajectories.


Q4. How does DeepSeek-V3.2 perform under the same evaluation setup?

A4. Under the same OpenHands evaluation setup as LiberCoder, DeepSeek-V3.2 achieves 42.7% pass@1 on Terminal-Bench 1.0.

Thank you for the detailed response.

A1. The TermiGen comparison from the same Qwen3-32B start (10.3 -> 32.4 vs 10.4 -> 27.5) is the most useful piece of context here. I note, however, that the margin over TerminalTraj-32B on TB1.0 (38.9 vs 35.3) only appears with the 48K-trajectory SWE initialization, and that on TB2.0 the ordering is reversed (19.5 vs 22.0). The data-efficiency framing around "291 trajectories" should therefore be stated as conditional on the 48K first stage. I would also keep in mind that the rows you compare against are leaderboard-reported numbers from different scaffolds, teachers and base models (TerminalTraj-32B starts from Qwen2.5-Coder-32B with a Qwen3-Coder-480B teacher), while each of your pass@1 scores is a single greedy run (Appendix B.3); greedy decoding does not make agent runs deterministic, since environment responses and small decoding differences vary between runs, so differences of a few points across papers are within what re-evaluation typically moves. With the comparison conditions made explicit as you propose, I still consider the level of comparison acceptable for this submission since the authors frame their data synthesis approach as complementary.

A2. The response names the task sources (SWE-smith, SWE-Gym, R2E-Gym) but not the model that produced the 48K trajectories. Please state it directly: were these trajectories released by those projects, or rolled out by you, and with which model? Since LiberCoder is listed under "Open-source Data Pipeline" in Table 4 and the headline 46.1/31.0 numbers depend on this stage, a proprietary teacher here would need to be disclosed.

A3 is satisfactory. On A4, please confirm that the 42.7% for DeepSeek-V3.2 was reproduced by you in the same evaluation harness and environment as LiberCoder, rather than taken from a reported figure, since environment differences are known to shift Terminal-Bench scores.

Thank you for the follow-up and for considering the comparisons acceptable with the conditions clarified. We address the remaining points below.


Q1. How should the comparison conditions and the data-efficiency claims be clarified?

A1. We agree that the comparison with TermiGen using the same Qwen3-32B backbone provides the most relevant context. Our CLI-Gym-only result in that comparison uses 291 trajectories without SWE initialization.

For the full LiberCoder results, we will explicitly state that training consists of 48K SWE initialization trajectories followed by 291 CLI-Gym trajectories, and qualify the data-efficiency discussion accordingly. In particular, the reported TB1.0 advantage over TerminalTraj-32B uses this two-stage setting, while TerminalTraj-32B scores higher on TB2.0. We also acknowledge the differences in scaffolds, teachers, base models, and evaluation protocols, as well as run-to-run variability even with greedy decoding. We will make these conditions explicit and interpret the cross-paper results as contextual comparisons supporting the complementary positioning of our approach.


Q2. Were the 48K SWE trajectories publicly released by the source projects or generated by the authors, and which teacher models produced them?

A2. Thank you for identifying this omission. We assembled the 48K initialization set from publicly released trajectory datasets, including the following releases. We reused trajectories generated and released by these projects.

Released datasetReported trajectory-generation models
SWE-smith-trajectoriesClaude 3.7 Sonnet
R2EGym-SFT-TrajectoriesClaude 3.5 Sonnet v2, as described in the R2E-Gym paper
OpenHands-SFT-TrajectoriesGPT-4o and Claude 3.5 Sonnet, as described in the SWE-Gym paper

Thus, the SWE initialization stage does include publicly released trajectories generated by proprietary teacher models. In Table 4, “Open-source Data Pipeline” was intended to describe the availability of the training data. We recognize that this wording leaves the teacher-model provenance unclear. We will clarify the terminology in the table caption and explicitly disclose these data sources and teacher models in Section 4.1.


Q4. Was the 42.7% DeepSeek-V3.2 result reproduced by the authors under the same evaluation harness and environment as LiberCoder?

A4. We confirm that we evaluated DeepSeek-V3.2 ourselves and obtained 42.7% pass@1 on Terminal-Bench 1.0, using the same OpenHands evaluation harness and benchmark environment as LiberCoder. This is a result from our own evaluation. We will state its provenance explicitly in the revised manuscript.

Reviewer_SbpJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CLI-Gym, a pipeline for synthesizing command-line/environment-repair tasks by "inverting" healthy environments: starting from 29 unit-test-verified Python repository images (from the SWE-smith set), a DeepSeek-V3.2 agent running in OpenHands is prompted to degrade the environment (dependencies, permissions, system libraries, configuration, or source), summarize its actions as a replayable Dockerfile, and the failing unit tests become the verifier; an LLM then writes a natural-language repair issue. From 4,066 inversion prompts the pipeline yields 1,655 verified tasks (81% involving an environment modification). The teacher solves 417 of them, and 291 trajectories survive length/shortcut filtering. A three-expert stratified audit characterizes task quality per funnel stage (e.g., 59.8% solvability and 70.0% realism on the full collection vs. ~98% solvability on the filtered subset). Fine-tuning Qwen3-32B and Qwen3-235B-A22B-Instruct (after a 48K-trajectory SWE warm-up) on the 291 trajectories improves Terminal-Bench 1.0 pass@1 from 10.3→38.9 and 25.0→46.1, and Terminal-Bench 2.0 from 5.7→19.5 and 18.1→31.0, under a fixed OpenHands scaffold. Ablations cover SWE vs. CLI-Gym supervision, trajectory filtering, repository diversity, data scaling, and hint-augmented issues.

Critical strengths
  1. A simple, novel, and practical route to environment-level agent data. Inverting a unit-test-verified healthy environment into a degraded one, and recording the degradation as a replayable Dockerfile with a test-based verifier, addresses a real gap: unlike code repair, environment states have no mineable history. The pipeline is largely automated, yields 1,655 tasks at modest storage cost (119 GB), and 81% of tasks involve non-source modifications, which is complementary to PR-mined SWE datasets.
  2. Large, controlled gains with very little data, and a transparent quality analysis. Under the same OpenHands scaffold, 291 trajectories improve both Qwen3 scales on both Terminal-Bench versions by 13–29 points, far exceeding what 48K generic SWE trajectories provide (Table 5). The paper is unusually candid about evidence boundaries: the expert audit (with item-level labels and κ≈0.72 in an anonymous artifact) explicitly shows the full collection's lower solvability/realism, and the claims are scoped to the filtered, teacher-solvable subset.
Critical weaknesses
  1. The experiments do not isolate the contribution of the CLI-Gym construction method from teacher distillation, and the evaluation is statistically thin. All improvements are measured relative to the untuned base model. There is no data-matched comparison (same student, scaffold, and trajectory count) against other CLI data sources (e.g., TerminalTraj or TermiGen trajectories) or against DeepSeek-V3.2 trajectories on non-inverted CLI tasks, and the teacher's own Terminal-Bench score under OpenHands is not reported. Hence it is unclear whether the gains come from environment inversion specifically or from distilling a strong teacher's OpenHands-format agent behavior. In addition, every pass@1 number is a single greedy run over 80/89 tasks with no variance estimate; one task equals 1.25 (TB1.0) / 1.12 (TB2.0) points, so several ablation conclusions rest on 1–3 tasks (e.g., Table 6: 32.4 vs. 33.8 and 36.4 vs. 38.9; Table 12: 23.0 vs. 22.8).
Questions for authors
  1. (Clarification) Figure 8: were SWE trajectories included? The endpoints coincide with the CLI-only row of Table 5. Please reconcile with the text on p. 9.
  2. (Clarification) What is the source of the 48K open-source SWE trajectories, and did you check them for overlap with Terminal-Bench tasks?
1 comment

Thank you for the careful review and for recognizing the value of our task-construction approach and controlled training results. Your questions about the source of the gains and the interpretation of the ablations have helped us clarify the experimental settings and the scope of the conclusions supported by our evidence.


Q1. How much of the improvement can be attributed to environment inversion versus teacher distillation?

A1. Environment construction and teacher distillation both contribute to our pipeline. Inversion determines the failures and recovery problems that the teacher encounters, while the teacher supplies the actions and reasoning used to solve them. Under the same OpenHands evaluation setup, DeepSeek-V3.2 achieves 42.7% pass@1 on Terminal-Bench 1.0.

Related evidence shows why teacher strength alone leaves substantial variation in student outcomes. Terminal-Lego compares teachers using the same set of tasks and the same student models. Claude Opus 4.6 scores 69.4% on Terminal-Bench 2.0 and produces a Qwen3-32B student scoring 15.5%, while DeepSeek-V3.2 scores 39.3% and produces a student scoring 20.6%. In a further experiment using the same Claude teacher and common task set, encouraging environment inspection raises the student score from 15.4% to 19.5%. These controlled results support the relevance of trajectory content and interaction behavior alongside teacher capability. Terminal-Lego (Table 1 and Section 4.3.5)

Our own experiments provide complementary evidence under a fixed teacher and scaffold. Figure 7 holds the number of training trajectories at 100 and varies repository diversity. The observed improvement associates the composition of the training tasks with transfer performance. Table 6 (TB1.0 pass@1) also shows that trajectory selection interacts with initialization.

SWE initializationRaw trajectories (417)Filtered trajectories (291)
No33.8%32.4%
Yes36.4%38.9%

These small differences are descriptive, but they motivate examining data selection jointly with the student's starting capabilities.

Table 5 provides an additional, narrower check on SWE initialization. CLI-Gym-only training raises Qwen3-32B from 10.3% to 32.4% and Qwen3-235B-A22B-Instruct from 25.0% to 39.5%. Thus, substantial gains are also observed when the SWE initialization stage is omitted.

Together, these observations support the training value of the generated tasks and selected teacher trajectories. Precisely assigning the gain to inversion, teacher capability, trajectory composition, and initialization requires further controlled experiments. We will make this scope explicit in the revised analysis and conclusions, and identify systematic comparisons with a fixed teacher, student, scaffold, and training budget as future work.


Q2. How should small ablation differences be interpreted given the single-run pass@1 evaluation?

A2. We agree that the small differences in Table 6 should be treated as descriptive point estimates. All evaluations use greedy decoding (temperature 0), so token selection does not involve random sampling. This reduces one source of run-to-run variation and makes the results useful reference points under the fixed evaluation setup, although it does not remove variability from environment execution.

In Table 12, the two settings with 104 trajectories yielded similar scores of 23.0 and 22.8. Adding hints increased the number of usable trajectories from 104 to 291. We will revise the discussion to report that increase in trajectory yield alongside the observed evaluation scores and qualify the conclusions drawn from the small score differences.


Q3. Did Figure 8 use SWE initialization?

A3. All settings in Figure 8 start from the base Qwen3-32B checkpoint without SWE initialization. The experiment varies only the number of filtered CLI-Gym trajectories used for fine-tuning. The zero-trajectory and full-data endpoints therefore correspond to the base model and CLI-Gym-only rows in Table 5, respectively. We will replace the phrase “holding the SWE trajectories fixed” and state the initialization explicitly in the text and caption.


Q4. Where did the 48K SWE trajectories come from, and do they overlap with Terminal-Bench?

A4. The 48K SWE initialization trajectories come from solving tasks drawn from SWE-smith, SWE-Gym, and R2E-Gym. There is no overlap with Terminal-Bench in either the training tasks or their source repositories. The SWE data focus on repository-level code repair, while Terminal-Bench includes a broader distribution of terminal workflows such as system administration and environment troubleshooting.

Reviewer_7hSdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CLI-Gym, which uses agent interactions to construct degraded states from repository environments with passing unit tests. It generates replayable Dockerfiles, natural-language issues, and executable verifiers. The authors construct 1,655 task instances and use 291 filtered successful repair trajectories for supervised fine-tuning. The results show consistent gains across two Qwen3 model scales and Terminal-Bench 1.0/2.0.

Critical strengths
  1. A concrete and reproducible pipeline for environment-level CLI tasks. CLI-Gym extends fault synthesis to dependencies, permissions, and runtime configuration. It records faults introduced through agent interaction as replayable, verifiable tasks. Environment modifications appear in 81.0% of the generated tasks. This gives the collection clear value as a training resource. (Sections 3.2–3.3)

  2. Clear training gains from a small set of generated trajectories. With the same OpenHands scaffold and SWE initialization, adding 291 CLI-Gym trajectories yields clear gains. For the two models, Terminal-Bench 1.0 pass@1 rises from 22.1% to 38.9% and from 28.6% to 46.1%, respectively. This demonstrates the value of this supervision as a complement to generic SWE data. (Table 5)

Critical weaknesses
  1. The added value of execution-feedback-guided task generation lacks direct validation. The ablations establish the training value of the generated trajectories. However, they do not directly compare agentic environment inversion with single-pass Dockerfile synthesis. The contribution of execution feedback during generation to task yield and quality therefore remains unclear. (Section 4.3; Appendix A.4)

  2. The quality of the full task collection limits its direct use. In the expert audit sample, mean pass rates are 70.0% for realism and 59.8% for solvability. These findings limit how readily the full collection can be used as a set of training environments. The paper does not quantify the main reasons for failed solvability assessments or their frequencies. (Table 3; Appendix A.5)

Questions for authors
  1. Could you compare single-pass Dockerfile synthesis with execution-feedback-guided generation on the same small set of prompts? Please use the same initial environments and generation model, with comparable budgets. Keep the final rebuild and testing rules identical. Report the yield of reproducible tasks, quality on audited samples, and actual generation costs.

  2. What are the main reasons that tasks fail the solvability audit, and how frequent is each reason? Appendices A.2 and A.5 describe test-runner startup failures differently. Please clarify the conditions under which these tasks enter the final collection. Report their number and their solvability based on an audited sample.

1 comment

Thank you for carefully examining our task-generation pipeline and its usability. We appreciate your recognition of the environment-level tasks and training gains, and your suggestions for strengthening the comparison and quality audit.


Q1. How does single-pass Dockerfile synthesis compare with interactive environment inversion?

A1. Defining a single-pass baseline requires deciding which environment information to provide. Relevant paths, file contents, and dependency relationships vary across repositories and often become apparent only after inspection. A comprehensive dump may exceed the context budget, while selecting task-specific files requires additional design. A generated Dockerfile can execute commands when built; interactive generation also lets the model observe their effects and revise its decisions.

Pilot comparison. We tested a static input that could be prepared across repositories: a read-only directory listing to depth three from the healthy image's root directory, plus Python library information. The model was asked to generate a Dockerfile in one response. None of the 10 attempts on different repositories produced a usable task (0/10). Difficulties included locating files, obtaining contents for targeted edits, and checking whether the intended failure occurred.

Execution feedback. Execution feedback allows the agent to retrieve information, inspect outcomes, run tests, and revise ineffective changes. A final rebuild checks whether the Dockerfile reproduces the degraded state. The pilot illustrates limitations of this static-input configuration; its small sample and information-selection choices limit conclusions about single-pass synthesis generally.


Q2. Why did tasks fail the solvability audit, and when are tasks with test-runner startup failures retained?

A2. The full collection's 59.8% solvability and 70.0% realism audit rates reveal substantial quality variation. Automatic verification establishes a reproducible degraded state and provides tests for checking repairs. Usability also requires sufficient diagnostic information and recovery resources, and a verifier aligned with the issue.

Audit findings

Three experts provided 480 solvability judgments for 160 generated tasks; 193 were marked as failures. We reviewed their notes, where recording a reason was optional. All percentages below use the 193 failed judgments as the denominator.

Reason recorded by the expertShare of failed judgments
Insufficient diagnostic clues in the issue10.4%
A dependency or file required for repair was unavailable6.7%
An independent fault unrelated to the issue blocked completion7.3%
The verifier could not detect the relevant problem9.8%
The expert considered the root cause too difficult to localize18.1%
No reason recorded47.7%

These reasons concern task information, recovery resources, unrelated faults, verifier coverage, and diagnostic difficulty. They help explain why a reproducible task may still be difficult to use as supplied. Reasons were recorded for 52.3% of failed judgments, so this is a partial breakdown. Each task received three judgments; the 59.8% rate summarizes expert judgments, not a task-level certification of the full collection.

Test-runner startup failures

Thank you for identifying the inconsistency between Appendices A.2 and A.5. A.5 is inaccurately worded: it should refer to deleting or modifying original unit tests, which causes rejection. Under A.2, environment damage that prevents the runner from starting can be retained if the original tests remain intact and rebuilding reproduces the failure. The original tests verify the repair. We will correct A.5 to make this distinction explicit. The collection contains 761 such startup-failure tasks (46.0% of 1,655). The audit includes 66 of them, with a solvability pass rate of 53.5% across 198 expert judgments.

Usability and future refinement

Direct use of the full collection therefore requires care, which motivates our quality auditing and trajectory filtering. We will clarify this scope in the revision. Future work could use audit findings to iteratively refine and reassess tasks to improve collection-wide quality. This would require balancing review, refinement, and verification costs against usable-task yield; its impact remains to be evaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.