- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The author presents CLI-Gym, a pipeline to synthesize executable environments for Terminal Bench from GitHub. The pipeline produces 1,655 TB-like environments from 29 Python repositories, though environment inversion (letting LLMs freely break a known-good environment such that a random subset of unit tests fail and the break direction differs from previous breaks). Rejection sampling from Deekseep-V3.2 yields 291 trajectories. Fine-tuning Qwen3-32B and Qwen3-235B-A22B-Instruct with the teacher trajectories and open source SWE trajectories yields competitive Terminal Bench 1.0/2.0 performance among open weights models.
Critical strengths
- Data efficiency: with only SFT 291 teacher trajectories from Deepseek V3.2, the model's Terminal Bench 1.0 performance raised significantly, across 2 model sizes and with and without SWE trajectory data training before the teacher trajectory SFT.
Critical weaknesses
- Lack of fair comparison with previous TB-focused data curation work: The authors only compared the benchmark scores of their models against other open or closed weight models, that may use different agentic harnesses and inference budgets. There is no controlled comparison (training on same backbone model and same token budget, or comparison on yield rate and quality) with other open TB-focused datasets such as TerminalTraj or TerminalGen. The authors therefore frame their data synthesis approach as complementary with other works rather than superior.
Questions for authors
- Can you specify details about the source and composition of the 48k SWE initialization trajectories used in Sec 4. ?
- Do the LiberCoder models hack the reward or use shortcuts in evaluation, as found in teacher trajectories?
- What is the performance of the teacher model Deepseek V3.2 on Terminal Bench in the evaluation environment and harness of this work?
5 comments
Thank you for the detailed response.
A1. The TermiGen comparison from the same Qwen3-32B start (10.3 -> 32.4 vs 10.4 -> 27.5) is the most useful piece of context here. I note, however, that the margin over TerminalTraj-32B on TB1.0 (38.9 vs 35.3) only appears with the 48K-trajectory SWE initialization, and that on TB2.0 the ordering is reversed (19.5 vs 22.0). The data-efficiency framing around "291 trajectories" should therefore be stated as conditional on the 48K first stage. I would also keep in mind that the rows you compare against are leaderboard-reported numbers from different scaffolds, teachers and base models (TerminalTraj-32B starts from Qwen2.5-Coder-32B with a Qwen3-Coder-480B teacher), while each of your pass@1 scores is a single greedy run (Appendix B.3); greedy decoding does not make agent runs deterministic, since environment responses and small decoding differences vary between runs, so differences of a few points across papers are within what re-evaluation typically moves. With the comparison conditions made explicit as you propose, I still consider the level of comparison acceptable for this submission since the authors frame their data synthesis approach as complementary.
A2. The response names the task sources (SWE-smith, SWE-Gym, R2E-Gym) but not the model that produced the 48K trajectories. Please state it directly: were these trajectories released by those projects, or rolled out by you, and with which model? Since LiberCoder is listed under "Open-source Data Pipeline" in Table 4 and the headline 46.1/31.0 numbers depend on this stage, a proprietary teacher here would need to be disclosed.
A3 is satisfactory. On A4, please confirm that the 42.7% for DeepSeek-V3.2 was reproduced by you in the same evaluation harness and environment as LiberCoder, rather than taken from a reported figure, since environment differences are known to shift Terminal-Bench scores.
Thank you for the follow-up and for considering the comparisons acceptable with the conditions clarified. We address the remaining points below.
Q1. How should the comparison conditions and the data-efficiency claims be clarified?
A1. We agree that the comparison with TermiGen using the same Qwen3-32B backbone provides the most relevant context. Our CLI-Gym-only result in that comparison uses 291 trajectories without SWE initialization.
For the full LiberCoder results, we will explicitly state that training consists of 48K SWE initialization trajectories followed by 291 CLI-Gym trajectories, and qualify the data-efficiency discussion accordingly. In particular, the reported TB1.0 advantage over TerminalTraj-32B uses this two-stage setting, while TerminalTraj-32B scores higher on TB2.0. We also acknowledge the differences in scaffolds, teachers, base models, and evaluation protocols, as well as run-to-run variability even with greedy decoding. We will make these conditions explicit and interpret the cross-paper results as contextual comparisons supporting the complementary positioning of our approach.
Q2. Were the 48K SWE trajectories publicly released by the source projects or generated by the authors, and which teacher models produced them?
A2. Thank you for identifying this omission. We assembled the 48K initialization set from publicly released trajectory datasets, including the following releases. We reused trajectories generated and released by these projects.
| Released dataset | Reported trajectory-generation models |
|---|---|
| SWE-smith-trajectories | Claude 3.7 Sonnet |
| R2EGym-SFT-Trajectories | Claude 3.5 Sonnet v2, as described in the R2E-Gym paper |
| OpenHands-SFT-Trajectories | GPT-4o and Claude 3.5 Sonnet, as described in the SWE-Gym paper |
Thus, the SWE initialization stage does include publicly released trajectories generated by proprietary teacher models. In Table 4, “Open-source Data Pipeline” was intended to describe the availability of the training data. We recognize that this wording leaves the teacher-model provenance unclear. We will clarify the terminology in the table caption and explicitly disclose these data sources and teacher models in Section 4.1.
Q4. Was the 42.7% DeepSeek-V3.2 result reproduced by the authors under the same evaluation harness and environment as LiberCoder?
A4. We confirm that we evaluated DeepSeek-V3.2 ourselves and obtained 42.7% pass@1 on Terminal-Bench 1.0, using the same OpenHands evaluation harness and benchmark environment as LiberCoder. This is a result from our own evaluation. We will state its provenance explicitly in the revised manuscript.
Thank you for thoughtfully reviewing our work and recognizing the data efficiency of the generated supervision. We appreciate your specific questions about fair comparisons, the SWE initialization data, and evaluation behavior, which have helped us identify where the experimental setup and its limitations need clearer explanation.
Q1. How should readers interpret the comparisons with TerminalTraj and TermiGen?
A1. Thank you for raising comparability. TermiGen-Qwen3-32B provides the closest published backbone comparison. The following TB1.0 pass@1 results use the same Qwen3-32B backbone; our CLI-Gym-only setting excludes SWE initialization.
TermiGen uses BashAgent and averages three runs at temperature 0.6, while we use OpenHands and greedy decoding. The similar starting scores and shared backbone provide context for the observed gains, although scaffold, sampling, and training differences limit causal attribution. TermiGen, Sections 4.1-4.2
TerminalTraj-32B uses Qwen2.5-Coder-32B-Instruct, trajectories generated with Qwen3-Coder-480B, and the Terminus-2 scaffold. Our full LiberCoder-32B setting uses Qwen3-32B with SWE initialization followed by training on CLI-Gym trajectories.
This gives mixed cross-benchmark rankings and illustrates the distinct strengths of the resulting systems. TerminalTraj, Sections 2.3.2 and 4, Table 2
Our central empirical conclusion is that the filtered trajectories produced by CLI-Gym provide useful supervision across two student scales and both benchmark versions under a fixed OpenHands scaffold. The within-study ablations support that conclusion, while a matched training comparison would further establish the relative value of different CLI data sources. We will make these comparison conditions explicit and retain the complementary positioning of the construction methods.
Q2. What is the source and composition of the 48K SWE initialization trajectories?
A2. The 48K trajectories come from solving tasks drawn from SWE-smith, SWE-Gym, and R2E-Gym, all based on open-source repositories. We use them to initialize repository navigation, tool use, and code repair. These training tasks and their source repositories are disjoint from Terminal-Bench. We will name these sources in Section 4.1.
Q3. Did LiberCoder use shortcuts during Terminal-Bench evaluation?
A3. We used the standard Terminal-Bench 1.0 verifier, which was hidden from the agent during task execution. We inspected one complete evaluation run for each of LiberCoder-235B-A22B and LiberCoder-32B, covering 80 trajectories per model and 160 in total. We observed no apparent shortcut behavior in the inspected trajectories.
Q4. How does DeepSeek-V3.2 perform under the same evaluation setup?
A4. Under the same OpenHands evaluation setup as LiberCoder, DeepSeek-V3.2 achieves 42.7% pass@1 on Terminal-Bench 1.0.