- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents FACET, a pipeline for synthesizing terminal tasks in the Harbor format from 71K agent skills. The pipeline has three stages: (1) Skills that can serve a common user goal are grouped together, the authors use LLM judge to keep only sensible combinations. (2) Each skill group is then expanded into a detailed scenario. The pipeline writes a text plan for solving the task and a draft of the task requirements. (3) The Docker environment is built first and repaired until it runs. Working from this real environment, the pipeline writes the instruction, then the solution script, and finally the tests. A task is kept only if its tests fail before the solution runs and pass after it.
The resulting dataset contains 6,078 tasks with an average of 22.77 pytest items per task. Using DeepSeek-V4-Pro with Terminus-2 as the teacher, the authors collect around 1.2K successful trajectories and fine-tune Qwen3.5-4B/9B/27B, reporting gains of +6.75 to +8.24 points on Terminal-Bench 2.1. The paper also compares three artifact-generation orders and analyzes check-level failure patterns.
Critical strengths
-
The paper identifies a central failure mode in terminal-task synthesis: when instructions, environments, solutions, and verifiers are generated separately, they easily become mutually inconsistent. FACET builds and repairs the environment first and generates all downstream artifacts against the same realized container state.
-
Beyond the main results, the paper compares different artifact-generation orders (Sec. 3.5) and compares the full pipeline against a variant without scenario reconstruction and a reproduced TerminalWorld-style pipeline (App. E).
Critical weaknesses
-
The SFT gains cannot be attributed to the quality of FACET’s tasks. The main results only compare base models with FACET-fine-tuned models. There is no comparison against other data sources under the same setting, e.g., fine-tuning on the same number of trajectories (1.2K), collected with the same teacher and scaffold, from Tmax or other existing datasets. Without such a comparison, it is unclear whether FACET tasks are better than existing ones, or whether the gains simply come from distilling a strong teacher.
-
Task quality is questionable. The teacher (Deepseek V4 Pro) solves only 20.94% of the tasks (1,270/6,066, Appendix A.2), while the same model scores 73.03 on Terminal-Bench 2.1 (Table 2). Terminal-Bench 2.1 tasks are human-written and manually verified, so that success rate is reliable. The much lower success rate on FACET tasks raises the concern that many tasks are flawed rather than simply harder.
-
The correctness of the tests is not verified. The tests are written from the output of the reference solution, which is itself LLM-generated and may be wrong. The paper claims more tests per task than other datasets, but provides no evidence that these tests are correct, and it contains no human audit of task quality. This may also explain W2: if the tests check many details that the instruction does not clearly specify, even a strong agent will fail.
-
The source skills are collected from public repositories, but the paper does not check for overlap between the synthesized tasks and Terminal-Bench 2.1.
-
Terminal-task synthesis is an active area with many recent works, yet the related work section is only two short paragraphs.
Questions for authors
Please refer to the weaknesses part.