- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem that existing training datasets for terminal agents are concentrated on software engineering with limited coverage of scientific and other domains. The authors introduce software-in-the-loop reconstruction, which reuses executable software workflows to generate training tasks and automatically verify solutions. Agents reconstruct the behavior of existing programs from public input-output examples, while the solutions are evaluated on hidden inputs. The authors construct SWR3000, a dataset of 3000 tasks from 500 workflows across 6 domains and introduce a hierarchical semantic verifier. Qwen3.8-Max solves 838 tasks and generates 1422 verified trajectories. Finetuning Qwen3.8-27B on these trajectories improves performance across 4 benchmarks, including Terminal-Bench 2 (from 47.94% to 53.56%), outperforming 4 other training methods when using the same amount of training data.
Critical strengths
- Important problem to tackle and good idea of reusing workflows -- the paper addressed the lack of training data for terminal agents outside software engineering domain, and reuising existing software workflows to generate tasks and verifier is a promising approach.
- Extensive benchmark and evaluation -- the paper constructs 3000 tasks across 6 domains. Finetuning improves performance across 4 benchmarks, Terminal-Bench 2, Terminal-Bench 4, Long-Horizon Terminal Bench, and SWR100, introduced by the paper. The authors also evaluate the verifier through mutation tests and a human audit, reporting 98% agreement with human judges
Critical weaknesses
-
Presentation and methodology -- the paper is written and presented in a way that is hard to follow, it takes substantial effort to understand how the benchmark actually works. For example, Figure 2 is overloaded with information, and Algorithm 1 introduced many abstract operations. The main paper does not explain how the original 500 workflows were collected and selected, or provide a concrete example of a generated task. Moreover, the paper lacks clear ablations showing how individual components (e.g., semantic verification, anti-shortcut checks) contribute to the final results to understand how important each part is.
-
Limited evaluation and training data diversity -- the 3000 tasks are generated only from 500 workflows with 6 variants per each workflow. The finetuning set contains 1422 successful trajectories from 838 solved tasks which are repeatedly samples, and the paper doesn't tell about the benefit of adding more unique training examples from simply repeating existing trajectories. Also, finetuning is evaluated only on one model, and the paper doesn't establish generalization to unseen workflows.
Questions for authors
- . How does performance change with training only on unique successful trajectories without repeating examples?
- . How were the original 500 workflows collected and selected? What criteria were used? What is their source?
- . Is it possible to evaluate models other than Qwen3.8-27B and see if the improvements hold for other models?
- . Why did you choose exactly 6 task variants per workflow? Did you evaluate whether increasing the number of variants improves performance?
8 comments
W2: We sincerely thank the reviewer for the careful reading and constructive feedback. We agree that data diversity, repeated sampling, and generalization are important for understanding the value of our approach. We address these concerns below.
1. Diversity Across Task Variants.
Although SWR3000 contains 500 workflows with six variants each, these variants are not simply repeated tasks. They share the same source workflow but differ in their input conditions, output behaviors, and the cases that agents must handle.
To measure these differences, we analyzed 100 workflows (600 tasks):
- 100/100 workflows contain output fields that vary across their six variants.
- 79/100 workflows contain categorical outcomes that appear in only one variant.
- 90/100 workflows contain input parameters that are fixed in one variant but varied in another.
For example, OpenSSL variants involve different certificate and signature verification outcomes, while CMake variants test different behaviors involving project(), find_package(), and build configurations. These differences can require agents to handle different software behaviors and develop different reconstruction strategies.
Thus, while the six variants do not represent independent software mechanisms, they provide different reconstruction challenges rather than simply repeating the same task.
2. Unique Training Examples vs. Repeated Sampling.
We agree that the benefit of adding unique training examples should be separated from repeatedly sampling existing trajectories. Our SFT corpus contains 1,422 successful trajectories from 838 solved tasks, which are sampled to construct the training dataset.
To study this question, we compare different numbers of unique trajectories under a matched training-token budget. We also evaluate training on all 1,422 successful trajectories without repetition, directly addressing the reviewer's question.
Table: Effect of Training Data Diversity and Repeated Sampling on TB2.
| Setting | Unique Trajectories | Training Tokens | TB2 Pass@1 (%) |
|---|---|---|---|
| Base (No SFT) | — | — | 47.94 ± 1.30 |
| SFT-250 | 250 | 88.48M | 48.31 ± 1.12 |
| SFT-500 | 500 | 88.48M | 49.06 ± 1.72 |
| SFT-838 | 838 | 88.48M | 50.56 ± 1.12 |
| Unique-Only (No Repetition) | 1,422 | 41.92M | 48.69 ± 0.65 |
| Full SWR | 1,422 | 88.48M | 53.56 ± 0.65 |
The results provide three findings.
First, increasing unique trajectories from 250 to 838 improves TB2 performance from 48.31% to 50.56%, suggesting that adding diverse training examples is more useful than repeatedly sampling a smaller set.
Second, using all 1,422 successful trajectories further improves performance to 53.56%, suggesting that multiple successful trajectories from the same tasks also provide useful training information.
Third, training on all 1,422 trajectories without repetition achieves 48.69%, compared with 53.56% when the trajectories are repeatedly sampled. This suggests that additional training on existing trajectories can provide further gains.
3. Generalization to Unseen Software Families.
To further examine generalization, we evaluated software families entirely absent from SFT training. Among the 46 software families in SWR3000, 20 had no successful Qwen3.8-Max trajectories across three attempts. Since only successful trajectories are used for SFT, none of these families contributed to training.
These 20 families contain 46 SWR100 evaluation tasks. On this subset, reconstruction SFT improves Pass@1 from 1.45% to 4.35% (+2.90 percentage points), with improvements observed in three scientific domains. Although the absolute success rates remain low, these results provide initial evidence that learned skills can transfer to unseen software families.
We also observe improvements on three external benchmarks with no training-source overlap: TB2 (47.94% → 53.56%), TB4 (0.51% → 3.54%), and LHTB (20.67% → 27.67%).
More broadly, our key insight is that software reconstruction provides a scalable way to generate diverse, verified training tasks. By reusing existing software workflows as reference solutions and generating expected outputs through execution, we reduce the need to manually develop solutions and verification targets for individual tasks. The large number of available scientific software tools also provides a natural way to expand this approach to many more workflows and domains.
Q1: We sincerely thank the reviewer for this important question. To understand the effect of repeated sampling, we conducted an additional experiment using all 1,422 unique successful trajectories without repetition, and compared it with our original SFT setting.
Table: Effect of Repeated Sampling on TB2.
| Setting | Unique Trajectories | Training Tokens | TB2 Pass@1 (%) |
|---|---|---|---|
| Base (No SFT) | — | — | 47.94 ± 1.30 |
| Unique-Only (No Repetition) | 1,422 | 41.92M | 48.69 ± 0.65 |
| Full SWR (With Repetition) | 1,422 | 88.48M | 53.56 ± 0.65 |
One possible explanation is that reconstruction trajectories contain long sequences of related actions, including software inspection, implementation, testing, error analysis, and code revision. These behaviors are difficult to learn because successful task completion requires the model to connect many decisions across a long interaction. Repeated sampling increases the weight of these successful trajectories during training, giving the model more opportunities to learn useful patterns of tool use, error recovery, and iterative improvement. However, the repeated-sampling setting also uses more training tokens. Therefore, the observed improvement may reflect both greater exposure to successful trajectories and a larger training budget, rather than repetition alone.
We are grateful to Reviewer szVL for the careful evaluation of our work and the insightful suggestions. We address each concern in detail below.
W1: We sincerely thank the reviewer for the careful reading and constructive feedback. We agree that the task-generation process and the role of each component should be explained more clearly.
1. Workflow Collection and Task Generation.
We constructed 500 workflows from 46 software families across six scientific domains, using existing software tools, APIs, documentation, and executable examples. Two contributors spent approximately 336 person-hours on software integration and workflow construction. Each workflow uses the original software to generate reference outputs. Approximately 20% of candidate workflows were rejected by automated checks due to execution failures, unstable outputs, insufficient input sensitivity, or shortcut vulnerabilities.
We will add a concrete CMake example showing the complete process: workflow execution → public examples → agent reconstruction → hidden verification. We will also simplify Figure 2 and Algorithm 1 and move implementation details to the appendix.
2. Ablation of Verification Components.
We agree that the contributions of semantic verification and anti-shortcut checks should be examined separately. We therefore compare the full task-generation pipeline with each component removed and evaluate the resulting tasks and downstream training performance.
Table : Ablation of Task Validation Components.
We use four metrics to evaluate the contributions of these components:
These metrics measure both task quality and downstream training performance. Removing semantic verification leads to more invalid tasks and reduces TB2 performance from 53.56% to 48.31%. Removing anti-shortcut checks allows shortcut-based solutions to pass more tasks and reduces TB2 performance to 50.19%. This suggests that semantic verification helps ensure task correctness, while anti-shortcut checks help prevent tasks from being solved without learning the underlying software behavior. We will include these results and a clearer explanation of the task-generation process in the revised paper.