acceptodds
Under review as a conference paper at ICLR 2027

Software-in-the-Loop Reconstruction for Verified Terminal-Agent Training

Abstract

Existing training datasets for terminal agents concentrate on software engineering, leaving scientific and other specialized domains comparatively underrepresented. Constructing tasks in these domains requires both reference solutions and verifiers that distinguish correct behavior from superficially valid outputs. Developing these components separately for each task limits the expansion of training data. We introduce software-in-the-loop reconstruction, which addresses both requirements by reusing executable software workflows. Each workflow serves as a reference solution and generates input-output examples under different configurations. An agent receives selected examples and reconstructs an editable program, whose outputs are compared with the workflow's outputs on hidden inputs. The same workflow therefore supports multiple tasks and supplies their verification targets, reducing the need to author solutions and expected outputs separately for each task. A hierarchical verifier combines domain-specific semantic comparisons with structural and predefined anti-shortcut checks, while public feedback supports iterative repair. We instantiate this method as SWR3000, comprising 3,000 tasks derived from 500 workflows across 46 software families and six domains. Across three attempts per task, Qwen3.8-Max solves 838 tasks and generates 1,422 verified trajectories, which we oversample to obtain 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean performance among four matched-token corpus controls on all four reported evaluations. These results support workflow reconstruction as a means of expanding terminal-agent training across domains by reusing executable reference solutions and their outputs for verification. Code and evaluation data are available at https://anonymous.4open.science/r/ICLR2027-Anonymous-Code-EE00.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Tasks, Task Synthesis, Terminal Agents, Software Reconstruction, Synthetic Training Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_szVLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles the problem that existing training datasets for terminal agents are concentrated on software engineering with limited coverage of scientific and other domains. The authors introduce software-in-the-loop reconstruction, which reuses executable software workflows to generate training tasks and automatically verify solutions. Agents reconstruct the behavior of existing programs from public input-output examples, while the solutions are evaluated on hidden inputs. The authors construct SWR3000, a dataset of 3000 tasks from 500 workflows across 6 domains and introduce a hierarchical semantic verifier. Qwen3.8-Max solves 838 tasks and generates 1422 verified trajectories. Finetuning Qwen3.8-27B on these trajectories improves performance across 4 benchmarks, including Terminal-Bench 2 (from 47.94% to 53.56%), outperforming 4 other training methods when using the same amount of training data.

Critical strengths
  • Important problem to tackle and good idea of reusing workflows -- the paper addressed the lack of training data for terminal agents outside software engineering domain, and reuising existing software workflows to generate tasks and verifier is a promising approach.
  • Extensive benchmark and evaluation -- the paper constructs 3000 tasks across 6 domains. Finetuning improves performance across 4 benchmarks, Terminal-Bench 2, Terminal-Bench 4, Long-Horizon Terminal Bench, and SWR100, introduced by the paper. The authors also evaluate the verifier through mutation tests and a human audit, reporting 98% agreement with human judges
Critical weaknesses
  • Presentation and methodology -- the paper is written and presented in a way that is hard to follow, it takes substantial effort to understand how the benchmark actually works. For example, Figure 2 is overloaded with information, and Algorithm 1 introduced many abstract operations. The main paper does not explain how the original 500 workflows were collected and selected, or provide a concrete example of a generated task. Moreover, the paper lacks clear ablations showing how individual components (e.g., semantic verification, anti-shortcut checks) contribute to the final results to understand how important each part is.

  • Limited evaluation and training data diversity -- the 3000 tasks are generated only from 500 workflows with 6 variants per each workflow. The finetuning set contains 1422 successful trajectories from 838 solved tasks which are repeatedly samples, and the paper doesn't tell about the benefit of adding more unique training examples from simply repeating existing trajectories. Also, finetuning is evaluated only on one model, and the paper doesn't establish generalization to unseen workflows.

Questions for authors
  • Q1\textbf{Q1}. How does performance change with training only on unique successful trajectories without repeating examples?
  • Q2\textbf{Q2}. How were the original 500 workflows collected and selected? What criteria were used? What is their source?
  • Q3\textbf{Q3}. Is it possible to evaluate models other than Qwen3.8-27B and see if the improvements hold for other models?
  • Q4\textbf{Q4}. Why did you choose exactly 6 task variants per workflow? Did you evaluate whether increasing the number of variants improves performance?
8 comments

We are grateful to Reviewer szVL for the careful evaluation of our work and the insightful suggestions. We address each concern in detail below.

W1: We sincerely thank the reviewer for the careful reading and constructive feedback. We agree that the task-generation process and the role of each component should be explained more clearly.

1. Workflow Collection and Task Generation.

We constructed 500 workflows from 46 software families across six scientific domains, using existing software tools, APIs, documentation, and executable examples. Two contributors spent approximately 336 person-hours on software integration and workflow construction. Each workflow uses the original software to generate reference outputs. Approximately 20% of candidate workflows were rejected by automated checks due to execution failures, unstable outputs, insufficient input sensitivity, or shortcut vulnerabilities.

We will add a concrete CMake example showing the complete process: workflow execution → public examples → agent reconstruction → hidden verification. We will also simplify Figure 2 and Algorithm 1 and move implementation details to the appendix.

2. Ablation of Verification Components.

We agree that the contributions of semantic verification and anti-shortcut checks should be examined separately. We therefore compare the full task-generation pipeline with each component removed and evaluate the resulting tasks and downstream training performance.

Table : Ablation of Task Validation Components.

SettingAccepted WorkflowsAccepted TasksInvalid Tasks (%)Shortcut Success (%)TB2 Pass@1 (%)
Full SWR5003,0000.00.053.56 ± 0.65
w/o Semantic Verification5583,3488.00.048.31 ± 1.12
w/o Anti-Shortcut Checks5423,2529.96.850.19 ± 1.72
Base (No SFT)————47.94 ± 1.30

We use four metrics to evaluate the contributions of these components:

  • Accepted Tasks: The number of tasks retained after applying the enabled validation checks.
  • Invalid Tasks (%): The percentage of accepted tasks found to be invalid in a separate task-quality audit, including incorrect reference behavior or tasks that can be solved through shortcuts.
  • Shortcut Success (%): The percentage of accepted tasks that can be solved by predefined shortcut methods, such as copying public outputs instead of reconstructing the software behavior.
  • TB2 Pass@1 (%): The downstream success rate of the SFT model trained on the accepted tasks, reported as mean ± standard deviation across three seeds.

These metrics measure both task quality and downstream training performance. Removing semantic verification leads to more invalid tasks and reduces TB2 performance from 53.56% to 48.31%. Removing anti-shortcut checks allows shortcut-based solutions to pass more tasks and reduces TB2 performance to 50.19%. This suggests that semantic verification helps ensure task correctness, while anti-shortcut checks help prevent tasks from being solved without learning the underlying software behavior. We will include these results and a clearer explanation of the task-generation process in the revised paper.

W2: We sincerely thank the reviewer for the careful reading and constructive feedback. We agree that data diversity, repeated sampling, and generalization are important for understanding the value of our approach. We address these concerns below.

1. Diversity Across Task Variants.

Although SWR3000 contains 500 workflows with six variants each, these variants are not simply repeated tasks. They share the same source workflow but differ in their input conditions, output behaviors, and the cases that agents must handle.

To measure these differences, we analyzed 100 workflows (600 tasks):

  • 100/100 workflows contain output fields that vary across their six variants.
  • 79/100 workflows contain categorical outcomes that appear in only one variant.
  • 90/100 workflows contain input parameters that are fixed in one variant but varied in another.

For example, OpenSSL variants involve different certificate and signature verification outcomes, while CMake variants test different behaviors involving project(), find_package(), and build configurations. These differences can require agents to handle different software behaviors and develop different reconstruction strategies.

Thus, while the six variants do not represent independent software mechanisms, they provide different reconstruction challenges rather than simply repeating the same task.

2. Unique Training Examples vs. Repeated Sampling.

We agree that the benefit of adding unique training examples should be separated from repeatedly sampling existing trajectories. Our SFT corpus contains 1,422 successful trajectories from 838 solved tasks, which are sampled to construct the training dataset.

To study this question, we compare different numbers of unique trajectories under a matched training-token budget. We also evaluate training on all 1,422 successful trajectories without repetition, directly addressing the reviewer's question.

Table: Effect of Training Data Diversity and Repeated Sampling on TB2.

SettingUnique TrajectoriesTraining TokensTB2 Pass@1 (%)
Base (No SFT)——47.94 ± 1.30
SFT-25025088.48M48.31 ± 1.12
SFT-50050088.48M49.06 ± 1.72
SFT-83883888.48M50.56 ± 1.12
Unique-Only (No Repetition)1,42241.92M48.69 ± 0.65
Full SWR1,42288.48M53.56 ± 0.65

The results provide three findings.

First, increasing unique trajectories from 250 to 838 improves TB2 performance from 48.31% to 50.56%, suggesting that adding diverse training examples is more useful than repeatedly sampling a smaller set.

Second, using all 1,422 successful trajectories further improves performance to 53.56%, suggesting that multiple successful trajectories from the same tasks also provide useful training information.

Third, training on all 1,422 trajectories without repetition achieves 48.69%, compared with 53.56% when the trajectories are repeatedly sampled. This suggests that additional training on existing trajectories can provide further gains.

3. Generalization to Unseen Software Families.

To further examine generalization, we evaluated software families entirely absent from SFT training. Among the 46 software families in SWR3000, 20 had no successful Qwen3.8-Max trajectories across three attempts. Since only successful trajectories are used for SFT, none of these families contributed to training.

These 20 families contain 46 SWR100 evaluation tasks. On this subset, reconstruction SFT improves Pass@1 from 1.45% to 4.35% (+2.90 percentage points), with improvements observed in three scientific domains. Although the absolute success rates remain low, these results provide initial evidence that learned skills can transfer to unseen software families.

We also observe improvements on three external benchmarks with no training-source overlap: TB2 (47.94% → 53.56%), TB4 (0.51% → 3.54%), and LHTB (20.67% → 27.67%).

More broadly, our key insight is that software reconstruction provides a scalable way to generate diverse, verified training tasks. By reusing existing software workflows as reference solutions and generating expected outputs through execution, we reduce the need to manually develop solutions and verification targets for individual tasks. The large number of available scientific software tools also provides a natural way to expand this approach to many more workflows and domains.

Q1: We sincerely thank the reviewer for this important question. To understand the effect of repeated sampling, we conducted an additional experiment using all 1,422 unique successful trajectories without repetition, and compared it with our original SFT setting.

Table: Effect of Repeated Sampling on TB2.

SettingUnique TrajectoriesTraining TokensTB2 Pass@1 (%)
Base (No SFT)——47.94 ± 1.30
Unique-Only (No Repetition)1,42241.92M48.69 ± 0.65
Full SWR (With Repetition)1,42288.48M53.56 ± 0.65

One possible explanation is that reconstruction trajectories contain long sequences of related actions, including software inspection, implementation, testing, error analysis, and code revision. These behaviors are difficult to learn because successful task completion requires the model to connect many decisions across a long interaction. Repeated sampling increases the weight of these successful trajectories during training, giving the model more opportunities to learn useful patterns of tool use, error recovery, and iterative improvement. However, the repeated-sampling setting also uses more training tokens. Therefore, the observed improvement may reflect both greater exposure to successful trajectories and a larger training budget, rather than repetition alone.

Reviewer_ivTcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the cost of building training environments for terminal agents in scientific and specialized domains. The authors propose software-in-the-loop reconstruction that runs an existing executable workflow on typed input configurations, and the runs are split into public input-output observations and hidden evaluation cases. An agent must write an editable program that reproduces the workflow's behavior on the hidden inputs. Acceptance is decided by a hierarchical verifier that combines field-wise semantic comparison, structural checks, and anti-shortcut checks. The execution yields SWR3000, comprising 3,000 tasks, across 500 workflows, 6 variants from 46 families.

Critical strengths

Software as a Source of Supervision: The core contribution of this work is to use existing scientific software as a source of supervision. An AI agent observes a few examples of how a program behaves, attempts to reconstruct that behavior by writing its own executable program, and is tested on examples it has never seen. Successful agent interactions become training data for improving future terminal agents.

Robust Semantic Verification: The paper introduces a hierarchical semantic verification framework that checks whether reconstructed programs produce correct outputs instead of only checking their structure or exact output matches. The verifier uses domain-specific comparisons, structural checks, and anti-shortcut checks to ensure that programs work correctly on unseen inputs. This makes the verification process suitable for different scientific tasks.

Evaluation: The authors provide comprehensive experiments across multiple scientific domains, reporting three-seed results and matched-token comparisons against four alternative training corpora, demonstrating improvements in downstream terminal-agent performance

Critical weaknesses

Workflow Setup Cost: Each new workflow requires an interface, input settings, and checks for correct outputs. The authors did not report the time or human effort needed for this process. So more explanation required to support the scalability claim.

Task Diversity: The 3,000 tasks come from 500 workflows, with six versions of each. These versions share the same underlying program, so the tasks are not fully independent. The variance are mostly in the input-output not in the overall task behaviours.

Limited Generalization: SWR100 tests held-out tasks but does not evaluate completely new workflows. Other benchmarks measure general terminal-agent skills rather than real scientific problem-solving ability.

Questions for authors

[Q1] How much manual effort was required to wrap the 500 workflows and develop the domain-specific semantic verification rules? How many candidate workflows failed the release checks before the final dataset was constructed?

[Q2] Do the reconstruction-SFT improvements persist on out-of-distribution (OOD) tasks when all tasks from selected workflows or entire software families are excluded from training?

7 comments

We sincerely appreciate Reviewer ivTc’s thoughtful comments and the time devoted to evaluating our work. We provide our detailed responses below.

W1: We sincerely thank the reviewer for this important question. We agree that understanding the human effort required for task construction is essential to evaluating scalability. We will clarify these details in Appendix A.

Building SWR3000 involved two core contributors working 8 hours per day over 21 working days (336 person-hours), divided into two stages:

1. Software Family Onboarding (15 days for 46 families). Each software family requires a one-time integration into our common reconstruct.py interface, including a reusable adapter and reference oracle. We build upon mature, well-documented software rather than developing implementations from scratch. For example, KiCad provides demonstration projects [1], OpenFOAM provides executable tutorial cases [2], and MuJoCo Menagerie provides ready-to-use robot models [3]. These existing resources substantially reduce development effort. Once integrated, each family can support multiple workflows without repeating this work.

2. Workflow Construction (6 days for 500 workflows). After onboarding a software family, constructing a workflow primarily requires defining its input parameters and valid ranges (median: 5 parameters). Subsequent steps, including scenario generation, reference execution, verifier construction, and validity checks, are automated. Most execution time is computational rather than human effort. Individual tasks require no additional manual authoring.

Table: Human effort for SWR3000 construction.

StageHuman WorkTotal Effort
Software family (46)Reusable adapters and reference oracles15 days
Workflow (500)Define input parameters and ranges6 days
Task (3,000)Automatically generated0
Total2 contributors, 8 hours/day336 person-hours

Scalability. The key insight is that a single scientific software package can support thousands of executable workflows, each generating numerous tasks from different real-world inputs obtained from public datasets, online resources, or experiments. For example, OpenFOAM can simulate diverse fluid dynamics problems by varying geometries, material properties, and boundary conditions, producing distinct tasks with verifiable results. Crucially, by framing tasks as the reconstruction of software behavior and its outputs, we can directly leverage the original software to generate reference results and verify reconstructions, substantially reducing the effort of manually developing reference solutions and verifiers. Thus, human effort is concentrated on reusable software integration and workflow construction rather than individual tasks. The six variants per workflow in SWR3000 represent only a small fraction of this much larger task space, highlighting the potential for substantial scaling without proportional increases in human effort.

References

[1] KiCad Documentation Contributors. Getting Started in KiCad, Version 9.0. https://docs.kicad.org/9.0/en/getting_started_in_kicad/getting_started_in_kicad.html

[2] OpenFOAM. OpenFOAM Tutorial Guide. https://www.openfoam.com/documentation/tutorial-guide

[3] Google DeepMind. MuJoCo Menagerie: A Collection of High-Quality Models for the MuJoCo Physics Engine. https://github.com/google-deepmind/mujoco_menagerie

W2: We sincerely thank the reviewer for this important observation. We agree that the 3,000 tasks should not be treated as 3,000 independent software programs, as every six variants share an underlying workflow. SWR3000 comprises 500 distinct workflows across 46 software families and six scientific domains. We will make this distinction explicit in the revised paper.

However, we would like to clarify that variants of the same workflow can require substantially different software behaviors, rather than merely different input–output values. Although they share the underlying software, each variant uses different input configurations and test scenarios, potentially exercising different execution paths, validation rules, and error-handling behaviors. We illustrate this with two examples from our released tasks:

  • OpenSSL (openssl_store_loader_policy_chain): Variant __01 tests certificate chains that are accepted, whereas __02 requires distinguishing different certificate rejection reasons. Variants __05 and __06 further examine certificate-chain ordering and its interaction with extended key usage (EKU) constraints. A reconstruction that always accepts certificates can pass __01 but fails the other variants, demonstrating that these tasks require different decision logic.
  • CMake (cmake_object_plugin_export_graph): Variant __01 tests configuration errors involving target_compile_definitions and install, while __05 introduces failures during project() initialization. In contrast, __06 evaluates downstream find_package behavior, including missing components and incompatible package versions. These variants exercise different stages of CMake's execution rather than simply changing input values.

We further conducted a systematic analysis of 100 workflows (600 tasks), randomly sampled from the 3,000 tasks in SWR3000:

  • 100/100 workflows: All 100 workflows have output fields that remain unchanged across the test cases of one variant but change across those of another (median: 11 out of 26 output fields).
  • 79/100 workflows: In 79 workflows, at least one variant produces a unique outcome, such as an error code or status, that does not appear in the other five variants.
  • 90/100 workflows: In 90 workflows, at least one input parameter remains fixed in one variant but takes multiple values in another.

These results indicate that task diversity extends beyond numerical input–output variation to differences in the software behaviors being tested. While sibling variants are not fully independent, they can require different reconstruction logic and expose distinct failure modes.

Importantly, our main downstream improvements are evaluated on TB2, TB4, and LHTB, which are independent of SWR3000. Thus, the dependence among training variants does not artificially increase the number of independent downstream evaluation cases.

We sincerely appreciate the reviewer's thoughtful feedback, which helps us present a more rigorous and transparent analysis of task diversity.

W3: We sincerely thank the reviewer for these thoughtful comments. We agree that it is important to examine whether our method works on unseen software and to clarify its role in scientific research.

Generalization to Unseen Software Families. As discussed in Section 4.1 and Appendix A, SWR100 mainly evaluates new tasks rather than entirely new software workflows. To further study generalization, we analyze performance on software families that were not included in SFT training.

Among the 46 software families in SWR3000, 20 families had no successful Qwen3.8-Max trajectories across three attempts. Since we only use successful trajectories for SFT, these families provide no training data. They cover all six scientific domains and contain 46 tasks in SWR100. We compare the Base and SFT models on these tasks using three seeds (42, 43, 44).

Table: Generalization to Unseen Software Families (Pass@1, mean ± std, %).

Scientific DomainFamilies (All / Unseen)SWR3000 TasksSWR100 TasksUnseen Software Families (Task Counts)Unseen Eval. TasksBaseSFTGain (pp)
Software & Formal Systems11 / 366019CMake (3), OpenSSL (3), Network PCAP (1)74.76 ± 8.2514.29 ± 0.00+9.52
Physical & Engineering Simulation9 / 158820OpenFOAM (2)216.67 ± 28.8733.33 ± 28.87+16.67
Spatial, Visual & 3D8 / 652819Blender (4), KiCad (3), GDAL (3), Map rendering (3), LiDAR (2), OpenSCAD (1)160.00 ± 0.002.08 ± 3.61+2.08
Life & Molecular Sciences7 / 446815Molecular chemistry (3), Cell microscopy (3), Structural biology (2), Medical imaging (1)90.00 ± 0.000.00 ± 0.000.00
Data & Document Workflows6 / 339614Font shaping (3), DuckDB (2), SQLite (1)60.00 ± 0.000.00 ± 0.000.00
Earth & Space Sciences5 / 336013Well-log (3), Climate (2), Seismology (1)60.00 ± 0.000.00 ± 0.000.00
Overall46 / 203,00010020 unseen software families461.45 ± 1.264.35 ± 2.17+2.90

As shown in Table, SFT improves the overall success rate from 1.45% to 4.35% (+2.90 percentage points) on software families that provided no SFT training data. Although the success rates remain low, these results suggest that the model can transfer some of the skills learned from other software families to new ones. These findings also support our results on TB2, TB4, and LHTB, which use different tasks and software from our training data.

Scientific Problem-Solving. We fully agree with the reviewer that being able to use scientific software does not mean an AI agent can solve complete scientific research problems. Our current work focuses on helping agents better understand, use, and reconstruct scientific software through terminal interactions. We believe this is an important step toward building AI agents that can assist with scientific research.

However, solving real scientific problems requires more than using software correctly. Agents also need strong knowledge of specific scientific fields, the ability to reason about scientific questions, and the ability to plan and complete much longer tasks. These challenges are beyond the main focus of our current work.

In future work, we plan to extend our method to longer and more complex scientific tasks, where agents need to use domain knowledge, work with multiple scientific tools, design experiments, and analyze results. We will make this distinction clearer in the revised paper.

Reviewer_TUAtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes "software-in-the-loop reconstruction" for building terminal-agent training tasks outside software engineering. An executable workflow serves as the reference solution: it is run on public and hidden input scenarios, and the agent sees only the public input–output pairs. The agent must write a reconstruct.py whose outputs on hidden scenarios match the workflow's under a hierarchical verifier that combines structural checks, field-wise semantic comparison with numeric tolerances, and anti-shortcut gates. Release gates require determinism, input sensitivity, an oracle pass, and the failure of three public-output replay controls. The resulting SWR3000 contains 3,000 tasks (500 workflows × 6 variants, 46 software families, 6 domains). Verifier reliability is assessed with large mutation stress tests and a post-freeze blinded human audit (2% false-accept and 2% false-reject on 200 cases). Qwen3.8-Max solves 22.8% of tasks at pass@1 and 27.9% at pass@3, producing 1,422 verified trajectories. These are oversampled to 3,000 entries for SFT of Qwen3.8-27B, which improves Terminal-Bench 2 from 47.94% to 53.56% (three seeds), with gains on TB4, LHTB, and SWR100. Under a matched 88.48M-token budget it outperforms four public terminal-agent corpora.

Critical strengths
  1. An appealing, scalable recipe in which one executable artifact supplies both the reference solution and the verification targets, backed by an unusually thorough verifier audit. Reusing workflows across scenario variants removes the need to hand-write a solution and a test per task. The verifier is stress-tested on 28,808 invalid mutations and 7,065 near-boundary valid controls, and then audited after freezing on a disjoint 100-task set with blinded double annotation (98% agreement with adjudicated labels). This level of verifier validation is rare in agent-data papers.

  2. Careful SFT evaluation. Results use three independently trained seeds, per-seed numbers are reported (SFT-3000 beats Base on every seed for all four evaluations), and the comparison against four public terminal-agent corpora holds the assistant-loss-token budget fixed. The paper is also explicit that SWR100 is in-domain and that oversampled entries are not independent trajectories.

Critical weaknesses

The causal attribution of the downstream gains is confounded, and there may be hidden-feedback leakage. The reconstruction trajectories come from Qwen3.8-Max, the same model family as the student, collected in Terminus-2, the standard Terminal-Bench harness. The four control corpora come from other teachers and scaffolds, and all of them yield little or no gain over Base (e.g., Nemotron-Terminal 47.19 vs. Base 47.94 on TB2). So the comparison cannot separate the value of reconstruction tasks from same-family distillation and harness/format alignment. A control using the same teacher and harness on non-reconstruction tasks (or on unverified reconstruction trajectories) is missing.

The tasks may not reconstruct real domain software behavior, which undermines the paper's main premise. The paper presents tasks as reconstructing "real software-in-the-loop" workflows across scientific domains. In the anonymous code, however, several SWR100 oracles are thin wrappers that pass parameter strings through arbitrary hash functions into minimal "smoke" calls (I inspected the code read-only and did not execute it). For example, offcode6__robotics_urdf_mass_inertia_repair__05/tests/private/offcode6_oracle.py contains no URDF mass/inertia logic: it builds a one-hinge MuJoCo "smoke" model whose gain, margin, and target are offset + scale·(1 + (Σ ord(c) mod 97)/97) of the parameter's string, and the declared axes initial_pose and joint_limit are not used by that function.

Questions for authors

(Clarification) How was SWR100 selected? Qwen3.8-Max gets 22.8% pass@1 on SWR3000 with a 100-turn/1-hour budget but only 3% on SWR100 with a 500-turn/3-hour budget.

3 comments

We sincerely thank Reviewer TUAt for the careful reading and constructive feedback. We address the comments as follows.

W1: We sincerely thank the reviewer for this important concern. We clarify our motivation and provide additional controlled comparisons.

1.We intentionally use Qwen3.8-Max as the teacher and Qwen3.8-27B as the student because models within the same family may share similar reasoning patterns, tool-use behaviors, and output formats. This similarity may reduce the distribution shift between training trajectories and the student's existing behavior, making it easier for the student to learn from successful trajectories. Our goal is to investigate whether high-quality trajectories generated by a stronger model can effectively improve the performance of a smaller model within the same family. We acknowledge that the benefits of same-family training have not been independently isolated.

2.To address the reviewer's concern, we compare three training sources using the same teacher (Qwen3.8-Max-0902), harness (Terminus-2), and rollout settings (100 turns and 1 hour per task). All experiments use full-parameter SFT of Qwen3.8-27B, with identical training configurations (global batch size 128, learning rate 1×10⁻⁵ to 1×10⁻⁶, and 5% warmup), 88.48M matched assistant-loss tokens, and three independent seeds (42, 43, 44):

  • SWR3000 (Ours): Successful reconstruction trajectories.
  • TMax-15K: New rollouts from publicly available terminal tasks.
  • Unverified SWR3000: Failed reconstruction trajectories.

We evaluate three independent training seeds (42, 43, 44) on two benchmarks.

Table: Controlled comparisons under identical teacher, harness, rollout settings, and matched training-token budgets (all results in %).

Training DataSeedTB2TB4
Base4247.190.00
4349.441.52
4447.190.00
BaseMean ± Std47.94 ± 1.300.51 ± 0.87
SWR3000 (Ours)4253.933.03
4352.814.55
4453.933.03
SWR3000 (Ours)Mean ± Std53.56 ± 0.653.54 ± 0.87
TMax-15K4247.193.03
4349.443.03
4450.560.00
TMax-15KMean ± Std49.06 ± 1.722.02 ± 1.75
Failed SWR30004247.190.00
4346.071.52
4447.191.52
Failed SWR3000Mean ± Std46.82 ± 0.651.01 ± 0.87

Under identical teacher, harness, rollout settings, and matched training-token budgets, SWR3000 achieves the best performance on both TB2 (53.56%) and TB4 (3.54%), outperforming TMax-15K by 4.50% and 1.52%, respectively. Failed SWR3000 performs substantially worse (46.82% and 1.01%), highlighting the importance of verified training trajectories. These results demonstrate the practical value of our generated data without attributing the gains solely to reconstruction tasks.

3.Clarifying the Objectives of Tables 4 and 5. Table 5 is intended to compare our generated data with existing publicly available terminal-agent training datasets, rather than isolate the contribution of reconstruction tasks. Our goal is to evaluate whether our approach can produce competitive training data. Under the same 88.48M assistant-loss-token budget, our data achieves the highest mean performance across all four benchmarks. Table 4 provides complementary evidence for the effectiveness of reconstruction trajectories as training data. Using only our successful reconstruction trajectories, we observe non-decreasing mean performance across all benchmarks as training entries increase from 750 to 1,500 and 3,000. This demonstrates that reconstruction trajectories provide useful training supervision, with additional training exposure leading to stronger downstream performance under our sampling procedure.

4.Hidden-Feedback Leakage. During trajectory collection, agents can only access public input–output examples and receive feedback from public tests. Hidden inputs, reference outputs, and private verifier results are never revealed to the agents. Only after an agent completes its task do we run the private verifier to determine whether the final reconstruction is correct. We then retain successful trajectories for SFT. Thus, hidden verification is used solely for selecting training data, not for guiding agents during rollouts. We will make this distinction clearer in the revised paper.

W2: We sincerely thank the reviewer for the careful examination of our implementation and for raising this important concern. We appreciate the opportunity to clarify the distinction between our lightweight anonymous release and the complete executable benchmark. As the reviewer correctly observed, the previously released offcode6_oracle.py contained a simplified MuJoCo smoke-test implementation rather than the intended software-based computation. This distinction is closely related to the substantial external dependencies required by scientific software workflows. A fully executable SWR100 package occupies 426 MB on disk (2,819 files), including 374 MB of reference artifacts. Although many artifacts are shared across tasks, the repository still requires 404 distinct blobs totaling 244 MB. For example, the URDF workflow depends on the MuJoCo Menagerie UR5e model, whose XML configuration and 20 associated mesh assets occupy approximately 33 MB. These dependencies require appropriate installation and asset staging, making a fully self-contained lightweight distribution challenging. To keep the anonymous release manageable, we initially omitted these larger external assets and provided a simplified smoke-test implementation. We sincerely apologize that this distinction was not sufficiently clear in our initial release.

We have now addressed these implementation issues and updated our anonymous repository accordingly. Specifically, we corrected the affected oracle implementations to invoke the corresponding scientific software rather than relying on simplified name-derived numerical transformations. We further conducted a systematic AST-based audit of 580 source files, confirming that no such arithmetic patterns remain. To improve reproducibility, we documented 815 reference artifacts, including their upstream sources, file sizes, and SHA-256 checksums, and introduced automated checks to detect missing or inconsistent dependencies. We also strengthened the verification procedures through strict categorical comparisons, numerical tolerance checks, and comprehensive structured-output validation. The updated repository includes reproducible validation scripts, enabling to independently inspect the implementations and verify their correctness.

Furthermore, we plan to publicly release all 3,000 SWR tasks following the double-blind review process, including task specifications, executable environments, reference workflows, input scenarios, and verifiers. We believe this release will provide valuable resources for independent reproducibility, systematic analysis of agent failures, and further research on scientific software reconstruction and scalable agent training.

We sincerely appreciate the reviewer's thorough investigation and constructive feedback.

Q1: We sincerely thank the reviewer for this important question. We would like to clarify that SWR100 was deliberately constructed as a challenging subset of SWR3000, rather than randomly sampled from the full dataset.

Specifically, among the 3,000 SWR3000 tasks, 2,162 were not solved by Qwen3.8-Max in any of its three attempts. We selected SWR100 exclusively from these consistently unsuccessful tasks, sampling one task per workflow while maintaining coverage across source pools. The resulting benchmark contains 100 tasks from 100 distinct workflows across 46 software families, with no verified successful Qwen3.8-Max trajectories from the original evaluation.

This selection procedure explains the substantial difference between 22.8% and 3%. The 22.8% measures performance on the full SWR3000 distribution, which includes both easier and harder tasks. In contrast, SWR100 is explicitly conditioned on three consecutive failures, making it substantially more challenging for Qwen3.8-Max. Although we increased the evaluation budget from 100 turns / 1 hour to 500 turns / 3 hours, the model solved only 3 of the 100 selected tasks. Its median interaction length of 356.5 turns further shows that the larger budget was actively utilized, but additional interaction did not necessarily lead to successful reconstruction. We will clarify the selection criteria, sampling procedure, and interpretation of these results in Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.