acceptodds
Under review as a conference paper at ICLR 2027

WorkBorne: Discovering Agentic Tasks from Real-World Artifact Transitions

33 views

Abstract

Developing terminal-based agents for long-horizon execution demands synthetic tasks that tightly couple complex instructions with authentic, interactive environments. However, current automated synthesis suffers from a fundamental asymmetry: instruction-oriented methods fabricate low-fidelity, synthetic environments, whereas resource-oriented methods treat real artifacts merely as static inputs, degrading task instructions into shallow, trivial operations. To address this limitation, we introduce WorkBorne, a synthesis framework that uses artifact transitions as the central primitive for modeling the full life-cycle of agentic tasks. In WorkBorne, each transition specifies who performs the task, which artifacts anchor the initial state and intended outcome, and the corresponding relationship between them. Specifically, WorkBorne first constructs coherent and diverse artifact corpora from iterative web retrieval, then discovers non-trivial transitions by identifying meaningful input-output artifact assemblies and the relations that connect them. Each discovered transition is finally instantiated into grounded agentic tasks. We construct 1,061 artifact corpora and 7,882 task environments, yielding 15,485 high-quality trajectories, with 53.6% exceeding 30 assistant turns. Experiments on Qwen3.5-27B and Qwen3.5-35B-A3B show that WorkBorne substantially improves over base models (i.e., improving Terminal-Bench 2.1 from 43.1% to 61.8% and ClawEval General score from 75.0 to 80.2) and achieves the strongest overall performance among other synthesis baselines (e.g., TMax ivison2026tmaxsimplerecipeterminal, RST li2026recursive, Terminal-Universe wu2026terminaluniverseturningagenttrajectories). Further analysis shows that transition is strongly associated with execution depth, while corpus breadth and transition depth provide complementary scaling gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agentic Task Synthesis, Long-Horizon Agents, Synthetic Data, Agent Training, Terminal Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cXXUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WorkBorne, a pipeline that synthesizes terminal-agent tasks by modeling each task as an "artifact transition" (a persona, real input artifacts, real output anchors, and the relation between them). Starting from 1,061 agent skills, it retrieves real artifacts from the web, discovers and validates transitions, and instantiates each into an instruction, a real-artifact environment, and rubrics. Kimi-K3 rollouts filtered by an Opus judge yield 15,485 trajectories. SFT of Qwen3.5-27B on these trajectories raises TB 2.1 from 43.1 to 61.8 and the ClawEval score from 75.0 to 80.2, outperforming prior synthesis methods. A matched comparison against instruction-first and input-only pipelines shows that transition modeling yields deeper execution and better downstream performance.

Critical strengths

1. Consistent controlled evidence for the central claim. Table 3 compares fabricated environments, input-only pipelines, and transition modeling. Environment size, execution depth, human-rated complexity, and TB 2.1 all rise in that order, and the full-scale 27B model reaches 61.8 on TB 2.1. Why it matters: this is my main reason for leaning toward acceptance. The paper's diagnosis, its fix, and the evidence line up.

2. Credible absolute numbers. The paper includes thorough contamination checks (Table 6), re-measures most baselines under the same protocol, and fully specifies training and prompts. Why it matters: this rules out leakage and mismatched evaluation as explanations for the gains.

Critical weaknesses

1. Evidence for generality is limited, and data quality is bounded by the teacher. All results use a single teacher (Kimi-K3), a single base family (Qwen3.5), and three benchmarks. Because only trajectories that Kimi-K3 completes are retained, the difficulty of the training data is capped by the teacher's ability, and the teacher's own scores are not reported. The contribution is a data-construction method, so its value depends on transferring beyond this specific setup and on teaching the student more than the teacher's existing behavior on these benchmarks.

Questions for authors

1. Teacher reference. Please report Kimi-K3's own TB 2.1 and ClawEval scores, so the student's results can be read relative to the teacher. Please also report how many tasks have no retained trajectory and whether they are counted among the 7,882.

2. Generality. Could you evaluate on additional agent benchmarks, such as ClawBench or SWE-bench-style tasks? If not, please narrow the claims about the method's generality.

3 comments
AuthorsOpenReview ↗

Response to Reviewer #cXXU (Part 1/2)

Hi Reviewer #cXXU, thank you for your thoughtful review and support of our work. Below, we address W1 and Q1-Q2 with Kimi-K3's reference scores, a full task-coverage audit, and new evaluations on ClawBench V2 and SWE-bench-Verified-Mini.

W1 & Q1. Teacher Reference and Task Coverage

"Please report Kimi-K3's own TB 2.1 and ClawEval scores, so the student's results can be read relative to the teacher. Please also report how many tasks have no retained trajectory and whether they are counted among the 7,882."

(1) Teacher reference. Table R1 reports Kimi-K3 alongside the base and fine-tuned student. As a 2.8T-parameter frontier model (~100× the student), Kimi-K3 remains stronger on every metric, as expected for SFT distillation.

Table R1. Student Performance Relative to the Kimi-K3 Teacher

ModelParamsTB LiteTB 2.1ClawEval Pass@3ClawEval Pass^3ClawEval Score
Kimi-K3 (Teacher)2.8T89.785.0†88.676.283.1
Qwen3.5-27B (Base)27B68.343.177.663.475.0
WorkBorne-27B27B84.361.882.066.580.2

† TB 2.1 with Terminus2, as reported by Artificial Analysis [1].

(2) Task coverage. Table R2 breaks down task coverage across the 15,485 retained trajectories. The 7,882 count includes all constructed environments, whereas the filtered SFT set in Table 2 covers 6,380 tasks. We will clarify this in the revision.

Table R2. Task Coverage of Retained Trajectories

StatisticCount
Constructed task environments7,882
Tasks with ≥ 1 retained trajectory6,380 (80.9%)
Tasks with no retained trajectory1,502 (19.1%)

(3) Is the data's value bounded by the teacher? While the retained trajectories are constrained by the rollout teacher, this does not impose a universal upper bound on student performance or WorkBorne's value. Two points stand out:

  • Task design matters under a fixed teacher. In the controlled ablation (§5.1), with the same teacher, base model, and budget, WorkBorne tasks yield 44.5 vs. 27.8 average turns and +3.4 / +4.5 points on TB 2.1 over Input-only / Instruction-first.
  • The task pool is not bound to the teacher. WorkBorne synthesizes tasks independently of the rollout model, so their difficulty and quality are not capped by the current teacher; stronger future models can re-roll the same environments to collect even better trajectories.
AuthorsOpenReview ↗

Response to Reviewer #cXXU (Part 2/2)

Q2. Generality on Additional Agent Benchmarks

"Could you evaluate on additional agent benchmarks, such as ClawBench or SWE-bench-style tasks? If not, please narrow the claims about the method's generality."

Thank you for this suggestion. Following your recommendation, we evaluated WorkBorne-27B on ClawBench V2 [4] (Codex and Hermes harnesses) and SWE-bench-Verified-Mini [5] (Claude Code harness) against the base model and two 27B synthesis baselines, FACET-27B [2] and RST-27B [3] (Table R3). For reference, on the official ClawBench V2 leaderboard (Hermes), DeepSeek-V4-Pro achieves 33.9 / 43.9 and DeepSeek-V4-Flash achieves 2.3 / 3.1 (Reward / Intercepted).

Table R3. Evaluation on Additional Benchmarks and Harnesses

ModelClawBench V2 (Codex) Reward↑ / Intercepted↑ClawBench V2 (Hermes) Reward↑ / Intercepted↑SWE-bench-Verified-Mini (Claude Code)
Qwen3.5-27B13.18 / 20.1612.40 / 18.6061.33
FACET-27B7.75 / 13.188.53 / 17.0560.67
RST-27B13.18 / 19.3814.73 / 19.3862.00
WorkBorne-27B15.50 / 24.8114.73 / 22.4864.67

In ClawBench V2, Intercepted↑ (higher is better) measures the proportion of correctly identified target access operations.

WorkBorne-27B matches or outperforms all baselines on every metric. It achieves the best reward on Codex (+2.3 pp over base), ties RST on Hermes Reward while leading on Intercepted (+3.1 pp), and improves SWE-bench-Verified-Mini by +3.3 pp. Together with Terminal-Bench (Terminus2) and ClawEval (claw-agent), the same model is now validated on five benchmarks across five harnesses.

We hope these results address your concerns, and we would be grateful if you would consider raising your assessment.

References

AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer #cXXU,

Thank you again for your supportive and thoughtful review. As the rebuttal period is coming to a close, we wanted to briefly follow up on our response and the uploaded revised manuscript.

In our response and revision, we added the Kimi-K3 teacher's reference scores (Appendix I, Table 18) and the full task-coverage statistics for retained trajectories (Appendix A, Table 6). We also evaluated WorkBorne-27B on two additional agent benchmarks(ClawBench V2 and SWE-bench-Verified-Mini) across the Codex, Hermes, and Claude Code harnesses to further demonstrate its generality (Appendix I, Table 17).

We hope the added results address your concerns and can be taken into account in your final assessment. Thank you again for your support and recognition of our work. Please let us know if any further questions remain.

Reviewer_BQuHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorkBorne, a framework for synthesizing long-horizon tasks for terminal-based agents from real-world artifacts. WorkBorne models each task as an artifact transition, which consists of a persona, a set of input artifacts, a set of output anchors that demonstrate the intended outcome, and the relation connecting them. The framework first builds artifact corpora through skill-guided iterative web retrieval. It then discovers transitions through artifact assembly, transition planning, and validation. Finally, it instantiates each transition into an aligned task instruction, an execution environment, and evaluation rubrics. Models fine-tuned on trajectories collected from these tasks achieve substantial gains on several agentic benchmarks and outperform existing synthesis methods. Extensive analyses further examine the impact of different synthesis strategies on task complexity and quality, providing empirical support for the design of WorkBorne.

Critical strengths
  • The artifact transition abstraction is intuitive and well motivated. It addresses a key limitation of instruction-oriented and resource-oriented methods, which struggle to balance task quality, complexity, and alignment with real-world scenarios.
  • The empirical gains are substantial. The fine-tuned models outperform all synthesis baselines and are competitive with much larger open-source models.
  • The analysis is informative. The controlled pipeline ablation, the human evaluation, and the scaling study on corpus breadth and transition depth offer practical guidance for future data synthesis.
Critical weaknesses
  • The main comparison with synthesis baselines is not controlled. The baselines differ in teacher models, data scale, and training recipes. It is therefore unclear how much of the gain comes from the synthesis framework rather than from the strong teacher model. Under the controlled setting, the gains are much smaller.
  • Trajectory filtering relies entirely on LLM-generated rubrics scored by an LLM judge, without execution-based verification. The human audit covers only a small sample and reports no inter-annotator agreement, so the correctness of the retained trajectories is not well established.
  • The cost of the pipeline is not reported. Synthesis involves multiple LLM agents across retrieval, assembly, planning, validation, instantiation, and judging, as well as a large number of rollouts. A cost comparison with baseline methods is needed to assess its scalability.
  • Experiments are limited to SFT on a single model family. It remains unclear whether the gains transfer to other base models or to RL training with the synthesized rubrics.
Questions for authors

Does the performance advantage of WorkBorne hold on mainstream harnesses such as Codex and Claude Code, compared to these baseline models?

6 comments
AuthorsOpenReview ↗

Response to Reviewer #BQuH (Part 1/2)

Hi Reviewer #BQuH, thank you for your constructive feedback and for recognizing our intuitive artifact transition abstraction, substantial empirical gains, and informative analyses. Below, we address W1-W4 and Q1 with new human agreement analyses, pipeline cost breakdowns, and mainstream harness evaluations.

W1. Controlled Comparison with Synthesis Baselines

"The main comparison with synthesis baselines is not controlled. ... It is therefore unclear how much of the gain comes from the synthesis framework rather than from the strong teacher model. Under the controlled setting, the gains are much smaller."

We appreciate this important point regarding controlled evaluation. First, because prior synthesis baselines use heterogeneous (often partially proprietary) pipelines and re-running full-scale synthesis and rollouts across all methods is prohibitively expensive, Table 2 compares publicly released models to demonstrate overall competitiveness. Even so, our closest competitors (FACET-27B [3], RST-27B [2], and Terminal-Universe-27B [4]) are all fine-tuned from the exact same Qwen3.5-27B backbone across diverse data scales (e.g., RST uses 37K tasks, while Terminal-Universe uses ~32K SFT trajectories, more than twice WorkBorne's 15.5K). Despite using less than half the data of Terminal-Universe, WorkBorne-27B achieves 61.8% on TB 2.1, outperforming FACET (47.6%), RST (46.8%), and Terminal-Universe (58.1%), and also consistently leads FACET-27B and RST-27B on the new benchmarks across Codex, Hermes, and Claude Code (Table R4).

Second, to strictly isolate our synthesis framework from teacher and recipe differences, Table 3 (§5.1) conducts a fully controlled comparison across paradigms. Specifically, we compare Instruction-first (fabricated environments, following Terminal-World [1]) → Input-only (real files as static inputs, mirroring Terminal-Universe [4] and CLI-Universe [5]) → WorkBorne (artifact transitions), all under the exact same Kimi-K3 teacher, Qwen3.5-27B base model, and 1K-trajectory training recipe. Under this controlled setting, WorkBorne shows a clear advantage across every dimension: execution depth jumps by 60.1% (25.9 → 27.8 → 44.5 turns), while TB 2.1 (47.2% → 48.3% → 51.7%) and TB Lite (68.3% → 69.0% → 71.1%) improve monotonically. Furthermore, our controlled scaling study (§5.3, Figure 4) under the same teacher shows that increasing transitions per corpus from 1 to 8 steadily expands the TB 2.1 gain from 52.8% to 59.6% (and TB Lite from 72.5% to 79.4%), smoothly bridging the 1K ablation and our full-scale results (61.8% / 84.3%).

W2. Trajectory Verification and Human Agreement

"Trajectory filtering relies entirely on LLM-generated rubrics scored by an LLM judge, without execution-based verification. The human audit covers only a small sample and reports no inter-annotator agreement ..."

Thank you for this valuable suggestion on validating our trajectory filtering. We report inter-annotator agreement (IAA) for our 50-task rubric audit (Appendix F) and add a new human verification study on 50 randomly sampled rollouts, covering both accepted and rejected trajectories (Table R1). Two experts independently inspected the full execution traces and final workspace artifacts, scoring each rollout with the task rubrics under the same acceptance threshold (≥8/10\ge 8/10). Against the averaged human judgments, the Opus-4.6 judge achieves 87.5% precision and 93.3% recall (κ=0.746\kappa = 0.746, 88.0% agreement).

Table R1. Inter-Annotator Agreement and Human-Judge Alignment

EvaluationHuman-HumanOpus-Human
Rubric Quality (50 tasks)κ=0.815\kappa = 0.815 (82.7% exact agreement)N/A
Trajectory Filtering (50 rollouts)κ=0.692\kappa = 0.692 (86.0% agreement)κ=0.746\kappa = 0.746 (88.0% agreement)

Regarding why we adopt a rubric-based LLM judge, WorkBorne synthesizes thousands of open-ended tasks spanning documents (98.0%), structured data (82.4%), code (80.0%), and native office/media files (59.7%). For such open-ended and diverse deliverables, hard-coded unit tests are both difficult to construct and inaccurate, as they easily reject valid alternative solutions. Instead, WorkBorne derives artifact-grounded rubrics directly from real output anchors and transition requirements, which specify observable evidence from execution outcomes, including final workspace artifacts, functional behavior, terminal execution outputs, and validation logs (Appendix I.7). We will add these complete IAA and human review results to Section 3.3 and Appendix F in the final version.

AuthorsOpenReview ↗

Response to Reviewer #BQuH (Part 2/2)

W3. Synthesis Cost and Scalability

"The cost of the pipeline is not reported. ... A cost comparison with baseline methods is needed to assess its scalability."

Across the full dataset (Table R2), WorkBorne costs $1.143/task after amortization. Since each corpus is reused across multiple transitions, one-time retrieval amortizes to $0.171/task. In the §5.1 controlled setting (Table R3), WorkBorne adds only 9.0% cost over Input-only while yielding 60.1% longer trajectories and +3.4 pp on TB 2.1, and reduces cost by 30.9% vs. Instruction-first by reusing 98.8% of workspace files from corpora.

Table R2. Synthesis Cost Breakdown of WorkBorne

StageAccounting UnitAvg. TokensAvg. Cost (USD)
Corpus RetrievalPer corpus1.429M$1.274
Corpus Construction (Amortized)Per task192.4K$0.171
Assembly, Transition Planning & InstantiationPer task608.1K$0.972
Total Task Synthesis (Amortized)Per task800.5K$1.143

Table R3. Cost and Performance Across Pipelines (§5.1 Controlled Setting)

MethodTokens / TaskCost / TaskAvg. TurnsTB 2.1
Instruction-first (following [1])1,154.3K$1.66125.947.2
Input-only (similar to [4, 5])736.3K$1.05227.848.3
WorkBorne798.0K$1.14744.551.7

W4. Generalization Across Base Models and Training Paradigms

"Experiments are limited to SFT on a single model family. It remains unclear whether the gains transfer to other base models or to RL training with the synthesized rubrics."

(1) Base models. We evaluate both dense (Qwen3.5-27B) and MoE (Qwen3.5-35B-A3B) models. Qwen3.5 is the standard backbone in recent terminal-agent synthesis work [2, 3, 4], enabling direct comparison without backbone confounding.

(2) SFT validation. As our contribution is a task synthesis framework, we follow prior work [2, 3, 4] in using SFT as a standard protocol for assessing data quality. WorkBorne improves TB 2.1 by +18.7 pp (dense) and +8.6 pp (MoE) with consistent ClawEval gains, and §5.1 isolates the contribution of transitions under identical training. We agree that other model families and RL with our synthesized rubrics are valuable directions and will discuss them as future work.

Q1. Performance Advantage on Mainstream Harnesses

Yes. WorkBorne trajectories are already collected across 14 harnesses, including Codex, Claude Code, and Hermes (Appendix A, Table 5). We further evaluated WorkBorne-27B against Qwen3.5-27B, FACET-27B [3], and RST-27B [2] on ClawBench V2 (Codex and Hermes) and SWE-bench-Verified-Mini (Claude Code). As shown in Table R4, WorkBorne-27B matches or outperforms all baselines on every metric. Together with Terminus2 and claw-agent (Table 2), WorkBorne-27B is now validated across five harnesses.

Table R4. Evaluation on Mainstream Harnesses

ModelClawBench V2 (Codex) Reward↑ / Intercepted↑ClawBench V2 (Hermes) Reward↑ / Intercepted↑SWE-bench-Verified-Mini (Claude Code) Resolved↑
Qwen3.5-27B13.18 / 20.1612.40 / 18.6061.33
FACET-27B [3]7.75 / 13.188.53 / 17.0560.67
RST-27B [2]13.18 / 19.3814.73 / 19.3862.00
WorkBorne-27B15.50 / 24.8114.73 / 22.4864.67

We hope these results address your concerns, and we would be grateful if you would consider raising your assessment.

References

  • [1] Terminal-World: Scaling terminal-agent environments via agent skills, arXiv'26
  • [2] Recursive synthesis for long-horizon terminal tasks, arXiv'26
  • [3] FACET: Preserving source intent and executable state in terminal task synthesis, arXiv'26
  • [4] Terminal-Universe: Turning agent trajectories into scalable terminal environments, arXiv'26
  • [5] CLI-Universe: Towards verifiable task synthesis engine for terminal agents, arXiv'26
AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer #BQuH,

Thank you again for your constructive review and insightful suggestions. As the rebuttal period is coming to a close, we wanted to briefly follow up on our response and the uploaded revised manuscript.

In our response and revision, we clarified the controlled pipeline and scaling comparisons under the same teacher (§5.1, §5.3), added inter-annotator agreement and new human verification of our rubric-based LLM judge (Appendix F, Table 15), and reported a full synthesis cost breakdown (Appendix E, Tables 11–12). We also added new evaluations on mainstream harnesses including Codex, Hermes, and Claude Code to further verify WorkBorne's effectiveness (Appendix I, Table 17).

We hope these new experiments and analyses address your concerns. We would be grateful if these updates could inform your final evaluation. Please let us know if any further questions remain!

Reviewer_AD1jOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper notes that the current efforts of synthesizing agents for terminal-based on horizon tasks suffer from not being able to use the trajectory artifacts in a trajectory context and trajectory relevant way. Instead they treat it as a static and non-binding artifacts. This lead to the synthesis of agents that are not effective despite the desire to develop agents that are effective on long trajectories. The paper present "WorkBorne" to address this persistent problem by gathering trajectory transitional artifacts and then goes extra mile of connecting the transitions using the input-output artifacts to extract the needed bindings for the agent synthesis.

Critical strengths

Developing an approach of tracking the transitional relationships for the agentic tasks in order to build effective terminal-based agents. Highly tedious and task specific effort.

Critical weaknesses

From my reading, it seems that the approach is super specific to tasks or class of tasks. Is that correct or am I missing something here. It was not easy to read.

Questions for authors

Is this going to be highly task specific?

6 comments
AuthorsOpenReview ↗

Response to Reviewer #AD1j (Part 1/2)

Hi Reviewer #AD1j, thank you very much for your time and feedback. We noticed your central concern is whether WorkBorne is restricted to specific tasks or a narrow class of tasks (W1 & Q1), and that parts of the paper were “not easy to read.”

To directly answer your question upfront: no, WorkBorne is general-purpose, both in the tasks it synthesizes and in the agent capabilities it trains. Below, we first explain why our task setting, open-web data collection, and results across five benchmarks are general-purpose (Part 1/2), and then walk through how our algorithm achieves this automatically in plain language (Part 2/2).

1. Why Terminal Agents Cannot Be Narrow & Why We Need Automated Synthesis

"From my reading, it seems that the approach is super specific to tasks or class of tasks. Is that correct or am I missing something here... Is this going to be highly task specific?"

Our goal is to train general-purpose terminal agents that operate in a command-line environment and handle the full spectrum of digital work across domains (e.g., software development, data analysis, system administration, document/spreadsheet processing, and automation). By definition, a terminal agent must be a generalist, so our training data cannot be narrow and must span diverse domains.

However, manually curating such broad coverage across domains, file types, and software tools does not scale. Individual annotators typically design tasks only around the few libraries and workflows they use in daily work, so covering the breadth of the terminal ecosystem would require hiring domain experts across countless fields at prohibitive cost. Therefore, automated synthesis is essential to scale up cross-domain task diversity.

2. Open-Web Diversity Across 1,061 Skills & 7,882 Environments

Before even considering our algorithm, the raw materials collected by WorkBorne ensure broad generality. WorkBorne retrieves real-world files in a free-form manner from the open web across 1,061 diverse skills and heterogeneous web sources, building 7,882 task environments without any domain-specific templates.

  • Wide Domain Coverage. Synthesized tasks naturally span automation, spreadsheet/slide editing, data analysis, and software engineering.
  • Rich Multi-Format Files. Across all 7,882 tasks, 98.0% involve documents, 82.4% involve structured data, 80.0% involve code/scripts, and 59.7% involve images (Figure 5d). Drawing from such varied open-web files naturally demands diverse problem-solving workflows and prevents the tasks from collapsing into a narrow class of operations.

3. Empirical Proof Across 5 Diverse Benchmarks

Most importantly, we use WorkBorne trajectories to train a general-purpose model (WorkBorne-27B) and evaluate this same model across five distinct benchmarks and five harnesses, including two new benchmarks added during the rebuttal.

Table R1. Performance of WorkBorne-27B Across Five Diverse Benchmarks

BenchmarkTask Domain / ScopeEvaluation HarnessQwen3.5-27B (Base)FACET-27BRST-27BWorkBorne-27B (Ours)
Terminal-Bench LiteGeneral CLI & System TasksTerminus268.3%77.4%71.9%84.3% (+16.0 pp)
Terminal-Bench 2.1Long-Horizon Complex Terminal TasksTerminus243.1%47.6%46.8%61.8% (+18.7 pp)
ClawEval (General)Open-Ended Digital Workflowsclaw-agent75.077.175.780.2 (+5.2)
ClawBench V2 (New)General Agent Tasks (Reward↑ / Intercepted↑)Codex13.18% / 20.16%7.75% / 13.18%13.18% / 19.38%15.50% / 24.81%
ClawBench V2 (New)General Agent Tasks (Reward↑ / Intercepted↑)Hermes12.40% / 18.60%8.53% / 17.05%14.73% / 19.38%14.73% / 22.48%
SWE-bench-Verified-Mini (New)Software EngineeringClaude Code61.33%60.67%62.00%64.67% (+3.34 pp)

As shown in Table R1, WorkBorne-27B consistently outperforms both the base model and competing synthesis methods (FACET-27B, RST-27B) across general CLI operations, open-ended digital workflows, and software engineering, confirming that WorkBorne is a general-purpose terminal task synthesis framework.

AuthorsOpenReview ↗

Response to Reviewer #AD1j (Part 2/2)

4. Why We Propose "Artifact Transition" (Algorithmic Motivation)

We appreciate the opportunity to clarify the core motivation behind our method in plain language. Why is WorkBorne able to synthesize such diverse, high-quality tasks from open-web files when prior automated methods struggle? Existing synthesis paradigms face a dilemma between environment realism and task complexity.

  • Instruction-first methods ask an LLM to invent task descriptions and fabricate files from scratch, resulting in toy-like, low-fidelity environments.
  • Input-only methods collect real files from the web but only use them as static starting inputs, which easily degrades instructions into shallow, one-step operations (e.g., "read this file and summarize it").

To automatically discover complex, multi-step tasks from open-ended real-world files without domain-specific rules, we observe that substantive computer work in any domain naturally follows a universal 3W pattern, in which Who works on What Input files toward What Output files.

Based on this observation, we propose Artifact Transition as a domain-agnostic primitive. Instead of humans designing tasks, our framework automatically discovers transition relationships between real-world files in any retrieved file pool and uses each transition as the skeleton to construct a challenging, authentic task and environment.

  • A Realistic Example (pomodoro-lights). In a retrieved file pool centered on study and focus tools (which includes desktop timer software, audio assets, configuration templates, and related utility files), WorkBorne assigns a smart-home enthusiast persona (Who), takes a standard Pomodoro timer application (Pomotroid) and an audio chime file (bell.wav) as Inputs, and selects a light-control application template as an Output Anchor. From this pairing, WorkBorne automatically discovers the functional Transition of building an integrated bridge platform that syncs study timer events with smart lighting and audio routines. The resulting Task requires the agent to develop and wire up a full-featured bridge program so that room lights dynamically adjust color/brightness and play chime notifications in real time as study or break sessions transition.

Anchoring each task at both ends with real input files and concrete output anchors naturally induces deep, multi-step execution, increasing average trajectory length from 27.8 to 44.5 turns over an input-only baseline under controlled comparisons (Table 3).

5. WorkBorne’s Core Pipeline in Three Simple Steps

As shown in Figure 1, WorkBorne operates as a fully automated, domain-agnostic three-step pipeline without any manual, per-task rule engineering.

  1. Step 1. Collect a pool of real-world files (Corpus Construction). Starting from any skill topic, a search agent automatically retrieves a coherent collection of real files from the open web (e.g., documents, data tables, media, configs, and code).
  2. Step 2. Automatically discover transitions from the files (Transition Discovery). From the collected file pool, an agent automatically selects a relevant user persona, pairs starting Input files with target Output Anchors, and infers the substantive Transition connecting them.
  3. Step 3. Synthesize the task and environment from the transition (Task Instantiation). Using the discovered transition as the blueprint, the framework automatically generates the user instruction, populates an executable workspace with the real files, and creates evaluation rubrics for trajectory collection.

6. Commitment to Improving Readability in the Revision

We sincerely appreciate your feedback on readability ("It was not easy to read"). In the revised manuscript, we will make two concrete improvements.

  1. Streamline Section 1 & Section 3.1. First explain why general-purpose terminal agents require automated open-web synthesis, and present the intuitive three-step overview (collect real files → automatically discover transitions → synthesize tasks and environments) and the 3W motivation in plain language before introducing formal notation.
  2. Include Intuitive Case Studies (such as pomodoro-lights) in Section 3, Figure 1 & Appendix. Make it immediately clear how transitions are automatically discovered from everyday file pools across diverse domains.

We hope this clear walkthrough and the expanded empirical results fully address your concerns, and we would be grateful if you would consider raising your assessment.

AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer #AD1j,

Thank you again for your time and helpful feedback. As the rebuttal period is coming to a close, we wanted to briefly follow up on our response and the uploaded revised manuscript.

In our response and revision, we clarified that WorkBorne is not a task-specific approach and why WorkBorne is a domain-agnostic, general-purpose synthesis framework, supported by empirical evaluations across five diverse benchmarks and five agent harnesses (Table 2; Appendix I, Table 17). We also improved the readability by adding an intuitive end-to-end case study (Appendix J, Figure 7).

We hope these clarifications address your concerns, and we would be very grateful if you could take these updates into account in your final assessment. Please let us know if any further questions remain!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and Responses

We sincerely thank all three reviewers for their careful reading and constructive feedback. As the discussion period approaches its close, we would like to consolidate our new experiments and analyses addressing the key concerns raised by the three reviewers (G1–G4). Full tables and detailed replies are in each reviewer’s thread, and all additions are marked in blue in the revised PDF.

G1. Domain generality, new benchmarks and mainstream harnesses (AD1j, cXXU, BQuH) WorkBorne is a domain-agnostic synthesis framework that retrieves real open-web files across 1,061 skills to build 7,882 environments spanning documents (98.0%), structured data (82.4%), code (80.0%), and images (59.7%). To test generality beyond Terminal-Bench and ClawEval, we evaluated the same WorkBorne-27B model on ClawBench V2 (Codex and Hermes harnesses) and SWE-bench-Verified-Mini (Claude Code harness) against Qwen3.5-27B, FACET-27B, and RST-27B. WorkBorne-27B matches or beats every baseline across all metrics: it achieves 15.50% / 24.81% (Reward / Intercepted) on Codex (+2.32 / +4.65 pp over base), 14.73% / 22.48% on Hermes (+3.10 pp on Intercepted), and 64.67% on SWE-bench-Verified-Mini (+3.34 pp over base), validating our model across five benchmarks and five harnesses.

G2. Controlled isolation of the synthesis framework, teacher reference and RL potential (BQuH, cXXU) Among 27B synthesis baselines on the same Qwen3.5-27B backbone, WorkBorne-27B reaches 61.8% on TB 2.1 using 15.5K trajectories, ahead of FACET-27B (47.6%), RST-27B (46.8%, 37K tasks), and Terminal-Universe-27B (58.1%, ~32K trajectories). When the teacher (Kimi-K3), base model, and 1K trajectory budget are held fixed (§5.1), moving from Instruction-first and Input-only to WorkBorne raises execution depth from 25.9 and 27.8 to 44.5 turns and TB 2.1 from 47.2% and 48.3% to 51.7%, while scaling transitions per corpus (§5.3) further reaches 59.6%. We also report Kimi-K3’s reference scores (89.7% TB Lite, 85.0% TB 2.1, 83.1 ClawEval) and task coverage: 6,380 of the 7,882 environments (80.9%) yield retained SFT trajectories.

G3. Human verification of rubric-based trajectory filtering (BQuH) For open-ended deliverables across documents, data, code, and media files, WorkBorne derives artifact-grounded rubrics from real output anchors to avoid rejecting valid alternative solutions. Our 50-task rubric audit shows strong inter-annotator agreement (κ=0.815\kappa = 0.815, 82.7% exact agreement). In a new human verification study on 50 randomly sampled rollouts covering both accepted and rejected trajectories, the Opus-4.6 rubric judge achieves 87.5% precision and 93.3% recall against averaged expert judgments (κ=0.746\kappa = 0.746, 88.0% agreement).

G4. Synthesis cost and scalability (BQuH)

Across all 7,882 tasks, WorkBorne costs USD 1.143 per task after amortization; reusing each corpus across multiple transitions reduces one-time web retrieval to USD 0.171 per task. Under the controlled comparison in Section 5.1, WorkBorne adds only 9.0% cost over Input-only (USD 1.147 vs. USD 1.052) while producing 60.1% longer trajectories and +3.4 pp on TB 2.1, and cuts cost by 30.9% compared to Instruction-first (USD 1.661) by reusing 98.8% of workspace files directly from retrieved corpora.

Changes to the paper (all made in the revised PDF, marked in blue):

  • Section 3.2 & Appendix J: end-to-end case study card (pomodoro-lights) illustrating persona selection, input/output anchors, transition discovery, and instruction instantiation (G1; p. 5 & Appendix J, Figure 7, pp. 24–25).
  • Section 4.2 & Appendix I: evaluations on ClawBench V2 (Codex, Hermes) and SWE-bench-Verified-Mini (Claude Code), plus the Kimi-K3 teacher comparison and discussion on future RL with the hard task set and rubrics (G1, G2; p. 7 & Appendix I, Tables 17–18, pp. 23–24).
  • Appendix A: task coverage breakdown across all 7,882 constructed environments (G2; Table 6, pp. 16–17).
  • Appendix E: stage-by-stage synthesis cost breakdown and cross-pipeline cost comparison (G4; Tables 11–12, pp. 19–20).
  • Appendix F: inter-annotator agreement and the 50-rollout human verification study for rubric-based filtering (G3; Table 15, pp. 20–21).

We hope this summary and the revised manuscript are helpful to the reviewers and AC as the discussion period concludes, and we remain happy to address any final questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.