- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes WorkBorne, a pipeline that synthesizes terminal-agent tasks by modeling each task as an "artifact transition" (a persona, real input artifacts, real output anchors, and the relation between them). Starting from 1,061 agent skills, it retrieves real artifacts from the web, discovers and validates transitions, and instantiates each into an instruction, a real-artifact environment, and rubrics. Kimi-K3 rollouts filtered by an Opus judge yield 15,485 trajectories. SFT of Qwen3.5-27B on these trajectories raises TB 2.1 from 43.1 to 61.8 and the ClawEval score from 75.0 to 80.2, outperforming prior synthesis methods. A matched comparison against instruction-first and input-only pipelines shows that transition modeling yields deeper execution and better downstream performance.
Critical strengths
1. Consistent controlled evidence for the central claim. Table 3 compares fabricated environments, input-only pipelines, and transition modeling. Environment size, execution depth, human-rated complexity, and TB 2.1 all rise in that order, and the full-scale 27B model reaches 61.8 on TB 2.1. Why it matters: this is my main reason for leaning toward acceptance. The paper's diagnosis, its fix, and the evidence line up.
2. Credible absolute numbers. The paper includes thorough contamination checks (Table 6), re-measures most baselines under the same protocol, and fully specifies training and prompts. Why it matters: this rules out leakage and mismatched evaluation as explanations for the gains.
Critical weaknesses
1. Evidence for generality is limited, and data quality is bounded by the teacher. All results use a single teacher (Kimi-K3), a single base family (Qwen3.5), and three benchmarks. Because only trajectories that Kimi-K3 completes are retained, the difficulty of the training data is capped by the teacher's ability, and the teacher's own scores are not reported. The contribution is a data-construction method, so its value depends on transferring beyond this specific setup and on teaching the student more than the teacher's existing behavior on these benchmarks.
Questions for authors
1. Teacher reference. Please report Kimi-K3's own TB 2.1 and ClawEval scores, so the student's results can be read relative to the teacher. Please also report how many tasks have no retained trajectory and whether they are counted among the 7,882.
2. Generality. Could you evaluate on additional agent benchmarks, such as ClawBench or SWE-bench-style tasks? If not, please narrow the claims about the method's generality.
3 comments
Response to Reviewer #cXXU (Part 2/2)
Q2. Generality on Additional Agent Benchmarks
"Could you evaluate on additional agent benchmarks, such as ClawBench or SWE-bench-style tasks? If not, please narrow the claims about the method's generality."
Thank you for this suggestion. Following your recommendation, we evaluated WorkBorne-27B on ClawBench V2 [4] (Codex and Hermes harnesses) and SWE-bench-Verified-Mini [5] (Claude Code harness) against the base model and two 27B synthesis baselines, FACET-27B [2] and RST-27B [3] (Table R3). For reference, on the official ClawBench V2 leaderboard (Hermes), DeepSeek-V4-Pro achieves 33.9 / 43.9 and DeepSeek-V4-Flash achieves 2.3 / 3.1 (Reward / Intercepted).
Table R3. Evaluation on Additional Benchmarks and Harnesses
| Model | ClawBench V2 (Codex) Reward↑ / Intercepted↑ | ClawBench V2 (Hermes) Reward↑ / Intercepted↑ | SWE-bench-Verified-Mini (Claude Code) |
|---|---|---|---|
| Qwen3.5-27B | 13.18 / 20.16 | 12.40 / 18.60 | 61.33 |
| FACET-27B | 7.75 / 13.18 | 8.53 / 17.05 | 60.67 |
| RST-27B | 13.18 / 19.38 | 14.73 / 19.38 | 62.00 |
| WorkBorne-27B | 15.50 / 24.81 | 14.73 / 22.48 | 64.67 |
In ClawBench V2, Intercepted↑ (higher is better) measures the proportion of correctly identified target access operations.
WorkBorne-27B matches or outperforms all baselines on every metric. It achieves the best reward on Codex (+2.3 pp over base), ties RST on Hermes Reward while leading on Intercepted (+3.1 pp), and improves SWE-bench-Verified-Mini by +3.3 pp. Together with Terminal-Bench (Terminus2) and ClawEval (claw-agent), the same model is now validated on five benchmarks across five harnesses.
We hope these results address your concerns, and we would be grateful if you would consider raising your assessment.
References
- [1] Artificial Analysis, Terminal-Bench evaluation results, https://artificialanalysis.ai
- [2] FACET: Preserving source intent and executable state in terminal task synthesis, arXiv'26
- [3] Recursive synthesis for long-horizon terminal tasks, arXiv'26
- [4] ClawBench: Can AI agents complete everyday online tasks?, arXiv'26
- [5] SWE-bench-Verified-Mini, https://github.com/mariushobbhahn/SWEBench-verified-mini
Brief Follow-up on Our Response
Dear Reviewer #cXXU,
Thank you again for your supportive and thoughtful review. As the rebuttal period is coming to a close, we wanted to briefly follow up on our response and the uploaded revised manuscript.
In our response and revision, we added the Kimi-K3 teacher's reference scores (Appendix I, Table 18) and the full task-coverage statistics for retained trajectories (Appendix A, Table 6). We also evaluated WorkBorne-27B on two additional agent benchmarks(ClawBench V2 and SWE-bench-Verified-Mini) across the Codex, Hermes, and Claude Code harnesses to further demonstrate its generality (Appendix I, Table 17).
We hope the added results address your concerns and can be taken into account in your final assessment. Thank you again for your support and recognition of our work. Please let us know if any further questions remain.
Response to Reviewer #cXXU (Part 1/2)
Hi Reviewer #cXXU, thank you for your thoughtful review and support of our work. Below, we address W1 and Q1-Q2 with Kimi-K3's reference scores, a full task-coverage audit, and new evaluations on ClawBench V2 and SWE-bench-Verified-Mini.
W1 & Q1. Teacher Reference and Task Coverage
(1) Teacher reference. Table R1 reports Kimi-K3 alongside the base and fine-tuned student. As a 2.8T-parameter frontier model (~100× the student), Kimi-K3 remains stronger on every metric, as expected for SFT distillation.
Table R1. Student Performance Relative to the Kimi-K3 Teacher
† TB 2.1 with Terminus2, as reported by Artificial Analysis [1].
(2) Task coverage. Table R2 breaks down task coverage across the 15,485 retained trajectories. The 7,882 count includes all constructed environments, whereas the filtered SFT set in Table 2 covers 6,380 tasks. We will clarify this in the revision.
Table R2. Task Coverage of Retained Trajectories
(3) Is the data's value bounded by the teacher? While the retained trajectories are constrained by the rollout teacher, this does not impose a universal upper bound on student performance or WorkBorne's value. Two points stand out: