- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents WorkForge, a pipeline that turns expert workflow blueprints into work-agent environments built from real files retrieved online. Facts extracted from each workspace are used to write the task, a hidden solution plan, and a mix of programmatic checks and LLM-judged rubrics. Trajectories from GLM 5.3 under Claude Code and OpenClaw are then used for SFT. Fine-tuned Qwen3.5 models improve on five work-agent benchmarks, and the paper adds analyses on data scale, trajectory length, harness choice and the verifier used for selection.
Critical strengths
- Anchoring both the task and its checks on facts pulled from a real workspace is a sensible design. The factor-premium example in the appendix shows rubrics tied to concrete file facts, which are harder to hallucinate than criteria written against a toy workspace.
- The gains are large and consistent across two models and five benchmarks, including APEX, which runs on a harness never used in training.
Critical weaknesses
- All trajectories come from GLM 5.3, which is not reported in Table 2, while GLM-5.2 is already strong on GDPval. So it is hard to tell how much of the gain comes from the environments and how much is plain distillation from a strong teacher. Using a Base model for the 35B-A3B runs makes this worse, since almost any agentic SFT data would give a large jump there.
- The main claim that real workspaces matter is not tested downstream. The Workspace Authenticity part only compares file counts and sizes; there is no model trained on the matched synthetic workspaces.
- It is unclear how much the verifiers actually matter. If I read it right, the main set has one trajectory per task and a sizable share of tasks in Figure 3(c) score near zero, so there seems to be no score-based filtering; the verifiers only play a role in the 1K ablation. I could not find any human agreement check for the rubrics or the judge, and the limitations section itself mentions limited expert calibration. The judge model is not named either.
- Four of the five benchmarks are run with the same harnesses used for training, and Figure 5 shows that harness match alone is worth a clear margin. Part of the main gains may come from learning harness conventions rather than work skills.Re-evaluate at least one benchmark under an unseen harness.
Questions for authors
- How does GLM 5.3 itself score on these five benchmarks? Was it also used for task design or judging?
- What happens if you train on the matched synthetic workspaces instead?
- Is the main training set filtered by verifier score? If so, what is the threshold?
- Did you spot-check any rubrics or judge scores by hand?
- A natural next step would be RL with these verifiers on top of SFT. This is a suggestion, not a condition for my score.
5 comments
Response to reviewer 96xU (Part 2/3)
W3, Q3 & Q4. Implementation details and quality check for agent environments
Thank you for catching these omissions. We apologize that several implementation details were missing from our initial submission; we have supplemented these implementation details in our revised manuscript, and additionally report a manual quality check below:
- Environment synthesis and task generation. Environment mapping, environment profiling, task design, and verifier design (fact verification and rubric generation) all use DeepSeek-V4-Pro.
- Verification module. Each task averages 21.9 atomic rubrics across six dimensions: 37.3% are programmatic checks instantiated from deterministic templates, and 62.7% are semantic criteria scored on a [0, 10] scale by Claude Sonnet 4.6 and normalized to [0, 1]. GLM 5.3 is used exclusively for trajectory rollout, ensuring that no trajectory is evaluated by its generator.
- Trajectory generation and filtering (Q3). WorkForge-16K is strictly score-filtered: from 35.7K sandboxed GLM 5.3 rollouts, we retain only well-formed trajectories with a hybrid programmatic and semantic verifier score ≥ 0.7, yielding the 16,665 training trajectories (46.7%). Figure 3(c) plots the pre-filtering distribution over all 35.7K candidates—the discarded 53.3% (including the low-score mass) are excluded from training, reflecting verifier strictness—and Table 3 confirms that this hybrid signal outperforms single-verifier filtering by +5.5–6.2 ALE-CLI points.
- Quality check of synthesized tasks and rubrics (Q4). Following your suggestion, we randomly sample 50 tasks from WorkForge-16K and manually inspect each task specification, the coverage of its rubric set, and all 1,093 associated rubrics. Each item is independently checked by two evaluators, with disagreements resolved by a third evaluator:
| Aspect | Sample | Validity | Agreement |
|---|---|---|---|
| Tasks | 50 tasks | 94.0% | 96.0% |
| Rubric sets (coverage) | 50 tasks | 92.0% | 90.0% |
| Rubrics (each item) | 1,093 rubrics | 94.5% | 94.3% |
Across three aspects, 92.0–94.5% of checked items are valid with 90.0–96.0% agreement, establishing two conclusions:
- Task and rubric validity: Nearly all sampled tasks are well-formed and solvable (47/50), with rubric sets comprehensively covering required deliverables (46/50)—confirming that grounding generation in observable workspace facts yields reliable tasks and rubrics at scale.
- Verification robustness: Invalid rubrics average only 1.2 per task out of 21.9 atomic rubrics (5.5%), ensuring that task-level verification scores are dominated by valid signals.
Response to reviewer 96xU (Part 3/3)
W4. Generalization effect across agentic harnesses
To verify that WorkForge imparts transferable work skills rather than overfitting to training harnesses, we first note that training solely on OpenClaw trajectories already improves ALE-CLI under Claude Code from 11.4 to 16.1 (Figure 5). Moreover, since existing benchmarks often tie their sandboxes to specific harnesses (e.g., WildClawBench is built around OpenClaw), we additionally evaluate AutomationBench and JobBench alongside APEX under their respective official harnesses (none of which appears in training), reporting WorkForge's gain (Δ) over each backbone:
| Model | APEX | AutomationBench | JobBench |
|---|---|---|---|
| Official Harness | ReAct Toolbelt | Search-and-Execute Loop | OpenCode |
| Qwen3.5-35B-A3B-Base (Δ) | +16.3 | +3.4 | +16.0 |
| Qwen3.5-35B-A3B (Δ) | +0.6 | +3.0 | +0.3 |
| Qwen3.5-27B (Δ) | +4.2 | +7.7 | +16.8 |
WorkForge consistently improves both base and post-trained backbones across all three unseen official harnesses, confirming that our synthesized environments impart genuine, generalizable work-agent capabilities rather than harness-specific conventions.
Q5. On-policy RFT and RL in WorkForge environments
Guided by your advice, we evaluate on-policy rejection fine-tuning (RFT) and reinforcement learning (RL) in WorkForge environments without any external teacher supervision:
- RFT setup. We sample trajectories directly from Qwen3.5-27B on WorkForge tasks (temperature 1.0), retain 4K error-free rollouts with a hybrid verifier score ≥ 0.7, and fine-tune Qwen3.5-27B.
- RL setup. Starting from WorkForge-A3B, we run multi-turn GRPO for 80 steps using the continuous hybrid verifier score in [0, 1] as the online reward (batch size 8, group size 8, temperature 0.9, lr 1e-6, clip [0.2, 0.28] without KL penalty, up to 256K tokens and 200 turns).
| Model | GDPval | JobBench | AutomationBench |
|---|---|---|---|
| Qwen3.5-27B | 79.4 | 17.0 | 9.2 |
| + RFT on WorkForge | 82.8 | 18.9 | 9.8 |
| WorkForge-A3B | 73.6 | 16.2 | 3.7 |
| + RL on WorkForge | 78.3 | 20.3 | 4.1 |
Both RFT and RL consistently improve over their respective initializations without any external teacher supervision. Within WorkForge environments, on-policy RFT lifts Qwen3.5-27B across all benchmarks—reaching 82.8 on GDPval. Meanwhile, multi-turn GRPO further boosts the already distilled WorkForge-A3B by +4.7 on GDPval and +4.1 on JobBench.
Thank you once again for your rigorous and constructive feedback, which has helped us make the manuscript substantially stronger. All references below have been cited and discussed in our revised manuscript. We are happy to discuss any remaining questions and would be grateful if you would consider raising your assessment.
References
[1] ClawGym: A Scalable Framework for Building Effective Claw Agents.
[2] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis.
Response to reviewer 96xU (Part 1/3)
Dear Reviewer 96xU,
We sincerely appreciate your detailed review. Your feedback helps us improve the paper quality. A detailed response and supplementary experiments are provided below.
The paper you have read and the responses we have provided are the result of our authors' utmost effort, dedication, and hard work. It holds great significance for us, and we sincerely hope you will reconsider our contribution.
W1 & Q1. Continual SFT of post-trained LLMs with GLM 5.3 teacher
Thank you for raising these important questions. We first report the teacher GLM 5.3 and apply WorkForge on top of the officially post-trained Qwen3.5-35B-A3B and Qwen3.5-27B, to test whether our gains are merely what any agentic SFT data gives a base model. Note that GLM 5.3 is used exclusively for trajectory rollout: task design uses DeepSeek-V4-Pro and semantic judging uses Claude Sonnet 4.6 (see W3 for full details). All backbones are fine-tuned on the full WorkForge-16K set under the same SFT recipe, so the base and post-trained runs differ only in their initialization. The results are summarized below:
W2 & Q2. Environment synthesis methods and real vs. synthetic workspaces
To further separate environment quality from GLM 5.3 distillation (W1) and directly test matched synthetic workspaces as you suggested (W2 & Q2), we conduct a controlled study fixing the teacher model (GLM 5.3 with high reasoning effort), rollout harness (OpenClaw), data volume (1K trajectories), student backbone (Qwen3.5-35B-A3B-Base), and SFT recipe (5 epochs, learning rate 1e-5, global batch size 64).
Under this setup, we compare WorkForge against previous environment synthesis methods, ClawGym [1] and EnvScaler [2], as well as a synthetic-workspace variant built from the same WorkForge scene blueprints, in which real-resource acquisition is replaced by DeepSeek-V4-Pro workspace synthesis (7.9 vs. 42.3 files/workspace; Figure 1). The results are summarized below:
WorkForge outperforms prior environment synthesis methods and the synthetic-workspace variant on 5 of 6 benchmarks and virtually ties on JobBench (12.2 vs. 12.4), establishing two key conclusions: