acceptodds
Under review as a conference paper at ICLR 2027

Scaling Verifiable Environments for Long-horizon Work Agents

11 views

Abstract

Work agents operate over digital artifacts to execute professional knowledge-intensive work, requiring training environments that support long-horizon interaction and trustworthy verification. However, hand-crafted environments incur prohibitive engineering overhead that prevents environment scaling, whereas synthesis methods sacrifice workspace complexity, realism, or grounded verifiability. To bridge this gap, we introduce WorkForge, a scalable synthesis framework for constructing verifiable work-agent environments from real-world resources. Starting from expert workflows, WorkForge first identifies the resources, decisions, and deliverables required by each workflow. It then retrieves relevant real-world files and organizes them into a workspace. WorkForge inspects the workspace to extract concrete, checkable facts about its content. These factual anchors fix which task types the workspace can support and how their outcomes can be verified. Therefore, WorkForge derives each task's instructions, solution plan, and complementary programmatic and semantic verifiers directly from these factual anchors, keeping verification traceable to observable workspace evidence. Furthermore, we construct 16.7K verifiable environments across 40 professional domains, with workspaces collectively covering 60 file types. Post-training Qwen3.5-35B-A3B-Base improves GDPVal from 45.5 to 73.6 and APEX Score from 5.0 to 21.3, while enabling Qwen3.5-27B to achieve highly competitive performance and outperform strong competitors. Our analyses confirm the efficacy of the proposed method and reveal consistent scaling behaviors across both data volume and interaction horizons.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language model, Agent, Environments Scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_96xUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents WorkForge, a pipeline that turns expert workflow blueprints into work-agent environments built from real files retrieved online. Facts extracted from each workspace are used to write the task, a hidden solution plan, and a mix of programmatic checks and LLM-judged rubrics. Trajectories from GLM 5.3 under Claude Code and OpenClaw are then used for SFT. Fine-tuned Qwen3.5 models improve on five work-agent benchmarks, and the paper adds analyses on data scale, trajectory length, harness choice and the verifier used for selection.

Critical strengths
  1. Anchoring both the task and its checks on facts pulled from a real workspace is a sensible design. The factor-premium example in the appendix shows rubrics tied to concrete file facts, which are harder to hallucinate than criteria written against a toy workspace.
  2. The gains are large and consistent across two models and five benchmarks, including APEX, which runs on a harness never used in training.
Critical weaknesses
  1. All trajectories come from GLM 5.3, which is not reported in Table 2, while GLM-5.2 is already strong on GDPval. So it is hard to tell how much of the gain comes from the environments and how much is plain distillation from a strong teacher. Using a Base model for the 35B-A3B runs makes this worse, since almost any agentic SFT data would give a large jump there.
  2. The main claim that real workspaces matter is not tested downstream. The Workspace Authenticity part only compares file counts and sizes; there is no model trained on the matched synthetic workspaces.
  3. It is unclear how much the verifiers actually matter. If I read it right, the main set has one trajectory per task and a sizable share of tasks in Figure 3(c) score near zero, so there seems to be no score-based filtering; the verifiers only play a role in the 1K ablation. I could not find any human agreement check for the rubrics or the judge, and the limitations section itself mentions limited expert calibration. The judge model is not named either.
  4. Four of the five benchmarks are run with the same harnesses used for training, and Figure 5 shows that harness match alone is worth a clear margin. Part of the main gains may come from learning harness conventions rather than work skills.Re-evaluate at least one benchmark under an unseen harness.
Questions for authors
  1. How does GLM 5.3 itself score on these five benchmarks? Was it also used for task design or judging?
  2. What happens if you train on the matched synthetic workspaces instead?
  3. Is the main training set filtered by verifier score? If so, what is the threshold?
  4. Did you spot-check any rubrics or judge scores by hand?
  5. A natural next step would be RL with these verifiers on top of SFT. This is a suggestion, not a condition for my score.
5 comments
AuthorsOpenReview ↗

Response to reviewer 96xU (Part 1/3)

Dear Reviewer 96xU,

We sincerely appreciate your detailed review. Your feedback helps us improve the paper quality. A detailed response and supplementary experiments are provided below.

The paper you have read and the responses we have provided are the result of our authors' utmost effort, dedication, and hard work. It holds great significance for us, and we sincerely hope you will reconsider our contribution.

W1 & Q1. Continual SFT of post-trained LLMs with GLM 5.3 teacher

Thank you for raising these important questions. We first report the teacher GLM 5.3 and apply WorkForge on top of the officially post-trained Qwen3.5-35B-A3B and Qwen3.5-27B, to test whether our gains are merely what any agentic SFT data gives a base model. Note that GLM 5.3 is used exclusively for trajectory rollout: task design uses DeepSeek-V4-Pro and semantic judging uses Claude Sonnet 4.6 (see W3 for full details). All backbones are fine-tuned on the full WorkForge-16K set under the same SFT recipe, so the base and post-trained runs differ only in their initialization. The results are summarized below:

ModelAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
GLM-5.354.063.548.889.353.464.0
Qwen3.5-35B-A3B18.031.423.663.94.413.6
 + WorkForge18.630.224.775.87.413.9
Qwen3.5-35B-A3B-Base5.018.611.445.50.30.2
 + WorkForge21.329.928.773.63.716.2
Qwen3.5-27B25.730.532.779.49.217.0
 + WorkForge29.938.533.982.416.933.8
  • Substantial improvements over backbones: WorkForge transfers practical work skills from GLM 5.3 across both base and post-trained checkpoints. While training from Base already surpasses the official A3B (e.g., 73.6 vs. 63.9 on GDPval), continual SFT on post-trained Qwen3.5-35B-A3B and Qwen3.5-27B delivers substantial further gains—nearly doubling 27B on JobBench (17.0→33.8) and AutomationBench (9.2→16.9), confirming that WorkForge imparts genuine work-agent capabilities into lightweight models.

W2 & Q2. Environment synthesis methods and real vs. synthetic workspaces

To further separate environment quality from GLM 5.3 distillation (W1) and directly test matched synthetic workspaces as you suggested (W2 & Q2), we conduct a controlled study fixing the teacher model (GLM 5.3 with high reasoning effort), rollout harness (OpenClaw), data volume (1K trajectories), student backbone (Qwen3.5-35B-A3B-Base), and SFT recipe (5 epochs, learning rate 1e-5, global batch size 64).

Under this setup, we compare WorkForge against previous environment synthesis methods, ClawGym [1] and EnvScaler [2], as well as a synthetic-workspace variant built from the same WorkForge scene blueprints, in which real-resource acquisition is replaced by DeepSeek-V4-Pro workspace synthesis (7.9 vs. 42.3 files/workspace; Figure 1). The results are summarized below:

MethodAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
Qwen3.5-35B-A3B-Base5.018.611.445.50.30.2
+ ClawGym17.620.820.570.14.210.2
+ EnvScaler14.321.819.170.93.212.4
+ Synthetic workspace13.815.117.364.44.28.0
+ WorkForge21.524.122.172.54.712.2

WorkForge outperforms prior environment synthesis methods and the synthetic-workspace variant on 5 of 6 benchmarks and virtually ties on JobBench (12.2 vs. 12.4), establishing two key conclusions:

  • Effectiveness of environment design: Under the same GLM 5.3 teacher, varying only the environment source still produces significant gaps (e.g., EnvScaler 14.3 vs. WorkForge 21.5 on APEX). Grounding tasks in complex, multi-file workspaces elicits deep exploration and cross-file reasoning, effectively transferring GLM 5.3's practical work skills into lightweight models.
  • Real vs. synthetic workspaces: Replacing real files with LLM-synthesized ones consistently degrades downstream performance—even dropping WildClaw below the untuned student (15.1 vs. 18.6). Realistic, multi-file workspaces are therefore essential for teaching agents to search and reason across complex files.
AuthorsOpenReview ↗

Response to reviewer 96xU (Part 2/3)

W3, Q3 & Q4. Implementation details and quality check for agent environments

Thank you for catching these omissions. We apologize that several implementation details were missing from our initial submission; we have supplemented these implementation details in our revised manuscript, and additionally report a manual quality check below:

  • Environment synthesis and task generation. Environment mapping, environment profiling, task design, and verifier design (fact verification and rubric generation) all use DeepSeek-V4-Pro.
  • Verification module. Each task averages 21.9 atomic rubrics across six dimensions: 37.3% are programmatic checks instantiated from deterministic templates, and 62.7% are semantic criteria scored on a [0, 10] scale by Claude Sonnet 4.6 and normalized to [0, 1]. GLM 5.3 is used exclusively for trajectory rollout, ensuring that no trajectory is evaluated by its generator.
  • Trajectory generation and filtering (Q3). WorkForge-16K is strictly score-filtered: from 35.7K sandboxed GLM 5.3 rollouts, we retain only well-formed trajectories with a hybrid programmatic and semantic verifier score ≥ 0.7, yielding the 16,665 training trajectories (46.7%). Figure 3(c) plots the pre-filtering distribution over all 35.7K candidates—the discarded 53.3% (including the low-score mass) are excluded from training, reflecting verifier strictness—and Table 3 confirms that this hybrid signal outperforms single-verifier filtering by +5.5–6.2 ALE-CLI points.
  • Quality check of synthesized tasks and rubrics (Q4). Following your suggestion, we randomly sample 50 tasks from WorkForge-16K and manually inspect each task specification, the coverage of its rubric set, and all 1,093 associated rubrics. Each item is independently checked by two evaluators, with disagreements resolved by a third evaluator:
AspectSampleValidityAgreement
Tasks50 tasks94.0%96.0%
Rubric sets (coverage)50 tasks92.0%90.0%
Rubrics (each item)1,093 rubrics94.5%94.3%

Across three aspects, 92.0–94.5% of checked items are valid with 90.0–96.0% agreement, establishing two conclusions:

  • Task and rubric validity: Nearly all sampled tasks are well-formed and solvable (47/50), with rubric sets comprehensively covering required deliverables (46/50)—confirming that grounding generation in observable workspace facts yields reliable tasks and rubrics at scale.
  • Verification robustness: Invalid rubrics average only 1.2 per task out of 21.9 atomic rubrics (5.5%), ensuring that task-level verification scores are dominated by valid signals.
AuthorsOpenReview ↗

Response to reviewer 96xU (Part 3/3)

W4. Generalization effect across agentic harnesses

To verify that WorkForge imparts transferable work skills rather than overfitting to training harnesses, we first note that training solely on OpenClaw trajectories already improves ALE-CLI under Claude Code from 11.4 to 16.1 (Figure 5). Moreover, since existing benchmarks often tie their sandboxes to specific harnesses (e.g., WildClawBench is built around OpenClaw), we additionally evaluate AutomationBench and JobBench alongside APEX under their respective official harnesses (none of which appears in training), reporting WorkForge's gain (Δ) over each backbone:

ModelAPEXAutomationBenchJobBench
Official HarnessReAct ToolbeltSearch-and-Execute LoopOpenCode
Qwen3.5-35B-A3B-Base (Δ)+16.3+3.4+16.0
Qwen3.5-35B-A3B (Δ)+0.6+3.0+0.3
Qwen3.5-27B (Δ)+4.2+7.7+16.8

WorkForge consistently improves both base and post-trained backbones across all three unseen official harnesses, confirming that our synthesized environments impart genuine, generalizable work-agent capabilities rather than harness-specific conventions.

Q5. On-policy RFT and RL in WorkForge environments

Guided by your advice, we evaluate on-policy rejection fine-tuning (RFT) and reinforcement learning (RL) in WorkForge environments without any external teacher supervision:

  • RFT setup. We sample trajectories directly from Qwen3.5-27B on WorkForge tasks (temperature 1.0), retain 4K error-free rollouts with a hybrid verifier score ≥ 0.7, and fine-tune Qwen3.5-27B.
  • RL setup. Starting from WorkForge-A3B, we run multi-turn GRPO for 80 steps using the continuous hybrid verifier score in [0, 1] as the online reward (batch size 8, group size 8, temperature 0.9, lr 1e-6, clip [0.2, 0.28] without KL penalty, up to 256K tokens and 200 turns).
ModelGDPvalJobBenchAutomationBench
Qwen3.5-27B79.417.09.2
 + RFT on WorkForge82.818.99.8
WorkForge-A3B73.616.23.7
 + RL on WorkForge78.320.34.1

Both RFT and RL consistently improve over their respective initializations without any external teacher supervision. Within WorkForge environments, on-policy RFT lifts Qwen3.5-27B across all benchmarks—reaching 82.8 on GDPval. Meanwhile, multi-turn GRPO further boosts the already distilled WorkForge-A3B by +4.7 on GDPval and +4.1 on JobBench.

Thank you once again for your rigorous and constructive feedback, which has helped us make the manuscript substantially stronger. All references below have been cited and discussed in our revised manuscript. We are happy to discuss any remaining questions and would be grateful if you would consider raising your assessment.

References

[1] ClawGym: A Scalable Framework for Building Effective Claw Agents.

[2] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis.

Reviewer_ZUipOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WorkForge, a way to create training environments for work agents using real files and resources from the web. It starts with workflows written by experts, retrieves relevant files from sources like Kaggle and GitHub, and uses agents to build workspaces and create tasks with verifiable answers. A teacher model then solves these tasks, and the resulting trajectories are used to fine-tune Qwen models. The authors report improvements across several work-agent benchmarks. They also find that adding more training data, using longer trajectories, and combining programmatic checks with an LLM judge lead to better results.

Critical strengths
  • Using real files and workspaces to create tasks is a good idea. It makes the tasks feel closer to actual work than environments built entirely from scratch. Checking the files before writing the task and its evaluation criteria is also a useful step.

  • The results are promising across several benchmarks. The 27B model also comes close to much larger open models on GDPval. If released, the dataset and pipeline could be useful for future work.

Critical weaknesses
  • The training data comes from a strong teacher model, so the gains could largely be due to distillation. The paper does not compare against another environment-generation method using the same base model, data size, and training setup. More importantly, the authors argue that real workspaces are better than synthetic ones, but the authenticity ablation only looks at file counts and sizes. There is no direct comparison of models trained on real and synthetic workspaces.

  • The paper relies on programmatic checks and an LLM judge, but does not report whether humans checked their reliability or how often their judgments are correct. The authors acknowledge this limitation themselves. Also, the verifiers are used to filter training examples rather than provide rewards during RL, so the paper does not fully demonstrate the benefits of verifiable environments. It is also unclear whether the retrieved resources overlap with the benchmark tasks.

Questions for authors
  • How does WorkForge compare with other environment-generation methods when the model, training data size, and training setup are kept the same?

  • Can you compare training on real workspaces against LLM-generated workspaces built from the same blueprints? This seems important for supporting the main claim.

  • Did you check whether the retrieved files or resources overlap with benchmark tasks, particularly GDPval?

  • Did human evaluators check any of the rubrics or LLM-judge scores? If so, how closely did they agree with the automated evaluations?

  • GDPval and ALE use Claude Code for evaluation, which is also used to generate much of the training data. Since Figure 5 suggests that the harness matters, how much of the improvement comes from learning to use this particular setup?

  • Have you tried using the verifiers as rewards during RL, rather than only for filtering SFT data?

6 comments
AuthorsOpenReview ↗

Response to reviewer ZUip (Part 1/3)

Dear Reviewer ZUip,

We sincerely thank you for your positive assessment and constructive questions.

Both our updated manuscript and the responses we present below carry the full commitment, care, and hard work of all authors. This work means a great deal to us, and we sincerely hope our efforts earn your continued recognition and support. Below, we address each point in turn.

W1, Q1 & Q2. Controlled study of environment synthesis methods and workspace authenticity

We appreciate your suggestions on comparing against prior environment-generation methods under identical settings (Q1) and directly testing real vs. LLM-generated workspaces built from the same blueprints (Q2). Guided by your feedback, we conduct a controlled study fixing the teacher model (GLM 5.3 with high reasoning effort), rollout harness (OpenClaw), data volume (1K trajectories), student backbone (Qwen3.5-35B-A3B-Base), and SFT recipe (5 epochs, learning rate 1e-5, global batch size 64).

Under this setup, we compare WorkForge against prior environment synthesis methods, ClawGym [1] and EnvScaler [2], as well as a synthetic-workspace variant built from the same WorkForge scene blueprints, in which real-resource acquisition is replaced by DeepSeek-V4-Pro workspace synthesis (7.9 vs. 42.3 files/workspace; Figure 1). The results are summarized below:

MethodAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
Student Model5.018.611.445.50.30.2
+ ClawGym17.620.820.570.14.210.2
+ EnvScaler14.321.819.170.93.212.4
+ Synthetic workspace13.815.117.364.44.28.0
+ WorkForge21.524.122.172.54.712.2

WorkForge outperforms prior environment synthesis methods and the synthetic-workspace variant on 5 of 6 benchmarks and virtually ties on JobBench (12.2 vs. 12.4), establishing two key conclusions:

  • Effectiveness of environment design: Under the same teacher, varying only the environment source still produces significant gaps (e.g., EnvScaler 14.3 vs. WorkForge 21.5 on APEX). WorkForge anchors long-horizon tasks in complex, multi-file workspaces—requiring the agent to execute deep exploration, cross-file reasoning, and artifact synthesis.
  • Real vs. synthetic workspaces: Replacing real files with LLM-synthesized ones consistently degrades downstream performance—even dropping WildClaw below the untuned student (15.1 vs. 18.6). Realistic, multi-file workspaces are therefore essential for teaching agents to search and reason across complex files.

W2 & Q3. Decontamination analysis

Thank you for raising this question. To verify that retrieved resources do not overlap with downstream evaluations, we perform systematic string-matching deduplication across all workspace files, task instructions, and training trajectories in WorkForge-16K against the reference artifacts and prompts of all evaluation benchmarks (with particular attention to GDPval). No file or task overlap is found, confirming that WorkForge's downstream gains stem from generalizable work-agent capabilities rather than benchmark leakage.

AuthorsOpenReview ↗

Response to reviewer ZUip (Part 2/3)

W2 & Q4. Implementation details and quality check for agent environments

Thank you for raising this important point. We apologize that several implementation details were missing from our initial submission; we have supplemented these implementation details in our revised manuscript, and additionally report a manual quality check below:

  • Environment synthesis and task generation. Environment mapping, environment profiling, task design, and verifier design (fact verification and rubric generation) all use DeepSeek-V4-Pro.
  • Verification module. Each task averages 21.9 atomic rubrics across six dimensions: 37.3% are programmatic checks instantiated from deterministic templates, and 62.7% are semantic criteria scored on a [0, 10] scale by Claude Sonnet 4.6 and normalized to [0, 1]. GLM 5.3 is used exclusively for trajectory rollout, ensuring that no trajectory is evaluated by its generator.
  • Trajectory generation and filtering. WorkForge-16K is strictly score-filtered: from 35.7K sandboxed GLM 5.3 rollouts, we retain only well-formed trajectories with a hybrid programmatic and semantic verifier score ≥ 0.7, yielding the 16,665 training trajectories (46.7%). Figure 3(c) plots the pre-filtering distribution over all 35.7K candidates—the discarded 53.3% (including the low-score mass) are excluded from training, reflecting verifier strictness—and Table 3 confirms that this hybrid signal outperforms single-verifier filtering by +5.5–6.2 ALE-CLI points.
  • Quality check of synthesized tasks and rubrics. Following your suggestion, we randomly sample 50 tasks from WorkForge-16K and manually inspect each task specification, the coverage of its rubric set, and all 1,093 associated rubrics. Each item is independently checked by two evaluators, with disagreements resolved by a third evaluator:
AspectSampleValidityAgreement
Tasks50 tasks94.0%96.0%
Rubric sets (coverage)50 tasks92.0%90.0%
Rubrics (each item)1,093 rubrics94.5%94.3%

Across three aspects, 92.0–94.5% of checked items are valid with 90.0–96.0% agreement, establishing two conclusions:

  • Task and rubric validity: Nearly all sampled tasks are well-formed and solvable (47/50), with rubric sets comprehensively covering required deliverables (46/50)—confirming that grounding generation in observable workspace facts yields reliable tasks and rubrics at scale.
  • Verification robustness: Invalid rubrics average only 1.2 per task out of 21.9 atomic rubrics (5.5%), ensuring that task-level verification scores are dominated by valid signals.
AuthorsOpenReview ↗

Response to reviewer ZUip (Part 3/3)

Q5. Generalization effect across agentic harnesses

Thank you for this question regarding the Claude Code setup. To verify that WorkForge imparts transferable capabilities rather than conventions specific to Claude Code, we note that training exclusively on OpenClaw trajectories already lifts ALE-CLI under Claude Code from 11.4 to 16.1 (+4.7) in Figure 5, as well as lifting GDPval from 45.5 to 72.5 and ALE-CLI to 22.1 in our 1K OpenClaw study above. More broadly, we evaluate across APEX, AutomationBench, and JobBench under their respective official harnesses (none of which appears in WorkForge training), reporting WorkForge's gain (Δ) over each backbone:

ModelAPEXAutomationBenchJobBench
Official HarnessReAct ToolbeltSearch-and-Execute LoopOpenCode
Qwen3.5-35B-A3B-Base (Δ)+16.3+3.4+16.0
Qwen3.5-35B-A3B (Δ)+0.6+3.0+0.3
Qwen3.5-27B (Δ)+4.2+7.7+16.8

WorkForge consistently improves both base and post-trained backbones across all three unseen official harnesses, confirming that our synthesized environments impart transferable work-agent capabilities rather than conventions specific to Claude Code.

W2 & Q6. On-policy RFT and RL in WorkForge environments

We appreciate your insightful suggestion to evaluate our verifiers beyond SFT data filtering. Guided by your feedback, we run on-policy rejection fine-tuning (RFT) and reinforcement learning (RL) in WorkForge environments without any external teacher supervision:

  • RFT setup. We sample trajectories directly from Qwen3.5-27B on WorkForge tasks (temperature 1.0), retain 4K error-free rollouts with a hybrid verifier score ≥ 0.7, and fine-tune Qwen3.5-27B.
  • RL setup. Starting from WorkForge-A3B, we run multi-turn GRPO for 80 steps using the continuous hybrid verifier score in [0, 1] as the online reward (batch size 8, group size 8, temperature 0.9, lr 1e-6, clip [0.2, 0.28] without KL penalty, up to 256K tokens and 200 turns).
ModelGDPvalJobBenchAutomationBench
Qwen3.5-27B79.417.09.2
 + RFT on WorkForge82.818.99.8
WorkForge-A3B73.616.23.7
 + RL on WorkForge78.320.34.1

Both RFT and RL consistently improve over their respective initializations without any external teacher supervision. Within WorkForge environments, on-policy RFT lifts Qwen3.5-27B across all benchmarks—reaching 82.8 on GDPval. Meanwhile, multi-turn GRPO further boosts the already distilled WorkForge-A3B by +4.7 on GDPval and +4.1 on JobBench.

Finally, as you noted regarding community impact, we will release the WorkForge pipeline together with the 16.7K verifiable environments, task packages, and verifiers to support future SFT and RL research on work agents. All references below have been cited and discussed in our revised manuscript. We are happy to clarify further if needed, and we would be grateful if you would consider raising your assessment.

References

[1] ClawGym: A Scalable Framework for Building Effective Claw Agents.

[2] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis.

Reviewer_TdjUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes WorkForge, a pipeline that builds work-agent environments from real files downloaded from the web, with tasks and rubrics grounded in facts extracted from each workspace. The authors build 16.7K environments across 40 domains, collect one GLM 5.3 trajectory per task, and fine-tune Qwen3.5-35B-A3B-Base and Qwen3.5-27B on them. They report gains on five benchmarks, eg GDPval from 45.5 to 73.6 for the base model.

Critical strengths
  1. The paper tackles an important and timely problem of training environments for long-horizon work agents.
  2. The evaluation is broad, with five recent benchmarks and many baselines.
Critical weaknesses
  1. Headline gains are compared from a base model: For example, GDPval 45.5 to 73.6 is mostly post-training from pretrained model. While a comparison with the post-trained Qwen3.5-35B-A3B is not required, WorkForge should also be applied on top of it to separate its effect from mid-training or sharpening effects. For instance, on Qwen3.5-27B the gains are small (+3.0 GDPval, +1.2 ALE) and from a single run.
  2. Gains are not separated from the GLM 5.3 teacher: The baselines differ in base model, teacher and harness. A same-teacher comparison with other task sources (eg NexForge, ClawGym, or the synthetic workspaces in Section 3.3) is missing, and could be run at the 1K scale of Table 3. NexForge is the closest prior work (real files, same base model, GDPval) and is not compared.
  3. Real vs synthetic workspaces is never tested: Section 3.3 only compares file counts and sizes. No model is trained on the synthetic workspaces.
  4. Verifiers are not validated or used: There is no human check of tasks, rubrics or the LLM judge, and the judge model is not stated. The main run uses all 16,665 trajectories (Appendix D.1), while only about 47% of tasks score 0.7 or higher (Figure 3c), even though Table 3 shows that filtering helps.

Minor:

  1. Baselines look under-evaluated: Nemotron-Terminal-32B gets 5.8 on GDPval, far below the untuned base model (45.5), while these models do much better in their own papers. They are run in the same harnesses used for WorkForge's training data, and Figure 5 shows that harness alignment alone is worth about 8 ALE points. So calling ClawGym-30A3 the "previous SOTA" (line 311) is misleading.
Questions for authors

See Weaknesses Above

5 comments
AuthorsOpenReview ↗

Response to reviewer TdjU (Part 1/3)

Dear Reviewer TdjU,

We sincerely thank you for your thorough review and constructive feedback. We have taken each suggestion seriously to make our study substantially more rigorous.

Guided by your feedback, we have conducted extensive new experiments—including controlled synthesis comparisons, continual training on post-trained models, harness generalization, and a manual quality check. We are deeply committed to the scientific rigor of this work and hope these comprehensive results merit a positive reassessment. Below, we address each of your concerns in full detail.

W1. Experiments on post-trained LLMs

We clarify the rationale behind our base-model setting and present the continual post-training results you suggested:

Experimental setting regarding the base LLM. Starting from Qwen3.5-35B-A3B-Base (following NexForge [1]) enables a clean comparison against the official Qwen3.5-35B-A3B from identical initial weights, whereas continuing SFT on an already post-trained checkpoint can introduce recipe mismatch and interference from unknown prior training.

Continual training on post-trained LLMs. Following your constructive advice, we apply WorkForge directly on top of the official post-trained Qwen3.5-35B-A3B, and report it alongside the post-trained Qwen3.5-27B, the teacher model GLM 5.3, and prior open-source agents:

ModelAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
GLM-5.354.063.548.889.353.464.0
NexForge-A3B [1]23.222.227.076.09.312.6
ClawGym-A3B [2]2.514.99.135.00.33.2
EnvScaler-8B [3]4.913.70.715.00.10.7
Qwen3.5-35B-A3B18.031.423.663.94.413.6
 + WorkForge18.630.224.775.87.413.9
Qwen3.5-35B-A3B-Base5.018.611.445.50.30.2
 + WorkForge21.329.928.773.63.716.2
Qwen3.5-27B25.730.532.779.49.217.0
 + WorkForge29.938.533.982.416.933.8

We highlight three takeaways from these comparisons:

  • Post-Train recipe: WorkForge-A3B (initialized from Base) already surpasses the official post-trained Qwen3.5-35B-A3B on 4 of 6 benchmarks. Furthermore, continual SFT on the official Qwen3.5-35B-A3B with the same recipe improves 5 of 6 benchmarks (+11.9 on GDPval to 75.8), while minor interference from prior alignment (e.g., on WildClaw) illustrates why Base provides a cleaner initialization.
  • Substantial and consistent gains on 27B: On Qwen3.5-27B, the narrower GDPval margin (+3.0) reflects its strong 79.4 baseline, whereas WorkForge nearly doubles performance on JobBench (17.0→33.8) and AutomationBench (9.2→16.9) alongside +8.0 on WildClaw. WorkForge-27B brings significant benefits, showing consistent gains across diverse tasks.
  • Comparison with open-source agents: WorkForge-27B leads all open-source baselines across all six benchmarks. On the 35B-A3B backbone, WorkForge-A3B substantially outperforms ClawGym-A3B and EnvScaler-8B, and achieves competitive performance with NexForge-A3B [1] using nearly 4× fewer trajectories (16.7K vs. 65.2K).
AuthorsOpenReview ↗

Response to reviewer TdjU (Part 2/3)

W2 & W3. Controlled study of environment synthesis methods and workspace authenticity

We conduct a controlled study fixing the teacher model (GLM 5.3 with high reasoning effort), rollout harness (OpenClaw), data volume (1K trajectories), student backbone (Qwen3.5-35B-A3B-Base), and SFT recipe (5 epochs, learning rate 1e-5, global batch size 64).

Under this setup, we compare WorkForge against ClawGym [2], EnvScaler [3], and a synthetic-workspace variant built from the exact same WorkForge scene blueprints, in which only real-resource acquisition is replaced by DeepSeek-V4-Pro workspace synthesis (7.9 vs. 42.3 files/workspace; Figure 1). Since NexForge's executable task packages are not publicly released for aligned 1K rollouts, we compare against its released model in W1 and discuss it in Section 4. The results are summarized below:

MethodAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
Student Model5.018.611.445.50.30.2
+ ClawGym17.620.820.570.14.210.2
+ EnvScaler14.321.819.170.93.212.4
+ Synthetic workspace13.815.117.364.44.28.0
+ WorkForge21.524.122.172.54.712.2

WorkForge outperforms prior environment synthesis methods and the synthetic-workspace variant on 5 of 6 benchmarks and virtually ties on JobBench (12.2 vs. 12.4), establishing two key conclusions:

  • Effectiveness of environment design: Under the same teacher, varying only the environment source still produces significant gaps (e.g., EnvScaler 14.3 vs. WorkForge 21.5 on APEX). WorkForge anchors long-horizon tasks in complex, multi-file workspaces—requiring the agent to execute deep exploration, cross-file reasoning, and artifact synthesis.
  • Real vs. synthetic workspaces: Replacing real files with LLM-synthesized ones consistently degrades downstream performance—even dropping WildClaw below the untuned student (15.1 vs. 18.6). Realistic, multi-file workspaces are therefore essential for teaching agents to search and reason across complex files.

W4. Implementation details and quality check

Thank you for pointing this out. We apologize that several implementation details were missing from our initial submission, which caused confusion regarding main-set filtering and the judge configuration. We have supplemented these implementation details in our revised manuscript, and additionally conduct a manual quality check below:

  • Environment synthesis and task generation. Environment mapping, environment profiling, task design, and verifier design (fact verification and rubric generation) all use DeepSeek-V4-Pro.
  • Verification module. Each task averages 21.9 atomic rubrics across six dimensions: 37.3% are programmatic checks instantiated from deterministic templates, and 62.7% are semantic criteria scored on a [0, 10] scale by Claude Sonnet 4.6 and normalized to [0, 1]. GLM 5.3 is used exclusively for trajectory rollout, ensuring that no trajectory is evaluated by its generator.
  • Trajectory generation and filtering. WorkForge-16K is strictly score-filtered: from 35.7K sandboxed GLM 5.3 rollouts, we retain only well-formed trajectories with a hybrid programmatic and semantic verifier score ≥ 0.7, yielding the 16,665 training trajectories (46.7%). Figure 3(c) plots the pre-filtering distribution over all 35.7K candidates—the discarded 53.3% (including the low-score mass) are excluded from training, reflecting verifier strictness—and Table 3 confirms that this hybrid signal outperforms single-verifier filtering by +5.5–6.2 ALE-CLI points.
  • Quality check of synthesized tasks and rubrics. Following your suggestion, we randomly sample 50 tasks from WorkForge-16K and manually inspect each task specification, the coverage of its rubric set, and all 1,093 associated rubrics. Each item is independently checked by two evaluators, with disagreements resolved by a third evaluator:
AspectSampleValidityAgreement
Tasks50 tasks94.0%96.0%
Rubric sets (coverage)50 tasks92.0%90.0%
Rubrics (each item)1,093 rubrics94.5%94.3%

Across three aspects, 92.0–94.5% of checked items are valid with 90.0–96.0% agreement, establishing two conclusions:

  • Task and rubric validity: Nearly all sampled tasks are well-formed and solvable (47/50), with rubric sets comprehensively covering required deliverables (46/50)—confirming that grounding generation in observable workspace facts yields reliable tasks and rubrics at scale.
  • Verification robustness: Invalid rubrics average only 1.2 per task out of 21.9 atomic rubrics (5.5%), ensuring that task-level verification scores are dominated by valid signals.
AuthorsOpenReview ↗

Response to reviewer TdjU (Part 3/3)

W5. Performance gap, harness generalization, and terminology revision

Thank you for these considerate observations. We address each part of this concern in order:

  • Performance gap in specialized competitors. Nemotron-Terminal-32B [4] is specialized for synthetic CLI tasks, whereas GDPval requires multi-file office deliverables (.xlsx, .pptx, .docx, .pdf) outside its target domain. This sacrifices general-purpose work-agent capabilities, accounting for its sharp drop on work-agent benchmarks.
  • Terminology revision. We appreciate your point that calling ClawGym-30A3 (35.0) "previous SOTA" on GDPval is imprecise; we have removed this phrasing in the paper.
  • Generalization effect across agentic harnesses. To verify that WorkForge imparts transferable capabilities rather than harness-specific conventions, we note that in Figure 5, training exclusively on OpenClaw trajectories still improves ALE-CLI under Claude Code from 11.4 to 16.1 (+4.7). Furthermore, we evaluate APEX, AutomationBench, and JobBench under their respective official harnesses (none of which is seen during WorkForge training), reporting WorkForge's gain (Δ) over each backbone:
ModelAPEXAutomationBenchJobBench
Official HarnessReAct ToolbeltSearch-and-Execute LoopOpenCode
Qwen3.5-35B-A3B-Base (Δ)+16.3+3.4+16.0
Qwen3.5-35B-A3B (Δ)+0.6+3.0+0.3
Qwen3.5-27B (Δ)+4.2+7.7+16.8

WorkForge consistently improves both base and post-trained backbones on these unseen official harnesses, confirming that our synthesized environments impart generalizable work-agent skills rather than harness-specific conventions.

We sincerely appreciate your time and guidance in helping us strengthen this work, and we are happy to provide any further details. All references below have been cited and discussed in our revised manuscript. If our responses address your concerns, we would be grateful if you would consider raising your assessment.

References

[1] NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs.

[2] ClawGym: A Scalable Framework for Building Effective Claw Agents.

[3] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis.

[4] On Data Engineering for Scaling LLM Terminal Capabilities.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response. Part [1/3]

Dear Reviewers, the Area Chair, the Senior Area Chairs and Program Chairs,

We sincerely thank you for your valuable time, thoughtful assessments, and constructive feedback. We are encouraged by your recognition of the importance of scaling long-horizon work-agent environments.

Following your suggestions, we have substantially strengthened the manuscript with new controlled comparisons, harness generalization studies, on-policy RL/RFT experiments, and a quality check. All new quantitative results are reported in the tables below, while updated figures and experimental details are incorporated into the revised manuscript. Below, we address the important questions.

#1 Controlled study of environment synthesis methods and workspace authenticity.

Following the suggestions of reviewers, we conduct a controlled study to isolate the effect of environment synthesis from teacher distillation. Specifically, we fix the teacher model (GLM 5.3 with high reasoning effort), rollout harness (OpenClaw), data volume (1K trajectories), student backbone (Qwen3.5-35B-A3B-Base), and SFT recipe (5 epochs, learning rate 1e-5, global batch size 64).

Under this setup, we compare WorkForge against prior environment synthesis methods, ClawGym [1] and EnvScaler [2], as well as a synthetic-workspace variant built from the same WorkForge scene blueprints, in which real-resource acquisition is replaced by DeepSeek-V4-Pro workspace synthesis (7.9 vs. 42.3 files/workspace; Figure 1). The results are summarized below:

MethodAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
Student Model5.018.611.445.50.30.2
+ ClawGym17.620.820.570.14.210.2
+ EnvScaler14.321.819.170.93.212.4
+ Synthetic workspace13.815.117.364.44.28.0
+ WorkForge21.524.122.172.54.712.2

WorkForge outperforms prior environment synthesis methods and the synthetic-workspace variant on 5 of 6 benchmarks and virtually ties on JobBench (12.2 vs. 12.4), establishing two key conclusions:

  • Effectiveness of environment design: Under the same teacher, varying only the environment source still produces significant gaps (e.g., EnvScaler 14.3 vs. WorkForge 21.5 on APEX). WorkForge anchors long-horizon tasks in complex, multi-file workspaces—requiring the agent to execute deep exploration, cross-file reasoning, and artifact synthesis.
  • Real vs. synthetic workspaces: Replacing real files with LLM-synthesized ones consistently degrades downstream performance—even dropping WildClaw below the untuned student (15.1 vs. 18.6). Realistic, multi-file workspaces are therefore essential for teaching agents to search and reason across complex files.

#2 Experiments of additional models on work-agent benchmarks.

Following the suggestions of three reviewers, we include additional competitors: (1) the teacher model GLM 5.3; (2) open-source agentic models, including ClawGym, EnvScaler and NexForge [3]; and (3) continual fine-tuning on the post-trained Qwen3.5-35B-A3B. The results are summarized below:

ModelAPEXWildClawALE-CLIGDPvalAutomationBenchJobBench
GLM-5.354.063.548.889.353.464.0
NexForge-A3B23.222.227.076.09.312.6
ClawGym-A3B2.514.99.135.00.33.2
EnvScaler-8B4.913.70.715.00.10.7
Qwen3.5-35B-A3B18.031.423.663.94.413.6
  + WorkForge18.630.224.775.87.413.9
Qwen3.5-35B-A3B-Base5.018.611.445.50.30.2
  + WorkForge21.329.928.773.63.716.2
Qwen3.5-27B25.730.532.779.49.217.0
  + WorkForge29.938.533.982.416.933.8

We highlight two takeaways from these comparisons:

  • Substantial improvements over backbones: WorkForge transfers practical work skills from GLM 5.3 across both base and post-trained checkpoints. While training from Qwen3.5-35B-A3B-Base already surpasses the official A3B (e.g., 73.6 vs. 63.9 on GDPval), continual SFT on post-trained Qwen3.5-35B-A3B and Qwen3.5-27B delivers substantial further gains—nearly doubling 27B on JobBench (17.0→33.8) and AutomationBench (9.2→16.9), confirming that WorkForge imparts genuine work-agent capabilities into lightweight models.
  • Comparison with open-source agents: WorkForge-27B leads all open-source baselines across all six benchmarks. On the 35B-A3B backbone, WorkForge-A3B substantially outperforms ClawGym-A3B and EnvScaler-8B, and achieves competitive performance with NexForge-A3B [3] using nearly 4× fewer trajectories (16.7K vs. 65.2K).
AuthorsOpenReview ↗

General Response. Part [2/3]

#3 Generalization effect across agentic harnesses.

To verify that WorkForge imparts transferable work-agent ability rather than overfitting to its training harnesses (Claude Code and OpenClaw), we note that training exclusively on OpenClaw trajectories already lifts ALE-CLI under Claude Code from 11.4 to 16.1 (Figure 5). More broadly, we evaluate across APEX, AutomationBench, and JobBench under their respective official harnesses (none of which appears in training), reporting WorkForge's gain (Δ) over each backbone:

ModelAPEXAutomationBenchJobBench
Official HarnessReAct ToolbeltSearch-and-Execute LoopOpenCode
Qwen3.5-35B-A3B-Base (Δ)+16.3+3.4+16.0
Qwen3.5-35B-A3B (Δ)+0.6+3.0+0.3
Qwen3.5-27B (Δ)+4.2+7.7+16.8

WorkForge consistently improves both base and post-trained backbones across all three unseen official harnesses (e.g., +16.8 on JobBench and +7.7 on AutomationBench for 27B). On these three benchmarks, all baselines in #1 and #2 are likewise evaluated under the same official harnesses, so WorkForge gains no advantage from its training harnesses in those comparisons.

#4 On-policy RFT and RL in WorkForge environments.

Following the reviewers' suggestions, we conduct on-policy rejection fine-tuning (RFT) and reinforcement learning (RL) in WorkForge environments without any external teacher supervision:

  • RFT setup. We sample trajectories directly from Qwen3.5-27B on WorkForge tasks (temperature 1.0), retain 4K error-free rollouts with a hybrid verifier score ≥ 0.7, and fine-tune Qwen3.5-27B.
  • RL setup. Starting from WorkForge-A3B, we run multi-turn GRPO for 80 steps using the continuous hybrid verifier score in [0, 1] as the online reward (batch size 8, group size 8, temperature 0.9, lr 1e-6, clip [0.2, 0.28] without KL penalty, up to 256K tokens and 200 turns).
ModelGDPvalJobBenchAutomationBench
Qwen3.5-27B79.417.09.2
 + RFT on WorkForge82.818.99.8
WorkForge-A3B73.616.23.7
 + RL on WorkForge78.320.34.1

Both RFT and RL consistently improve over their respective initializations without any external teacher supervision. Within WorkForge environments, on-policy RFT lifts Qwen3.5-27B across all benchmarks—reaching 82.8 on GDPval. Meanwhile, multi-turn GRPO further boosts the already distilled WorkForge-A3B by +4.7 on GDPval and +4.1 on JobBench.

AuthorsOpenReview ↗

General Response. Part [3/3]

#5 Implementation details of environment synthesis, verification, and trajectory filtering.

We apologize that several implementation details were missing from our initial submission. Following the reviewers' suggestions, we have supplemented these details in the revised manuscript and summarize them below:

  • Environment synthesis and task generation. Environment mapping, environment profiling, task design, and verifier design (fact verification and rubric generation) all use DeepSeek-V4-Pro.
  • Verification module. Each task averages 21.9 atomic rubrics across six dimensions: 37.3% are programmatic checks instantiated from deterministic templates, and 62.7% are semantic criteria scored on a [0, 10] scale by Claude Sonnet 4.6 and normalized to [0, 1]. GLM 5.3 is used exclusively for trajectory rollout, ensuring that no trajectory is evaluated by its generator.
  • Trajectory generation and filtering. WorkForge-16K is strictly score-filtered: from 35.7K sandboxed GLM 5.3 rollouts, we retain only well-formed trajectories with a hybrid programmatic and semantic verifier score ≥ 0.7, yielding the 16,665 training trajectories (46.7%). Figure 3(c) plots the pre-filtering distribution over all 35.7K candidates—the discarded 53.3% (including the low-score mass) are excluded from training, reflecting verifier strictness—and Table 3 confirms that this hybrid signal outperforms single-verifier filtering by +5.5–6.2 ALE-CLI points.

#6 Quality check of synthesized tasks and rubrics.

Following the suggestions of Reviewers TdjU, ZUip, and 96xU, we randomly sample 50 tasks from WorkForge-16K and manually inspect each task specification, the coverage of its rubric set, and all 1,093 associated rubrics. Each item is independently checked by two evaluators, with disagreements resolved by a third evaluator. The results are summarized below:

AspectSampleValidityAgreement
Tasks50 tasks94.0%96.0%
Rubric sets (coverage)50 tasks92.0%90.0%
Rubrics (each item)1,093 rubrics94.5%94.3%

Across three aspects, 92.0–94.5% of checked items are valid with 90.0–96.0% agreement, establishing two conclusions:

  • Task and rubric validity: Nearly all sampled tasks are well-formed and solvable (47/50), with rubric sets comprehensively covering required deliverables (46/50)—confirming that grounding generation in observable workspace facts yields reliable tasks and rubrics at scale.
  • Verification robustness: Invalid rubrics average only 1.2 per task out of 21.9 atomic rubrics (5.5%), ensuring that task-level verification scores are dominated by valid signals.

References

[1] ClawGym: A Scalable Framework for Building Effective Claw Agents.

[2] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis.

[3] NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.