acceptodds
Under review as a conference paper at ICLR 2027

WorkGenesis: Building the Worlds That Teach Agents to Work

Abstract

The ability of Large Language Model (LLM) agents to complete daily and professional work is receiving increasing attention. Training such agents requires realistic work scenarios. Expert-authored occupational work is costly and slow to produce, while unconstrained synthesis often yields tasks with weak factual grounding or internally inconsistent requirements. To bridge this gap, we introduce WorkGenesis, a framework that constructs executable occupational work from real-world artifacts through two core technical innovations: (1) Evidence-Based Work Construction, which grounds each unit of work in real-world evidence by retrieving public files guided by O*NET occupational knowledge and synthesizing the surrounding context, companion materials, work request, and itemwise rubric around them; and (2) Execution-Guided Consistency Verification, which renders a reference deliverable inside the constructed work, attributes every unsatisfied rubric item to the agent, the task, or the rubric, and uses task and rubric defects as feedback to iteratively repair the work until it passes the audit. Experimental results demonstrate that WorkGenesis-35B, trained with simple supervised fine-tuning (SFT) on only 20K units of work synthesized by WorkGenesis, achieves the highest scores among all comparable-scale baselines on the five reported metrics across GDPvalAA-v2, APEX-Agents-AA, and JobBench (31.00 versus 24.79 average score), and even surpasses frontier models such as the 1.6T DeepSeek-V4-Pro-Preview. These results show that WorkGenesis provides scalable training data for working agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, agentic work

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4CAbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorkGenesis is a framework for constructing occupational training tasks grounded in real-world files. Guided by O*NET occupational information, it retrieves public files and synthesizes supporting materials, task instructions, and evaluation rubrics. An execution-based verification loop then attributes unmet rubric criteria to agent, task, or rubric errors and iteratively repairs task and rubric defects while preserving the original source files. The authors construct 20K tasks covering 65 occupations, collect execution trajectories from GLM-5.2, and use them to fine-tune Qwen3.6-35B-A3B. The resulting WorkGenesis-35B improves over the base model across all five reported metrics on GDPval-AA, APEX-Agents, and JobBench, supporting the training utility of the generated data.

Critical strengths

A practical framework for constructing occupational training data. WorkGenesis combines public source files, synthesized supporting materials, task instructions, and rubrics into executable work scenarios. Its coverage of 65 occupations and diverse file formats makes it a useful contribution toward scaling training data for agents that must produce concrete professional deliverables.

Consistent downstream improvements across multiple benchmarks. Fine-tuning Qwen3.6-35B-A3B on the generated trajectories improves all five reported metrics across GDPval-AA, APEX-Agents, and JobBench. For example, APEX-Agents pass@1 increases from 13.4% to 25.2%, and JobBench Main rises from 14.99 to 25.19. These results provide meaningful evidence of the data’s training utility, even though the contributions of individual construction components remain unresolved.

Critical weaknesses

Limited methodological novelty and insufficient evidence for the distinctive components. The framework largely combines established ideas in grounded data synthesis and execution-based verification. Its most distinctive components—error attribution and targeted repair—are not evaluated against simpler alternatives, such as filtering failed tasks or directly revising them without explicit attribution. The downstream improvements establish the utility of the overall training pipeline, but do not demonstrate the added value of these components, weakening the paper’s central methodological claim.

Insufficient evidence of the constructed tasks’ real-world grounding and quality. The paper reports file-format diversity but not the relative proportions of retrieved and synthetic materials, or whether real-world files supply essential task evidence rather than background context. Moreover, reaching zero task/rubric defects under the framework’s own model-based audit does not independently establish professional realism or correctness. Although manual inspection is mentioned, its scope and findings are insufficiently documented. These gaps make it difficult to assess how reliably the framework produces the high-quality occupational tasks it claims.

Questions for authors

Value of error attribution and targeted repair. A controlled comparison with simpler alternatives—filtering failed tasks or revising them without explicit attribution—would help establish the contribution of this design. Using the same teacher and comparable training budgets, does targeted repair lead to better task quality and downstream performance?

Real-world grounding. Please report the proportions of retrieved and synthetic materials and their distribution across tasks. A few representative examples would also help clarify whether real-world files provide essential evidence for completing the tasks or mainly serve as background context.

Independent quality validation. Please provide more detail on the manual inspection, including the sample size, reviewers’ expertise, and findings. A blinded comparison before and after repair would help assess whether the system correctly attributes failures, resolves genuine defects, and preserves task difficulty.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4CAb(1/2)

Thank you for your insightful review and constructive feedback. Below is our point-by-point response to your questions and comments.


Q1 & W1: Value of error attribution and targeted repair: comparison with simpler alternatives and downstream impact.

Response:
Thank you for this suggestion. To establish the contribution of error attribution and targeted repair, we have supplemented (1) a controlled comparison against filtering failed tasks and (2) a Repair-vs-No-Repair ablation of Execution-Guided Consistency Verification.

1. Comparison with Simpler Alternatives (Filtering Failed Tasks)

From our initial synthesized task pool, we selected 5K tasks that exhibited zero Rubric or Task defects prior to repair (Filtering), and compared SFT performance on Qwen3.6-35B-A3B against 5K and 20K tasks produced by the full WorkGenesis pipeline (Table 1).

Table 1: SFT comparison between failed-task filtering and full WorkGenesis on Qwen3.6-35B-A3B.

MethodTraining DataGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Filtering (Pass-0 Clean Only)5K126881.1120.6467.16
WorkGenesis (Ours)5K130181.8622.5668.93
WorkGenesis (Ours)20K135286.6625.1970.06

As shown in Table 1, under the same 5K data budget, WorkGenesis consistently outperforms simple filtering across all metrics (+33 Elo on GDPvalAA-v2, +1.92 on JobBench Main, and +1.77 on JobBench Easy), and scales further to 20K tasks. This is because naive filtering systematically discards complex, multi-constraint tasks, which naturally require iterative refinement to align prompts, files, and rubrics, leaving behind disproportionately simple tasks, whereas revising without explicit attribution risks altering valid Agent-level challenges. Therefore, Execution-Guided Consistency Verification with explicit attribution is essential for preserving complex occupational scenarios and scaling diverse, high-quality training data.

2. Repair-vs-No-Repair Experiment

Targeted repair also leads to substantially stronger downstream performance, as demonstrated by our Repair-vs-No-Repair ablation (Table 2).

As shown in Table 2, under the same 20K SFT budget on Qwen3.6-35B-A3B, training on unverified tasks without repair already yields clear gains over the base model, while incorporating Execution-Guided Consistency Verification brings substantial further improvements on every benchmark (+98 Elo and +3.61 Rubric points on GDPvalAA-v2, and +5.33 on JobBench Main). This confirms that our verification and repair loop effectively resolves inconsistencies in synthesized tasks, improving scenario quality and, consequently, the quality of the resulting training trajectories.

Table 2: Repair-vs-No-Repair SFT comparison on Qwen3.6-35B-A3B (20K tasks).

ModelConsistency VerificationGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.6-35B-A3B (Base)N/A105382.6114.9956.02
WorkGenesis-35B (w/o Verification)No125483.0519.8667.20
WorkGenesis-35B (w/ Verification)Yes135286.6625.1970.06

Q2 & W2: Real-world grounding: proportions of retrieved and synthetic materials, distribution across tasks, and representative examples.

Response:

1. Proportions and Sector-Level Distribution of Real-World vs. Synthetic Materials

By construction (Section 3.2), every WorkGenesis task is grounded in at least one retrieved real-world anchor file, while synthetic companion files are added as needed to complete the scenario. Across the entire 20K dataset, retrieved real-world files account for 59.77% of all reference files, while synthetic companion materials account for 40.23%. Given the 65 distinct occupations, Table 3 reports the detailed breakdown across all 11 O*NET occupational sectors, showing that real-world artifacts consistently form the majority of reference materials in every sector.

Table 3: Proportions of synthetic companion files and retrieved real-world files across O*NET sectors.

Occupational SectorSynthetic Files (%)Real-World Files (%)
Finance and Insurance33.35%66.65%
Government36.45%63.55%
Health Care and Social Assistance35.26%64.74%
Information43.06%56.94%
Manufacturing47.44%52.56%
Professional, Scientific, and Technical Services38.13%61.87%
Real Estate and Rental and Leasing36.59%63.41%
Retail Trade47.79%52.21%
Wholesale Trade45.17%54.83%
Educational Services40.12%59.88%
Mining, Quarrying, and Oil and Gas Extraction46.78%53.22%
Overall40.23%59.77%
AuthorsOpenReview ↗

Response to Reviewer 4CAb(2/2)

2. Representative Examples of Real-World File Roles

In WorkGenesis, retrieved real-world files serve either as authoritative domain/regulatory context or as direct core evidence that the deliverable must faithfully process, while synthetic files supply complementary scenario-specific and operational records when needed. The two case studies in Appendix C illustrate both roles:

  • Task A (Mechanical Engineering — Normative & Technical Context): Among the 12 attachments, the 3 real-world files are official OSHA 3071, OSHA 3080, and OSHA 3170 booklets, which the prompt designates as regulatory safeguarding and JHA reference context (requiring no external web search, with OSHA 3170 cited to justify control choices). Meanwhile, the plant-specific decision evidence, Line 4 layout, dimensions, observation logs, stop-test data, production/maintenance constraints, cost estimates, and release gates, comes from 9 synthesized ME-042 operational files.
  • Task B (Film and Video Editing — Indispensable Core Evidence): This task contains only 1 reference file, a real National Park Service archival interview video (nps_acadia_shirley_beccue_source.mp4) with zero synthetic attachments. The deliverable must preserve Shirley Beccue's exact spoken words and meaning from the source footage, adhere strictly to its visual/audio content and technical specifications (duration, resolution, frame rate, audio, and burned-in captions), and introduce no external narration, music, stock footage, or factual claims. Here, the real artifact directly determines both the deliverable content and its acceptance criteria.

Since occupational attachments span diverse binary and multimodal formats that are difficult to display exhaustively in static pages, we will publicly release the full WorkGenesis training dataset upon publication, including all synthesized task prompts, reference attachments, evaluation rubrics, and complete execution trajectories, enabling readers to inspect the constructed tasks directly.


Q3 & W2: Independent quality validation: details on manual inspection, root-cause attribution accuracy, defect resolution, and difficulty preservation.

Response:
First, we apologize for the ambiguity caused by the phrase "manual inspection" in the AI Use Statement section. That phrase referred to our manual spot-checks of intermediate pipeline outputs while developing and iterating on WorkGenesis's system architecture and prompts, rather than a formal manual audit across the final 20K synthesized tasks. We will revise this wording in the final version to avoid any misunderstanding.

Second, we fully agree on the importance of independently validating WorkGenesis's attribution quality. To be honest, since our tasks span 65 specialized occupations, it is difficult for AI researchers alone to serve as reliable domain-expert judges across every professional field. We therefore randomly sampled 300 tasks synthesized by WorkGenesis and used GPT-6-Astra (xhigh) within Codex to independently inspect the tasks and diagnose unsatisfied rubric items as an Expert Reference, comparing its classifications against WorkGenesis's attributions (Table 4).

As shown in Table 4, WorkGenesis achieves near-perfect agreement with the Expert Reference on solver capability deductions (Agent), while exceeding 91% across Precision, Recall, and F1 on both Rubric and Task categories, reaching an overall Macro F1-score of 94.55%. Combined with Figure 6 (where Rubric and Task defects are eliminated across verification rounds while Agent-attributed items remain stable, preserving task difficulty) and the downstream SFT gains in Tables 1–2, this demonstrates that WorkGenesis accurately attributes failures, resolves genuine scenario defects, and preserves task difficulty.

Table 4: Diagnostic accuracy of WorkGenesis root-cause attribution against the Expert Reference across 300 sampled tasks.

Root-Cause LevelExpert SharePrecision (%)Recall (%)F1-Score (%)
Agent55.80%98.499.398.85
Rubric28.40%93.692.192.84
Task15.80%92.191.891.95
Overall (Macro-Avg)100.00%94.794.494.55

Thank you again for your valuable comments and suggestions. We will incorporate all of the above clarifications, dataset statistics, and experimental results into the final version of the paper.

Thanks for the authors' reply. I am curious about the individual contributions of the synthetic files and the real-world files. Specifically, what would the performance be when using only synthetic files or only real-world files? Would these two sources contribute similarly?

Reviewer_FymjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorkGenesis, a framework for synthesizing agent tasks and training data for professional work scenarios. It constructs tasks, supporting materials, and evaluation rubrics around publicly available real-world files, then uses feedback from agent execution to identify and repair inconsistencies, aiming to produce more realistic training scenarios. The authors use the resulting interaction trajectories for supervised fine-tuning and report improved performance over the base model across several professional-work benchmarks.

Critical strengths
  1. The paper describes the task construction and verification pipeline in detail, with module-specific prompts provided in the appendix to clarify how the framework operates.

  2. The detailed case studies illustrate the task requests, reference materials, evaluation rubrics, and deliverables, making the synthesized work scenarios easier to understand.

Critical weaknesses
  1. Lack of comparisons with existing data construction methods and RL experiments. The paper does not sufficiently compare its approach with prior task and trajectory synthesis methods or the task construction pipeline used in JobBench. This leaves the limitations of existing approaches insufficiently established, weakening the motivation. Moreover, the experiments are limited to SFT and do not examine whether the proposed data better support subsequent agentic RL than existing datasets.

  2. Lack of benchmark contamination analysis. The occupational roles covered by the training data overlap with those in the evaluation benchmarks. Given the use of public-web retrieval and large-scale task synthesis, it is important to check whether benchmark materials or tasks highly similar to test examples enter the training data. The reported deduplication primarily operates within the synthesis pipeline, without a reported contamination analysis against the evaluation sets.

  3. Lack of an ablation on consistency verification. The paper uses GLM-5.2 to generate training trajectories but does not compare downstream training performance with and without consistency verification. It therefore remains unclear how much the verification and repair steps contribute beyond directly performing SFT on trajectories generated by the same model on the initial, unrepaired tasks.

Questions for authors
  1. How does WorkGenesis differ from existing task and trajectory synthesis methods, including JobBench’s construction pipeline? What gap does it address?

  2. Why are the experiments limited to SFT? Do the synthesized data better support subsequent agentic RL than other alternative trajectory-generation methods?

  3. Given the overlapping occupational roles and large-scale web-based synthesis, could benchmark materials or near-duplicate test tasks enter the training data? Has contamination been checked?

  4. How much does consistency verification improve downstream performance compared with directly using GLM-5.2 trajectories from the initial, unrepaired tasks under matched SFT budgets?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Fymj(1/2)

Thank you very much for your insightful review and constructive feedback. Below is our point-by-point response to your questions and comments.


Q1 & W1: How does WorkGenesis differ from existing task and trajectory synthesis methods, including JobBench’s construction pipeline? What gap does it address?

Response:
Compared with existing automated synthesis methods, prior approaches fall into two main categories that do not fully address realistic occupational work:

  1. Instruction and trajectory synthesis methods—such as Self-Instruct, PersonaHub, and Magpie primarily synthesize text prompts and responses from compact seeds without real-world reference files or executable workspaces.
  2. Agentic experience and environment synthesis methods—such as DreamGym, Qwen-AgentWorld, and Echoverse focus on simulating interactive state transitions via world models or synthesized web/database sandboxes for GUI and tool navigation.

In contrast, WorkGenesis grounds every task in real-world public files guided by O*NET occupational knowledge, synthesizes cross-referenced companion attachments and itemized rubrics around them, and applies execution-guided root-cause attribution to iteratively repair task and rubric inconsistencies while keeping real anchor files.

Compared with JobBench, as documented in its official paper, JobBench is designed as an evaluation benchmark and relies heavily on recruited human domain experts throughout construction, primarily for:

  1. Task Drafting: Domain experts and annotators manually write task sketches for each occupation, designing the workplace scenario, reasoning challenges, expected deliverables, and grading criteria.
  2. Manual Refinement: Human reviewers manually revise task instructions, remove low-quality rubric items, and provide iterative quality-control feedback.

Consequently, JobBench incurs high human-annotation costs and its throughput is constrained by expert availability, making it difficult to scale to large training sets.

Therefore, WorkGenesis bridges the gap between scalable automated synthesis and high-fidelity expert-authored occupational benchmarks: by automating both real-evidence-grounded workspace construction and execution-guided consistency repair, it enables low-cost, large-scale generation of realistic, executable, and internally consistent occupational training environments without human-expert bottlenecks.


Q2 & W1: Why are the experiments limited to SFT? Do the synthesized data better support subsequent agentic RL than other alternative trajectory-generation methods?

Response:
WorkGenesis-synthesized data effectively support agentic RL. In our initial submission, our main experiments focused on SFT primarily due to the high computational and wall-clock overhead of long-horizon multi-turn agentic RL. To address your concern, we have supplemented two sets of Rubric-Based Agentic RL experiments during the rebuttal period.

Specifically, given time and compute constraints, we ran GRPO experiments on Qwen3.5-9B using 1K WorkGenesis tasks, with a batch size of 16 and 8 rollouts per task. Table 1 compares training Qwen3.5-9B on WorkGenesis data under RL, SFT, and SFT+RL, while Table 2 compares RL using WorkGenesis tasks with versus without the Execution-Guided Consistency Verification module.

Table 1: Rubric-Based Agentic RL (GRPO) and SFT results on WorkGenesis (Qwen3.5-9B).

ModelTraining DataGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.5-9B (Base)N/A74265.713.7132.28
+ RL1K83969.279.5342.11
+ SFT20K120477.6417.3165.93
+ SFT + RL19K SFT + 1K RL124580.0219.0866.55

Table 2: Ablation of Execution-Guided Consistency Verification under Rubric-Based Agentic RL (Qwen3.5-9B, 1K tasks).

ModelConsistency VerificationGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.5-9B (Base)N/A74265.713.7132.28
+ RL (w/o Verification)No76566.925.2235.92
+ RL (w/ Verification)Yes83969.279.5342.11
AuthorsOpenReview ↗

Response to Reviewer Fymj(2/2)

As shown in Table 1, applying RL on just 1K tasks brings substantial gains across all benchmarks (+97 Elo and +3.56 Rubric points on GDPvalAA-v2, along with +5.82 and +9.83 points on JobBench Main and Easy). Moreover, under a matched 20K task budget, performing SFT on 19K tasks followed by RL on the remaining 1K tasks (+ SFT + RL) further improves over full 20K SFT by +41 Elo points, with consistent gains across GDPvalAA-v2 Rubric and both JobBench splits. These results demonstrate that WorkGenesis is effective not only for SFT, but also for rubric-based RL: because our verification loop ensures rubric validity and task completeness, the resulting rubrics provide reliable reward signals to further strengthen working agents.

In Table 2, we compare RL on 1K WorkGenesis tasks against RL on their initial, unverified counterparts (prior to Execution-Guided Consistency Verification). While both settings improve over the base model, consistency-verified tasks substantially outperform unverified raw tasks across all metrics, confirming the necessity and effectiveness of Execution-Guided Consistency Verification for RL.


Q3 & W2: Given the overlapping occupational roles and large-scale web-based synthesis, could benchmark materials or near-duplicate test tasks enter the training data? Has contamination been checked?

Response:
We carefully accounted for potential benchmark contamination when designing WorkGenesis and enforced three layers of safeguards throughout data construction:

  1. Zero Benchmark Information in Synthesis Prompts: None of the prompts in WorkGenesis contain any information, names, task templates, or examples from our downstream evaluation benchmarks.
  2. Search-Level Domain Blocklisting: In our web retrieval tools, we explicitly blocklisted dataset hubs and benchmark-hosting platforms (e.g., huggingface.co, github.com, etc.) to prevent accidentally retrieving benchmark tasks or attachments.
  3. File SHA-256 and URL Verification Before Acceptance: As described in Section 3 of our paper, WorkGenesis tracks both the SHA-256 hash and download URL of every real-world anchor file used. Before any synthesized task is accepted into the dataset, we run a final check on its file SHA-256 hashes and URLs against evaluation benchmark assets, ensuring that original benchmark materials cannot enter our dataset even via external mirrors or non-standard sources.

Through these three safeguards, we confirm that no benchmark materials or contaminated test tasks exist in the WorkGenesis training data. We will add a dedicated decontamination subsection clarifying these mechanisms in the final version of the paper.


Q4 & W3: How much does consistency verification improve downstream performance compared with directly using GLM-5.2 trajectories from the initial, unrepaired tasks under matched SFT budgets?

Response:
As shown in Table 3, under the same 20K SFT budget on Qwen3.6-35B-A3B, training on unverified tasks without repair already yields clear gains over the base model, while incorporating Execution-Guided Consistency Verification brings substantial further improvements on every benchmark (+98 Elo and +3.61 Rubric points on GDPvalAA-v2, and +5.33 on JobBench Main). This confirms that our verification and repair loop effectively resolves inconsistencies in synthesized tasks, improving scenario quality and, consequently, the quality of the resulting training trajectories.

Table 3: Repair-vs-No-Repair SFT comparison on Qwen3.6-35B-A3B (20K tasks).

ModelConsistency VerificationGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.6-35B-A3B (Base)N/A105382.6114.9956.02
WorkGenesis-35B (w/o Verification)No125483.0519.8667.20
WorkGenesis-35B (w/ Verification)Yes135286.6625.1970.06

Thank you again for your valuable comments and suggestions. We will incorporate all of the above comparisons, clarifications, and experimental results into the final version of the paper.

AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer Fymj,

Thank you again for your thoughtful review. As the end of the discussion period approaches, we would like to kindly follow up on our rebuttal, particularly the additional experiments and analyses addressing your central concerns.

We have provided four new clarifications and empirical results directly addressing W1–W3 and Q1–Q4:

  • Comparison with prior pipelines and JobBench (Q1 & W1): We clarified how WorkGenesis differs from text-only prompt synthesis, simulated GUI/sandbox environments, and expert-authored benchmarks like JobBench.
  • Rubric-based Agentic RL (Q2 & W1): On Qwen3.5-9B, GRPO on 1K WorkGenesis tasks yields +97 Elo on GDPvalAA-v2 and +5.82 on JobBench Main; 19K SFT + 1K RL further improves over 20K SFT by +41 Elo, and consistency-verified tasks substantially outperform unverified tasks in RL (+74 Elo).
  • Benchmark decontamination (Q3 & W2): We detailed our three-layer safeguard, including zero benchmark information in prompts, search-level domain blocklisting of dataset hubs, and pre-acceptance file SHA-256 and URL checks against evaluation benchmark assets.
  • Consistency verification ablation (Q4 & W3): Under a matched 20K SFT budget on Qwen3.6-35B-A3B, verification and repair bring +98 Elo and +3.61 Rubric points on GDPvalAA-v2 and +5.33 on JobBench Main over unrepaired tasks.

We would greatly appreciate your thoughts on whether these additional results address your concerns. We would also be grateful if you could consider updating your assessment in light of the new evidence.

Thank you for your time and consideration.

Sincerely and respectfully,

The Authors

Reviewer_F83sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Real world work data is difficult and expensive to collect.

The authors create WorkGenesis, a pipeline to create synthetic professional assignments. This pipeline is grounded in real artifacts using occupational knowledge from O*Net, a well known professional work ontology. Authors share construction prompts and example tasks in the appendix. A synthetic assignment contains both the context needed to do the job as well as the rubric to grade the job.

A point of novelty is in the repairing of these synthetic assignments.

  1. Agent attepts the work
  2. the pipeline does root cause analysis and places the error in agent, task, or rubric
  3. if the error is on the task/rubric, those get repaired

The rest of the paper is an exercise in a sort of distillation, where GLM-5.2 acts as the teacher and Qwen3.6-35B as the fine tuned student. GLM 5.2 completes 20k WorkGenesis assignment, and those trajectories are used to fine tune the smaller 35B model. The 35B model is not able to surpass the teacher overall (8.68 for original Qwen, 31.00 for WorkGenesis-35B, and 37.80 for GLM-5.2), but does beat other models of similar size on popular real work datasets.

Critical strengths

Work Genesis: The actual synthetic data pipeline. The authors create a detailed pipeline to create synthetic professional assignments (task, context, and rubrics). The appendix is quite detailed, and contains enough prompts and details to build a meaingful independent implementation. Especially clever is the repair part of the pipeline, that identifies a first attempt agent error as being part of the rubric or task and then repairs those accordingly.

WorkGenesis-35B: a distillation fine tuning from a teacher GLM-5.2 on 20k work genesis synthetic jobs, the trajectories of which are used to fine tune a qwen 3.6 35B. This model generalizes to do much better on other professional work datasets, beating other models of this class while also being a soft proof of the utility of the work genesis dataset. To be clear, there's clear evidence that the trained model is best in its class for professional work.

Critical weaknesses

Distillation serves as a softer proof of the value of the data, as it is unclear how much of the added value comes from the dataset itself. A stronger proof would have been something like rubric-based RL (though computationally expensive), since the Work Genesis dataset has rubrics.

Though clever, the repair aspect was not strongly inspected. There is no diagnostic accuracy of the root cause analysis (identifying the error at the agent, task, or rubric level) nor a repair-vs-no-repair experiment.

Questions for authors

Would the authors be willing to release the WorkGenesis pipeline or the 20k dataset as a public dataset? This would allow others to reproduce the authors claims fully as well as expand experiments to understand how much signal is provided by this synthetic dataset. A commitment to such would be a fast flip from weak acceptance to clear acceptance for me.

3 comments
AuthorsOpenReview ↗

Response to Reviewer F83s(1/2)

Thank you for your insightful review and constructive feedback. Below is our point-by-point response to your questions and comments.


Q1: Would the authors be willing to release the WorkGenesis pipeline or the 20k dataset as a public dataset?

Response:
Yes, upon publication, we will publicly release the WorkGenesis training dataset, including all synthesized task prompts, required reference attachments, evaluation rubrics, and complete execution trajectories used for model training. We observe that publicly available training data for realistic professional work remain scarce, which is one of the primary motivations behind WorkGenesis. We hope this open dataset will facilitate future community research on working agents.


W1: A stronger proof would have been something like rubric-based RL (though computationally expensive), since the WorkGenesis dataset has rubrics.

Response:
We agree that rubric-based reinforcement learning provides a stronger and more direct validation of the dataset's intrinsic value, and we have conducted additional Rubric-Based Agentic RL experiments during the rebuttal period.

Due to time and computational constraints, we ran GRPO experiments on Qwen3.5-9B using 1K WorkGenesis tasks, with a batch size of 16 and 8 rollouts per task. Table 1 compares training Qwen3.5-9B on WorkGenesis data under RL, SFT, and SFT+RL, while Table 2 compares RL using WorkGenesis tasks with versus without the Execution-Guided Consistency Verification module.

Table 1: Rubric-Based Agentic RL (GRPO) and SFT results on WorkGenesis (Qwen3.5-9B).

ModelTraining DataGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.5-9B (Base)N/A74265.713.7132.28
+ RL1K83969.279.5342.11
+ SFT20K120477.6417.3165.93
+ SFT + RL19K SFT + 1K RL124580.0219.0866.55

Table 2: Ablation of Execution-Guided Consistency Verification under Rubric-Based Agentic RL (Qwen3.5-9B, 1K tasks).

ModelConsistency VerificationGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.5-9B (Base)N/A74265.713.7132.28
+ RL (w/o Verification)No76566.925.2235.92
+ RL (w/ Verification)Yes83969.279.5342.11

As shown in Table 1, applying RL on just 1K tasks brings substantial gains across all benchmarks (+97 Elo and +3.56 Rubric points on GDPvalAA-v2, along with +5.82 and +9.83 points on JobBench Main and Easy). Moreover, under a matched 20K task budget, performing SFT on 19K tasks followed by RL on the remaining 1K tasks (+ SFT + RL) further improves over full 20K SFT by +41 Elo points, with consistent gains across GDPvalAA-v2 Rubric and both JobBench splits. These results demonstrate that WorkGenesis is effective not only for SFT, but also for rubric-based RL: because our verification loop ensures rubric validity and task completeness, the resulting rubrics provide reliable reward signals to further strengthen working agents.

In Table 2, we compare RL on 1K WorkGenesis tasks against RL on their initial, unverified counterparts (prior to Execution-Guided Consistency Verification). While both settings improve over the base model, consistency-verified tasks substantially outperform unverified raw tasks across all metrics, confirming the necessity and effectiveness of Execution-Guided Consistency Verification for RL.

AuthorsOpenReview ↗

Response to Reviewer F83s(2/2)

W2: There is no diagnostic accuracy of the root cause analysis (identifying the error at the agent, task, or rubric level) nor a repair-vs-no-repair experiment.

Response:
We agree with this point and have supplemented both (1) a controlled Repair-vs-No-Repair experiment and (2) an evaluation of the Diagnostic Accuracy of Root-Cause Analysis.

1. Repair-vs-No-Repair Experiment

As shown in Table 3, under the same 20K SFT budget on Qwen3.6-35B-A3B, training on unverified tasks without repair already yields clear gains over the base model, while incorporating Execution-Guided Consistency Verification brings substantial further improvements on every benchmark (+98 Elo and +3.61 Rubric points on GDPvalAA-v2, and +5.33 on JobBench Main). This confirms that our verification and repair loop effectively resolves inconsistencies in synthesized tasks, improving scenario quality and, consequently, the quality of the resulting training trajectories.

Table 3: Repair-vs-No-Repair SFT comparison on Qwen3.6-35B-A3B (20K tasks).

ModelConsistency VerificationGDPvalAA-v2 (Elo)GDPvalAA-v2 (Rubric)JobBench (Main)JobBench (Easy)
Qwen3.6-35B-A3B (Base)N/A105382.6114.9956.02
WorkGenesis-35B (w/o Verification)No125483.0519.8667.20
WorkGenesis-35B (w/ Verification)Yes135286.6625.1970.06

2. Diagnostic Accuracy of Root-Cause Analysis

To be honest, since our tasks span 65 specialized occupations, it is difficult for AI researchers alone to serve as reliable domain-expert judges across every professional field. We therefore randomly sampled 300 tasks synthesized by WorkGenesis and used GPT-6-Astra (xhigh) within Codex to independently inspect the tasks and diagnose unsatisfied rubric items as an Expert Reference, comparing its classifications against WorkGenesis's attributions (Table 4).

As shown in Table 4, WorkGenesis achieves near-perfect agreement with the Expert Reference on solver capability deductions (Agent), while exceeding 91% across Precision, Recall, and F1 on both Rubric and Task categories, reaching an overall Macro F1-score of 94.55%. This demonstrates that WorkGenesis reliably diagnoses the root causes of defects that require scenario or rubric repair.

Table 4: Diagnostic accuracy of WorkGenesis root-cause attribution against the Expert Reference across 300 sampled tasks.

Root-Cause LevelExpert SharePrecision (%)Recall (%)F1-Score (%)
Agent55.80%98.499.398.85
Rubric28.40%93.692.192.84
Task15.80%92.191.891.95
Overall (Macro-Avg)100.00%94.794.494.55

Thank you again for your valuable suggestions and support. We will incorporate all of the above experiments and analyses into the final version of the paper.

AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer F83s,

Thank you again for your thoughtful and encouraging review. As the end of the discussion period approaches, we would like to kindly follow up on our earlier rebuttal addressing your questions on open-sourcing, rubric-based RL, and verification diagnostics.

We have provided a clear open-source commitment and three new empirical analyses directly addressing Q1, W1, and W2:

  • Public dataset release commitment (Q1): Upon publication, we will publicly release the full 20K WorkGenesis dataset, including all synthesized task prompts, reference attachments, evaluation rubrics, and complete execution trajectories.
  • Rubric-based Agentic RL (W1): GRPO experiments on Qwen3.5-9B show that 1K RL tasks bring +97 Elo on GDPvalAA-v2 and +5.82 on JobBench Main, 19K SFT + 1K RL outperforms full 20K SFT by +41 Elo, and consistency-verified tasks substantially outperform unverified tasks in RL (+74 Elo).
  • Repair-vs-No-Repair SFT ablation (W2): Under a matched 20K SFT budget on Qwen3.6-35B-A3B, targeted verification and repair yield +98 Elo and +3.61 Rubric points on GDPvalAA-v2 and +5.33 on JobBench Main over unrepaired tasks.
  • Root-cause diagnostic accuracy (W2): Across 300 randomly sampled tasks audited against an independent Expert Reference, WorkGenesis achieves a 94.55% Macro F1-score across the Agent, Rubric, and Task root-cause levels.

We would greatly appreciate your thoughts on whether these commitments and results address your concerns. We would also be grateful if you could reconsider your assessment in light of the new evidence and experiment results.

Thank you for your time and consideration.

Sincerely and respectfully,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.