- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WorkGenesis is a framework for constructing occupational training tasks grounded in real-world files. Guided by O*NET occupational information, it retrieves public files and synthesizes supporting materials, task instructions, and evaluation rubrics. An execution-based verification loop then attributes unmet rubric criteria to agent, task, or rubric errors and iteratively repairs task and rubric defects while preserving the original source files. The authors construct 20K tasks covering 65 occupations, collect execution trajectories from GLM-5.2, and use them to fine-tune Qwen3.6-35B-A3B. The resulting WorkGenesis-35B improves over the base model across all five reported metrics on GDPval-AA, APEX-Agents, and JobBench, supporting the training utility of the generated data.
Critical strengths
A practical framework for constructing occupational training data. WorkGenesis combines public source files, synthesized supporting materials, task instructions, and rubrics into executable work scenarios. Its coverage of 65 occupations and diverse file formats makes it a useful contribution toward scaling training data for agents that must produce concrete professional deliverables.
Consistent downstream improvements across multiple benchmarks. Fine-tuning Qwen3.6-35B-A3B on the generated trajectories improves all five reported metrics across GDPval-AA, APEX-Agents, and JobBench. For example, APEX-Agents pass@1 increases from 13.4% to 25.2%, and JobBench Main rises from 14.99 to 25.19. These results provide meaningful evidence of the data’s training utility, even though the contributions of individual construction components remain unresolved.
Critical weaknesses
Limited methodological novelty and insufficient evidence for the distinctive components. The framework largely combines established ideas in grounded data synthesis and execution-based verification. Its most distinctive components—error attribution and targeted repair—are not evaluated against simpler alternatives, such as filtering failed tasks or directly revising them without explicit attribution. The downstream improvements establish the utility of the overall training pipeline, but do not demonstrate the added value of these components, weakening the paper’s central methodological claim.
Insufficient evidence of the constructed tasks’ real-world grounding and quality. The paper reports file-format diversity but not the relative proportions of retrieved and synthetic materials, or whether real-world files supply essential task evidence rather than background context. Moreover, reaching zero task/rubric defects under the framework’s own model-based audit does not independently establish professional realism or correctness. Although manual inspection is mentioned, its scope and findings are insufficiently documented. These gaps make it difficult to assess how reliably the framework produces the high-quality occupational tasks it claims.
Questions for authors
Value of error attribution and targeted repair. A controlled comparison with simpler alternatives—filtering failed tasks or revising them without explicit attribution—would help establish the contribution of this design. Using the same teacher and comparable training budgets, does targeted repair lead to better task quality and downstream performance?
Real-world grounding. Please report the proportions of retrieved and synthetic materials and their distribution across tasks. A few representative examples would also help clarify whether real-world files provide essential evidence for completing the tasks or mainly serve as background context.
Independent quality validation. Please provide more detail on the manual inspection, including the sample size, reviewers’ expertise, and findings. A blinded comparison before and after repair would help assess whether the system correctly attributes failures, resolves genuine defects, and preserves task difficulty.
7 comments
Response to Reviewer 4CAb(2/2)
2. Representative Examples of Real-World File Roles
In WorkGenesis, retrieved real-world files serve either as authoritative domain/regulatory context or as direct core evidence that the deliverable must faithfully process, while synthetic files supply complementary scenario-specific and operational records when needed. The two case studies in Appendix C illustrate both roles:
- Task A (Mechanical Engineering — Normative & Technical Context): Among the 12 attachments, the 3 real-world files are official
OSHA 3071,OSHA 3080, andOSHA 3170booklets, which the prompt designates as regulatory safeguarding and JHA reference context (requiring no external web search, withOSHA 3170cited to justify control choices). Meanwhile, the plant-specific decision evidence, Line 4 layout, dimensions, observation logs, stop-test data, production/maintenance constraints, cost estimates, and release gates, comes from 9 synthesizedME-042operational files. - Task B (Film and Video Editing — Indispensable Core Evidence): This task contains only 1 reference file, a real National Park Service archival interview video (
nps_acadia_shirley_beccue_source.mp4) with zero synthetic attachments. The deliverable must preserve Shirley Beccue's exact spoken words and meaning from the source footage, adhere strictly to its visual/audio content and technical specifications (duration, resolution, frame rate, audio, and burned-in captions), and introduce no external narration, music, stock footage, or factual claims. Here, the real artifact directly determines both the deliverable content and its acceptance criteria.
Since occupational attachments span diverse binary and multimodal formats that are difficult to display exhaustively in static pages, we will publicly release the full WorkGenesis training dataset upon publication, including all synthesized task prompts, reference attachments, evaluation rubrics, and complete execution trajectories, enabling readers to inspect the constructed tasks directly.
Q3 & W2: Independent quality validation: details on manual inspection, root-cause attribution accuracy, defect resolution, and difficulty preservation.
Response:
First, we apologize for the ambiguity caused by the phrase "manual inspection" in the AI Use Statement section. That phrase referred to our manual spot-checks of intermediate pipeline outputs while developing and iterating on WorkGenesis's system architecture and prompts, rather than a formal manual audit across the final 20K synthesized tasks. We will revise this wording in the final version to avoid any misunderstanding.
Second, we fully agree on the importance of independently validating WorkGenesis's attribution quality. To be honest, since our tasks span 65 specialized occupations, it is difficult for AI researchers alone to serve as reliable domain-expert judges across every professional field. We therefore randomly sampled 300 tasks synthesized by WorkGenesis and used GPT-6-Astra (xhigh) within Codex to independently inspect the tasks and diagnose unsatisfied rubric items as an Expert Reference, comparing its classifications against WorkGenesis's attributions (Table 4).
As shown in Table 4, WorkGenesis achieves near-perfect agreement with the Expert Reference on solver capability deductions (Agent), while exceeding 91% across Precision, Recall, and F1 on both Rubric and Task categories, reaching an overall Macro F1-score of 94.55%. Combined with Figure 6 (where Rubric and Task defects are eliminated across verification rounds while Agent-attributed items remain stable, preserving task difficulty) and the downstream SFT gains in Tables 1–2, this demonstrates that WorkGenesis accurately attributes failures, resolves genuine scenario defects, and preserves task difficulty.
Table 4: Diagnostic accuracy of WorkGenesis root-cause attribution against the Expert Reference across 300 sampled tasks.
| Root-Cause Level | Expert Share | Precision (%) | Recall (%) | F1-Score (%) |
|---|---|---|---|---|
| Agent | 55.80% | 98.4 | 99.3 | 98.85 |
| Rubric | 28.40% | 93.6 | 92.1 | 92.84 |
| Task | 15.80% | 92.1 | 91.8 | 91.95 |
| Overall (Macro-Avg) | 100.00% | 94.7 | 94.4 | 94.55 |
Thank you again for your valuable comments and suggestions. We will incorporate all of the above clarifications, dataset statistics, and experimental results into the final version of the paper.
Thanks for the authors' reply. I am curious about the individual contributions of the synthetic files and the real-world files. Specifically, what would the performance be when using only synthetic files or only real-world files? Would these two sources contribute similarly?
Response to Reviewer 4CAb(1/2)
Thank you for your insightful review and constructive feedback. Below is our point-by-point response to your questions and comments.
Q1 & W1: Value of error attribution and targeted repair: comparison with simpler alternatives and downstream impact.
Response:
Thank you for this suggestion. To establish the contribution of error attribution and targeted repair, we have supplemented (1) a controlled comparison against filtering failed tasks and (2) a Repair-vs-No-Repair ablation of Execution-Guided Consistency Verification.
1. Comparison with Simpler Alternatives (Filtering Failed Tasks)
From our initial synthesized task pool, we selected 5K tasks that exhibited zero
RubricorTaskdefects prior to repair (Filtering), and compared SFT performance onQwen3.6-35B-A3Bagainst 5K and 20K tasks produced by the full WorkGenesis pipeline (Table 1).Table 1: SFT comparison between failed-task filtering and full WorkGenesis on
Qwen3.6-35B-A3B.Filtering(Pass-0 Clean Only)WorkGenesis(Ours)WorkGenesis(Ours)As shown in Table 1, under the same 5K data budget, WorkGenesis consistently outperforms simple filtering across all metrics (+33 Elo on GDPvalAA-v2, +1.92 on JobBench Main, and +1.77 on JobBench Easy), and scales further to 20K tasks. This is because naive filtering systematically discards complex, multi-constraint tasks, which naturally require iterative refinement to align prompts, files, and rubrics, leaving behind disproportionately simple tasks, whereas revising without explicit attribution risks altering valid
Agent-level challenges. Therefore, Execution-Guided Consistency Verification with explicit attribution is essential for preserving complex occupational scenarios and scaling diverse, high-quality training data.2. Repair-vs-No-Repair Experiment
Targeted repair also leads to substantially stronger downstream performance, as demonstrated by our Repair-vs-No-Repair ablation (Table 2).
As shown in Table 2, under the same 20K SFT budget on
Qwen3.6-35B-A3B, training on unverified tasks without repair already yields clear gains over the base model, while incorporating Execution-Guided Consistency Verification brings substantial further improvements on every benchmark (+98 Elo and +3.61 Rubric points on GDPvalAA-v2, and +5.33 on JobBench Main). This confirms that our verification and repair loop effectively resolves inconsistencies in synthesized tasks, improving scenario quality and, consequently, the quality of the resulting training trajectories.Table 2: Repair-vs-No-Repair SFT comparison on
Qwen3.6-35B-A3B(20K tasks).Qwen3.6-35B-A3B(Base)WorkGenesis-35B(w/o Verification)WorkGenesis-35B(w/ Verification)Q2 & W2: Real-world grounding: proportions of retrieved and synthetic materials, distribution across tasks, and representative examples.
Response:
1. Proportions and Sector-Level Distribution of Real-World vs. Synthetic Materials
By construction (Section 3.2), every WorkGenesis task is grounded in at least one retrieved real-world anchor file, while synthetic companion files are added as needed to complete the scenario. Across the entire 20K dataset, retrieved real-world files account for 59.77% of all reference files, while synthetic companion materials account for 40.23%. Given the 65 distinct occupations, Table 3 reports the detailed breakdown across all 11 O*NET occupational sectors, showing that real-world artifacts consistently form the majority of reference materials in every sector.
Table 3: Proportions of synthetic companion files and retrieved real-world files across O*NET sectors.