acceptodds
Under review as a conference paper at ICLR 2027

Synthesizing Occupational Tasks at Scale for Productivity Agents

Abstract

Synthesizing training data for productivity agents requires more than generating task instructions. The tasks must reflect the work people actually perform in their occupations, and their input materials must provide the information needed to meet the task requirements. We introduce SynthWorker to synthesize occupational tasks and their input materials at scale. SynthWorker leverages news articles relevant to each occupation and sector to provide work context for task generation. Within this context, we use a task hierarchy as the unit of synthesis, constructing shared input materials through web retrieval and synthesis and refining task instructions based on their contents. During execution, intermediate deliverables become additional input materials for dependent tasks, reflecting how real work builds on earlier results. Sharing the materials reduces average input and output token use per task by and compared with constructing materials separately for each task. We synthesize 100K tasks across 44 occupations in nine sectors and use a teacher model to solve them in dependency order. Fine-tuning the 4B and 9B Qwen3.5 base models on the resulting trajectories yields SynthWorker-4B and SynthWorker-9B, improving GDPval win rates by 39.9 and 44.0 percentage points over their respective base models. These gains extend to the subset of APEX-Agents requiring file deliverables, where mean scores increase by 17.6 and 20.9 percentage points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Synthetic Data Generation, LLM Agents, Productivity Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1z22OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses occupational tasks execution by agents. These are tasks performed by people in a specific occupation, for example, preparing an annual budget from spendings or other material. Each occupational task has instructions and input data.

The paper constructs a synthetic dataset of occupational tasks and their input materials. Construction starts with a predefined list of occupations (from an existing dataset, GDPval). Then, news articles from a large corpus of 171M articles are mined, and relevant articles for each occupation are retrieved. Retrieval is done with a TF-IDF based regression model, trained on articles labeled with an LLM (to save costs, instead of embedding the entire corpus).

From each relevant article, an LLM then creates a task, represented as a hierarchy, with the main task at the root, and smaller tasks, forming a decomposition of the main task, as child nodes. The hierarchy contains synthesized per-task instructions for each of its task nodes, and synthesized input materials, shared between all tasks in the hierarchy.

The empirical evaluation is in a teacher-student setup. A teacher model runs on the synthesized tasks, produces outputs and trajectories. The trajectories are the used to fine tune smaller student models.

The results compare the teacher model (Qwen3.6-27B), with two student models (Qwen3.5-4B-Base, Qwen3.5-9B-Base) fine-tuned with the teacher’s trajectories. Also in the comparison are the instruction tuned versions of the students. This is done on two datasets, GDPval and APEX-Agents. Results show the fine-tuned models are better than the base models and the instruction tuned models, and close the gap with the teacher model. The teacher remains significantly better than the student models.

The results also include experiments with varying fine-tuning dataset sizes, and results describing the retrieval performance.

Critical strengths
  • The idea of synthesizing tasks along with their input material is novel and was unexplored. The paper demonstrated the validity of this approach, and its application in a teacher-student setup.
Critical weaknesses
  • The teacher-student setup is not well-motivated. It is described in the introduction as the method, but it is not explained why this setup is chosen. The related works section also does not cover any existing work in this context. What are the common approaches to the teacher-student setup? Is SFT (the approach taken) the common approach? Are there alternatives? If there are, how do they perform?
  • There are no shared or exemplified tasks from the dataset synthesis, and I did not see any estimation of the quality of the synthetic data. It is shown to be valuable in the empirical evaluation, but beyond that there is no independent evaluation. What are the tasks? Is forming them from news articles required? (could they have been synthesized without the retrieval?) Is the decomposition to tasks reasonable? do the input material reflect real work? How much of that material is found on the web? Or synthesized? There are many missing details. Also, the entire process is not reproducible apart from a few shared prompts.
Questions for authors
  • Q1. What was the cost of producing the trajectories? (how many GPUs required, how many GPU-hours? Can you estimate?)
  • Q2. Similarly, what was the compute required for fine-tuning the students? How many GPUs used? For how long?
  • Q3. What are the common approaches to the teacher-student setup? Is SFT (the approach taken) the common approach? Are there alternatives? If there are, how do they perform?
  • Q4. What are the synthesized tasks? Can upi share examples? Are the tasks going to be open-sources?
  • Q5. Is forming taskse from news articles required? (could they have been synthesized without the retrieval?)
  • Q6. Is the decomposition to tasks reasonable?
  • Q7. Do the input material reflect real work?
  • Q8. How much of that material is found on the web? How much is synthesized?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1z22 (1/2)

Dear Reviewer 1z22,

We appreciate the careful review and the recognition that synthesizing tasks together with their input materials is a novel direction. We respond to each question below.

Q1: Compute Budget for Teacher Solving and Student Training

Both stages ran on standard 8×A100-80GB nodes. For trajectory generation, the Qwen3.6-27B teacher generated 3.74B output tokens on our 100K tasks, about 37K per task, and the running times of all tasks sum to 51.4K agent-hours. For student training, each model was trained on 14 nodes with 112 GPUs in total for 3 epochs over 27.3B tokens. Training took about 17 hours for the 4B model and about 41 hours for the 9B model, or about 1.9K and 4.6K GPU-hours. Both runs used a global batch size of 252, a learning rate of 3e-5 and a maximum sequence length of 131,072 with sample packing.

Q2: Choice of Training Setup

The contribution of this work is the training data, so we use the most common recipe, supervised fine-tuning on trajectories from a stronger teacher [1, 2], to attribute the gains to the tasks themselves. SFT is also the most general choice, since it only needs trajectories and works with any strong teacher, including GLM-5.2 from a different model family. Rejection sampling [3, 4] and RL [6] instead need a reward to judge each output, and on-policy distillation [5] needs the teacher's token-level probabilities, which usually requires a shared tokenizer. Adapting these methods to open-ended occupational tasks is an important research question beyond the scope of this work, and our tasks can directly serve as their training queries.

Q3: Role of News Retrieval

News retrieval gives each task a concrete setting drawn from real events. We tested whether it is needed in an earlier experiment. Following GLAN [7], which synthesizes tasks from a taxonomy instead of retrieved text, we built an occupation tree and a sector tree over the same 44 occupations and 9 sectors and paired their nodes to produce 77K tasks. Trained on these tasks, Qwen3.5-4B-Base reaches 38.1% win-or-tie on GDPval. Trained on fewer of our tasks, the same model reaches 54.6% with 75K and already 47.0% with 25K. Without retrieval, task diversity is limited and hard to scale further.

Q4: Quality Control of the Synthetic Data

Each task is an occupational assignment with an instruction, a set of input files and the deliverables it expects. The pipeline enforces several rules while it builds the tasks. For the decomposition, each subtask must take the same or a shorter horizon than its parent, a task can only use the outputs of sibling tasks it explicitly depends on, and dependency graphs with cycles are rejected. For the input materials, every file a task refers to must exist and have a known source, and the construction agent has to fix any detected error before the materials are accepted. The agent then rewrites the task instructions to match the contents of the files. These rules hold across all 100K tasks.

AuthorsOpenReview ↗

Response to Reviewer 1z22 (2/2)

Q5: Analysis of the Input Materials

The pipeline tracks the origin of every material file. Of the 67K files, 15.0% were downloaded from the web as they are, 60.3% were synthesized or modified from content retrieved from the web, 22.6% were synthesized without retrieval, and 2.0% are generated images or videos. Most files are therefore grounded in the web, drawing on 14,980 distinct domains. The most frequent domains are sources that professionals consult in their work, such as sec.gov, gov.uk, cdc.gov, fda.gov and irs.gov, and most files are Excel workbooks, PDFs and Word documents, the formats that professionals work with. The input materials therefore reflect real work in both their content and their form.

Q6: Release Plan

We are currently undergoing internal compliance review and plan to release the SynthWorker code and the SynthWorker-4B and SynthWorker-9B models upon completion of this process. The revised manuscript will also include complete example hierarchies from different sectors, the full prompt templates and the hyperparameters of the retrieval loop, so that each stage of the pipeline can be reproduced.

We are grateful for the detailed questions, which helped us present the dataset more fully. The revised manuscript will include these analyses, and we are happy to clarify anything further during the discussion period.

Sincerely,

The Authors

References

  1. Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., & Tang, J. (2024). AgentTuning: Enabling Generalized Agent Abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024.
  2. Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., & Yao, S. (2023). FireAct: Toward Language Agent Fine-tuning. arXiv preprint arXiv:2310.05915.
  3. Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., & Zhang, Y. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. In Proceedings of the 42nd International Conference on Machine Learning.
  4. Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025). SWE-smith: Scaling Data for Software Engineering Agents. arXiv preprint arXiv:2504.21798.
  5. Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos Garea, S., Geist, M., & Bachem, O. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations.
  6. Gunjal, A., Wang, A., Lau, E., Nath, V., Liu, B., & Hendryx, S. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv preprint arXiv:2507.17746.
  7. Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., ... & Wei, F. (2024). Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models. arXiv preprint arXiv:2402.13064.
Reviewer_fy9MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors create a synthetic data anchored on real news stories. Essentially, they mine news stories for occupation-related stories. From the story, they generate a work assignment hierarchy, each parent being a broader task that decomposes into smaller child tasks. Each node has its own instructions and expected deliverables, as well as dependencies. AFter that, they build out a shared file corpus for this tree of tasks with some revisions for consistency and structural executability.

This dataset then has a 27B model run on it to produce trajectories that a 4B and 9B model uses to fine tune. This increases their performance in real world tasks against other popular real world professional task benchmarks.

Critical strengths

Professional work synthetic dataset: Authors present a novel idea of using news stories to construct occupation/sector informed task hierarchies. These hierarchies can construct several inter-dependent tasks that mimic real world settings because they are based on real news articles. Further revision rounds aims to optimize synthetic files as well as structural executability of the dataset. Prompts are included to enable readers to generally replicate this pipeline.

SynthWorker is a distilled model from a larger 27B model. It learns from the trajectories of 100k different tasks from the synthetic dataset attempted by a 27B model. Both the 4B and 9B version of SynthWorker shows large improvements and best in class results for common professional work datasets.

Critical weaknesses

Lack of reproducibility from no code, data, nor SynthWorker model shared.

The dataset would have stronger confirmed signal and utility if SynthWorker was trained on only the dataset via rubric-based RL. Additionally, you could also show matched-budget comparisons against alternative (as similar field as possible) datasets with similar size as well as computation on the distillation.

Questions for authors

Would the authors be willing to share code, data, or model weights to the community?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fy9M

Dear Reviewer fy9M,

Thank you for the detailed review. We are glad that you found building task hierarchies from real news novel and noted the gains of the distilled models. Our responses to each point follow.

Q1: Reproducibility and Release Plan

We agree that reproducibility matters for this line of work. We are currently undergoing internal compliance review and plan to release the SynthWorker code for retrieval, task hierarchy generation and material construction, as well as the SynthWorker-4B and SynthWorker-9B models, upon completion of this process. We will also include examples of complete task hierarchies with their input materials in the revised manuscript.

Q2: Source of the Training Signal

The main gap in training productivity agents is the tasks themselves. Unlike coding, where public repositories provide abundant training tasks, occupational work starts from input materials and produces files, and such tasks are rarely available in public data. We therefore kept the training setup standard to isolate the contribution of the tasks. Rubric-based RL is a natural use of the same tasks. Because the tasks are open-ended, there is usually no single correct deliverable, and the reward has to come from task-specific rubrics or a judge model. Designing such rewards reliably is a research question of its own, and our tasks can serve directly as the query pool for that work.

Q3: Comparison with an Alternative Dataset at Matched Scale

We agree that a matched comparison is the clearest test of a dataset. In an earlier experiment, we built an alternative dataset in the same field and at a similar size. Following GLAN [1], which synthesizes tasks from a taxonomy instead of retrieved text, we built an occupation tree and a sector tree over the same 44 occupations and 9 sectors and paired their nodes to produce 77K tasks. Trained on this dataset, Qwen3.5-4B-Base reaches 38.1% win-or-tie on GDPval. We found that the taxonomy limits task diversity and is hard to scale much beyond this size, which motivated news retrieval. Trained on 75K of our tasks, the same model reaches 54.6%, and with only 25K it already reaches 47.0%, so our tasks lead even with fewer tasks to solve and train on.

Thank you again for the constructive suggestions. The points above will be reflected in the revised manuscript, and we would be glad to discuss any remaining questions.

Sincerely,

The Authors

References

  1. Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., ... & Wei, F. (2024). Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models. arXiv preprint arXiv:2402.13064.
Reviewer_a6CnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors of this paper introduces SynthWorker, a framework for generating large-scale synthetic occupational tasks and their associated input materials to train productivity agents. The SynthWorker generates realistic workplace assignments/tasks at scale by using relevant news articles for each occuption and sector. The paper introduces a task heirarcy approach, decompsing broader task into finer tasks using the retrieved materials from the web. During the execution of the agent, the intermediate deliverables are reused by dependent tasks. The authors generated 100,000 tasks across 44 occupations and nine sectors. Authors also fine-tuned Qwen3.5-4B and Qwen3.5-9B models showing substantial improvements on GDPVal, ranging from 40-44% over base models. The work demonstrates promising scalability and training utility.

Critical strengths
  1. The paper is well-motivated, highlighting the weakness of current synthetic data and task generation pipeline for Agents at scale. The choice to focus on generating real-world occupational tasks at scale from news information available on Web is an interesting approach. This avoids accumulating seed tasks/information that is typical of synthetic data/tasks generation pipeline.

  2. The authors use an heirarchical approach to generate tasks, specifically, decomposing broader task into fine-grained independent tasks. Related tasks share materials, and the intermediate deliverables are reused by dependent tasks. This allow authors to gain on input and output token cost by multiple factors, relative to creating independent tasks.

  3. Authors perform systematic experiments, fine-tuning small models 4B and 9B sized and evaluating on GDPVal. They demonstrate significant improvement in the performance, ranging from 40% (4B) to 44% (9B) relative to base model.

Critical weaknesses
  1. The ablation studies are limited. The method combines several components: occupation-conditioned retrieval, hierarchical decomposition, shared input materials, dependency-aware execution, and teacher-generated trajectories. However, the principal training experiments compare models trained on the complete SynthWorker pipeline against pretrained or instruction-tuned baselines. The results establish that training on SynthWorker data helps, but do not establish which design choices caused the gains.

  2. Realism and task coverage are asserted but not enough validation. The relevance criterion in Appendix A.1 is very broad. An article may qualify because it discusses an occupation's wider professional environment, even if it does not describe an actual occupational activity. In addition, 80% of task fall within 1 day time-horizon, which is too simplistic. This is also reflected in the strong performance by relatively small teacher model (27B) on GDPVal. It will be good to see how these tasks are performed by larger (> 100B) or frontier models (I think the performance will be very strong, which may indicate relatively simple tasks.)

  3. Valuation does not sufficiently rule out confounding and benchmark contamination. The paper uses GPT-5.5 for synthetic data construction and benchmark grading. I am wondering the possibility of dataset contamination.

  4. Did the authors generate multimodal tasks, which is imperative in sectors like manufacturing, retail, healthcare etc.? Would like to know the fractio of tasks that had image or videos as material inputs. Also some measure of the complexities of such tasks.

Questions for authors

See the weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer a6Cn (1/2)

Dear Reviewer a6Cn,

We thank the reviewer for the thorough and constructive review, and for recognizing the news-grounded design of our task synthesis, the token economics of sharing input materials across a hierarchy, and the strength of the fine-tuning results. The questions raised are valuable and have helped us strengthen the paper. We address each concern below.

Q1: Component Ablations and Task Diversity

The components of SynthWorker play two different roles. Retrieval shapes the content and diversity of the tasks, while the remaining components make it feasible to build them at scale. To test the role of retrieval, we compared it in an earlier experiment against GLAN [1], which synthesizes tasks from a taxonomy instead of retrieved text. We built an occupation tree and a sector tree over the same 44 occupations and 9 sectors and paired their nodes to produce 77K tasks. Trained on these tasks, Qwen3.5-4B-Base reaches 38.1% win-or-tie on GDPval. Trained on fewer of our tasks, the same model reaches 54.6% with 75K and already 47.0% with 25K. We attribute this gap to task diversity. With only a taxonomy, the synthesizer has nothing external to condition on and falls back on its most frequent patterns, which prior work also observes in synthetic data generation [2]. Retrieving news gives the synthesizer this external input. The other components, namely hierarchical decomposition, shared materials and dependency-aware execution, serve the second role. Because the tasks in a hierarchy share their input materials, construction uses 6.7x fewer input tokens and 9.5x fewer output tokens per task (Section 5.2), which makes building 100K tasks affordable.

Q2: Task Realism, Horizons and Difficulty

Realism is hard to measure directly, so we test it by whether training on the tasks transfers to benchmarks built from professionals' real work. On GDPval, win-or-tie rises from 9.9% to 57.0%, with gains also on APEX-Agents. The relevance criterion is broad because the article only supplies the setting, while the task is generated for the target occupation, and it still rejects articles that mention the occupation only in passing. Most input materials also come from the web. Of the 67K material files, 75.4% carry a source URL, across 14,980 distinct domains led by sources that professionals consult in their work, such as sec.gov, gov.uk, cdc.gov and fda.gov. The pipeline also requires that every file a task refers to exists and has a known source, and the construction agent rewrites each task instruction to match the contents of its files.

On task horizons, hour and day tasks make up most of the release, and the horizon label separates tasks clearly. The median teacher trajectory produces 14.1K output tokens on hour tasks and 50.0K on month tasks, and month tasks also use 6.76 input files on average against 1.80 for hour tasks. Horizon can therefore serve as a field for selecting tasks. The default composition of the release is not something training has to inherit. An early training stage can draw mostly on hour and day tasks, and a later stage can shift weight toward week and month tasks. On larger models, the paper already uses GLM-5.2, a model well above 100B parameters, as a second teacher on the same 10K tasks. The 4B student trained on its trajectories reaches a 33.9% GDPval win rate, against 28.0% with Qwen3.6-27B as the teacher. A stronger model therefore brings a larger gain from the same tasks.

AuthorsOpenReview ↗

Response to Reviewer a6Cn (2/2)

Q3: Contamination and the Shared Use of GPT-5.5

Our pipeline does not use either benchmark at any stage, and the only inputs it needs are the names of occupations and sectors. We also checked overlap between our 100K training tasks and the public test tasks of both benchmarks. No test task matches a training task exactly or shares any 13-word span with one, and the highest TF-IDF cosine similarity between a test task and its nearest training task is 0.351 for GDPval and 0.376 for APEX-Agents. For input materials, none of our files was collected from a benchmark source, and none shares a file name with a GDPval reference or deliverable file. This check is lexical and does not rule out paraphrased overlap, but it shows that no benchmark task appears in our data. On the shared use of GPT-5.5, its role in our pipeline is to write tasks and input materials. The solutions the student learns from come from the teacher model, so the outputs GPT-5.5 grades at evaluation are not text it produced. All models in our comparison are also graded by the same judge, so any preference of the judge applies to every model in the table.

Q4: Multimodal Tasks and Material Inputs

Yes, the release includes multimodal tasks. Images enter tasks either as input files or during execution, when the agent downloads or generates them, and 8.9% of tasks process an image at some point. The input materials further include sector-specific formats, including 433 DXF drawings and 20 STEP models for engineering and manufacturing work and 160 DICOM studies for healthcare. Image tasks are also more complex. They take 38.5 teacher turns on average against 22.6 for other tasks, and the gap holds within every horizon, from 27.6 against 14.3 at hour to 50.1 against 41.5 at month. We agree that media inputs remain a small share overall, and extending them is a natural direction for future work.

We sincerely appreciate the reviewer's thoughtful feedback, which has helped us strengthen the paper. We will incorporate the analyses and clarifications above into the revised manuscript. We hope our responses have addressed the concerns raised, and we welcome any further discussion.

Sincerely,

The Authors

References

  1. Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., ... & Wei, F. (2024). Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models. arXiv preprint arXiv:2402.13064.
  2. Zhang, J., Yu, S., Chong, D., Sicilia, A., Tomz, M. R., Manning, C. D., & Shi, W. (2025). Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity. arXiv preprint arXiv:2510.01171.
Reviewer_a6CnOpenReview ↗

Reply to Authors' response

I thank the authors for the detailed clarification. It helps in alleviating the weakness mentioned earlier to great extent. The main premise of the paper, that of building synthetic environment to fine-tune LLM/Agent using web-based occupational information, though interesting is naive and at most captures the generalist tasks in occupation. From my experience, creating synthetic environment for enterprise needs much more details including information about the applications/SW, workflows, and other artifacts. In light of this, I will retain my previous score not because of any inherent weakness in the paper, but due to simplicity of the approach.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and thoughtful feedback. We appreciate the opportunity to address their concerns and will incorporate the additional comparisons and analyses into the revised manuscript.

  1. Roles of the synthesis components. News retrieval provides varied occupational settings for task generation. The same 4B model achieved a GDPval win-or-tie rate of 54.6% after training on 75K SynthWorker tasks, compared with 38.1% on 77K tasks generated from a taxonomy covering the same occupations. Task hierarchies let related tasks share input materials and pass deliverables to dependent tasks. Sharing materials reduces input and output token use per task by factors of 6.7 and 9.5 during material construction.

  2. Occupational relevance and difficulty. About three quarters of the input files draw on web sources, with the SEC, CDC, and FDA among the most frequent. We revise each assignment to match the information in its materials. The horizon labels also track execution effort. On tasks with a month horizon, the teacher takes more turns and generates over three times as many output tokens at the median as on tasks with an hour horizon. These labels can therefore be used to filter for more demanding tasks or adjust the training mix.

  3. Effectiveness of the synthesized tasks. Standard SFT on teacher trajectories demonstrates the value of SynthWorker tasks as training data. GDPval win rates increased by 39.9 and 44.0 percentage points over the 4B and 9B base models, respectively, and both models also improved on APEX-Agents. The same tasks can serve as training queries for rubric-based RL, a direction we leave for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.