- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses occupational tasks execution by agents. These are tasks performed by people in a specific occupation, for example, preparing an annual budget from spendings or other material. Each occupational task has instructions and input data.
The paper constructs a synthetic dataset of occupational tasks and their input materials. Construction starts with a predefined list of occupations (from an existing dataset, GDPval). Then, news articles from a large corpus of 171M articles are mined, and relevant articles for each occupation are retrieved. Retrieval is done with a TF-IDF based regression model, trained on articles labeled with an LLM (to save costs, instead of embedding the entire corpus).
From each relevant article, an LLM then creates a task, represented as a hierarchy, with the main task at the root, and smaller tasks, forming a decomposition of the main task, as child nodes. The hierarchy contains synthesized per-task instructions for each of its task nodes, and synthesized input materials, shared between all tasks in the hierarchy.
The empirical evaluation is in a teacher-student setup. A teacher model runs on the synthesized tasks, produces outputs and trajectories. The trajectories are the used to fine tune smaller student models.
The results compare the teacher model (Qwen3.6-27B), with two student models (Qwen3.5-4B-Base, Qwen3.5-9B-Base) fine-tuned with the teacher’s trajectories. Also in the comparison are the instruction tuned versions of the students. This is done on two datasets, GDPval and APEX-Agents. Results show the fine-tuned models are better than the base models and the instruction tuned models, and close the gap with the teacher model. The teacher remains significantly better than the student models.
The results also include experiments with varying fine-tuning dataset sizes, and results describing the retrieval performance.
Critical strengths
- The idea of synthesizing tasks along with their input material is novel and was unexplored. The paper demonstrated the validity of this approach, and its application in a teacher-student setup.
Critical weaknesses
- The teacher-student setup is not well-motivated. It is described in the introduction as the method, but it is not explained why this setup is chosen. The related works section also does not cover any existing work in this context. What are the common approaches to the teacher-student setup? Is SFT (the approach taken) the common approach? Are there alternatives? If there are, how do they perform?
- There are no shared or exemplified tasks from the dataset synthesis, and I did not see any estimation of the quality of the synthetic data. It is shown to be valuable in the empirical evaluation, but beyond that there is no independent evaluation. What are the tasks? Is forming them from news articles required? (could they have been synthesized without the retrieval?) Is the decomposition to tasks reasonable? do the input material reflect real work? How much of that material is found on the web? Or synthesized? There are many missing details. Also, the entire process is not reproducible apart from a few shared prompts.
Questions for authors
- Q1. What was the cost of producing the trajectories? (how many GPUs required, how many GPU-hours? Can you estimate?)
- Q2. Similarly, what was the compute required for fine-tuning the students? How many GPUs used? For how long?
- Q3. What are the common approaches to the teacher-student setup? Is SFT (the approach taken) the common approach? Are there alternatives? If there are, how do they perform?
- Q4. What are the synthesized tasks? Can upi share examples? Are the tasks going to be open-sources?
- Q5. Is forming taskse from news articles required? (could they have been synthesized without the retrieval?)
- Q6. Is the decomposition to tasks reasonable?
- Q7. Do the input material reflect real work?
- Q8. How much of that material is found on the web? How much is synthesized?
2 comments
Response to Reviewer 1z22 (2/2)
Q5: Analysis of the Input Materials
The pipeline tracks the origin of every material file. Of the 67K files, 15.0% were downloaded from the web as they are, 60.3% were synthesized or modified from content retrieved from the web, 22.6% were synthesized without retrieval, and 2.0% are generated images or videos. Most files are therefore grounded in the web, drawing on 14,980 distinct domains. The most frequent domains are sources that professionals consult in their work, such as sec.gov, gov.uk, cdc.gov, fda.gov and irs.gov, and most files are Excel workbooks, PDFs and Word documents, the formats that professionals work with. The input materials therefore reflect real work in both their content and their form.
Q6: Release Plan
We are currently undergoing internal compliance review and plan to release the SynthWorker code and the SynthWorker-4B and SynthWorker-9B models upon completion of this process. The revised manuscript will also include complete example hierarchies from different sectors, the full prompt templates and the hyperparameters of the retrieval loop, so that each stage of the pipeline can be reproduced.
We are grateful for the detailed questions, which helped us present the dataset more fully. The revised manuscript will include these analyses, and we are happy to clarify anything further during the discussion period.
Sincerely,
The Authors
References
- Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., & Tang, J. (2024). AgentTuning: Enabling Generalized Agent Abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024.
- Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., & Yao, S. (2023). FireAct: Toward Language Agent Fine-tuning. arXiv preprint arXiv:2310.05915.
- Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., & Zhang, Y. (2025). Training Software Engineering Agents and Verifiers with SWE-Gym. In Proceedings of the 42nd International Conference on Machine Learning.
- Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., & Yang, D. (2025). SWE-smith: Scaling Data for Software Engineering Agents. arXiv preprint arXiv:2504.21798.
- Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos Garea, S., Geist, M., & Bachem, O. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations.
- Gunjal, A., Wang, A., Lau, E., Nath, V., Liu, B., & Hendryx, S. (2025). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv preprint arXiv:2507.17746.
- Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., ... & Wei, F. (2024). Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models. arXiv preprint arXiv:2402.13064.
Response to Reviewer 1z22 (1/2)
Dear Reviewer 1z22,
We appreciate the careful review and the recognition that synthesizing tasks together with their input materials is a novel direction. We respond to each question below.
Q1: Compute Budget for Teacher Solving and Student Training
Both stages ran on standard 8×A100-80GB nodes. For trajectory generation, the Qwen3.6-27B teacher generated 3.74B output tokens on our 100K tasks, about 37K per task, and the running times of all tasks sum to 51.4K agent-hours. For student training, each model was trained on 14 nodes with 112 GPUs in total for 3 epochs over 27.3B tokens. Training took about 17 hours for the 4B model and about 41 hours for the 9B model, or about 1.9K and 4.6K GPU-hours. Both runs used a global batch size of 252, a learning rate of 3e-5 and a maximum sequence length of 131,072 with sample packing.
Q2: Choice of Training Setup
The contribution of this work is the training data, so we use the most common recipe, supervised fine-tuning on trajectories from a stronger teacher [1, 2], to attribute the gains to the tasks themselves. SFT is also the most general choice, since it only needs trajectories and works with any strong teacher, including GLM-5.2 from a different model family. Rejection sampling [3, 4] and RL [6] instead need a reward to judge each output, and on-policy distillation [5] needs the teacher's token-level probabilities, which usually requires a shared tokenizer. Adapting these methods to open-ended occupational tasks is an important research question beyond the scope of this work, and our tasks can directly serve as their training queries.
Q3: Role of News Retrieval
News retrieval gives each task a concrete setting drawn from real events. We tested whether it is needed in an earlier experiment. Following GLAN [7], which synthesizes tasks from a taxonomy instead of retrieved text, we built an occupation tree and a sector tree over the same 44 occupations and 9 sectors and paired their nodes to produce 77K tasks. Trained on these tasks, Qwen3.5-4B-Base reaches 38.1% win-or-tie on GDPval. Trained on fewer of our tasks, the same model reaches 54.6% with 75K and already 47.0% with 25K. Without retrieval, task diversity is limited and hard to scale further.
Q4: Quality Control of the Synthetic Data
Each task is an occupational assignment with an instruction, a set of input files and the deliverables it expects. The pipeline enforces several rules while it builds the tasks. For the decomposition, each subtask must take the same or a shorter horizon than its parent, a task can only use the outputs of sibling tasks it explicitly depends on, and dependency graphs with cycles are rejected. For the input materials, every file a task refers to must exist and have a known source, and the construction agent has to fix any detected error before the materials are accepted. The agent then rewrites the task instructions to match the contents of the files. These rules hold across all 100K tasks.