- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the composition of synthetic tool-use data affects the performance of LLM agents. The authors introduce SynthAPI, a pipeline that generates executable environments with apps, APIs, database states, tasks, verifiers, and oracle solutions. Rather than only scaling the total amount of data, they systematically vary factors such as the number of apps, tasks per app, database variations, and distractor complexity. Using more than 458K generated tasks and 849 trained models, they derive scaling laws that relate these data properties to downstream performance. Their main finding is that as the training budget grows, increasing tool and database diversity becomes more useful than simply adding more tasks. They then use these findings to build a larger training setup, which improves performance on AppWorld, BFCL, ASTRA-Bench, and a SynthAPI holdout set.
Critical strengths
- The paper conducts a large-scale and systematic study with more than 458,000 generated tasks and 849 trained models.
- SynthAPI provides explicit control over key data-generation factors, including app diversity, task diversity, database variation, and task complexity.
- The paper derives practical scaling-law insights and shows that the resulting training recipe improves performance across multiple tool-use benchmarks.
Critical weaknesses
Although the overall direction of the work is interesting, I have one main concern and several questions that I would appreciate the authors addressing.
-
The authors use an oracle solver and executable verifier to filter generated tasks, which provides a useful check of task executability. However, this does not necessarily establish the semantic quality of the tasks or the correctness of the generated verifiers at scale. For example, a task may be technically solvable while still being unrealistic, ambiguous, poorly specified, or paired with an imperfect verifier. Given that 458,752 tasks are generated and the paper states that the generated tasks are otherwise relatively unsupervised, I believe a more systematic evaluation of task and verifier quality would strengthen the analysis and the conclusions drawn from the scaling experiments.
-
The evaluation mainly focuses on settings where interaction with an active user is not a primary challenge. I am curious whether a model fine-tuned on environments containing 10,472 APIs, including 6,266 write and 4,206 read APIs, can transfer its learned tool-use behavior to environments where the agent must interact with a simulated user and reason about user-provided information or policies. Evaluating on benchmarks such as -bench or VITA-Bench could provide useful evidence in this direction.
-
The paper studies the number of apps and task allocation, but another interesting dimension is the number of tools exposed within each environment. Is it more effective to train on environments containing many tools, where tool discovery itself becomes challenging, or on smaller environments with fewer tools? A controlled experiment varying the number of available tools while keeping the training budget fixed could help clarify how tool-set size and tool discovery affect generalization.
-
The paper evaluates only the BFCL multi-turn split. I would be interested in seeing performance on the broader BFCL benchmark, including its single-turn and other tool-calling settings. This could help determine whether large-scale SynthAPI training improves general tool-calling capabilities beyond long-horizon agentic interaction.
-
The appendix reports statistics such as tokens per action, but I would also find it useful to see the distribution of the number of interaction turns or steps and the total number of tokens per trajectory. Given the scale of the experiments, these statistics would make the training-data composition and computational cost easier to understand.
-
The paper evaluates transfer to benchmarks with tools and tasks that differ from the SynthAPI training environments, but I would appreciate a more explicit analysis of what determines this transfer. For example, if a model is trained on environments containing read tools and write tools, how does performance change when evaluated on environments with different APIs, schemas, or tool-set sizes? A controlled analysis along this dimension would strengthen the claims about generalizable tool-use scaling.
-
The related-work discussion could be expanded to cover several closely related works on scalable agent/tool-use training and environment generation, including AReaL, VULCAN, and Simia.
If the authors can address my main concern regarding the quality of the generated tasks and provide clear answers to the questions above, I would be willing to increase my score. However, if these concerns are not adequately addressed, I may reconsider and lower my score.
References
AReaL: https://arxiv.org/abs/2505.24298
VULCAN: https://openreview.net/forum?id=Ht4HB4pIQq - https://github.com/microsoft/vulcan
Questions for authors
Please read the weakness.
6 comments
Q4. I would be interested in seeing performance on the broader BFCL benchmark, including its single-turn and other tool-calling settings.
We added evaluations on the BFCL single-turn benchmark in the same evaluation setting as the main paper. SynthAPI-Big substantially improves the four measured tool-calling categories for the 4B and 9B models, while results for the 27B model are mixed. We see reduced performance on the irrelevance category as this tests the ability of the model to not call any tool, while all the generated SynthAPI data the agent is trained with always requires at least one tool call.
| Model | Checkpoint | Python Simple | Multiple | Parallel | Parallel Multiple | Irrelevance |
|---|---|---|---|---|---|---|
| 4B | Base | 53.0 | 55.0 | 46.0 | 32.0 | 89.6 |
| 4B | SynthAPI | 85.5 | 88.0 | 82.5 | 82.4 | 62.9 |
| 9B | Base | 83.8 | 82.9 | 78.0 | 62.0 | 84.2 |
| 9B | SynthAPI | 87.8 | 89.0 | 84.0 | 84.9 | 69.2 |
| 27B | Base | 89.5 | 92.0 | 89.5 | 85.5 | 84.9 |
| 27B | SynthAPI | 89.5 | 89.5 | 88.4 | 85.9 | 66.2 |
Q5. I would also find it useful to see the distribution of the number of interaction turns or steps and the total number of tokens per trajectory
We now updated Appendix B with histograms of both statistics. There is an average of 13.4 steps per trajectory with a maximum of 32. There are an average of 13,139 total processed tokens per trajectory with a maximum of 61,475.
Q6. Add a more explicit analysis of what determines transfer to new tools and tasks. If a model is trained on (X) environments containing (Y) read tools and (Z) write tools, how does performance change when evaluated on environments with different APIs, schemas, or tool-set sizes? A controlled analysis along this dimension would strengthen the claims about generalizable tool-use scaling.
We agree that independently characterizing these effects is an important direction, but for future work. In SynthAPI, tools are generated jointly with applications, so app diversity and tool composition are coupled. Disentangling their effects requires a separate study with independent control over tool types and differences between train and test tools.
As a limited analysis using our existing runs, we refitted the scaling law using tool counts in place of app count (A). Extrapolation MAE increases by 3.1% relative to the original formulation. Since the tool count is highly correlated with the app count, similar predictive performance is consistent with tool count serving as a proxy for app count in this dataset.
Q7. The related-work discussion could be expanded to cover several closely related works on scalable agent/tool-use training and environment generation, including AReaL, VULCAN, and Simia.
We expanded the related work section to include AReaL, VULCAN, and Simia. VULCAN and SynthAPI both use code-backed executable environments, while VULCAN constructs tools from supplied APIs. Simia uses LLMs to simulate tool responses. SynthAPI’s focus is the controlled study of how to allocate a synthetic-data budget across applications, tasks, database states, and distractor complexity. AReaL provides infrastructure for efficient RL training and is complementary to SynthAPI as it could help scale RL training in SynthAPI's scale of generated environments.
I would like to thank the authors for conducting additional experiments and providing further clarifications. However, I still have a few questions and concerns:
-
Could the authors report the number of training samples used to fine-tune the SynthAPI models in Q2?
-
Could the authors explain why SynthAPI-9B performs worse than its base model? Is there any justification for this performance drop?
-
The SynthAPI-tuned models also appear to perform worse on BFCL single-turn tasks, particularly on the Irrelevance metric. This raises concerns that the models may have primarily learned to call tools rather than determine when tool use is appropriate. Such behavior could limit their reliability in real-world settings, where verifying the need for a tool call and justifying its use are just as important as executing it. Could the authors comment on this trade-off?
I also wonder how we can meaningfully discuss laws when the models are only applicable to a narrow range of tool-calling agents, such as those evaluated on AppWorld-style tasks.
Q1. A more systematic evaluation of task and verifier quality would strengthen the analysis and the conclusions drawn from the scaling experiments.
We now ran every task and verifier through an LLM and confirmed overall high quality synthetic environments. Specifically, we use Gemini-3.1-Pro with full access to all the task information as a binary classifier to determine if a task is: ambiguous or an unrealistic instruction posed by a user. We define ambiguity as insufficient information in the instruction and initial state to determine the task’s requirements. We also classify each verifier by: accept invalid states, rejects valid states, or is hackable by the agent. Invalid acceptance means accepting a plausible completion that fails a stated requirement because the verifier does not check every criterion. Hackability means allowing an orthogonal action to bypass the intended task.
The table below reports the % of tasks and verifiers that fall into each of these categories.
As shown above, 98.5% of instructions are clearly specified and 91.9% of verifiers have none of the 3 verifier issues. The 13.2% realism flags typically arise from combining sensible operations into requests that users would be unlikely to make naturally. This identifies a mismatch in request realism, while the downstream benchmark gains demonstrate that the overall dataset still supports transferable tool-use learning. Seeding generation with real user requests could further improve naturalness. The 7.4% hackable tasks is judged by the powerful Gemini-3.1-Pro model with access to the verifier code and could be addressed during data collection with an LLM judge.
We added full details about this audit including full prompts to Appendix A.4.
Q2. Evaluating on benchmarks such as tau-bench or VITA-Bench could provide useful evidence in the direction of interacting with a simulated user and reasoning about user-provided information or policies
We evaluated the SynthAPI-Big models zero-shot on Tau2Bench, averaging over four trials per task. The 27B model improves overall performance by 4.3 percentage points, while the 9B model declines by 1.8 points. These results show mixed transfer to user-interactive environments. The improvement at 27B is encouraging, but the decline at 9B shows the limitation that SynthAPI currently contains no tasks requiring interaction with an active user, so this remains a limitation of the training distribution.
Q3. A controlled experiment varying the number of available tools while keeping the training budget fixed could help clarify how tool-set size and tool discovery affect generalization.
We added an experiment varying how many distractor tools are exposed during training. Tools used in the teacher trajectory are always available, and we additionally retain 0%, 25%, 50%, or 100% of the remaining tools. Thus, 100% is the setting from the main paper where all tools are available. Each setting trains Qwen3.5-4B with 16 apps and 512 tasks in the setting of Figure 2a, followed by evaluation on SynthAPI Holdout and AppWorld.
The below table shows small differences, spanning two percentage points on each benchmark. This preliminary finding suggests limited sensitivity to distractor availability during training.