acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Tool Agents via Controllable Environment Synthesis

Abstract

We explore how to build performant large language model (LLM) tool-use agents that complete complex, multi-step tasks. Specifically, we study how data composition in synthetically generated tool-use environments affects the training of such agents. To do so, we propose the SynthAPI pipeline, which explicitly controls the diversity of generated environments along the axes of tools, tasks, and the databases on which the tools operate. By systematically varying SynthAPI's generation controls across 849 trained models of four different sizes, we derive a scaling law that predicts downstream tool-agent generalization based on the scale and composition of the synthesized environments. These predictable relationships reveal several lessons for scaling tool-agent data. As data budgets grow, scaling tool and database diversity becomes more effective than increasing the number of tasks, and increasing data complexity yields larger gains. Scaling according to these findings yields agents that outperform models trained on data from prior work in large-scale tool-agent and synthetic environment generation across four evaluation benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, synthetic environment generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cb4pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the composition of synthetic tool-use data affects the performance of LLM agents. The authors introduce SynthAPI, a pipeline that generates executable environments with apps, APIs, database states, tasks, verifiers, and oracle solutions. Rather than only scaling the total amount of data, they systematically vary factors such as the number of apps, tasks per app, database variations, and distractor complexity. Using more than 458K generated tasks and 849 trained models, they derive scaling laws that relate these data properties to downstream performance. Their main finding is that as the training budget grows, increasing tool and database diversity becomes more useful than simply adding more tasks. They then use these findings to build a larger training setup, which improves performance on AppWorld, BFCL, ASTRA-Bench, and a SynthAPI holdout set.

Critical strengths
  1. The paper conducts a large-scale and systematic study with more than 458,000 generated tasks and 849 trained models.
  2. SynthAPI provides explicit control over key data-generation factors, including app diversity, task diversity, database variation, and task complexity.
  3. The paper derives practical scaling-law insights and shows that the resulting training recipe improves performance across multiple tool-use benchmarks.
Critical weaknesses

Although the overall direction of the work is interesting, I have one main concern and several questions that I would appreciate the authors addressing.

  1. The authors use an oracle solver and executable verifier to filter generated tasks, which provides a useful check of task executability. However, this does not necessarily establish the semantic quality of the tasks or the correctness of the generated verifiers at scale. For example, a task may be technically solvable while still being unrealistic, ambiguous, poorly specified, or paired with an imperfect verifier. Given that 458,752 tasks are generated and the paper states that the generated tasks are otherwise relatively unsupervised, I believe a more systematic evaluation of task and verifier quality would strengthen the analysis and the conclusions drawn from the scaling experiments.

  2. The evaluation mainly focuses on settings where interaction with an active user is not a primary challenge. I am curious whether a model fine-tuned on environments containing 10,472 APIs, including 6,266 write and 4,206 read APIs, can transfer its learned tool-use behavior to environments where the agent must interact with a simulated user and reason about user-provided information or policies. Evaluating on benchmarks such as τ\tau-bench or VITA-Bench could provide useful evidence in this direction.

  3. The paper studies the number of apps and task allocation, but another interesting dimension is the number of tools exposed within each environment. Is it more effective to train on environments containing many tools, where tool discovery itself becomes challenging, or on smaller environments with fewer tools? A controlled experiment varying the number of available tools while keeping the training budget fixed could help clarify how tool-set size and tool discovery affect generalization.

  4. The paper evaluates only the BFCL multi-turn split. I would be interested in seeing performance on the broader BFCL benchmark, including its single-turn and other tool-calling settings. This could help determine whether large-scale SynthAPI training improves general tool-calling capabilities beyond long-horizon agentic interaction.

  5. The appendix reports statistics such as tokens per action, but I would also find it useful to see the distribution of the number of interaction turns or steps and the total number of tokens per trajectory. Given the scale of the experiments, these statistics would make the training-data composition and computational cost easier to understand.

  6. The paper evaluates transfer to benchmarks with tools and tasks that differ from the SynthAPI training environments, but I would appreciate a more explicit analysis of what determines this transfer. For example, if a model is trained on XX environments containing YY read tools and ZZ write tools, how does performance change when evaluated on environments with different APIs, schemas, or tool-set sizes? A controlled analysis along this dimension would strengthen the claims about generalizable tool-use scaling.

  7. The related-work discussion could be expanded to cover several closely related works on scalable agent/tool-use training and environment generation, including AReaL, VULCAN, and Simia.

If the authors can address my main concern regarding the quality of the generated tasks and provide clear answers to the questions above, I would be willing to increase my score. However, if these concerns are not adequately addressed, I may reconsider and lower my score.


References

AReaL: https://arxiv.org/abs/2505.24298

VULCAN: https://openreview.net/forum?id=Ht4HB4pIQq - https://github.com/microsoft/vulcan

Simia: https://arxiv.org/abs/2511.01824

Questions for authors

Please read the weakness.

6 comments

Q1. A more systematic evaluation of task and verifier quality would strengthen the analysis and the conclusions drawn from the scaling experiments.

We now ran every task and verifier through an LLM and confirmed overall high quality synthetic environments. Specifically, we use Gemini-3.1-Pro with full access to all the task information as a binary classifier to determine if a task is: ambiguous or an unrealistic instruction posed by a user. We define ambiguity as insufficient information in the instruction and initial state to determine the task’s requirements. We also classify each verifier by: accept invalid states, rejects valid states, or is hackable by the agent. Invalid acceptance means accepting a plausible completion that fails a stated requirement because the verifier does not check every criterion. Hackability means allowing an orthogonal action to bypass the intended task.

The table below reports the % of tasks and verifiers that fall into each of these categories.

Ambiguous TaskUnrealistic TaskVerifier Accepts InvalidVerifier Rejects ValidVerifiers HackableAny Verifier Issue
1.5%13.2%1.2%2.3%7.4%8.1%

As shown above, 98.5% of instructions are clearly specified and 91.9% of verifiers have none of the 3 verifier issues. The 13.2% realism flags typically arise from combining sensible operations into requests that users would be unlikely to make naturally. This identifies a mismatch in request realism, while the downstream benchmark gains demonstrate that the overall dataset still supports transferable tool-use learning. Seeding generation with real user requests could further improve naturalness. The 7.4% hackable tasks is judged by the powerful Gemini-3.1-Pro model with access to the verifier code and could be addressed during data collection with an LLM judge.

We added full details about this audit including full prompts to Appendix A.4.

Q2. Evaluating on benchmarks such as tau-bench or VITA-Bench could provide useful evidence in the direction of interacting with a simulated user and reasoning about user-provided information or policies

We evaluated the SynthAPI-Big models zero-shot on Tau2Bench, averaging over four trials per task. The 27B model improves overall performance by 4.3 percentage points, while the 9B model declines by 1.8 points. These results show mixed transfer to user-interactive environments. The improvement at 27B is encouraging, but the decline at 9B shows the limitation that SynthAPI currently contains no tasks requiring interaction with an active user, so this remains a limitation of the training distribution.

ModelAirlineRetailTelecomOverall
Base-27B0.6900.4190.3950.501
SynthAPI-27B0.7250.4470.4610.544
Base-9B0.6750.3970.4040.492
SynthAPI-9B0.6720.3810.3710.474

Q3. A controlled experiment varying the number of available tools while keeping the training budget fixed could help clarify how tool-set size and tool discovery affect generalization.

We added an experiment varying how many distractor tools are exposed during training. Tools used in the teacher trajectory are always available, and we additionally retain 0%, 25%, 50%, or 100% of the remaining tools. Thus, 100% is the setting from the main paper where all tools are available. Each setting trains Qwen3.5-4B with 16 apps and 512 tasks in the setting of Figure 2a, followed by evaluation on SynthAPI Holdout and AppWorld.

The below table shows small differences, spanning two percentage points on each benchmark. This preliminary finding suggests limited sensitivity to distractor availability during training.

Distractor RatioSynthAPI-HoldoutAppWorld
0%0.500.44
25%0.500.43
50%0.510.44
100%0.520.45

Q4. I would be interested in seeing performance on the broader BFCL benchmark, including its single-turn and other tool-calling settings.

We added evaluations on the BFCL single-turn benchmark in the same evaluation setting as the main paper. SynthAPI-Big substantially improves the four measured tool-calling categories for the 4B and 9B models, while results for the 27B model are mixed. We see reduced performance on the irrelevance category as this tests the ability of the model to not call any tool, while all the generated SynthAPI data the agent is trained with always requires at least one tool call.

ModelCheckpointPython SimpleMultipleParallelParallel MultipleIrrelevance
4BBase53.055.046.032.089.6
4BSynthAPI85.588.082.582.462.9
9BBase83.882.978.062.084.2
9BSynthAPI87.889.084.084.969.2
27BBase89.592.089.585.584.9
27BSynthAPI89.589.588.485.966.2

Q5. I would also find it useful to see the distribution of the number of interaction turns or steps and the total number of tokens per trajectory

We now updated Appendix B with histograms of both statistics. There is an average of 13.4 steps per trajectory with a maximum of 32. There are an average of 13,139 total processed tokens per trajectory with a maximum of 61,475.

Q6. Add a more explicit analysis of what determines transfer to new tools and tasks. If a model is trained on (X) environments containing (Y) read tools and (Z) write tools, how does performance change when evaluated on environments with different APIs, schemas, or tool-set sizes? A controlled analysis along this dimension would strengthen the claims about generalizable tool-use scaling.

We agree that independently characterizing these effects is an important direction, but for future work. In SynthAPI, tools are generated jointly with applications, so app diversity and tool composition are coupled. Disentangling their effects requires a separate study with independent control over tool types and differences between train and test tools.

As a limited analysis using our existing runs, we refitted the scaling law using tool counts in place of app count (A). Extrapolation MAE increases by 3.1% relative to the original formulation. Since the tool count is highly correlated with the app count, similar predictive performance is consistent with tool count serving as a proxy for app count in this dataset.

Q7. The related-work discussion could be expanded to cover several closely related works on scalable agent/tool-use training and environment generation, including AReaL, VULCAN, and Simia.

We expanded the related work section to include AReaL, VULCAN, and Simia. VULCAN and SynthAPI both use code-backed executable environments, while VULCAN constructs tools from supplied APIs. Simia uses LLMs to simulate tool responses. SynthAPI’s focus is the controlled study of how to allocate a synthetic-data budget across applications, tasks, database states, and distractor complexity. AReaL provides infrastructure for efficient RL training and is complementary to SynthAPI as it could help scale RL training in SynthAPI's scale of generated environments.

I would like to thank the authors for conducting additional experiments and providing further clarifications. However, I still have a few questions and concerns:

  1. Could the authors report the number of training samples used to fine-tune the SynthAPI models in Q2?

  2. Could the authors explain why SynthAPI-9B performs worse than its base model? Is there any justification for this performance drop?

  3. The SynthAPI-tuned models also appear to perform worse on BFCL single-turn tasks, particularly on the Irrelevance metric. This raises concerns that the models may have primarily learned to call tools rather than determine when tool use is appropriate. Such behavior could limit their reliability in real-world settings, where verifying the need for a tool call and justifying its use are just as important as executing it. Could the authors comment on this trade-off?

I also wonder how we can meaningfully discuss laws when the models are only applicable to a narrow range of tool-calling agents, such as those evaluated on AppWorld-style tasks.

Reviewer_1P7zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SynthAPI, a pipeline for generating executable application environments, stateful tool-use tasks, verifiers, and supervised agent trajectories. It controls the number of applications, tasks per application, initial database states, and distractor data. Across 849 SFT runs with four Qwen3.5 model sizes, the authors study how these factors affect transfer and fit a predictive scaling model. They hold out the highest-compute runs for extrapolation and use the analysis to select data recipes that improve downstream tool-agent performance. The controlled sweep is the main contribution. The stronger interpretation as a compute-optimal scaling law needs more precise cost accounting and validation outside the sampled allocation structure.

Critical strengths
  1. The scale and control of the study are strong. The authors train 849 models rather than comparing a few hand-picked recipes, and the fit uses all fitting-set runs rather than only the displayed frontier. The high-compute holdout is a useful test of budget extrapolation. Appendix E.3 also reproduces the qualitative application-versus-task trend with SimSAT, giving some evidence beyond the main synthesis pipeline.

  2. The synthetic environments are executable and stateful, and the synthesis procedure checks that the verifier rejects the initial state and accepts an oracle solution. The paper evaluates transfer on external benchmarks as well as held-out SynthAPI applications. The fixed-task-count recipe comparison in Table 1 is informative: for the 4B model, the selected recipe reaches 45.0 on AppWorld Test-C versus 38.4 for the alternative allocation. The large-scale runs also show substantial transfer gains, although not every metric improves monotonically.

Critical weaknesses
  1. The joint law needs a test on unseen factor combinations. The high-compute holdout tests budget extrapolation, while Equation 2 combines application count, database breadth, and distractor depth as ASD. That form assumes these factors can substitute for one another. The sequential sweeps and SimSAT results support the qualitative trends, but do not independently test this assumption. The authors appropriately acknowledge validation in one pipeline; within that scope, held-out combinations and uncertainty estimates would make the predicted allocations more convincing.

  2. The compute estimate needs an unambiguous token definition. Section 4.3 uses approximately 6ML FLOPs and calls L supervision tokens. Section 4.1 distinguishes 721 million supervised tokens from 5.9 billion total tokens, and Appendix B masks non-agent tokens from the loss. If L includes only loss-bearing assistant tokens, the estimate misses computation on input context. This could affect comparisons when distractors and tool observations change sequence length. Please clarify the accounting and check the preferred allocations using all processed tokens or measured FLOPs; fixed task counts alone do not match training compute.

Questions for authors
  1. Does L in the compute estimate include all processed tokens, including masked prompt and tool-observation tokens? Please report processed-token counts or measured training costs for the Opt/SubOpt comparisons, and check whether the inferred optimal allocation changes under that accounting.

  2. Can the fitted law predict held-out factor combinations, for example different A, S, and D settings with comparable products, rather than only higher-compute runs? How much does prediction error change if these factors receive separate coefficients?

  3. Does the maximum used for score normalization include held-out runs? If so, please check whether fitting-only normalization changes the predictions, or report holdout errors in raw success rates. This is a validation detail, not evidence against the raw gains. Confidence intervals over application samples or training seeds would also help assess recipe stability where score differences are small.

  4. The AI Use Statement mentions manual rollout checks for reward hacking. Beyond those and the initial-state/oracle checks, can you report the audit sample size and quantitative verifier error rates? Wrong but plausible action sequences, alternative valid solutions, and reward-hacking attempts would be useful tests. The privileged-state oracle establishes feasibility, but does not by itself establish that the verifier accepts exactly the intended outcomes.

2 comments

Q1. Does L in the compute estimate include all processed tokens, including masked prompt and tool-observation tokens? Please report processed-token counts or measured training costs for the Opt/SubOpt comparisons, and check whether the inferred optimal allocation changes under that accounting.

In the paper, L includes only supervised assistant tokens and excludes masked prompt and tool-observation tokens.

Opt and SubOpt train on an equal number of tasks and trajectories, as data collection is based on tasks rather than tokens. The dataset for SubOpt has 1.361B processed tokens and 166M supervision tokens while the dataset for Opt has 1.445B processed tokens and 185M supervision tokens which is a 6.2%/11.4% increase respectively. However, at 32,768 tasks, the selected recipe already reaches the largest values studied for all three diversity dimensions: (A=64), (S=4), and (D=4). This budget lies well beyond the observed crossovers favoring greater diversity, so the recommendation within our evaluated ranges remains unchanged. Changing the compute accounting does not change the recipe selected for this fixed-task comparison. We updated the paper to explicitly distinguish this task-matched comparison from a compute-matched comparison: Opt processes 6.2% more tokens.

We also added a version of our scaling law from Figure 3 where training FLOPs are now estimated with respect to all processed tokens. In this accounting, the data composition aware scaling law achieves a held-out MAE of 0.0281 as opposed to the original 0.0285.

Q2a. Can the fitted law predict held-out factor combinations, for example different A, S, and D settings with comparable products, rather than only higher-compute runs?

We added an evaluation of leaving out one of the (A,S,D) product factor runs. For each observed A,S,DA,S,D product, we remove one run with that product and fit the scaling law on the remaining runs and repeat for all possible runs to leave out. Averaged over all fits, the held-out combination MAE is 0.040 compared to 0.0398 MAE on random held-out runs. So the interpolation to unseen combinations performs similarly to interpolating to random held out runs. Note that our original experiments about compute extrapolation, also measure generalization to new (A,S, D) combinations for larger A⋅S⋅DA\cdot S \cdot D products.

Q2b. How much does prediction error change if the A, S, and D factors receive separate coefficients?

We added separate coefficients to the A, S, and D factors in the scaling law from Eq. 2. This reduces average interpolation MAE from 0.0398 to 0.0381 (4.3% improvement), while the average extrapolation MAE worsens from 0.0285 to 0.0298 (4.6% degradation). We updated Appendix E.1 to discuss this alternative scaling model.

Q3a. Does the maximum used for score normalization include held-out runs? Does fitting-only normalization change the predictions?

Yes, our score normalization uses held-out runs. Specifically, runs are normalized by the maximum scoring run per benchmark so all scores are upper bounded by 1. We re-run with in-fit normalization and find the prediction only slightly changes with extrapolation MAE slightly changes from 0.0285 to 0.0295.

Q3b. Confidence intervals over application samples or training seeds would also help assess recipe stability where score differences are small.

We repeated the 4B app vs. task allocation experiment from Figure 2a with two additional seeds, varying both the training and data-sampling seeds. Figure 13 in Appendix E.5 now reports means and standard deviations across three runs. The standard deviation is generally low with an average of 1.6 points on both AppWorld and SynthAPI-Holdout (at most 3.3) and 1.5 points on the combined scale (at most 3.7), which is 3–5% of the score. We observe the same crossover behavior where higher app count is better with increasing total task count appears consistently in all 3 seeds.

Q4. The AI Use Statement mentions manual rollout checks for reward hacking. Beyond those and the initial-state/oracle checks, can you report the audit sample size and quantitative verifier error rates?

We now ran every task and verifier through an LLM and confirmed overall high quality synthetic environments. Specifically, we use Gemini-3.1-Pro with full access to all the task information as a binary classifier to determine if a task is: ambiguous or an unrealistic instruction posed by a user. We define ambiguity as insufficient information in the instruction and initial state to determine the task’s requirements. We also classify each verifier by: accept invalid states, rejects valid states, or is hackable by the agent. Invalid acceptance means accepting a plausible completion that fails a stated requirement because the verifier does not check every criterion. Hackability means allowing an orthogonal action to bypass the intended task.

The table below reports the % of tasks and verifiers that fall into each of these categories.

Ambiguous TaskUnrealistic TaskVerifier Accepts InvalidVerifier Rejects ValidVerifiers HackableAny Verifier Issue
1.5%13.2%1.2%2.3%7.4%8.1%

As shown above, 98.5% of instructions are clearly specified and 91.9% of verifiers have none of the 3 verifier issues. The 13.2% realism flags typically arise from combining sensible operations into requests that users would be unlikely to make naturally. This identifies a mismatch in request realism, while the downstream benchmark gains demonstrate that the overall dataset still supports transferable tool-use learning. Seeding generation with real user requests could further improve naturalness. The 7.4% hackable tasks is judged by the powerful Gemini-3.1-Pro model with access to the verifier code and could be addressed during data collection with an LLM judge.

We added full details about this audit including full prompts to Appendix A.4.

Reviewer_wD8ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SynthAPI, a pipeline that uses an LLM to build synthetic tool-use environments. Each app consists of executable Python code defining a database schema and APIs that read and modify the database. For each app the pipeline also generates tasks, starting database states, success verifiers and oracle solutions. The key idea is control over four properties of the training data: how many apps there are, how many tasks each app gets, how many different starting databases each task has, and how much distractor data those databases contain. The authors collect GLM-5.2 solutions to these tasks and fine-tune Qwen3.5 models from 2B to 27B on them, training 849 models in total while sweeping these settings. They find that with small budgets it works best to concentrate tasks in a few apps. As the budget grows, adding more apps, more database variants and more distractor data matters more than simply generating more tasks. They fit a scaling law in which app diversity, database diversity, and distractor complexity determine a predicted performance ceiling, while tasks per app control progress toward that ceiling. The authors report low prediction error on held-out runs with compute budgets up to 16× larger than the largest used for fitting. Models trained on the full dataset outperform the strongest reported baselines by 7.6–11.6 percentage points on AppWorld Test-Challenge and 2.0–3.7 points on BFCL, although these comparisons use different training regimes and do not isolate the effect of the synthesis method.

Critical strengths
  • An important and useful research question: Understanding how to allocate synthetic-data budget and what should be the split between new tools, more tasks and more database variants is important. The finding that diversity matters more as budgets grow is also useful.
  • Rigorous, large-scale setup and strong empirical results: The study systematically varies data composition across 849 trained models spanning four model sizes, providing substantial evidence for the benefits of tool and database diversity. Consistent gains across three external benchmarks, particularly the substantial improvements on AppWorld, demonstrate the practical value of the resulting training recipe.
Critical weaknesses
  • Unclear generality across model families: all scaling experiments use Qwen3.5 students distilled from a single GLM-5.2 teacher. The fitted parameters already vary widely across sizes within this one family, so it’s unclear whether the law, its coefficients or the recommended allocations hold for other model families or teachers. Even a small replication on a second family, at one or two sizes, would substantially strengthen the claim.
  • While the scaling laws are interesting, the qualitative finding that tool and database diversity become more valuable as budgets grow is largely expected. A key contribution is therefore precise, actionable allocation guidance. However, the reported crossover points lack uncertainty estimates, and the paper does not clearly report repetitions across training seeds or independently sampled app subsets, making their robustness difficult to assess. Furthermore, the generated training tasks are restricted to single-app interactions and involve a median of two functionalities. Although the resulting agents transfer to multi-app benchmarks, this does not establish that the fitted law or recommended allocations remain valid when training on more complex, multi-app or longer-horizon workflows. The evidence therefore leaves uncertainty about how reliably practitioners could use the law to choose allocations in their own settings.
Questions for authors

Please see weaknesses.

2 comments

W1. Unclear generality across model families: even a small replication on a second family, at one or two sizes, would substantially strengthen the claim.

We replicated the app vs. task allocation experiment from Figure 2a on IBM-Granite-4.2-3B and 8B. Both models reproduce the qualitative finding that concentrating tasks in fewer apps works better at small task budgets, while increasing app diversity becomes beneficial at larger budgets. Specifically, for the 8B model, 8 apps does best at 128-512 tasks and A=64 beyond 4096 tasks. For the 3B model, 8 apps does best from 128-256 tasks, A=32 512-2048, and A=64 beyond 2048. For both models, the fitted performance ceiling increases as app count increases from 8 to 64. We added these results with the plots and full details to a new Appendix E.4.

W2. The paper does not clearly report repetitions across training seeds or independently sampled app subsets, making their robustness difficult to assess.

We repeated the 4B app vs. task allocation experiment from Figure 2a with two additional seeds, varying both the training and data-sampling seeds. The seed variation also samples different app subsets. Figure 13 in Appendix E.5 now reports means and standard deviations across three runs. The standard deviation is generally low with an average of 1.6 points on both AppWorld and SynthAPI-Holdout (at most 3.3) and 1.5 points on the combined scale (at most 3.7), which is 3–5% of the score. We observe the same crossover behavior where higher app count is better with increasing total task count appears consistently in all 3 seeds.

Thanks for the response and new experiments. They address most of my concern and therefore i maintain my positive assessment and score at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.