- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Davinci-Tool and Davinci-ToolData, an execution-grounded pipeline for scaling multi-step tool-use supervision. Each data instance preserves a restorable initial state, tool trajectory, outcome evidence, and executable verification result. Execution-Grounded Task Synthesis (EGTS) adapts existing environments by exploring feasible outcomes, synthesizing tasks from observed evidence, restoring the initial state, and collecting task-conditioned replays that must reproduce the target outcome. Executable Environment Synthesis (EES) instead builds and audits executable domains, then generates state-grounded tasks and verifiers either within a single domain (EES-Single) or across composed domains with shared entities, policies, and dependencies (EES-Cross).
The authors collect 30,395 verified trajectories spanning 2,260 invoked tool names. They first apply SFT to Qwen3-Coder-30B-A3B-Instruct, then construct 1,150 same-context preference pairs from restorable EES states and use executable downstream outcomes to supervise DPO. Davinci-Tool-30B achieves extensive improvement on three benchmarks after SFT and further DPO.
Critical strengths
-
A coherent execution-grounded data substrate that supports both demonstrations and preferences.
-
Strong empirical gains with broad coverage. For example, Davinci-Tool improves the controlled Qwen3-Coder-30B backbone by over 30 points on the three reported benchmark averages after SFT, and DPO adds further gains on all three aggregates.
Critical weaknesses
-
The methodological novelty and data provenance are not sufficiently clear. A growing body of work has already explored executable-environment-based synthesis for tool-use agents, including EnvScaler as well as AWM [1], ScaleEnv [2], AgentWorld [3], and EnvFactory [4], several of which are not cited or compared. As a result, it is difficult to identify the unique design contribution of Davinci-Tool beyond prior pipelines. In addition, the paper does not clearly explain where the authored systems used by EGTS or the synthesized domains used by EES come from, nor whether decontamination against the three evaluation benchmarks is performed. Given the very large gains (over 30 absolute points on all three benchmarks), stronger analysis of data provenance, overlap, and novelty is necessary.
-
The experimental design leaves several important gaps. The paper does not report Davinci-ToolData SFT+DPO results for Qwen3-8B and Qwen3-14B, nor corresponding MCP-Atlas results for these models, making it difficult to assess whether the full training recipe generalizes across scales. This is especially important because Qwen3-8B/14B are mixed reasoning models, while Qwen3-Coder-30B-A3B-Instruct is an instruct model, yet the paper does not explain how the data or training format is adapted across these model types. In addition, the proposed executable verification naturally appears suitable for online RL, which is highly relevant to recent agent-training work, but the paper only uses it for DPO. A clearer justification or RL-based comparison would strengthen the contribution. Finally, the results on Airline domain in -Bench are missing without any reasons.
[1] Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning, 2026.
[2] ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training, 2026.
[3] Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence, 2026.
[4] EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL, 2026.
Questions for authors
-
Do the constructed SFT data contains reasoning content? If not, how do you use in Qwen3-8B/14B? Are those results without reasoning?
-
Why do you not present the results on Airline domain of -Bench?
-
Have you analyze the overlaps between your data and benchmark environments?
-
Why do you not conducting experiments on RL?
7 comments
Response to Reviewer Umjt (2/4)
Response to Critical Weakness 2: Data quality, model coverage, and post-training scope
Evidence for the reported gains. Our primary goal is to build high-quality, executable-verified agentic training data and demonstrate its value for supervised learning. SFT accounts for most of the improvement over Qwen3-Coder-30B. It raises performance by 32.7 points on τ²-bench, 30.8 on BFCL v4, and 35.9 on MCP-Atlas. We thank the reviewer for the attention to evaluation reliability, which led us to recheck the original score files and correct the base BFCL score from 30.4 to 29.9 in the revised Table 2. Decision-level DPO adds 3.1, 2.1, and 3.4 points, respectively. We also evaluated the collection teacher, DeepSeek-V4-Pro, under the same protocols. The trained 30B model scores 62.75 on BFCL, above the teacher’s 54.88, and reaches 88.16 versus 90.35 on τ²-bench. In our additional frontier comparison under the same protocols, it also exceeds Kimi-K3 (61.38) and stays close to GLM-5.2 (66.38) on BFCL.
| Model / training corpus | SFT trajectories | τ²-bench Avg. | BFCL v4 | MCP-Atlas |
|---|---|---|---|---|
| Qwen3-Coder-30B (base) | n/a | 52.40 | 29.88 | 16.90 |
| Qwen3-Coder-30B + Toucan-1.5M SFT | ~1.5M | 77.70 | 41.90 | 22.50 |
| Davinci-Tool-30B-SFT | 30,395 | 85.09 | 60.63 | 52.81 |
| Davinci-Tool-30B (SFT + DPO) | 30,395 | 88.16 | 62.75 | 56.18 |
| DeepSeek-V4-Pro (collection teacher) | n/a | 90.35 | 54.88 | 66.29 |
The Toucan-1.5M SFT baseline offers another comparison of data effectiveness: using the same 30B backbone and SFT settings, our substantially smaller verified corpus yields +7.4 / +18.7 / +30.3 points on τ²-bench, BFCL, and MCP-Atlas. These results are consistent with our data construction process. The teacher is used for exploration and collection. EGTS guides exploration with progressive tool exposure and reusable experience, then collects new task-conditioned replays. Only trajectories that satisfy execution and outcome checks enter SFT. EES supplies additional verified demonstrations from audited domains. This process turns teacher-guided exploration into verified, task-grounded supervision and helps explain the strong BFCL results.
Data source complementarity. We also trained four SFT models with 8,000 trajectories each, the same initialization, and the same training settings. The mixed dataset is best or tied for best on all four reported metrics. For example, on MCP-Atlas it scores 55.06, compared with 48.31 for the strongest single source. Compared with EES-Cross, the mixture improves BFCL by 11.0 points (paired McNemar p < 0.001) and MCP-Atlas by 6.75 points (p = 0.031). The different strengths of EGTS, EES-Single, and EES-Cross support the value of combining their supervision.
| SFT data (8,000 trajectories) | τ² Retail | τ² Telecom | BFCL v4 | MCP-Atlas |
|---|---|---|---|---|
| EGTS | 64.91 | 89.47 | 58.13 | 40.45 |
| EES-Single | 64.91 | 97.37 | 55.00 | 41.57 |
| EES-Cross | 73.68 | 96.49 | 47.50 | 48.31 |
| Three-source mixture | 74.56 | 97.37 | 58.50 | 55.06 |
Preference learning evidence. Across three DPO training seeds, the mean gains over SFT are +2.05 on τ²-bench, +1.83 on BFCL, and +3.75 on MCP-Atlas. The average Retail gain is +5.26 with a task-bootstrap 95% CI of [1.75, 9.65]. We also compared outcome-verified preferences with DeepSeek-V4-Pro relabeling on the same 1,150 decision pairs using the same SFT backbone and DPO settings:
| Post-SFT comparison (seed 42) | τ² Retail | BFCL v4 | MCP-Atlas |
|---|---|---|---|
| SFT only | 73.68 | 60.63 | 52.81 |
| DPO, DeepSeek preference labels | 75.44 | 60.63 | 56.18 |
| DPO, verified-outcome labels | 79.82 | 62.75 | 56.18 |
The verified-label model improves Retail by 4.38 points and BFCL by 2.12 points over the LLM-relabeled counterpart. Along with the multi-seed gains, this supports using executable outcomes for decision-level preference supervision.
Response to Reviewer Umjt (3/4)
Response to Question 1: Reasoning format and smaller-model transfer
Question 1: Do the constructed SFT data contains reasoning content? If not, how do you use in Qwen3-8B/14B? Are those results without reasoning?
Our smaller-model experiments assess the quality and transferability of Davinci-ToolData as SFT supervision. The demonstrations contain no explicit reasoning traces, and Qwen3-8B/14B use non-thinking mode with model-compatible tool-call serialization. Both backbones show substantial SFT gains on τ²-bench and BFCL, demonstrating that the data benefits models beyond the primary 30B backbone.
| Model | τ²-bench Avg. | BFCL v4 | MCP-Atlas diagnostic* |
|---|---|---|---|
| Qwen3-8B (base, non-thinking) | 32.5 | 15.4 | 6.75 |
| Qwen3-8B + Davinci-ToolData SFT | 74.6 | 33.9 | 6.74 |
| Qwen3-14B (base, non-thinking) | 33.4 | 21.1 | 2.25 |
| Qwen3-14B + Davinci-ToolData SFT | 76.8 | 41.4 | 3.37 |
MCP-Atlas scores were returned by the official evaluator at the original 40,960-token limit. Unfinished tasks, including context-overflow runs, count as failures. These are not extended-context evaluations.
We applied SFT followed by DPO to our primary 30B model to further demonstrate that the same environment and verifier framework can produce both verified demonstrations and decision-level preferences. This prioritizes the paper's data-construction contribution while keeping post-training costs manageable. Extending full SFT+DPO to 8B/14B is a natural direction beyond the current experiments.
For MCP-Atlas, the 8B and 14B base and SFT checkpoints used a 40,960-token context window, without YaRN. The benchmark includes extended interactions with real MCP servers, during which tool schemas and accumulated messages can exceed the available window (Bandi et al., 2026). We inspected the runs and confirmed context overflow in some trajectories. The official evaluator scored unfinished tasks as failures. Other failures arose from the smaller models' difficulty completing complex tool workflows. The values in the table are therefore scores under the original context-limited evaluation setting, rather than scores from an extended-context comparison. Extending the window for four checkpoints would also require additional serving adaptation and evaluation resources. Given our time and compute budget, we prioritized the complete τ²-bench and BFCL transfer evaluations. We will include these MCP-Atlas scores and the precise evaluation setting in the revised manuscript.
Response to Question 2: Choice of τ²-bench domains
Question 2: Why do you not present the results on Airline domain of τ²-Bench?
When selecting our τ²-bench evaluation scope, we considered reported concerns about the Airline benchmark. The Claude Opus 4.5 System Card documented policy loopholes in Airline and showed that corrections changed its reported score from 70.1 to 87.8. Subsequent Claude Opus 4.6 and Claude Sonnet 4.6 reports omitted Airline because these fixes had not yet been incorporated into the upstream benchmark. We also noted that the Gemini 3.1 Pro model card presents Retail and Telecom in its main τ²-bench comparison, while the MiniMax M2.5 model card separately reports Telecom. Given these findings and the differences between Airline evaluation versions, we chose Retail and Telecom for our evaluation. Retail measures stateful customer-service workflows, while Telecom evaluates dual-control interactions in which both the agent and user can use tools. We use this two-domain scope for every model and report the macro average of Retail and Telecom.
Response to Reviewer Umjt (1/4)
We sincerely thank the reviewer for the thorough and constructive assessment. We especially appreciate your recognition of the shared execution-grounded foundation for demonstrations and preferences, as well as the substantial gains across all three benchmarks. Your careful attention to methodological novelty, data provenance, and evaluation reliability has helped us present these contributions more clearly.
We clarify the relationship between EES and EnvScaler, distinguish our new design choices from prior pipelines, and provide an overlap audit. We also report teacher and frontier-model evaluations, matched-count data-source comparisons, and additional DPO analyses. Together, these results help explain where the gains come from and how the pipeline is evaluated.
Response to Critical Weakness 1: Methodological contribution and data provenance
Our central contribution is a pipeline for constructing high-quality, executable-verified tool-use data from different environment sources. EGTS integrates 11 independently authored systems, including API-Bank, AppWorld, and Toolathlon (full inventory in the appendix), and converts explored outcomes into verified, task-conditioned replays. They share one collection and verification pipeline, with a median system-specific adapter size of about 1.1K lines. EES-Single starts from 191 executable EnvScaler domains (Song et al., 2026). We test their tool implementations, repair issues found during execution, and construct verified tasks on the audited environments. EES-Cross then composes these single-domain environments into 246 cross-domain settings with shared entities, policies, and state dependencies. Its trajectories show a 14.50% domain-switch ratio, with 6.39% involving cross-domain entity transfer. These steps extend the available verified supervision from individual environments to coordinated, multi-domain workflows.
We appreciate the references to EnvScaler (Song et al., 2026), ScaleEnv (Tu et al., 2026), Agent World Model (Wang et al., 2026), Agent-World (Dong et al., 2026), and EnvFactory (Xu et al., 2026). These works contribute to executable environment construction and agent training. Our work connects audited single-domain environments to explicit cross-domain composition, verified replay from authored systems, and the reuse of states and verifiers for SFT and decision-level DPO.