acceptodds
Under review as a conference paper at ICLR 2027

Verifiable State Transitions as Unified Supervision for Training Stateful Tool-Use Agents

Abstract

Training stateful tool-use agents requires evaluating not only whether they invoke the correct tools but also whether their actions produce the intended changes in persistent environments. However, supervision based primarily on final task outcomes does not explicitly assess intermediate execution, while tool-call matching alone cannot verify the actual effects of executed actions. Reliably assessing solution quality therefore requires jointly evaluating tool-call correctness and the resulting state transitions. We introduce VESTA (Verifiable Execution and State Transitions for Agents), a unified training framework that derives execution-grounded supervision from verifiable state transitions. VESTA constructs executable environments from API specifications and synthesizes tasks in an executable-first manner, recording reference tool calls and expected environment states at each task turn as state-transition contracts. These contracts enable joint verification of tool-call correctness and state changes, supporting SFT trajectory selection, policy-conditioned RL task selection, and RL reward computation without learned or LLM-based reward judges. Experiments across three Qwen3 model scales on BFCL-v3 Multi-Turn, ACEBench-Agent, and -Bench show improvements over the corresponding base models in all nine model–benchmark comparisons, with RL further improving over SFT in every setting. On Qwen3-4B, VESTA achieves absolute gains of 12.63, 20.00, and 8.76 points on the three benchmarks, respectively. Across model scales, VESTA outperforms EnvScaler in seven of nine comparisons and AWM in all six available comparisons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-Use Agents, Executable Environments, State-Transition Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w1SzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VESTA, a framework for training stateful tool-use agents with execution-derived supervision. Starting from API specifications, VESTA constructs replay-validated executable environments and workflow skeletons, then records reference tool calls and normalized pre/post states as state-transition contracts. These contracts are reused for SFT trajectory selection, RL rewards, and policy-conditioned RL task selection. The authors retain 131 environments, train Qwen3 models, and evaluate on three tool-use benchmarks. VESTA improves all 9 model–benchmark averages over the corresponding base models and generally outperforms EnvScaler and AWM, although those baseline comparisons are not fully controlled.

Critical strengths
  1. Reusable execution-grounded supervision. VESTA derives tool calls and state transitions from executable workflows and reuses them for SFT selection, RL rewards, and task sampling. The RL reward combines tool-call matching with state verification and guards against unintended state changes. And evaluations on three model sizes and three benchmarks validate the effectiveness and corresponding designs.
Critical weaknesses
  1. The novelty and effectiveness of the proposed design are not sufficiently established. Stateful executable environment construction and task synthesis have already been explored in prior work such as EnvScaler, ScaleEnv, AgentWord and EnvFactory, while the paper does not clearly articulate what is fundamentally new beyond these pipelines. The main additional component—matching predicted tool calls against a single reference sequence alongside state verification—is also not fully convincing, since valid tool-use trajectories are often non-unique and a single reference may unnecessarily restrict RL exploration. A clearer comparison with prior methods and stronger motivation/ablation of this design are needed. Also, the authors only compare with AWM and EnvScaler in their experiments, without comparing ScaleEnv, AgentWord and EnvFactory, which should also be very related work and have already been revealed in their introduction.

  2. The presentation and technical specification need improvement. The current figures are overly tuple-heavy and make it difficult to identify the paper’s unique contributions. The use of “offline” and “online” rollout is also potentially misleading, as the main distinction appears to be whether a user simulator is involved rather than the conventional meaning of online learning. In addition, important implementation details—such as how predicted and reference tool calls are aligned before computing multiset F1—are underspecified.

Questions for authors
  1. What is the key technical novelty of VESTA relative to EnvScaler, ScaleEnv, EnvFactory, and other executable-environment construction pipelines? Please provide a more explicit component-level comparison and clarify which parts of the performance gains can be attributed specifically to the proposed state-transition contracts rather than the broader environment/task-generation pipeline.

  2. How sensitive is the method to using a single reference tool-call sequence? Since multiple tool-call trajectories may be valid for the same task, can the authors compare reference-call matching against multiple-reference matching, or a less restrictive trajectory-level criterion?

  3. How are predicted tool calls matched to reference calls when computing multiset F1, especially when there are repeated calls, reordered operations, or multiple semantically equivalent argument choices?

  4. RL only seems not stable according to Tab 3. Can you provide more detailed analysis on it?

Reviewer_3U3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs synthetic executable environments for training stateful tool-use agents. While similar environment construction has been explored in prior work, the main contribution of VESTA is to generate and execute reference trajectories and record the corresponding state transitions. These reference transitions are then used as verifiable supervision for SFT data selection, RL task selection, and RL reward computation. In particular, the RL reward jointly evaluates reference tool calls and the resulting state changes.

Critical strengths

The use of reference state transitions provides a direct and verifiable training signal without relying on LLM-based reward models or judges. The same execution-derived supervision can also be reused across different stages of the training pipeline, including SFT filtering and RL. The experiments show consistent improvements across different model scales and tool-use benchmarks.

Critical weaknesses

The overall idea of constructing synthetic executable environments for agent training has been extensively explored in prior work, making the novelty of the environment construction itself limited. The main novelty appears to be recording state transitions along reference trajectories and using them for supervision.

More importantly, the workflow styles used to synthesize reference trajectories are manually defined. Although these templates can be instantiated with different tools, entities, and arguments to generate many concrete trajectories, the high-level task structures are still constrained by a small set of predefined patterns. This may substantially limit the structural diversity and scalability of the synthesized tasks, especially for more complex or previously unseen workflows.

Questions for authors

I noticed that many performance numbers in this paper, such as the performance of qwen-3 4b base model on BFCL v3 and other baselines, is larger than those in the publised papers or publised benchmarks. May I ask the reason?

I suggest to add more discussion about the comparison between this paper and other tool-use data synthesis pipeline paper, especially those build RL data/env papers. Here are some examples.

Qwen-AgentWorld: Language World Models for General Agents

Controllable and verifiable tool-use data synthesis for agentic reinforcement learning

SAP: State-Guided Data Synthesis with Argument Provenance for Multi-Turn Tool Use

Reviewer_aCavOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper presents VESTA a post-training framework that synthesis API-based excutable tool-use envrionments, produced metadata supports SFT selection and policy-conditioned RL task selection, and rewards combining tool-call matching with state verification. The paper retains 131 environments and trains three Qwen3 series models with 3,600 SFT tasks and 2,400 RL tasks per model. Evaluation on BFCL-v3 Multi-Turn, ACEBench-Agent, and tau2-Bench shows improvements in all nine model-benchmark averages over the base models. It outperforms the released EnvScaler checkpoints in most comparisons and the released AWM checkpoints in all of them. Ablation studies cover offline versus online rollouts, reward components, task selection, and environment and workflow scaling.

Critical strengths
  1. The paper is complete, carefully engineered and well documented. Its appendices specify environment construction, contract extraction, all three scoring functions, the workflow catalog, prompts and evaluation settings in enough detail to follow every step. The scope of paper is also cleared that the environments aim for internal consistency rather than equivalence to the real APIs, and that the baseline comparison reflects complete training pipelines.
  2. The reward is transparent. Treating read-only and mutating turns differently and adding a guard against off-target state changes is a interesting design, and reusing a single artifact for SFT filtering, task selection and the RL reward is a clean idea.
  3. The gains over the base models are large and robust. On BFCL multi-turn, every model scale gains roughly eight to thirteen points, which corresponds to dozens of additional solved tasks out of eight hundred. I checked the tables for internal consistency, and every reported average matches its subset columns.
Critical weaknesses
  1. VESTA motivates by arguing that final-outcome supervision and tool-call matching are each insufficient, so that tool calls and the resulting state transitions must be verified jointly, and most of the gain can be explained by SFT distillation. No experiment tests this against an outcome-level alternative within the same pipeline. The SFT trajectories come from a stronger model of the same family whose own benchmark scores are not reported, and there is no control that uses unfiltered, randomly filtered or final-state-filtered trajectories from the same teacher. The SFT selector also lacks the properties the paper argues for. About three quarters of the SFT data are selected by the online score, which pools tool-call recall and therefore does not penalize extra calls, and its final-state term by the paper's own description does not constrain unchanged fields. As a result, the value of contract-based selection is not separated from plain distillation of a stronger model on in-domain synthetic tasks.

  2. The validity of the evaluation is weakened because the training and evaluation user simulators coincide and the baseline protocol is unspecified. The same Qwen3 instruct model serves as the user simulator that generated the online SFT dialogues and as the user simulator in the ACEBench multi-turn and τ²-Bench evaluations. Adaptation to that simulator's behavior is an uncontrolled alternative explanation for part of the online SFT and τ²-Bench gains, and the paper does not discuss it. Separately, the evaluation appendix documents chat templates, thinking mode and tool-call parsers only for the Qwen3 models, not for the released EnvScaler and AWM checkpoints, and no published baseline number is reproduced. Some patterns suggest a protocol mismatch: AWM falls below its own base model in five of six comparisons, and the largest EnvScaler model solves exactly the same number of τ²-Bench tasks as its base model.

Questions for authors

Q1. Could you report paired task-level confidence intervals, for example via bootstrap or McNemar tests, for the comparison between the final model and its SFT initialization and for the ablation rows? Multiple τ²-Bench trials following the benchmark's usual pass^k protocol, and if possible a few training seeds for the middle-scale model, would also help. Please give pooled τ²-Bench success counts alongside the macro-averages.

Q2. Could you re-evaluate the middle-scale models on τ²-Bench with a user simulator from a different model family? Which chat template, thinking-mode setting and tool-call parser did you use for the EnvScaler and AWM checkpoints, and do your base-model and baseline numbers match the published ones?

Q3. On a stratified sample of RL rollouts, how often does the RL reward, or the SFT selection threshold, disagree with an independent judgment of task success, in the form of correct but low-scored or wrong but high-scored trajectories? How often does the absolute-state guard zero out later turns after a single early error?

Q4. Similar to other papers on tool-using agents that incorporate a post-training phase, VESTA adopts a "SFT+RL" approach. Could you explain why this combined strategy outperforms using Reinforcement Learning alone (direct RL) within the synthetic training tasks you have constructed? No addtional run needs for answering this question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.