acceptodds
Under review as a conference paper at ICLR 2027

StateForge: Synthesizing Verifiable Long-Horizon Data for Tool-Using Agents via State-Change Supervision

Abstract

Synthesizing long-horizon training data for tool-using agents requires reachable goals and verifiable outcomes. Successful tool calls do not guarantee task completion, while reference-call matching can reject valid alternative executions. We propose StateForge, a framework that unifies data synthesis, supervised fine-tuning (SFT), and reinforcement learning (RL) through state-change supervision. StateForge constructs executable environments with explicit state spaces and tools for querying and transforming states. Executing dependency-guided tool workflows yields reachable state-change targets, which ground task synthesis and task–target consistency checks. For each task, three trajectory synthesis modes preserve the same target: Direct provides complete goals, Progressive reveals goals in stages, and Recovery introduces recoverable environment perturbations. This shared target supports exact state-change matching to filter SFT trajectories and RL rewards that credit required changes and penalize unintended modifications. StateForge produces 544 environments and 1,697 tasks, with an average of 69 tool calls per task. With only 4B parameters, StateForge achieves state-of-the-art performance among specialized agent models on representative tool-use benchmarks, attaining 91.17% macro-average success on -Bench, 43.33% accuracy on DeepPlanning Shopping, and a 27.25% overall score on BFCL-v3 Multi-Turn.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Tool Use Agents, Agentic Data Synthesis, Trajectory Synthesis

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_sMxZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StateForge generates executable environments and tasks defined by target state changes. It produces direct, progressive, and recovery trajectories, then uses state-based checks for supervised training and reinforcement learning. Experiments show improvements on several agent benchmarks.

Critical strengths

1)Checking final state changes allows different valid action sequences. The same outcome-based framework supports task generation, trajectory filtering, and reward design. 2)Results across two model sizes and comparisons using the same backbone support the value of the generated data beyond the synthetic environments.

Critical weaknesses

1)Training corpora and trajectory mixtures differ in size and token exposure. Equal training epochs do not ensure equal training budgets, so the contribution of the state-based design remains partly unclear. 2)Exact state checks depend on correct generated task specifications. The paper provides limited independent evidence about specification errors, and some synthesis and reinforcement learning settings need clearer reporting.

Questions for authors

1)Do the gains remain when supervised tokens and training updates are matched across data sources? 2)What error rate does an independent audit find in the generated tasks and target states?

2 comments

Thank you for recognizing outcome-based verification and the results across two backbone sizes. We have added a comparison and ablation studies with matched training-set sizes.

W1 / Q1: Matching data size and training exposure

We added two comparisons to examine whether the reported gains persist after controlling training-set size and, for the interaction-mixture ablation, training updates. These comparisons partially address your concern.

First, we control training-set size across data sources. Using the same Qwen3.5-4B backbone and shared SFT settings, we compare StateForge and EnvFactory with 29,463 training records each.

Model / SFT training dataNumber of SFT training recordsτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)
EnvFactory SFT29,46360.0077.1985.0974.0911.67
StateForge-4B-SFT29,46372.0081.8997.3083.7342.50
StateForge-4B-SFT34,782 (full)78.0078.9099.1085.3342.50

StateForge retains an advantage on both τ² Macro and Shopping. For example, Shopping accuracy increases from 11.67% to 42.50%. This shows that the cross-corpus advantage persists at equal record counts.

Second, we control both record counts and training updates in the interaction-mixture ablation. Each condition uses 10,000 training records and the same number of optimizer updates.

Interaction modeτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)BFCL-MT (%)
Direct70.0076.4389.0978.5138.3324.50
Direct + Progressive76.0079.1993.1082.7641.6726.50
Full74.0078.2498.2383.4942.5027.13

Adding Progressive improves all three aggregate benchmark scores, and adding Recovery further improves them. Thus, the benefits of mixing interaction modes persist when record counts and update counts are held constant.

These results address the effects of training-set size and, within the mixture ablation, update count.

W2: Task-specification errors and reporting of synthesis and RL settings

We agree that the effectiveness of exact state checks depends on the correctness of the tasks themselves. Task construction includes two layers of verification: reference execution establishes a reachable target from the initial state; a bidirectional task–target audit then checks whether the requested effects are fully covered, the target effects are justified, the entities are unambiguous, and the required information is accessible. Exact trajectory verification subsequently compares the actual outcome against this accepted specification. These checks are used to control task-specification quality.

Regarding the request for clearer reporting of synthesis and RL settings, we will consider open-sourcing the task and trajectory synthesis code, as well as the RL training code, after the paper is accepted.

Q2: Independent audit of generated tasks and target states

We have not yet conducted a dedicated statistical assessment of this error rate. Based on a rough estimate of the synthesis process, more than 35% of tasks exhibit incomplete alignment among the task requirements, evaluation criteria, and target states during the auditing stage and are consequently discarded.

Dear Reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_ZMSXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

StateForge is about generating verifiable, long-horizon training data for tool-using agents. Its main idea is to use one execution-derived target state change throughout the pipeline: to build tasks, filter SFT data, and compute RL rewards. The results look very promising, a 4B model reports state-of-the-art results among specialized agent models. Synthesized interaction data provides gains beyond model scaling. Moreover, RL with state-changing supervision provides further gains.

Critical strengths
  • Strong results, the tasks are long. Trained on the same backbone with the same SFT recipe, StateForge data is the only corpus that beats the base model.
  • Progressive and Recovery modes add a real training signal, and the ablation supports them.
Critical weaknesses
  • There's no comparison of exact state-change filtering against simpler filters, such as "every call succeeded" or rubric-based filtering, on the same trajectories.
  • There's also no comparison of R_goal against a binary reward, a rubric reward or a reference-matching reward for RL.
Questions for authors

Q1: What does exact state-change filtering add over simpler filters?

Q2: What does the graded R_goal reward add?

Q3: Is Progressive data valuable because of how it's built, or just because it adds more turns?

3 comments

Thank you for recognizing the downstream results and the value of the interaction modes. We have added comparisons of filtering strategies and reward designs, a trajectory-mixture ablation with matched data sizes and training steps, and a comparison between long and short trajectories.

W1 / Q1: What does exact state-change filtering add over simpler filters?

The value of exact state-change filtering is that it selects demonstrations based on whether they achieve the intended task effects, while allowing different valid execution paths. This provides an outcome-based criterion for constructing SFT data, as successful tool calls do not necessarily imply successful task completion: an agent may omit a required change, or introduce unintended side effects. Rubric-based judgments may also overlook such discrepancies. Conversely, a trajectory may encounter a temporary tool error and subsequently recover to the correct outcome. Exact state-change matching checks whether all required changes are applied to the correct entities and whether unrelated state remains unchanged.

To evaluate this value, we apply tool-call success filtering, rubric-only filtering, and exact state-change matching to the same candidate trajectory pool, construct three equally sized training sets, and perform SFT on the same Qwen3.5-4B backbone.

Filtering strategyτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)
Tool-call success72.0078.0096.1082.0334.17
Rubric-only76.0078.6798.5984.4236.67
Exact state-change matching78.0078.9099.1085.3342.50

These results support the training value of exact state-change filtering. Compared with tool-call success filtering, exact state-change matching improves τ² Macro by 3.30 percentage points and Shopping accuracy by 8.33 percentage points; compared with rubric-only filtering, the improvements are 0.91 and 5.83 percentage points, respectively. Thus, under the same candidate pool and training-set size, verifying the exact state-change yields SFT data that produces better downstream performance.

W2 / Q2: What does the graded R_goal reward add?

The graded RgoalR_{\mathrm{goal}} reward provides two additional benefits: it distinguishes partial progress among unsuccessful executions and explicitly accounts for unintended state changes. Together, these properties provide more informative outcome supervision while encouraging the agent to complete the target without modifying unrelated state.

These benefits follow from how the reward evaluates execution outcomes. A binary exact-match reward assigns zero to every execution that falls short of complete success, treating partial completion and complete failure identically. Graded feedback preserves this distinction by measuring how much of the target has been achieved. However, target completion alone does not account for unintended modifications. Our full reward, M/(N+U)M/(N+U), incorporates both aspects: NN is the number of required state-change paths, MM counts those realized with the correct values, and UU measures weighted unintended changes.

To evaluate these benefits, we compare four reward variants using the same Qwen3.5-4B SFT checkpoint, training tasks, and RL training budget: Binary-Exact (BE) rewards only exact success; Target-Only (TO) uses M/NM/N without penalizing unintended changes; Rubric uses an adapted code-augmented rubric reward; and Full uses M/(N+U)M/(N+U). This controlled comparison examines whether the reward design contributes beyond applying RL training alone.

Modelτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)Shopping Match (%)
StateForge-SFT78.0078.9099.1085.3342.5078.57
StateForge-BE74.0090.5498.2187.5836.6775.64
StateForge-TO58.0072.0096.2675.4228.3373.63
StateForge-Rubric78.0086.4996.1586.8843.3380.03
StateForge-Full84.0089.50100.0091.1743.3377.84

The results support the value of combining graded feedback with penalties for unintended modifications. Compared with binary exact matching, the full reward improves τ² Macro by 3.59 percentage points and Shopping accuracy by 6.66 points. It also outperforms the target-only reward on both benchmarks, supporting the importance of accounting for side effects. Overall, these comparisons support the additional training value of more informative, side-effect-aware outcome supervision.

Q3: Progressive construction versus additional interaction turns

Progressive provides structured training coverage for handling incremental requirements while preserving prior progress. New requirements are disclosed only after the current cumulative target has been achieved, and subsequent actions must preserve previously completed effects. This exposes agents to coordinating new requirements with existing results. The following Shopping evaluation trajectory from a model trained with Progressive data illustrates compatible behavior; intermediate retrieval steps and product details are omitted:

User: Buy four products, each satisfying its respective attribute and review constraints,
with a total budget of RMB 3,438–3,684.

Agent → Check each product's requirements and retain candidates for each item:
Product ①: RMB 349 / 429; product ②: RMB 529;
product ③: RMB 689 / 899 / 1,299; product ④: RMB 1,499.

Agent → Check the global budget:
The lowest-priced combination totals RMB 3,066, below the budget's lower bound.
Keep the other selections and switch product ③ to the RMB 1,299 candidate.
The new combination totals RMB 3,676, satisfying the budget requirement.

Agent → Add the four products to the cart and verify the final cart.
Tool → The cart contains four products, totaling RMB 3,676.

This trajectory illustrates how the agent first satisfies local requirements and then adjusts its selection to meet a global constraint while preserving the other valid choices. This behavior is consistent with the training coverage provided by Progressive.

To evaluate its training value, we conducted a trajectory-mixture ablation on Qwen3.5-4B, with 10,000 training records and the same number of training steps in every condition:

Mixtureτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)BFCL-MT (%)
Direct70.0076.4389.0978.5138.3324.50
Direct + Progressive76.0079.1993.1082.7641.6726.50
Full74.0078.2498.2383.4942.5027.13

Adding Progressive improves performance across τ² Macro, Shopping, and BFCL-MT under matched training-record counts and update counts. These results support the training value of including Progressive data.

To examine the role of interaction length, we also constructed Short and Long training subsets with the same number of training records and performed SFT separately on Qwen3.5-4B:

Training subsetAverage interaction turnsτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)
Short7.862.0077.0097.6578.8833.33
Long15.272.2176.7896.1981.7335.83

Long achieves higher τ² Macro and Shopping accuracy, indicating that longer trajectories can also provide training benefits in this setting.

Overall, Progressive provides structured examples of handling incremental requirements and adapting subsequent actions. The trajectory-mixture ablation supports the practical value of including these examples. However, further analysis is needed to disentangle the effects of trajectory length from those of Progressive's staged disclosure mechanism.

Dear Reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_1j7tOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StateForge, a unified framework for task construction from state transitions, data synthesis under shared state-transition goals, Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL), focusing on reachable goals and verifiable outcomes in agentic AI. StateForge abstracts the environment into a finite state space, expresses reachable goals in natural language, and evaluates whether the agent has achieved those goals. For environment synthesis, it pre-constructs an explicit environment state space from multiple task-oriented resources, represents dependencies as a Directed Acyclic Graph (DAG), and performs and audits reference execution and goal extraction. Trajectory synthesis is conducted through direct, progressive, and recovery approaches. Subsequently, candidate trajectories are sampled and filtered through validity and efficiency checks for SFT, while RL is performed based on state transitions.

Critical strengths

NA

Critical weaknesses

W.1. Most of all, the paper is incremental.

The problem of requiring verifiable rewards and reachable goals is a well-known problem from the early stage of agentic AI [1-3]. The proposed solution of each step is not so different from the previous works.

  • [2, 4] deal with graph-based synthesis of tasks, corresponding to the overall process of task and trajectory generation.
  • [5] proposed noise injection in tool-call training, covering StateForge’s scaling, Progressive, and Recovery.
  • [6] proposed trajectory filtering, corresponding to StateForge’s validation and SFT conversion.

Also, viewing an agentic workflow as a state machine is nothing new [7]; graph-based construction automatically implies this process; a finite state-action MDP naturally forms a graph MDP .

Please state more than “combining all,” as written in the related work (line number 126), or viewing the environment as states.

W.2. The benchmark is oversaturated, making it difficult to see how much the framework is worth quantifiably

As shown in Table 1, Qwen3.5-4B already shows very close scores to StateForge-4B-SFT, and so does the 9B case; it is difficult to see whether all of this process is the result of RL or any other procedure.


Refs.

[1] Trivedi, Harsh, et al. "Appworld: A controllable world of apps and people for benchmarking interactive coding agents." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] Prabhakar, Akshara, et al. "Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay." Advances in Neural Information Processing Systems 38 (2026).

[3] Yao, Shunyu, et al. "$\tau $-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv preprint arXiv:2406.12045 (2024).

[4] Fang, Runnan, et al. "Towards general agentic intelligence via environment scaling." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[5] Chen, Yuxin, et al. "Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments." arXiv preprint arXiv:2605.27209 (2026).

[6] Zeng, Aohan, et al. "Agenttuning: Enabling generalized agent abilities for llms." Findings of the Association for Computational Linguistics: ACL 2024. 2024.

[7] Guan, Jian, et al. "Amor: A recipe for building adaptable modular knowledge agents through process feedback." Advances in Neural Information Processing Systems 37 (2024): 126118-126148.

Questions for authors

Q.1 Please explain the unique contribution that the paper is proposing

Q.2 I think graph-based synthesis of environments does not consider the semantics of the task, which is most important for language model-based agentic AI. How are tasks derived semantically?

For instance in Appendix C.1. the task is

Identify the top-ranked Amsterdam hotel, check the amenity catalog, and remove and restore a verified hotel–
amenity association. 

This task seems so artificial that an actual user would not request it in the real world.

9 comments

Thank you for asking us to clarify the contributions and the source of task semantics. Below, we explain the relationship to each cited work and analyze the performance improvements over the corresponding backbones.

W1 / Q1: Unique contributions and their relationship to prior work

Graph-based task synthesis, noise injection in tool-call training, and trajectory filtering, as highlighted in the review, describe the pipeline at a relatively high level. These high-level descriptions do not specify how the components are implemented. For example, while both methods employ trajectory filtering, this does not imply that their filtering mechanisms are equivalent. We therefore explain the methodological differences between StateForge and the seven cited studies below.

The core contribution of StateForge is a unified target-supervision mechanism based on execution outcomes, with three components: (1) execution-driven target construction, which extracts the state change Δ=Diff(S0,Sg)\Delta=\mathrm{Diff}(S_0,S_g) from a reference execution to obtain a reachable target; (2) target-constrained interaction construction, in which Direct, Progressive, and Recovery share the same final target; and (3) unified supervision across training stages, in which the same target specification is used for exact outcome verification and filtering in SFT, as well as for graded reward computation in RL.

The differences from related work are as follows:

  1. AppWorld [1]: Its state-based tests allow different correct solutions and check for unintended modifications. StateForge uses the execution-derived target as an input to task auditing and trajectory construction, and reuses the same comparison rules for SFT trajectory filtering and graded RL feedback.

  2. APIGen-MT [2]: It uses API dependency graphs, semantic validation, and trajectory filtering based on states and outputs. StateForge explicitly defines disclosure conditions in terms of verified cumulative path–value targets, adds persistent-state interventions and verifies that the original target remains reachable from the actual perturbed state. We do not claim execution diffs or multi-turn synthesis alone as standalone contributions.

  3. τ-bench [3]: It evaluates task success against annotated final database states. StateForge instead derives reachable targets through execution and uses them for task synthesis, interaction construction, and training supervision.

  4. AgentScaler [4]: It uses dependency-guided workflows and terminal-state filtering, including successful trajectories with intermediate tool errors. StateForge adds state-gated Progressive disclosure, Recovery interventions verified by witnesses from perturbed states, and graded RL feedback from the shared target.

  5. NoisyAgent [5]: It perturbs interactions and tool observations using an adaptive noise curriculum. StateForge instead perturbs persistent environment state and verifies recovery to the original target from the perturbed state, including unintended side effects.

  6. AgentTuning [6]: It emphasizes high-quality interaction trajectories and hybrid instruction fine-tuning. StateForge instead constructs executable task-effect specifications and reuses them for interaction synthesis, exact SFT filtering, and graded RL feedback.

  7. AMOR [7]: Its finite-state machine organizes transitions among reasoning modules and supports module-level process feedback. In StateForge, state refers to the persistent state of the external environment, and the target specifies how that state should change. Transitions among reasoning modules do not, by themselves, specify the database effects required by a task, when cumulative outcomes permit disclosure of the next stage, or whether a state intervention leaves the target reachable.

Regarding graphs or MDPs, representing tool dependencies or environment transitions does not itself define task-effect specifications. StateForge additionally extracts and audits execution-derived targets and uses them to govern Progressive disclosure, Recovery reachability, and SFT/RL supervision.

In summary, our contribution is not graph-based synthesis, noise injection, or trajectory filtering in isolation, but a unified execution-derived target-supervision mechanism that connects reachable task construction, target-constrained interaction variation, strict SFT outcome verification, and graded RL feedback.

W2: Benchmark headroom and gains over the backbone

Thank you for raising the concerns about benchmark saturation and the quantitative value of the framework. In conducting our experiments and reproducing related methods, we found that further improvements can be difficult when the backbone already achieves high scores on some tasks. Discussions with researchers working on similar problems have echoed this observation.

We chose these widely used benchmarks to examine whether StateForge can provide additional benefits over strong backbones under established evaluation protocols. Although limited headroom on some tasks makes further improvement more challenging, evaluating under these conditions helps assess the practical value of our contribution.

We address your concern from two perspectives: the remaining room for improvement across benchmarks and the gains of StateForge over the corresponding backbones. The table below summarizes the scores (%) at each training stage from Table 1 of the paper.

Modelτ² Airlineτ² Retailτ² Telecomτ² MacroShopping accuracyBFCL-MT
Qwen3.5-4B Base76.0077.20100.0084.4035.8325.25
StateForge-4B-SFT78.0078.9099.1085.33 (+0.93; +1.1%)42.50 (+6.67; +18.6%)27.00 (+1.75; +6.9%)
StateForge-4B-RL84.0089.50100.0091.17 (+6.77; +8.0%)43.33 (+7.50; +20.9%)27.25 (+2.00; +7.9%)
Qwen3.5-9B Base80.0086.80100.0088.9340.8325.00
StateForge-9B-SFT84.0089.4798.2590.57 (+1.64; +1.8%)45.83 (+5.00; +12.2%)26.63 (+1.63; +6.5%)
StateForge-9B-RL84.0091.23100.0091.74 (+2.81; +3.2%)46.67 (+5.84; +14.3%)26.75 (+1.75; +7.0%)

(1) Are the benchmarks already saturated?

The Qwen3.5 backbones already achieve a perfect score on Telecom, so we acknowledge a ceiling effect on this subset. However, the remaining subsets and benchmarks retain headroom and should be examined individually.

Unsolved tasks remain in Airline and Retail, while Shopping and BFCL-MT also leave substantial room for improvement. As shown above, the complete StateForge pipeline improves the 4B model over its backbone on all four of these evaluations. These results show that the benchmarks remain capable of capturing performance improvements. Therefore, saturation on Telecom does not imply that the benchmarks as a whole are saturated.

(2) How much does StateForge improve over the backbone?

We agree that the SFT gains over Base on τ² Macro are small. However, this observation does not extend uniformly to the other benchmarks. On Shopping and BFCL-MT, SFT improves performance at both model scales, and the 4B model after SFT outperforms the 9B Base model on both benchmarks. These improvements are already present before RL, providing evidence for the training value of the synthetic data itself.

The complete StateForge pipeline further improves performance through outcome-supervised RL. At both model scales, the final models outperform their corresponding backbones on τ² Macro, Shopping, and BFCL-MT. Together, the SFT results demonstrate benefits from the synthetic data, while the final results quantify the overall gains of the complete framework. We therefore acknowledge the limited SFT gains on τ² while considering the evidence across benchmarks and the complete training pipeline when assessing StateForge’s empirical value.

Q2: Task semantics and the plausibility of the hotel example

Thank you for raising the questions about the source of task semantics and the plausibility of the hotel example. We address these concerns in two parts.

(1) Task semantics are derived from both the dependency graph and the environment context, and undergo semantic auditing. The dependency graph primarily captures argument passing, entity resolution, and state dependencies, while the environment provides domain entities, tool functionality, and operational rules that give these operations concrete domain meaning. As described in §4.1, we generate task descriptions using the environment context, initial state, reference execution, and resulting target state changes, rather than simply converting a sequence of tool calls into natural language. In addition, Appendix A.2.3 uses a bidirectional semantic audit to check consistency between the task description and the target state change. Candidates that fail these checks are rejected to reduce semantic ambiguity and unjustified operations.

(2) The hotel example. The example is a hotel-data audit task with clearly specified entities, query scope, and operational constraints. More generally, agentic tasks may use specialized, complex, and precisely constrained descriptions, because they may originate as execution instructions delegated by a main agent to a sub-agent. The plausibility of a task should therefore depend more on whether its entities, goals, and constraints are clear and semantically grounded than on whether an ordinary user would issue the request in exactly those words.

The works cited in your review also contain examples of corrective operations and specialized, multi-step workflows. For example, in τ-bench [3], the user Chen Smith explicitly asks to undo a previous order cancellation and restore delivery of the items in the original order. This illustrates that undoing and restoring can reflect clear user intent. A similar task appears in AppWorld: a user has mistakenly attached a résumé to an email draft and asks to replace it with a profile picture from the file system before sending the email. AgentScaler presents an engineering scheduling task that first queries the current project timeline and construction schedule, then generates a DIY task-level plan. This task combines domain-specific queries with planning and specifies the workflow and operational requirements.

The distinction between artificially constructed task descriptions and real-world requests is not, by itself, sufficient to determine semantic plausibility; what matters more is whether the operations have clear grounding in the environment, a coherent goal, and consistent constraints. These criteria also apply to our hotel example.

Reviewer_FbmhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StateForge, a framework for synthesizing verifiable long-horizon tool-use data through state-change supervision. It introduces three interaction modes and uses shared state-change targets for SFT trajectory filtering and RL rewards. Experiments on several representative benchmarks demonstrate improvements over the corresponding 4B and 9B backbones.

Critical strengths
  • The three interaction modes (Direct, Progressive, and Recovery) capture realistic challenges in tool-use scenarios, including evolving user requirements and unexpected environment changes. I also appreciate the idea of connecting data synthesis, SFT trajectory filtering, and RL optimization through a shared state-change target, which makes the overall training pipeline more coherent.

  • Deriving task objectives from successfully executed workflows is a sensible way to improve task reachability. The bidirectional consistency check between task descriptions and target state changes is particularly useful for identifying missing or unjustified effects in synthetic tasks.

Critical weaknesses
  • While StateForge introduces fine-grained state-change matching and explicit penalties for unintended modifications, its advantages over existing verification and reward strategies remain unclear. EnvFactory has already investigated combining trajectory-based and final-state rewards, while Agent World Model compares code-only, LLM-only, and code-augmented verification strategies. In contrast, StateForge does not provide controlled comparisons with these alternatives or isolate the contributions of its proposed reward components. Therefore, it is difficult to determine whether the observed RL improvements arise from the specific state-change reward design or from RL training itself.

  • Limited improvements from RL. On DeepPlanning Shopping, RL improves accuracy by only approximately 0.83 percentage points for both the 4B and 9B models, while the gains on BFCL Multi-Turn are merely 0.25 and 0.13 percentage points, respectively. Given the benchmark sizes, these improvements correspond to a net gain of only approximately one successful task per model on Shopping, and two and one successful tasks on BFCL, respectively.

  • Moreover, RL leads to lower reported performance on some subsets, including Shopping L3 and BFCL Missing Parameters. Although the 4B model achieves a more substantial improvement on τ²-Bench, this gain drops to +1.17 points for the 9B model. These results suggest that the benefits of RL are inconsistent across model scales and task types, and their robustness remains insufficiently established.
Questions for authors

Q1: Please refer to W1.

Q2: Could the authors provide more direct evaluations to demonstrate the benefits of Progressive and Recovery?

4 comments

Thank you for recognizing the value of the shared state-change target and the three interaction modes. We understand that your concerns primarily focus on RL. We would like to clarify that the main contribution of this paper is a unified framework built around execution-derived state-change targets, rather than RL itself.

W1 / Q1: Reward comparisons and component ablations

The table below summarizes the AWM and EnvFactory results reported in the paper, together with our additional reward ablations using Qwen3.5-4B. StateForge-BE (Binary-Exact) uses a binary reward based on exact outcome matching; StateForge-TO (Target-Only) uses only the target completion ratio, without penalizing unintended modifications; StateForge-Rubric uses an adapted rubric-based reward; and StateForge-Full uses the complete StateForge reward. All four StateForge RL variants use the same SFT checkpoint, training data, and RL configuration.

Modelτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)Shopping Match (%)
AWM-14B31.5063.6017.7637.629.1749.33
EnvFactory-8B44.0043.8613.1633.674.1728.39
StateForge-SFT78.0078.9099.1085.3342.5078.57
StateForge-BE74.0090.5498.2187.5836.6775.64
StateForge-TO58.0072.0096.2675.4228.3373.63
StateForge-Rubric78.0086.4996.1586.8843.3380.03
StateForge-Full84.0089.50100.0091.1743.3377.84

StateForge-BE assigns zero reward to outcomes that do not achieve exact success, whereas StateForge-Full provides graded feedback based on target completion and unintended modifications. The lower performance of StateForge-BE supports the value of graded feedback, while the degradation of StateForge-TO highlights the importance of penalizing unintended modifications. With the SFT checkpoint, training data, and RL configuration held fixed, StateForge-Full achieves the best overall performance, suggesting that the observed gains stem in part from our specific reward design rather than additional RL training alone.

W2: Magnitude of the additional RL gains

We agree that the additional gains from RL on DeepPlanning Shopping and BFCL-MT are relatively limited. However, we believe that these results should be considered alongside the improvements already achieved during SFT and the potentially diminishing returns from further optimization.

ModelShopping accuracy (%)Shopping gain over Base: pp (relative %)BFCL-MT (%)BFCL-MT gain over Base: pp (relative %)
Qwen3.5-4B (Base)35.83—25.25—
StateForge-4B-SFT42.50+6.67 (+18.6%)27.00+1.75 (+6.9%)
StateForge-4B-RL43.33+7.50 (+20.9%)27.25+2.00 (+7.9%)
Qwen3.5-9B (Base)40.83—25.00—
StateForge-9B-SFT45.83+5.00 (+12.2%)26.63+1.63 (+6.5%)
StateForge-9B-RL46.67+5.84 (+14.3%)26.75+1.75 (+7.0%)

First, most of the performance gains were already achieved during SFT, with RL providing additional improvements. The gains from RL vary across tasks: the additional gains on Shopping and BFCL-MT are relatively limited, whereas on τ² Macro, the 4B and 9B models improve over SFT by a further 5.83 and 1.17 percentage points, respectively. Importantly, the complete StateForge training pipeline achieves substantial improvements over the original backbones. For the 4B model, Shopping accuracy, BFCL-MT, and τ² Macro improve by 7.50, 2.00, and 6.77 percentage points, respectively. For the 9B model, the corresponding improvements are 5.84, 1.75, and 2.81 percentage points. These results demonstrate that the overall effectiveness of StateForge should be evaluated across both SFT and RL stages, rather than solely based on the incremental gains from RL.

Furthermore, scaling Qwen3.5 from 4B to 9B does not improve BFCL-MT performance, suggesting that further improvements on this benchmark may face diminishing returns.

W3: Inconsistent RL gains across task subsets and model scales

We agree with the reviewer's observation. We emphasize that the RL component of our work primarily focuses on reward design based on execution-derived state changes, rather than on the RL optimization algorithm itself. We use standard GRPO for policy optimization. Prior studies have identified potential issues with GRPO, including policy entropy collapse and optimization bias [1][2]. Moreover, the difficulty of long-horizon credit assignment in multi-turn agent tasks may further increase training instability [3]. These factors may explain some of the performance fluctuations. In future work, we will further investigate the limitations of RL in complex agent tasks and explore ways to improve its robustness across tasks and model scales.

References

[1] Yu et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.

[2] Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025.

[3] Wang et al. (2025). RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073.

Q2: Experimental evidence for Progressive and Recovery

We evaluate Progressive and Recovery through benchmark performance and execution behavior, targeting constraint coordination and autonomous error recovery, respectively. We first present a trajectory-mixture ablation on Qwen3.5-4B, with each condition using 10,000 training records and the same number of training steps.

SFT trajectory mixtureτ² Airline (%)τ² Retail (%)τ² Telecom (%)τ² Macro (%)Shopping accuracy (%)BFCL-MT (%)
Direct70.00 (0.0%)76.43 (0.0%)89.09 (0.0%)78.51 (0.0%)38.33 (0.0%)24.50 (0.0%)
Direct + Progressive76.00 (+8.6%)79.19 (+3.6%)93.10 (+4.5%)82.76 (+5.4%)41.67 (+8.7%)26.50 (+8.2%)
Full: Direct + Progressive + Recovery74.00 (+5.7%)78.24 (+2.4%)98.23 (+10.3%)83.49 (+6.3%)42.50 (+10.9%)27.13 (+10.7%)

Compared with Direct-only training, adding Progressive improves τ² Macro, Shopping accuracy, and BFCL-MT; further adding Recovery yields additional gains on all three metrics.

To illustrate how these benefits are reflected in actual execution behavior, we present two excerpts from Shopping benchmark evaluations. The first illustrates step-by-step reasoning under multiple constraints, and the second illustrates autonomous repair after an execution error. Intermediate retrieval steps and product details are omitted.

Case 1: Local and global constraint coordination (Progressive).

User:
Find four products that satisfy the following requirements, respectively:
① Camel color, with delivery time, sales volume, and the number of two-star reviews meeting the specified conditions;
② Women's Off The Wall corduroy pants, with stock and reviews meeting the specified conditions;
③ A size-S winter item, with the total number of reviews and the number of three-star reviews meeting the specified conditions;
④ Size-36 Patagonia Snowdrifter shoes, with the rating and the number of two-star reviews meeting the specified conditions.
Total budget: RMB 3,438–3,684.

Agent → Search for each item and verify its attributes and reviews:
Requirement ①: Retain two candidates priced at RMB 349 and RMB 429.
Requirement ②: Find pants meeting the conditions, priced at RMB 529.
Requirement ④: Find shoes meeting the conditions, priced at RMB 1,499.
Requirement ③: Find three winter-item candidates priced at RMB 689 / 899 / 1,299.

Agent → Reconcile the candidates with the total budget:
The lowest-priced combination is 349 + 529 + 689 + 1,499 = RMB 3,066, below the budget's lower bound.
Keep the other selections and switch the winter item to the RMB 1,299 candidate.
The new combination totals RMB 3,676, satisfying the budget requirement.

Agent → add_product_to_cart(...) × 4
Agent → get_cart_info()
Tool → The cart contains four products, totaling RMB 3,676.

This case illustrates task decomposition and global constraint coordination, where intermediate results guide subsequent decisions to satisfy both local requirements and the shared budget.

Case 2: Autonomous error recognition and repair (Recovery).

Initial state:
The cart contains four products, totaling RMB 3,676.
One required product has product_id 5f97c618 and is priced at RMB 349.

User:
Check whether the cart satisfies the requirements; if the task is already complete, stop.

Agent → delete_product_from_cart(product_id="5f97c618")
Tool → Three products remain in the cart, totaling RMB 3,327; a required product is missing.

Agent [verbatim]:
"Wait - I deleted `5f97c618` by accident. Let me add it back."

Agent → add_product_to_cart(product_id="5f97c618")
Tool → The cart is restored to four products, totaling RMB 3,676.

Agent → Confirm that the cart has been restored and the task is complete.

This case illustrates autonomous error recognition and recovery: without user intervention, the agent detects its own mistake and restores the intended target state.

Taken together, the controlled ablation demonstrates overall performance gains from adding Progressive and Recovery data, while the benchmark cases illustrate concrete behaviors in step-by-step problem solving and constraint coordination, and error recognition and autonomous repair, respectively. These forms of evidence support the training value of both interaction modes.

Dear Reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.