acceptodds
Under review as a conference paper at ICLR 2027

Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

Abstract

Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or contain actions that are unjustified by the information visible to the agent at decision time. We identify two coupled requirements for effective trajectory supervision: compositional coverage, so that important observation–action dependencies are represented, and decision grounding, so that each supervised action is supported by the learner-visible interaction history. We introduce Trajectory2Gym, which constructs executable interactions around under-supported compositional structures and derives supervision only from grounded decisions in their realized executions. Structural-support selection allocates the resulting experience, while parent-normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight. This reframes trajectory generation as the deliberate construction of learnable interaction experience, rather than the collection of successful trajectories. Across five Qwen3 scales, Trajectory2Gym improves Tau1, Tau2, and BFCL at every tested scale, increasing the equal-weight three-benchmark mean by 16.9–29.8 percentage points. The gains are strongest in stateful and multi-turn interaction, where later decisions depend on earlier observations and feedback. Our results suggest that scaling tool-use agents depends not only on collecting more successful trajectories, but on constructing interactions whose dependencies are represented and whose decisions are learnable from the agent’s visible history.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool-use agents, trajectory synthesis, agent post-training, decision grounding, compositional interaction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_yzAFOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Trajectory2Gym is a training mechanism designed to improve student-model supervision by addressing two main challenges: composition coverage and decision grounding. The authors evaluate the method across five open-weight Qwen3 models and report strong performance on the τ-bench, τ²-bench, and BFCL benchmarks.

Critical strengths

The main advantage of the paper is its comparison against most open-weight model competitors and its evaluation on three major tool-calling benchmarks.

Critical weaknesses

I appreciate the authors’ effort and the overall direction of the work. However, I have several major concerns that I would appreciate the authors addressing.

  1. My main concern is potential data contamination. The paper does not clearly explain which task sets were used as the starting point for generating the training trajectories. In Appendix A.2, the authors state that “BFCL follows its released training-side and evaluation membership.” However, to my knowledge, BFCL does not provide a standard public training split corresponding to its evaluation benchmark. The authors should therefore clarify exactly which BFCL tasks were used for training, where these tasks came from, and how they were separated from the evaluation set. The situation is also unclear for τ-bench and τ²-bench. For τ-bench Retail, the authors appear to use the training split released by the benchmark authors. However, for the Airline domain, the paper indicates that 50 airline tasks were used, and these appear to come from the benchmark evaluation tasks. If this interpretation is correct, the model may effectively have been trained on part of the test set. I have a similar concern regarding τ²-bench. The paper refers to a τ²-bench training set, but after carefully reviewing the benchmark, I could not identify an independently constructed training dataset. Instead, it appears that benchmark tasks may have been partitioned or adapted into smaller subsets. If the training and evaluation tasks originate from the same underlying task pool, there may be substantial semantic or structural similarity between them. This is particularly concerning because many tasks in these environments are already highly similar in terms of user goals, tool sequences, and database states. As a result, training on tasks derived from the evaluation pool could significantly inflate performance. Because of this issue, I currently find it difficult to interpret the results in Table 2 as a fair comparison. Several competing methods were not trained on the benchmark evaluation tasks, while Trajectory2Gym may have access to tasks that overlap with or are derived from them. The authors should provide a precise description of the task-level train/test separation and ideally quantify task similarity or overlap between the training and evaluation sets.

  2. The paper compares Trajectory2Gym against several recent synthetic data generation methods for which open-source models have been released. However, the experiments do not appear to use the original training task sets from these methods to isolate the contribution of Trajectory2Gym itself. I suggest two controlled experiments. First, the authors could use the AReaL [1] training tasks as a fixed task set and generate training trajectories from exactly the same tasks using the original or standard trajectory generation procedure without Trajectory2Gym and the proposed Trajectory2Gym method. The resulting models could then be evaluated on τ²-bench and τ-bench. This would provide a much cleaner comparison because the underlying training tasks would remain fixed and only the trajectory construction or selection method would change. Second, the authors could use the Simia/τ training data as another fixed task source and repeat the same controlled experiment. Such experiments would provide stronger evidence that the gains come from Trajectory2Gym rather than differences in the underlying task distributions.

  3. I respectfully disagree with the statement that trajectory quality is generally determined “by task completion or per-call execution.” I think this characterization is somewhat too broad. Some recent benchmarks already employ LLM-based judges to evaluate more nuanced aspects of agent behavior. For example, VITA-Bench uses LLM-based evaluation, and methods such as VULCAN [2] use LLM-as-a-judge signals for identifying and selecting higher-quality trajectories. More importantly, a failed task does not necessarily imply an incorrect agent trajectory. Given an initial state and a user request, the task may be inherently unsatisfiable, conflict with domain policies, or require the agent to refuse or explain that the requested action cannot be performed. In such cases, recognizing that no valid action should be taken can itself represent the correct behavior. Therefore, trajectory quality should not be defined only through whether the final user request was completed or whether individual tool calls succeeded. The paper would benefit from acknowledging these cases and distinguishing between task failure caused by incorrect agent behavior and task non-completion caused by an invalid, impossible, or policy-violating request.

  4. Another concern is whether the reported improvements reflect genuine generalization or partial overfitting to the task distributions used during training. Given the concerns above regarding possible overlap between training and evaluation tasks, I believe an evaluation on harder out-of-distribution task sets would substantially strengthen the paper. In particular, I suggest evaluating the trained models on τ-Trait [3] and Thinking-Box [4]. These benchmarks use environments closely related to τ-bench-style tool-use settings but introduce more challenging task constructions and therefore provide a useful test of whether the learned behavior generalizes beyond the original benchmark distributions. Strong performance on these benchmarks would provide considerably stronger evidence that Trajectory2Gym improves general agent capabilities rather than primarily adapting models to the structure of the evaluation tasks.

  5. AReaL [1] and VULCAN [2] are closely related to this work but are missing from the related work section. The authors should discuss both and clarify how Trajectory2Gym differs from them.

  6. Figure 1 is not sufficiently clear.

  7. The authors should clearly state which model is used as the user agent during trajectory generation and during evaluation on the τ benchmarks. They should also use the standard benchmark names consistently: “τ-bench” instead of “Tau1” and “τ²-bench” instead of “Tau2.”

References

[1] AReaL: https://arxiv.org/abs/2601.22607
Training data: https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data

[2] VULCAN: https://openreview.net/forum?id=Ht4HB4pIQq&noteId=SS4dHupwle
Repository: https://github.com/microsoft/vulcan

[3] τ-Trait: https://arxiv.org/abs/2510.04491
Repository: https://github.com/collinear-ai/tau-trait/tree/main

[4] Thinking-Box: https://arxiv.org/abs/2608.19741

Questions for authors

Please read the weaknesses.

Reviewer_kVitOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Trajectory2Gym, a training-data construction and selection framework for tool-use agents. It argues that rejection sampling based only on task success is not enough and that effective supervision also needs examples of multi-step interaction patterns (compositional coverage) and actions grounded by the information available to the agent at decision time (decision grounding). The authors evaluate five Qwen3 model sizes on Tau1, Tau2, and BFCL. The complete procedure improves all three benchmark aggregates at every tested scale, increasing the equal-weight mean across Tau1, Tau2, and BFCL by 16.9–29.8 percentage points over the corresponding Qwen3 initialization.

Critical strengths

None

Critical weaknesses
  1. The paper is extremely difficult to follow even after several readings and contains several instances of abstract terminology which do not sound intuitive at all. Here are a few examples: a. The abstract states that "Structural-support selection allocates the resulting experience, while parent- normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight." At this point in the paper, the reader does not have sufficient context to understand what is being selected, what constitutes a “semantic interaction,” what "parent-normailzied" means or how training weights are assigned. b. In Section 3: "we take the perspective of a developer who records the complete log during collection but trains a student that acts only through the native interface." This is again extremely hard to follow. c. Section 3 also describes coverage as providing “sufficient support for the observation–action dependencies and multi-step structures the policy should acquire,” without making clear what these structures look like or what constitutes sufficient support. Although the paper introduces mathematical notation, the correspondence between that notation and the implemented procedure remains unclear. The paper would greatly benefit from explicitly defining each stage’s inputs, outputs, and operations, and by also adding an example showing how an initial task becomes an executed trajectory, a set of supervised targets, and a weighted training example.

  2. The paper provides very little implementation detail and lacks controlled ablations needed to justify its main design choices. Although Appendix B reports task and trajectory counts, the exact input prompt set, the relationship between benchmark tasks and generated requests, and the generator/teacher configurations/prompts are not specified. Moreover, the targeted-expansion ablation adds newly generated trajectories but does not keep total training tokens constant. Thus, it does not isolate the benefit of targeting underrepresented interaction patterns from the benefit of additional training data. In addition, several conditions in Table 3, particularly “existing candidate pool” and “loose success-derived targets,” lack precise details, which makes it unclear what their relationship is to standard success sampled SFT.

Questions for authors

Clarifiaction Questions:

  1. Could you provide one clearly worked example showing the initial task, the trajectory from teacher and students perspective, the assigned interaction pattern, the supervised actions, and the resulting training weights?

  2. What is the exact input task/prompt set used for collection? Please specify the source datasets, splits, and counts

  3. Which models generate the requests and collect trajectories, and what prompts and configurations do they use?

Reviewer_J8N2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that successful trajectories are not necessarily good supervision. It asks for two properties: compositional coverage (cross-step dependencies are represented often enough) and decision grounding (each supervised action is justified by what the agent could see at that point). The pipeline generates tasks for under-represented capability paths, executes them with a teacher in the native environments, masks out actions that are not executable or whose identifier-like arguments are not supported by visible history (failed calls stay as context), selects trajectories with a per-scale support depth, and trains with SFT where each trajectory gets equal weight regardless of how many rendered examples it produces. On five Qwen3 sizes, SFT on this data gives large gains on Tau1, Tau2 and BFCL (e.g. 8B Tau2 25.0 to 74.2), and the paper reports four matched 8B ablations.

Critical strengths

The motivation is sound and practical. Treating failed calls as context without imitating them, and masking actions whose arguments come from collector-only information, are sensible choices for agent SFT. The parent-normalization observation (multi-turn interactions that render into many rows dominate a row-averaged loss) is a useful practical point, and the study covers five model sizes.

Critical weaknesses
  1. Most of the gain is in-domain SFT from an unnamed teacher, not the proposed components. Table 3 shows that simple success-filtered SFT ("loose success-derived targets") already reaches a mean of 65.22, against 40.06 for the Qwen3-8B initialization and 48.12 for the best public 8B checkpoint. Decision grounding adds 2.75 on top. Different configurations of the full method range from 67.97 to 72.63 (the "existing candidate pool" row is 69.08 and depth 2 is 72.28, both above the headline), and changing only the loss weighting moves Tau2 from 52.50 to 74.17. So the component gains of 2-3.6 points, each from a single run, are within the variation between configurations. The teacher and generator models are never named. If the teacher is a frontier model, the result that 30B reaches 84.17 on Tau2 mostly reflects distillation.

  2. Training and evaluation share environments, and in one case tasks. Tau1 Airline is transductive by the paper's own description: the same 50 tasks are used to construct training interactions and for evaluation, and they count toward every Tau1 average and overall mean. BFCL has no official training split, yet the paper refers to "released training-side memberships". The case studies show strong behavior on BFCL multi-turn backends, so it's unclear whether training data was built on those backends. If it was, the +7 to +41 multi-turn gains and the comparison to GPT-5.5 are in-domain. Most of the public checkpoints used for comparison were not trained on these environments, and the closed-source numbers come from reported results that may be on different task sets.

Smaller points. Every task is run once and Tau2 Airline has 20 tasks. Key details (the capability vocabulary, how paths are canonicalized, how identifier-like arguments are detected) are left to a "released configuration". The propositions in Appendix E are standard facts. Live Simple drops 6-13 points at four of five scales. There is no evaluation on environments not used for training.

Questions for authors
  1. Which models are the teacher and the task generator, and what do they score on Tau1/Tau2/BFCL?
  2. Where does the BFCL training data come from? Were tasks constructed on the BFCL multi-turn backends?
  3. Please report results without Tau1 Airline, or only on tasks strictly disjoint from training.
  4. Please add a plain success-filtered SFT baseline with the same teacher and environments at all five scales, so the contribution of grounding, selection and normalization can be separated from in-domain distillation.
  5. What is the run-to-run variance of one fixed configuration (training seed and evaluation repeats)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.