acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Data Generation for Long-Horizon Tool Calling

Abstract

Real-world tasks routinely require long sequences of tightly interdependent tool calls, yet existing benchmarks for evaluating agentic systems rarely reflect this complexity. While current benchmarks capture realistic multi-turn interactions and underspecified inputs, they focus on shallow execution trajectories with limited inter-function dependencies. We examine current data generation approaches for tool-calling benchmarks and argue that they inherently bias datasets toward low-complexity examples. We propose BRIT(Back translating and Refining Iteratively for Trajectory generation), which makes two systematic modifications compatible with existing pipelines: (1) generating trajectories before queries to optimize complexity explicitly, and (2) applying iterative refinement instead of filtering to preserve complex instances. We provide theoretical analysis showing two complementary results. First, filtering during post-processing inevitably produces a light-tailed complexity distribution, regardless of the complexity distribution of the trajectories initially generated. Second, when the generator already produces a heavy-tailed complexity distribution, per-node refinement budgets that grow logarithmically with trajectory complexity are sufficient to preserve that heavy tail after post-processing. Empirically, BRIT substantially increases the complexity of generated datapoints. Our work advocates for including high-complexity instances in tool-calling evaluation and provides practical, scalable methods for achieving this goal.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-tailed Distribution, Synthetic Data Generation, Tool Calling Systems and Agents, Data Complexity, Iterative Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_A1GqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that existing tool-use benchmarks that filter out invalid trajectories, have a bias towards low complexity examples. It proposes BRIT, which makes two modifications to synthetic data generation pipeline: Back Translating and Iterative refinement. Additionally the paper also provides theory arguing that filtering yields a light-tailed complexity distribution, and that a refinement budget growing logarithmically with complexity preserves a heavy tail if the generator already produces one. Experiments report higher complexity of trajectories produced by BRIT, compared to baselines.

Critical strengths
  • The paper explains well why filtering invalid trajectories leads to a higher number of low complexity trajectories in tool-use datasets.
  • The proposed modifications are simple and easy to integrate.
Critical weaknesses
  • Novelty is limited. Both the components of BRIT are borrowed from prior work. Back-Translation is similar to Back-instruct, which was proposed in TaskBench [1], and also closely related to back and forth translation in Magnet [2]. The feedback-based refinement was already explored in APIGen-MT.
  • More importantly, the long horizon claims in the title and paper, are not supported by the evidence the paper provides. BRIT averages 7.4 calls with a median dependency depth of 2 (Table 1), and there’s no comparison against long-horizon benchmarks (Eg: MCP-Bench [3], Toolathlon [4], UltraHorizon [5]). Downstream tests use APIBank and ACEBench, which the authors themselves note are largely simple (Sec. 5). This is a serious mismatch between the stated goal and the provided evidence.
  • One of the papers main motivation abt generating the trajectory before the query (Back Translation) makes little difference in its experiments. The query-first version (BRIT-direct) performs close to BRIT in Tables 1,7,8. Trajectory-first uses slightly more distinct tools, but the length of the chains are similar. Moreover, the authors themselves attribute most of the gain to prompting the model for complexity.
  • The experiments compare BRIT with simple baselines that have different tool pools, generators, and target use cases. Comparision against more complex approaches, that overlap with BRIT in terms of approach (TaskBench, Magnet, ToolMind [6]) is missing. Also, TaskBench's data is public.
  • The paper uses gpt-4o as a judge and refiner. No human study of complexity is performed as in TaskBench.
  • The theory has questionable assumptions (2.1 and 2.2), that assume that node errors and error correction is independent. This is false, since this is a DAG of tool calls, and an error earlier in the trajectory can lead to higher probability of errors downstream.
  • There is a filler placeholder in the appendix (lines 823-824), suggesting that the paper was not proofread before submitting ( [the top X% of samples / the stable region of the Hill plot]).

[1] TaskBench: Benchmarking Large Language Models for Task Automation

[2] Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation

[3] MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

[4] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

[5] UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios

[6] Toolmind technical report: A large-scale, reasoning-enhanced tool-use datase

Questions for authors

See weaknesses

Reviewer_NdbTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that synthetic tool-calling pipelines are biased toward simple examples because they write the query first and then filter out invalid samples. It proposes BRIT, which generates the trajectory before the query and refines invalid samples instead of discarding them. The authors back this with a light-tail/heavy-tail analysis and show that BRIT produces more complex trajectories.

Critical strengths

The paper clearly diagnoses a real problem, that filtering systematically removes complex examples, and offers a practical, modular fix supported by acceptance-rate data.

Critical weaknesses

It is unclear what drives the gains or whether they matter. The query-first ablation reaches similar complexity, so the benefit seems to come from prompting for complexity rather than from back-translation. The paper also never shows that BRIT data is harder for current models, and the downstream gains are small and inconsistent.

Questions for authors
  1. How do strong current models perform on BRIT tasks compared with existing benchmarks?

  2. What does back-translation add beyond complexity-targeted prompting?

  3. How reliable are the LLM simulator and validity judge on long trajectories?

  4. Do the independence and fixed fix-rate assumptions hold empirically?

  5. Are the fine-tuning results robust across seeds?

Reviewer_2P8jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper first investigates the issue that current synthetic tool-calling datasets usually have short and relatively simple trajectories. To address this issue, this work proposes a framework named BRIT (Back translating and Refining Iteratively for Trajectory generation). The proposed BRIT generates trajectories in advance of user queries and iteratively refines them to correct invalid examples. Experimental results show that BRIT can effectively increase the complexity of trajectories.

Critical strengths

This work has these critical strengths:

  • The paper first identifies an actual issue in existing methods and it is well motivated.
  • The two modification strategies in BRIT are easy to follow.
Critical weaknesses

This work has these following critical weaknesses:

  • The current experimental results are not convincing. Current results show that complexity-targeted generation is successful, but it does not show that back-translation is the core mechanism.
  • The novelty of this paper is limited. Many core designs are based on previous works. For example, MAGNET [1] already translates function-signature paths into queries and executable calls.
  • There are many errors in current bibliography. It is important to fix them:
    • The author names of 'Retrieval-Augmented Generation for Large Language Models: A Survey' is not fully correct.
    • The author names and orders of Gorilla OpenFunctions v2 [3] are not correct.
    • The last author of 'HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios' [4] is missing.

[1] Yin, Fan, et al. "Magnet: Multi-turn tool-use data synthesis and distillation via graph translation." In ACL 2025.
[2] https://arxiv.org/abs/2312.10997
[3] https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html
[4] https://aclanthology.org/2025.findings-acl.175/

Questions for authors

Please mainly refer to the weaknesses part.

There are also some typos:

  • Line 153 - Line 154: 'fig. 2' appears twice.
  • Line 224 - Line 226: 'a necessary and a sufficient criteria' - > 'a necessary and sufficient'.
  • Line 406: 'our generation procedures produces' -> 'our generation procedure produces' or 'our generation procedures produce'.
  • Line 823 - Line 825: there is a place holder content '[the top X% of samples/the stable region of the Hill plot]'.
  • Line 1006: "Is our theoretical findings applicable" -> "Are our theoretical findings applicable".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.