- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that existing tool-use benchmarks that filter out invalid trajectories, have a bias towards low complexity examples. It proposes BRIT, which makes two modifications to synthetic data generation pipeline: Back Translating and Iterative refinement. Additionally the paper also provides theory arguing that filtering yields a light-tailed complexity distribution, and that a refinement budget growing logarithmically with complexity preserves a heavy tail if the generator already produces one. Experiments report higher complexity of trajectories produced by BRIT, compared to baselines.
Critical strengths
- The paper explains well why filtering invalid trajectories leads to a higher number of low complexity trajectories in tool-use datasets.
- The proposed modifications are simple and easy to integrate.
Critical weaknesses
- Novelty is limited. Both the components of BRIT are borrowed from prior work. Back-Translation is similar to Back-instruct, which was proposed in TaskBench [1], and also closely related to back and forth translation in Magnet [2]. The feedback-based refinement was already explored in APIGen-MT.
- More importantly, the long horizon claims in the title and paper, are not supported by the evidence the paper provides. BRIT averages 7.4 calls with a median dependency depth of 2 (Table 1), and there’s no comparison against long-horizon benchmarks (Eg: MCP-Bench [3], Toolathlon [4], UltraHorizon [5]). Downstream tests use APIBank and ACEBench, which the authors themselves note are largely simple (Sec. 5). This is a serious mismatch between the stated goal and the provided evidence.
- One of the papers main motivation abt generating the trajectory before the query (Back Translation) makes little difference in its experiments. The query-first version (BRIT-direct) performs close to BRIT in Tables 1,7,8. Trajectory-first uses slightly more distinct tools, but the length of the chains are similar. Moreover, the authors themselves attribute most of the gain to prompting the model for complexity.
- The experiments compare BRIT with simple baselines that have different tool pools, generators, and target use cases. Comparision against more complex approaches, that overlap with BRIT in terms of approach (TaskBench, Magnet, ToolMind [6]) is missing. Also, TaskBench's data is public.
- The paper uses gpt-4o as a judge and refiner. No human study of complexity is performed as in TaskBench.
- The theory has questionable assumptions (2.1 and 2.2), that assume that node errors and error correction is independent. This is false, since this is a DAG of tool calls, and an error earlier in the trajectory can lead to higher probability of errors downstream.
- There is a filler placeholder in the appendix (lines 823-824), suggesting that the paper was not proofread before submitting ( [the top X% of samples / the stable region of the Hill plot]).
[1] TaskBench: Benchmarking Large Language Models for Task Automation
[2] Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
[3] MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
[4] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
[5] UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios
[6] Toolmind technical report: A large-scale, reasoning-enhanced tool-use datase
Questions for authors
See weaknesses