acceptodds
Under review as a conference paper at ICLR 2027

Beyond Functional Correctness: Optimizing Multi-turn Tool-Augmented Agent Trajectories to Reduce Tool Usage

Abstract

Large language model (LLM) agents are increasingly evaluated on multi-turn tool use, where success depends not only on calling the correct APIs but also on keeping interaction trajectories compact. Existing alignment methods largely optimize for correctness, while efficiency-oriented approaches mainly target retrieval or single-query settings, leaving open how to reduce excess tool calls in sustained, stateful dialogues without sacrificing task success. We propose an iterative Direct Preference Optimization (DPO) framework that improves both accuracy and efficiency for multi-turn tool agents. For each query, we sample multiple successful trajectories, rank them by tool-call overhead, and convert best–worst discrepancies into step-level preference pairs via two complementary rules: one detecting redundant or corrective detours, and another recovering divergences under parallel tool calling. A greedy decoding filter retains only pairs the current policy has not yet mastered, and the sample–construct–optimize loop is repeated to expose deeper inefficiencies inside turns. When this cross-trajectory loop plateaus, we further mine preferences from the greedy successful trajectory via validated pruning—ablating removable tool calls under the BFCL multi-turn checker—and apply an additional DPO stage with cleaned prefixes. On BFCLv3 multi-turn subsets, LoRA-based DPO followed by this prune stage raises an 8B BitAgent model to 79.5% average accuracy, surpassing the previous overall state of the art (xLAM-2-70b-fc-r, 77.38%), while reducing excess tool calls on the Base subset from 2082 to 25. Trained only on Base, the same policy generalizes to Miss-Param, Missing-Func, and Long-Context subsets with fewer excess tool calls (and often higher accuracy), and also lowers total tokens and inference steps, indicating that compact trajectories often coincide with correct multi-turn behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-turn tool use, DPO, agent efficiency, function calling, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vHBDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that multi-turn tool-using agents should be judged on efficiency as well as correctness: even successful trajectories often contain redundant or corrective tool calls that inflate interaction cost. It proposes an iterative DPO pipeline that samples several successful trajectories per query, ranks them by tool-call overhead, and turns best-vs-worst differences into step-level preference pairs via two rules (detour detection and first-divergence under parallel tool calls), keeping only pairs the current greedy policy fails. When cross-trajectory contrasts flatten out, an intra-trajectory pruning stage removes tool calls that the BFCL checker confirms are unnecessary and runs a final DPO stage on cleaned prefixes. On BFCL v3 multi-turn subsets, an 8B model trained this way surpasses a much larger prior best model while sharply cutting excess tool calls; trained only on the Base subset, the policy transfers to the harder subsets with fewer calls, tokens, and inference steps.

Critical strengths
  1. Reducing extra tool calls in agent trajectory is genuine problem, especially in improving the latency concerns for agentic AI applications. This paper addresses exactly that.
  2. The small model, once trained, seems to significantly reduce the number of tool calls on BFCL v3.
  3. Upon training, accuracy on some harder sets also improves in addition to efficiency.
Critical weaknesses
  1. The evaluation scope is limited to BFCL v3. In order for the technique to be considered generalizable, empirical analysis on multiple benchmarks is required.
  2. The baseline (untrained model) is a weaker choice. The paper mentions a few other efficiency-focused methods that should be considered and compared as baselines.
  3. Per-component isolation ablations are missing.
  4. The pipeline seems to be applied differently for different backbones. Authors should provide analysis on how tedious hand-tuning of the technique is.
  5. The method is heavily dependent on the BFCL checker. But in many cases where there are a very high number of tools, there can be many solution paths possible; such an oracle checker might not be directly applicable in such cases.
  6. Error bars and CIs are not reported. This makes it hard to gauge where the improvement comes from.
Questions for authors
  1. Does the technique generalize to other environments like TauBench or ToolSandbox?
  2. Is there a variant of the technique for other environments where oracle checker might not be available?
  3. How sensitive is the excess-call reduction to the choice of reference/ground-truth trajectory used to define "excess"?
1 comment

To compare our method with other baselines on matched data, We split the 200 base samples in BFCL multi-turn into training and evaluation by stratified sampling over the involved tool classes, yielding 150 training samples and 50 evaluation samples, with no overlap. The miss_func, miss_param, and long_context subsets share the same ids as the base samples. Once an id is held out for evaluation, its counterparts in these augmented subsets are also excluded from training and trajectory mining. Evaluation is conducted on the 50 base samples together with their three augmented subsets (200 instances in total), while trajectory mining uses only the 150 base samples.

We compare our method with two iterative baselines, trajectory-level SFT and trajectory-level DPO. In each iteration, we draw K=6 samples at temperature T=0.7. Trajectory-level SFT fine-tunes on the sampled trajectory with the fewest tool calls. Trajectory-level DPO contrasts the trajectory with the most tool calls against the one with the fewest. For both baselines, user messages and tool results are masked, so the loss is computed only on assistant outputs. Each method is trained for 4 iterations. All methods are evaluated on the same held-out set of 200 instances. The table below reports accuracy. Due to the time limit, the held-out evaluation is reported for BitAgent, the backbone on which the paper’s main gains were claimed. The same split for xLAM, and a regeneration check of pruned trajectories, will be included in the revision.

methodbasemiss_funcmiss_paramlong_contextoverall
Base (before training)40/5028/5035/5035/50138/200
Trajectory-level SFT44/5041/5039/5045/50169/200
Trajectory-level DPO43/5032/5036/5037/50148/200
Step-wise DPO(Our Method)44/5040/5039/5047/50170/200

The following table reports the inference cost on the same test set. Inference steps and tokens are measured during evaluation. Excess tools are the generated tool calls beyond the ground-truth calls.

methodinference stepstokensexcess tools
Base (before training)2,40025,630,6992,272
Trajectory-level SFT1,79915,343,569732
Trajectory-level DPO2,24024,997,0831,781
Step-wise DPO(Our Method)1,78015,305,385671

τ-bench is difficult for small open-source models. On retail, xLAM reaches 65/115, above 50%, but remains below mainstream closed-source models. On airline, xLAM-2-8b-fc-r reaches only 35.2, too low for efficiency optimization to be meaningful, so we focus on τ-bench retail. Under the time limit we ran one iteration. Accuracy rises from 65/115 to 70/115. The gain is on the 76 task ids that were not used to build training pairs, from 32/76 to 40/76. On the 30 unseen tasks solved by both the original and the updated model, the total trajectory length falls from 950 to 936. More results will be put in the revision soon.

The validity check used for pruning is the official BFCL multi-turn checker, the tool the benchmark itself uses to decide whether an answer is correct. It is bound to the scenario of each task. For a given example, the checker instantiates the simulated tool backends from that example's initial state and involved tool classes, executes both the proposed calls and the ground-truth calls, and applies two tests after every turn. The resulting backend state must match the ground-truth state, and every ground-truth tool return for the current turn must already be covered by the returns obtained so far, including returns from earlier turns. A call is removed only when both tests still pass. A limitation of our method is that we do not provide this evaluation ourselves. Deciding whether a backend state is sufficient to solve the task is a criterion that should be supplied by the benchmark authors, so the pruning stage applies to benchmarks that provide such a checker.

Excess is the model's tool-call count minus the call count of the benchmark's annotated ground-truth trajectory, on the same samples. It measures how many extra calls remain relative to that reference, not the target of training. During training we rank sampled trajectories against each other by their own tool-call counts. On a fixed set of samples, the ground-truth count cancels out, so the reported reduction in excess equals the reduction in the model's own calls. A longer or shorter reference trajectory would shift the absolute excess of every method by the same amount, and would not change the reduction.

Reviewer_d9ydOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes an iterative DPO framework to improve the accuracy and efficiency of multi-turn tool-using agents by reducing redundant tool calls. The method samples multiple successful trajectories for each task, identifies efficient and inefficient tool-use behaviors, and constructs step-level preference pairs using rule-based divergence detection. The model is then iteratively optimized with DPO, followed by an additional preference optimization stage that prunes unnecessary tool calls while preserving task correctness through programmatic verification. Experiments are conducted on two 8B models, BitAgent-Bounty and Llama-xLAM, using the BFCL v3 Base Multi-Turn subset for preference optimization. The authors report improved accuracy and reduced excess tool calls across four BFCL v3 multi-turn subsets, with the optimized BitAgent achieving 79.5% average accuracy and substantially fewer redundant tool calls.

Critical strengths
  • The work tackles with a practical problem of reducing tool call redundancy in multi-turn interactions. The proposed step-level preference construction and verifier-guided trajectory pruning provide a reasonable approach to identifying inefficient tool-use behaviors while attempting to preserve task correctness.
Critical weaknesses
  • The authors explicitly state in Section 4 that they perform iterative DPO on the BFCL v3 Base Multi-Turn subset, while Table 1 reports performance on the same benchmark subset. Appendix C further confirms that preference pairs are constructed from trajectories sampled on Base. However, the paper does not describe any held-out train/test split or establish that the training and evaluation instances are disjoint. This raises a serious concern that the reported improvements are obtained through direct optimization on benchmark evaluation tasks. Moreover, the other three evaluation subsets are augmented variants of the Base tasks, making them insufficient to establish generalization to genuinely unseen tasks. I therefore consider this a fundamental flaw in the experimental methodology that undermines the major conclusion of this paper.

  • The proposed method primarily combines existing techniques, including iterative DPO, step-level preference learning, and trajectory pruning. However, the paper does not provide comparisons against straightforward alternatives such as SFT on efficient successful trajectories, standard trajectory-level DPO, or efficiency-aware RL.

Questions for authors

None

1 comment

Thank you for the comment on our evaluation setup. We would like to acknowledge that the numbers reported in the original experiments are the model's leaderboard scores. The leaderboard evaluates the full test set, which is not separated from the samples used for trajectory mining and training, so those scores do not establish the effectiveness of the method. This is a limitation of the evaluation.

To address it, we no longer use the leaderboard scores as the main result. We split the 200 base samples in BFCL multi-turn into training and evaluation by stratified sampling over the involved tool classes, yielding 150 training samples and 50 evaluation samples, with no overlap. The miss_func, miss_param, and long_context subsets share the same ids as the base samples. Once an id is held out for evaluation, its counterparts in these augmented subsets are also excluded from training and trajectory mining. Evaluation is conducted on the 50 base samples together with their three augmented subsets (200 instances in total), while trajectory mining uses only the 150 base samples.

We compare our method with two iterative baselines, trajectory-level SFT and trajectory-level DPO. In each iteration, we draw K=6 samples at temperature T=0.7. Trajectory-level SFT fine-tunes on the sampled trajectory with the fewest tool calls. Trajectory-level DPO contrasts the trajectory with the most tool calls against the one with the fewest. For both baselines, user messages and tool results are masked, so the loss is computed only on assistant outputs. Each method is trained for 4 iterations. All methods are evaluated on the same held-out set of 200 instances. The table below reports accuracy. Due to the time limit, the held-out evaluation is reported for BitAgent, the backbone on which the paper’s main gains were claimed. The same split for xLAM, and a regeneration check of pruned trajectories, will be included in the revision.

methodbasemiss_funcmiss_paramlong_contextoverall
Base (before training)40/5028/5035/5035/50138/200
Trajectory-level SFT44/5041/5039/5045/50169/200
Trajectory-level DPO43/5032/5036/5037/50148/200
Step-wise DPO(Our Method)44/5040/5039/5047/50170/200

The following table reports the inference cost on the same test set. Inference steps and tokens are measured during evaluation. Excess tools are the generated tool calls beyond the ground-truth calls.

methodinference stepstokensexcess tools
Base (before training)2,40025,630,6992,272
Trajectory-level SFT1,79915,343,569732
Trajectory-level DPO2,24024,997,0831,781
Step-wise DPO(Our Method)1,78015,305,385671

Our main contribution is to locate inefficient tool use on these interactive tasks and fine-tune the policy to avoid the points that have already been identified. Using the benchmark checker only as a validity test, we group removable behavior into four types and build a different step-level preference for each. Autonomy covers trailing calls that do not affect completion, with the preferred action being a natural-language reply. Redundancy covers successful but unnecessary calls. Error covers avoidable failed attempts. Repair covers a compensating detour that is removable only when its calls are deleted together. Each case becomes a preference on the cleaned history, excluding the deleted calls and their observations, and a further DPO stage trains the policy toward that local decision.

Reviewer_nUZvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an iterative DPO framework that curbs unnecessary tool calls in multi-turn LLM agents. The method compares successful trajectories that differ in tool-use cost and turns those differences into step-level preference pairs. Once training plateaus, a final pruning stage removes any tool call whose deletion still lets the trajectory pass the benchmark checker. Experiments with two 8B models on BFCL v3 show lower tool usage, token consumption, and inference steps, along with substantial accuracy gains for BitAgent.

Critical strengths

This paper tackles a practical problem: agents that succeed can still waste resources through redundant calls and avoidable recovery loops. Its method builds targeted, step-level preferences (including for parallel tool calls) and filters out pairs the current policy already satisfies, so training focuses on remaining errors. The final pruning stage is carefully designed, verifying that deletions preserve success and using cleaned histories rather than observations from removed calls. Results are promising. For BitAgent, Base accuracy rises from 78.0% to 85.0%, excess tool calls drop from 2,082 to 25, and token usage falls from 15.2M to 10.8M. Both backbones show reduced excess tool use, and preferences built only from Base also improve efficiency on the missing-parameter, missing-function, and long-context variants, with especially strong accuracy gains for BitAgent.

Critical weaknesses

The main concern is evaluation validity: preference pairs come from BFCL Base, which is also used for reporting, and no clearly disjoint train/validation/test split is described. This makes the Base gains, and the state-of-the-art claim, hard to read as evidence of generalization. Novelty is also limited, since the framework combines existing pieces (iterative DPO, rule-based preferences, validated pruning), and it lacks comparisons with simpler alternatives such as SFT on compact successful trajectories, trajectory-level DPO, or tool-cost-penalized RL. Without these, the value of the proposed design is unclear. The pruning stage relies heavily on the BFCL checker, which does not show that the agent can still produce later actions and answers without the information from deleted calls. The paper should show that compressed trajectories remain executable.

Questions for authors

How does the method compare with simpler alternatives, such as SFT on compact successful trajectories or trajectory-level DPO, under matched data and compute budgets?

1 comment

Thank you for the comment on our evaluation setup. We would like to acknowledge that the numbers reported in the original experiments are the model's leaderboard scores. The leaderboard evaluates the full test set, which is not separated from the samples used for trajectory mining and training, so those scores do not establish the effectiveness of the method. This is a limitation of the evaluation.

To address it, we no longer use the leaderboard scores as the main result. We split the 200 base samples in BFCL multi-turn into training and evaluation by stratified sampling over the involved tool classes, yielding 150 training samples and 50 evaluation samples, with no overlap. The miss_func, miss_param, and long_context subsets share the same ids as the base samples. Once an id is held out for evaluation, its counterparts in these augmented subsets are also excluded from training and trajectory mining. Evaluation is conducted on the 50 base samples together with their three augmented subsets (200 instances in total), while trajectory mining uses only the 150 base samples.

We compare our method with two iterative baselines, trajectory-level SFT and trajectory-level DPO. In each iteration, we draw K=6 samples at temperature T=0.7. Trajectory-level SFT fine-tunes on the sampled trajectory with the fewest tool calls. Trajectory-level DPO contrasts the trajectory with the most tool calls against the one with the fewest. For both baselines, user messages and tool results are masked, so the loss is computed only on assistant outputs. Each method is trained for 4 iterations. All methods are evaluated on the same held-out set of 200 instances. The table below reports accuracy. Due to the time limit, the held-out evaluation is reported for BitAgent, the backbone on which the paper’s main gains were claimed. The same split for xLAM, and a regeneration check of pruned trajectories, will be included in the revision.

methodbasemiss_funcmiss_paramlong_contextoverall
Base (before training)40/5028/5035/5035/50138/200
Trajectory-level SFT44/5041/5039/5045/50169/200
Trajectory-level DPO43/5032/5036/5037/50148/200
Step-wise DPO(Our Method)44/5040/5039/5047/50170/200

The following table reports the inference cost on the same test set. Inference steps and tokens are measured during evaluation. Excess tools are the generated tool calls beyond the ground-truth calls.

methodinference stepstokensexcess tools
Base (before training)2,40025,630,6992,272
Trajectory-level SFT1,79915,343,569732
Trajectory-level DPO2,24024,997,0831,781
Step-wise DPO(Our Method)1,78015,305,385671

The validity check used for pruning is the official BFCL multi-turn checker, the tool the benchmark itself uses to decide whether an answer is correct. It is bound to the scenario of each task. For a given example, the checker instantiates the simulated tool backends from that example's initial state and involved tool classes, executes both the proposed calls and the ground-truth calls, and applies two tests after every turn. The resulting backend state must match the ground-truth state, and every ground-truth tool return for the current turn must already be covered by the returns obtained so far, including returns from earlier turns. A call is removed only when both tests still pass. A limitation of our method is that we do not provide this evaluation ourselves. We believe deciding whether a backend state is sufficient to solve the task is a criterion that should be supplied by the benchmark authors.

Our main contribution is to locate inefficient tool use on these interactive tasks and fine-tune the policy to avoid the points that have already been identified. Using the benchmark checker only as a validity test, we group removable behavior into four types and build a different step-level preference for each. Autonomy covers trailing calls that do not affect completion, with the preferred action being a natural-language reply. Redundancy covers successful but unnecessary calls. Error covers avoidable failed attempts. Repair covers a compensating detour that is removable only when its calls are deleted together. Each case becomes a preference on the cleaned history, excluding the deleted calls and their observations, and a further DPO stage trains the policy toward that local decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.