acceptodds
Under review as a conference paper at ICLR 2027

FLRT: Learning Efficient Tool Use from Tool-Sufficient States

Abstract

Tool-augmented reasoning substantially extends the capability of large language models to solve complex problems, but frequent tool use also incurs additional interaction and execution costs. We observe that tool necessity changes dynamically along a reasoning trajectory: a model may rely on external tools to acquire critical information early on, yet become capable of completing the task independently at later states. To characterize this transition, we introduce Fork Exploration, which disables future tool access from real intermediate states and observes whether the model can still reach a correct solution, yielding a state-level signal of tool sufficiency. Building on this signal, we propose FLRT, a two-stage training framework that learns more efficient tool-use policies through fork-guided supervision and fork-aware reinforcement learning. Experiments across three mathematical reasoning benchmarks show that FLRT improves average accuracy over the base model while reducing tool calls by about one-third. Further analysis shows that FLRT primarily reduces tool interactions after the model has already acquired sufficient information, suggesting that explicitly modeling within-trajectory changes in tool necessity is an effective approach to improving the efficiency of tool-augmented reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Tool Use, Mathematical Reasoning, Reinforcement Learning, Reasoning Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jvq1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FLRT, which uses Fork Exploration to disable future tools at real intermediate states and regenerate continuations, observing whether the model can answer correctly on its own to identify tool redundancy. This signal is used to construct SFT data and optimize tool use through RL based on answer correctness, call counts, and redundancy. Fork Exploration is not needed at deployment. Using GPT-OSS-20B and Python tools on AIME24, AIME25, and HMMT, mean accuracy increases from 76.30% to 79.77%, while the average number of calls decreases from 8.90 to 6.13.

Critical strengths
  1. A novel, interpretable state-level signal. Fork tests whether existing information suffices without further tools and generates successful continuations for training, providing more targeted supervision than call-count penalties alone.
  2. Higher accuracy with fewer calls. FLRT reaches 79.77% accuracy with about 31% fewer calls, versus 76.30% for Base and 59.62% without tools. Ablations offer preliminary support for the Fork signal beyond call counts.
Critical weaknesses
  1. Fork reliability. Single-sample continuations and stopping at the first failed fork may make redundancy estimates and RL rankings noisy. Repeated sampling and exhaustive-scan comparisons are needed.

  2. Attribution of training gains. Missing controls include Fork-ranked RL directly from Base and equal-data SFT without Fork. Incomplete checkpoint/configuration mappings and absent multi-seed results further limit the ablations. Matched-budget comparisons are needed.

  3. Practical efficiency. Fewer calls do not establish end-to-end speedups. The reported RL overhead of about 12% excludes Fork data construction and SFT; full training costs, deployment token counts, and latency remain unreported.

  4. Limited baseline coverage. Comparisons with EAPO, SMART, and full Probe&Prefill are missing; RTA is only a prompting baseline. These methods should be compared using the same model, tool environment, and comparable training budgets.

Questions for authors
  1. How stable are redundancy estimates and RL rankings under repeated Fork sampling or exhaustive scanning?
  2. How do direct Fork-ranked RL and SFT without Fork perform under matched budgets? Please clarify checkpoint/configuration mappings and report multi-seed results.
  3. What are the full training costs and deployment token counts, tool execution time, and end-to-end latency?
Reviewer_yZKfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies redundant tool calls in interleaved tool-augmented reasoning. Fork Exploration restores a pre-call state of a correct trajectory, disables all future tools, and regenerates the rest of the solution; scanning backward from the last call gives the length r(τ) of the redundant suffix. FLRT then trains in two stages: SFT on original prefixes joined with successful tool-free continuations, followed by GRPO whose reward ranks correct candidates by N_tool + α·r. With GPT-OSS-20B on AIME25, AIME24 and HMMT, FLRT reaches 79.77% mean accuracy with 6.13 calls per trajectory, compared with 76.30% and 8.90 calls for the base model.

Critical strengths

Careful scoping. The claims are stated cautiously and the limitations are explicit (Appendix D). The paper compares against a sweep of uniform per-call penalties rather than a single setting (Figure 3, L284) and reports the training-time overhead of online forking (+12.00% per update, Table 5, L459–461).

Critical weaknesses

W1. The manuscript indicates that several reported analyses cannot currently be traced to their underlying data. The paper states that "original per-trajectory records and script bindings for this historical subset remain pending" (L357–358). It describes the case study as "descriptive summaries ..., not verbatim recovered traces" (L374–375), notes that "the complete original case traces have not yet been recovered" (Figure 4 caption, L406–407), and says that "run-level binding of initialization checkpoints, rollout settings, training updates, and checkpoint-selection rules remains incomplete" (L447–449; see also L728–729). The numbers are also inconsistent across tables. The Base in Table 2 (82.36%, 9.43 calls, L331) is a "historical pooled analysis set" (L335), not the Base of Table 1 (82.50%, 9.7042 calls, L258). FLRT's redundancy at α = 1 is 45.53% in Table 2 (L333) but 46.96% in Table 4 (L438, L441). Section 5.2 uses yet another subset (7.14 vs. 5.98 mean calls, L354–355). Sections 5–6 are the main evidence that FLRT reduces redundant calls specifically, rather than calls in general, so this analysis needs to be rerun on the same, recoverable trajectories as Table 1. W2. Scale and statistical support are limited. All experiments use one model (GPT-OSS-20B, L220), one tool (Python), and mathematical benchmarks only (L725–726), with no multi-seed results (L225, L447, L722–723). The pilot study (L108–110) uses one trajectory per problem, and each fork outcome is a single sample (L126–127). The observation that all 25 correct trajectories contain a tool-sufficient state (L052, L109–110) is a weak test: the reverse scan starts before the last call (L131), and later calls are mostly verification (L362–364). A trajectory qualifies if skipping only its final call still leads to a correct answer once. Reporting the distribution of r(τ)/N_tool in the pilot and the success rate over several samples per fork would make the observation informative. Results on at least one more model or tool type would show whether the finding generalizes.

Questions for authors

N/A

Reviewer_k8j2OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FLRT, a two-stage training framework for reducing redundant tool calls in tool-augmented reasoning. Its core procedure, Fork Exploration, generates continuations from intermediate states of successful trajectories with all subsequent tools disabled, testing whether the model can complete the task using the information already available. Successful continuations provide supervision for Fork-guided SFT, while a reverse scan estimates contiguous redundant tool-call suffixes for Fork-ranked RL, which ranks correct trajectories using both total tool calls and measured redundancy within a GRPO framework. Experiments with GPT-OSS-20B on AIME24, AIME25, and HMMT show improved average accuracy and approximately 31% fewer tool calls than the base model, without requiring additional fork exploration at deployment.

Critical strengths
  • The paper has a clear motivation and addresses the practical problem of redundant tool calls. It focuses on the tendency of models to continue calling tools even after the available information is sufficient to complete the task. The problem is clearly defined and practically relevant.
  • The method is simple, intuitive, and easy to understand and implement. The overall pipeline is clear, and the two training stages are naturally connected. It requires no additional complex model components or fork exploration at deployment time, making the approach practical.
  • The experimental results demonstrate improvements in both accuracy and tool-use efficiency. Across the three mathematical reasoning benchmarks evaluated, the method reduces redundant calls while preserving effective tool use, achieving a favorable balance between task performance and tool-call cost.
Critical weaknesses
  • The paper adopts a somewhat narrow definition of redundant tool calls, focusing on a contiguous redundant suffix at the end of a trajectory. However, redundant calls may also occur at the beginning or in the middle of a trajectory, even when necessary calls remain near the end. Although the authors acknowledge this limitation, the current experiments do not establish how comprehensively the metric captures overall tool redundancy or whether training with this signal reduces redundant behavior earlier in the trajectory. I would appreciate further clarification and analysis of this issue.
  • Tool redundancy in tool-integrated reasoning (TIR) also occurs in knowledge-based question answering and scientific reasoning, yet the evaluation is limited to mathematics. Prior work has highlighted the trade-off between task accuracy and tool-call cost. I therefore suggest extending the evaluation to other domains, such as knowledge-based question answering or scientific reasoning. Adding one or two representative benchmarks would help assess the broader applicability of FLRT.
  • Existing studies have optimized tool use through approaches such as knowledge-boundary modeling and behavior calibration, including IKEA, Tool-Light, ET-Agent, AutoTIR, and SMART. Although the paper includes baselines such as OTC-GRPO, comparisons with these related approaches remain limited. I suggest adding representative baselines under consistent model, tool, and training settings to better establish the advantages of FLRT. Given computational constraints, FLRT and the selected baselines could also be evaluated using a smaller model.
  • The paper does not provide sufficient detail about the sources, scale, and composition of the training data. In particular, the numbers and mixing proportions of the three types of trajectories used during SFT are missing.
Questions for authors
  • How does FLRT disable subsequent tool calls when generating a continuation from an intermediate state? If the model still produces a tool-call request, how is it handled? Please provide detailed examples illustrating the implementation.
  • The ablation study shows a performance decline after Fork-guided SFT, although the supervised training dataset retains only trajectories that successfully solve the tasks. How do the authors explain this decline?
  • Could the authors clarify how final-answer correctness is evaluated (e.g., exact matching, symbolic equivalence checking, or an LLM judge) ?
  • Could the authors compare how rollout output token counts evolve during RL training for FLRT and standard GRPO? In particular, could they provide learning curves showing task performance against the cumulative number of output tokens generated during training rollouts?
  • The behavioral analysis shows that the suffixes identified as redundant contain many verification-related tool calls. However, a single successful continuation with subsequent tools disabled does not establish that these verification calls have no value in improving the probability of success. Many previous works show that they may help the model detect errors and correct its answer in other samples. Could the current redundancy criterion also penalize such beneficial verification, thereby weakening the model’s self-verification ability? I would appreciate a clear explanation addressing this concern.
Reviewer_Brz5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method to make LLMs more efficient in using tools. The key point is to find out during training which tool calls were redundant, and then use that to make the model better. They restart a correct solution just before a tool call with future tools turned off, and if the model still gets the right answer, the later calls count as redundant. SFT then trains on these shorter tool-free endings, and RL favors correct answers that use fewer and less redundant calls. The result is fewer tool calls with slightly higher accuracy on math benchmarks.

Critical strengths

Measurable benefits. Ideally this technology would stack with other methods that strive to improve reasoning efficiency.

Critical weaknesses

The main weakness is the evidence behind the numbers. There are no multi-seed results, numbers across tables come from different runs, and the authors say some records and details aren’t fully tied down, including how checkpoints were selected.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.