- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LOTS, a method for constructing and continuously refining a task-specific tool space for tasks an LLM agent performs repeatedly. Existing tool selection methods often rely on tool descriptions or the model’s pre-existing knowledge; they often fail to fully leverage the actual outcomes of tool execution and require repeating the selection process for every new request.
LOTS keeps the model’s final generated response fixed and measures the change in the response’s log-likelihood when the observed results from a specific tool are removed. By aggregating these "leave-one-out" attribution scores across multiple requests belonging to the same task, the method identifies high-scoring tools to form a persistent, task-specific tool space. When subsequent requests of the same type arise, the method can reuse this existing tool space and further update it with results from new executions.
Critical strengths
1. Efficient Tool Selection Leveraging Actual Tool Outputs : The primary strength of this paper lies in its approach: rather than relying solely on tool descriptions or semantic relevance, it measures the contribution of actual tool outputs, observed during execution, to the model’s final response. Specifically, the method of holding the final response constant and observing changes in likelihood when evidence from a given tool is removed is highly intuitive; furthermore, it can be applied without the need for additional model training.
2. Consistent Experimental Results Across Diverse Settings : The authors evaluate their method across three benchmarks: compositional tool use, multi-turn function calling, and executable ReAct agents. In the TGB benchmark, the method achieved higher accuracy across all eight tested models compared to using the full tool library; it also demonstrated positive results in the BFCL and GTA-Atomic benchmarks.
Critical weaknesses
1. Discrepancy between the claim of "label-free tool optimization" and the actual evaluation setup: A core feature of LOTS is that it estimates the importance of tools based on the model’s own outputs, without requiring ground-truth labels. However, the paper notes that the final tool budget, , is selected based on accuracy on a "fitting set." In other words, while the likelihood-based ranking itself does not utilize correctness labels, such labels are indeed used during the construction of the final tool space. Although the paper acknowledges this distinction and provides comparisons under a fixed budget, it remains unclear whether the primary experimental results can be replicated in a truly label-free deployment scenario. Specifically, further validation is required to determine the extent to which the task-specific selection of contributes to overall performance, and whether similar performance can be maintained if is determined without access to correctness labels.
2. Insufficient validation of "continual self-evolution" and actual task utility The paper posits that agents can continuously improve through persistent, task-specific tool spaces. Yet, in the primary sequential experiments, the task families or API classes are pre-defined, with tool spaces constructed and reused for each specific task. While these results demonstrate the utility of task-specific tool configurations, they do not sufficiently verify whether an agent can reliably adapt to real-world environments where task requirements shift or task boundaries are ill-defined.
Although the appendix includes an analysis of unsupervised task clustering, these experiments rely on synthetic task families with relatively clear-cut distinctions. Furthermore, likelihood-based attribution measures only the model’s reliance on its own generated responses—not the correctness of those responses or their actual utility. Because a model might assign a high score to a specific tool even when its output is incorrect, we need to examine more directly how tool-space quality evolves as errors accumulate over multiple updates. While the current experiment effectively demonstrates the benefits of tool-space optimization, it has limitations in proving the feasibility of reliable self-evolution in more general environments.
Questions for authors
Q1. Label-free tool selection: While LOTS uses likelihood-based scoring without correctness labels, the tool budget K is selected using fitting accuracy. I appreciate the fixed-budget results in Appendix, but how would K be determined when ground-truth answers are unavailable? Could the authors evaluate a fully label-free selection strategy to clarify how much the choice of K contributes to the reported gains?
Q2. Adaptation to changing tool requirements: The sequential experiments show that LOTS can refine tool spaces over time, but the task requirements remain largely stable. How would LOTS adapt if an existing task started requiring previously pruned tools? Figure 4 shows some recovery of missing tools, but it is unclear how reliably this works when tool requirements change during deployment.
Q3. Likelihood contribution versus task correctness: A tool that increases the likelihood of a generated answer may not necessarily improve its correctness. Although Appendix E.5. examines incorrect outputs, it is unclear whether these findings hold under repeated tool-space updates. Could incorrect outputs accumulate misleading attribution scores and eventually reinforce suboptimal tool selections?
Q4. Sensitivity to task grouping LOTS assumes that requests can be grouped into recurring tasks. Appendix D.1 shows that merging task families can hurt performance, particularly with smaller tool budgets. How robust is LOTS to imperfect task grouping, especially when semantically similar requests require different tools? Can the method recover from incorrect task assignments over time?
3 comments
Part 2/2 — Response to Reviewer D9rM
----- W2 / Q3. Can incorrect outputs accumulate misleading scores and reinforce poor selections? -----
This is a valid failure mode. A tool can receive a positive likelihood score for supporting an incorrect answer, so the score alone cannot prevent misleading evidence from influencing later selections. We tested whether incorrect fitting answers produce poor tool spaces and whether selected spaces are more vulnerable to erroneous tool outputs. These experiments address parts of the concern, but do not establish resistance to error amplification over repeated updates.
First, we constructed tool rankings separately from traces with correct and incorrect final answers, then evaluated the selected spaces at the same task-specific budget. Correctness labels are used only to form these diagnostic subsets; they are not inputs to the likelihood score. This tests whether an incorrect final answer necessarily makes its execution history unsuitable for tool selection.
| Model | Correct fitting answers | LOTS, all traces (%) | Correct-only ranking (%) | Wrong-only ranking (%) |
|---|---|---|---|---|
| Qwen2.5-7B | 575 / 800 | 82.41 | 82.75 | 80.72 |
| Llama-3.1-8B | 609 / 800 | 85.75 | 86.09 | 85.16 |
| Mistral-7B | 360 / 800 | 65.81 | 65.91 | 62.53 |
| Qwen3-4B | 395 / 800 | 68.34 | 62.88 | 68.03 |
| Qwen3-8B | 476 / 800 | 82.28 | 80.34 | 70.88 |
The wrong-only rankings still select useful spaces, but the effect varies by model. For example, Llama-3.1-8B achieves 85.16% with the wrong-only ranking versus 85.75% with all traces, whereas Qwen3-8B drops from 82.28% to 70.88%. Across 100 model–family–subset cases, including both correct-only and wrong-only subsets, 79 are within one percentage point of the all-trace result. Nine lose more than five points: four correct-only and five wrong-only. Thus, incorrect answers do not invariably destroy the selection signal, but the Qwen3-8B result shows that selection can deteriorate substantially.
Second, we tested erroneous tool outputs while keeping the selected spaces fixed. In an eight-model TGB experiment, a required tool returns an incorrect value on corrupted requests. Among requests that both LOTS and Keep all answer correctly under clean conditions, success under corruption is 17.17% for LOTS and 16.64% for Keep all. Both perform poorly under corruption, with similar aggregate success rates. This comparison tests sensitivity during execution; because the spaces are fixed, it does not test whether corrupted evidence leads to progressively worse selections.
The remaining question is whether errors reinforce themselves when new scores change the tool space and that space determines later execution histories. Neither diagnostic follows this feedback loop over repeated updates. We therefore treat long-term error amplification as an unresolved limitation, rather than infer stability from the fixed-history or fixed-space results.
We will incorporate these diagnostics and their limitations into the paper, clarify that likelihood attribution measures reliance on evidence rather than correctness, and explicitly state that resistance to error amplification over repeated updates has not been established.
----- W2 / Q4. How sensitive is LOTS to imperfect task grouping, and can it recover from incorrect assignments? -----
Grouping quality and tool budget jointly affect performance. In the tested misrouting experiment, the space recovers after routing is corrected externally; LOTS does not itself identify or repair the assignment error.
Text-only grouping matches oracle-grouping performance on the four-model TGB evaluation. Separate text-clustering diagnostics recover the evaluated TGB families and BFCL API classes.
Controlled merging into five groups reduces accuracy by up to five points at the fitted budgets, compared with 14.4–32.9 points at a fixed two-tool budget. A merged group can require more tools than the budget permits, making this both a grouping issue and a budget constraint.
For recovery, another family's requests are misrouted into a task's history during rounds 2–4, and correct routing resumes from round 5. Required tools are present within one corrected round in all 45 cells. In all 30 cells with exploration, accuracy from the second corrected round is within three points of the pre-misrouting level. Exploration also causes larger damage while misrouting continues, illustrating its sensitivity to the quality of incoming history.
We will incorporate the budget and recovery controls into the paper, state the dependence on supplied or inferred task groups, and distinguish recovery after external routing correction from autonomous correction of task identity.
Dear Reviewer D9rM,
As the author–reviewer discussion period draws to a close on Nov. 18, we would appreciate any further thoughts you may have on whether our response addresses your main concerns.
Thank you again for your time and consideration!
Response to Reviewer D9rM
Thank you for recognizing the value of scoring actual tool outputs. We clarify the distinction between label-free scoring and budget selection, and report additional analyses of changing requirements, incorrect histories, and task grouping.
----- W1 / Q1. Is the complete tool-selection procedure label-free when K is selected using fitting accuracy? -----
The likelihood score is label-free, while the submitted accuracy-based budget selection uses correctness labels. Fully label-free deployment can use a fixed budget, with a measurable performance trade-off. These properties should be reported separately.
We evaluated budget policies on TGB across eight models. The full accuracy curve over K∈{1,…,15} is used to evaluate the selected budgets; only the explicitly labeled oracle uses test accuracy to choose K. For off-model calibrated rules, the global constant is chosen on seven models and evaluated on the remaining model. Avoiding target-model tuning does not by itself establish that the entire calibration procedure is supervision-free, so we distinguish these results from fixed-budget deployment.
Fixed K=3 retains a six-point gain over Keep all. Two labeled requests per task raise the average to 74.61%, compared with 79.30% under the submitted 80-request budget selection. The mass-coverage rule reaches 73.39% without target-model calibration, but its off-model tuning should not be described as globally supervision-free. These results quantify the contribution of budget selection.
We will incorporate the two-label and fixed-budget results as additional operating points, reserve “label-free” for score estimation and label-independent configuration choices, and retain the correct labeling of the submitted budget protocol.
----- W2 / Q2. Can LOTS recover when an existing task begins to require previously pruned tools? -----
Yes, with exploration enabled. In our task-shift experiment, LOTS recovered the previously pruned required tools in all 27 evaluated cases, including 25 within 40 requests.**
We tested precisely this scenario across five TGB models: the task identity remains fixed, but its required tool set changes at round 4, making previously pruned tools necessary. LOTS revises the existing task configuration using subsequent execution evidence. Mean post-change success reaches 68.2% at ε=0.1 and 70.6% at ε=0.25; recovery is rare without exploration. Recovery is possible because pruning excludes a tool from the current serving space, but does not delete it from the registry. Exploration re-exposes tools outside that space. Their newly observed outputs update the running likelihood scores, allowing previously pruned tools to re-enter the top-K configuration. Thus, the configuration is revisable rather than permanently restricted to the tools retained during initial fitting.
We will add this experiment and clarify the recovery condition: LOTS can recover when exploration supplies fresh evidence about newly required tools, as demonstrated under the evaluated abrupt shifts.