- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2(−1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for cost-aware skill retrieval and a two-stage reference method that constructs candidate routes and selects skills based on anticipated execution costs. The problem is practically relevant, and the experiments demonstrate promising savings. However, benchmark generalizability and the attribution of these gains require stronger evidence.
Critical strengths
The study evaluates actual downstream execution rather than retrieval metrics alone. The paired-success analysis and cardinality-matched Top-M baseline provide useful controls. Reporting full and context-adjusted token consumption alongside selection overhead also improves transparency.
Critical weaknesses
-
The novelty and independent contribution of cost-aware reasoning need to be more clearly established. The authors should further discuss the connections between the proposed method and existing studies, including:
- SayCan: Feasibility-aware skill selection;
- Voyager: Reuse of executable skills;
- AeroWeaver: Experience-guided skill selection.
These related studies would help clarify how the proposed cost optimization objective differs from broader notions of skill utility and what additional contributions the proposed method provides.
-
Stronger experimental evidence is needed to demonstrate the preservation of task performance. In Table 9, the success rates of all three retrievers decrease compared with the Top-10 baseline. For example, the success rate of BM25 drops from 71.43% to 65.71%. Furthermore, conducting only three trials per task is insufficient to establish equivalent task performance across different methods.
-
The experimental configuration and efficiency claims require further clarification. In the downstream task evaluation, the paper uses the union of three skill selection results, whereas the selection-level experiments are based on individual runs. In addition, the estimated end-to-end latency assumes that skill selection can be performed in parallel, and the calculation formula does not explicitly account for the latency introduced by skill retrieval itself.
Questions for authors
-
Do the gains persist against a relevance-only LLM selector with comparable inference resources?
-
How does a single selection compare with the three-run union in success rate and total cost?
-
Do the final retrieval results retain their cost advantage when comparisons are restricted to tasks successfully completed under both conditions?
6 comments
Response to W1–W3: Related Work, Task Performance, and Experimental Clarifications
We thank the reviewer for the constructive suggestions. We have made corresponding additions and revisions to address the three major concerns, as detailed below.
Summary of Revisions
-
Related Work: We have added discussions of representative works closely related to our research problem in the revised manuscript, further clarifying the role of execution cost within the broader concept of Skill Utility, as well as the research objectives and contributions of our Cost-Aware Skill Retrieval Benchmark.
-
Abstract, Section 1, and Discussion: We no longer claim that task performance is maintained and instead explicitly characterize the results as a Success–Cost Trade-off.
-
We have added a new subsection on Multi-Sample Skill Set Aggregation, clarifying that this mechanism is an integral component of the Reference Method and that different values of B represent different configurations of the same method. We also compare the selection results and overhead under B = 1, 3, and 5.
-
Appendices G and H: We have added downstream execution experiments for single selection and paired analyses on jointly successful samples, and improved the calculation and reporting of End-to-End Cost.
W1: Related Work and Positioning of the Reference Method
We acknowledge that execution cost is indeed one dimension of the broader concept of Skill Utility. Prior studies have explored related issues from different perspectives, including task feasibility, knowledge expansion, Skill reuse, and execution experience. Building on these studies, our work focuses on a more specific problem: when multiple Skill sets can all support task completion, how can we prioritize those with lower expected downstream execution costs?
Our primary contribution is the construction of a dedicated Cost-Aware Skill Retrieval Benchmark, which provides a reproducible evaluation basis for this problem. The Reference Method serves as an initial exploration, demonstrating the feasibility and practical value of this research direction through explicit cost-aware decision-making and downstream execution experiments, while also illustrating the benchmark's capability to support the study and evaluation of related methods.
W2: Task Performance Degradation and the Number of Repeated Trials
We agree with the reviewer's concern. The original statement "maintaining comparable task performance" was not sufficiently precise, and three repeated trials are insufficient to establish comparable task performance.
In the revised manuscript, we explicitly characterize the results as a Success–Cost Trade-off: compared with Top-10, Cost-Aware Union exhibits an average Pass Rate reduction of 3.17 pp, while reducing End-to-End Time and End-to-End Full Tokens by 14.89% and 51.38%, respectively. We now report the changes in task success rate alongside the execution cost savings and no longer claim that our method maintains comparable task performance.
W3: Experimental Configuration and End-to-End Cost Accounting
Regarding the experimental configuration, the original manuscript did not sufficiently clarify the relationship between single Skill Selection and three-sample Union. In the revised manuscript, Section 4.3 explicitly clarifies that Multi-Sample Skill Set Aggregation is an integral component of the Reference Method, and that single selection and multi-sample Union are configurations of the same Reference Method with different sampling numbers B. When B = 1, Union reduces to the single selection result; when B > 1, it takes the union of multiple independently generated selection results executed in parallel. The overall workflow is illustrated in Figure 4. Section 5.4 now consistently reports the selection results and overhead under different B configurations, while Appendix H.3 further provides downstream execution evaluations of single selection.
Regarding cost accounting, multiple Skill Selection runs are executed in parallel by design, and Selection Latency is determined by the longest parallel branch. We have also explicitly incorporated initial Retrieval Latency, Selection Latency, and Agent Execution Time into End-to-End Time and improved the corresponding cost calculations.
Response to Q1–Q2: Additional Experiments and Analyses
To address the reviewer's question, we introduced a Utility-Aware Union baseline in Appendix H.2. This variant uses an LLM to select Skills associated with candidate routes solely based on task suitability, without an explicit objective of optimizing execution costs.
Both methods reuse exactly the same Top-10 Retrieval results and three sets of Planner outputs, with identical LLM call structures, B = 3 Union Aggregation, and downstream execution settings. The only difference lies in the Selector objective, which is changed from minimizing expected execution cost to maximizing task suitability, ensuring comparable inference resource consumption. The actual Selection Overhead is also included in the end-to-end cost. The experimental workflow is illustrated in Figure 5, and the complete prompts are available in our anonymized repository.
Experimental Results.
Compared with Utility-Aware Union, Cost-Aware Union exhibits a clearer preference for lower-cost Skills and achieves better downstream execution efficiency. The following percentages represent cost reductions achieved by Cost-Aware Union relative to Utility-Aware Union.
| Metric | BM25 | Qwen |
|---|---|---|
| Standard Skill Hits (Utility-Aware → Cost-Aware) | 51 → 54 | 48 → 53 |
| High-Cost Skill Hits (Utility-Aware → Cost-Aware) | 46 → 37 | 49 → 33 |
| Pass Rate (Utility-Aware → Cost-Aware) | 71.43% → 65.71% | 67.62% → 74.29% |
| Agent Execution Time Reduction | 39.25% | 39.89% |
| End-to-End Time Reduction | 32.36% | 25.04% |
| End-to-End Full Tokens Reduction | 43.99% | 34.87% |
1. Skill Selection Preference (Tables 12 and 15). Cost-Aware Union retains more Standard Skills and fewer High-Cost Skills across both retrievers. The Cardinality-Matched Top-M comparison further supports this observation.
2. Downstream Execution Performance (Table 17). Even after accounting for Selection Overhead, Cost-Aware Union substantially reduces end-to-end execution time and token consumption. However, the two methods do not exhibit a consistent advantage in Pass Rate across the two retrievers.
3. Paired Analysis on Successful Cases (Table 18). Among the jointly successful samples (59 for BM25 and 67 for Qwen), Cost-Aware Union still reduces End-to-End Time by 22.58% and 31.30%, respectively, and End-to-End Full Tokens by 34.87% and 49.26%, respectively.
These results demonstrate that, given the same candidate routes and comparable inference resources, explicitly accounting for execution costs provides additional execution efficiency gains beyond selection based solely on task suitability.
Q2: Comparison Between Single Selection and Three-Sample Union
We added a Cost-Aware w/o Union experiment in Appendix H.3, which performs single selection without Union Aggregation. Specifically, this experiment reuses the three independent Cost-Aware Route Skill Selection results generated by the full method at B = 3, before Union Aggregation, and uses each result separately for multiple downstream executions (Figure 6).
Table 19 reports the Pass Rate and full end-to-end costs of w/o Union and B = 3 Union:
| Retriever | Configuration | Pass Rate ↑ | E2E Time ↓ | E2E Full Tokens ↓ |
|---|---|---|---|---|
| BM25 | Top-10 | 71.43% | 248.46s | 833.86K |
| BM25 | Top-M | 51.11% | 345.51s | 663.42K |
| BM25 | w/o Union | 59.68% | 300.28s | 413.43K |
| BM25 | Union (B = 3) | 65.71% | 222.06s | 346.42K |
| Qwen | Top-10 | 77.14% | 269.67s | 814.16K |
| Qwen | Top-M | 56.83% | 319.96s | 864.95K |
| Qwen | w/o Union | 63.49% | 281.54s | 572.46K |
| Qwen | Union (B = 3) | 74.29% | 257.72s | 428.56K |
The results demonstrate two key findings:
-
Single selection is already effective. Compared with Top-M using the same number of Skills, w/o Union improves Pass Rate by 8.57 and 6.66 pp on BM25 and Qwen, respectively, while reducing both end-to-end execution time and Full Tokens. However, compared with Top-10, its lower Skill coverage also leads to a more noticeable loss in task success rate.
-
B = 3 improves the success–cost trade-off. Compared with w/o Union, Union improves Pass Rate by 6.03 and 10.80 pp on BM25 and Qwen, respectively, while reducing End-to-End Time by 26.05% and 8.46%, respectively. Full Tokens are also further reduced. Compared with Top-10, B = 3 still exhibits some reduction in task success rate but achieves substantially lower end-to-end execution costs.
Overall, single selection already provides cost-aware benefits but incurs a noticeable loss in task success rate. Multi-Sample Union further improves Skill coverage and task success rate while reducing end-to-end execution costs, achieving a better trade-off between task success and execution cost, albeit with some remaining reduction in task success rate compared with Top-10.
Author Response Summary
We thank the reviewer for the constructive suggestions. Below, we address the three Critical Weaknesses (W) and three Questions for Authors (Q), summarizing the corresponding revisions and key findings.
W1: Related Work and Positioning of Our Contributions
We have expanded the discussion of how our work relates to and differs from existing utility-aware Skill selection approaches. Execution cost is an important dimension of the broader concept of Skill Utility. Our work focuses on how to identify and prioritize lower-cost alternatives when multiple Skill sets can all support task completion.
Our primary contribution is the construction of a dedicated Cost-Aware Skill Retrieval Benchmark, which provides a reproducible evaluation basis for this problem. Meanwhile, through explicit cost-aware decision-making in our Reference Method and downstream execution experiments, we demonstrate the feasibility of this research direction and validate the evaluation value of the benchmark.
W2: Trade-off Between Task Performance and Execution Cost
We have revised the presentation of our results to explicitly characterize them as a Success–Cost Trade-off. Rather than claiming that comparable task performance is maintained, we now report the observed reductions in task success rate alongside the corresponding execution cost savings. We have also added task-level bootstrap confidence intervals for Pass Rate differences, clarified that the current experiments do not establish equivalent task performance, and revised the corresponding conclusions in the manuscript (Appendix G.3).
W3: Experimental Configuration and End-to-End Cost Accounting
The original manuscript did not sufficiently clarify the experimental configuration and cost accounting. First, Multi-Sample Union is an integral component of our Reference Method. Single selection and multi-sample Union are configurations of the same method with different sampling numbers B, designed to mitigate the instability in Skill coverage associated with single selection. We have clarified this mechanism in Section 4.3 and expanded the relevant evaluations in Section 5.4 and Appendix H.3 to better align the experimental protocols.
Regarding cost accounting, we have clarified that multiple Skill Selection runs are executed in parallel, with latency determined by the longest parallel branch rather than the sum of individual execution times. We have also explicitly incorporated initial Retrieval, Selection, and Agent Execution latency into End-to-End Time and revised the corresponding formulas.
Q1: Comparison with a Relevance-Only Selector under Comparable Inference Budgets
We introduced a Utility-Aware Union baseline in Appendix H.2. This variant changes only the Selector objective to select routes based on task suitability and relevance, without explicitly considering execution costs, while keeping the remaining settings consistent with the Cost-Aware method and maintaining comparable inference overhead. The results demonstrate that explicit cost-aware decision-making yields additional efficiency gains beyond task-suitability- and relevance-based selection. Compared with Utility-Aware Union, Cost-Aware Union reduces End-to-End Time by 25%–32% and Full Tokens by 35%–44%, while retaining its cost advantage even on samples where both methods successfully complete the tasks.
Q2: Comparison Between Single Selection and Three-Sample Union
We added Skill coverage analyses for Standard and High-Cost Skills under B = 1, 3, and 5 (Section 5.4), together with downstream ablation experiments comparing single selection with three-sample Union (Appendix H.3). The upstream recall results show that Union recovers some Standard Skills missed by single selection. The downstream execution results show that single selection already provides certain cost benefits but incurs a more noticeable loss in task success rate, whereas three-sample Union further improves task success rate while reducing end-to-end execution costs, achieving a better trade-off between task success and execution cost.
Q3: Cost Advantages on Jointly Successful Samples
We added a paired analysis of jointly successful samples comparing Top-10 with Cost-Aware Union in Appendix G.3. The results show that, when considering only cases where both methods successfully complete the same tasks, End-to-End Time is still reduced by 17.47%–31.79%, and token consumption by 53.39%–64.01% across the three Retrievers. These results demonstrate that the cost advantages persist even after accounting for all selection overhead and excluding the effects of unsuccessful executions.
Detailed experimental settings, results, and explanations for each issue are provided in the corresponding responses below.