acceptodds
Under review as a conference paper at ICLR 2027

Beyond Functional Relevance: Benchmarking Cost-Aware Skill Retrieval for Efficient Agent Execution

Abstract

As the scale of reusable Skills continues to grow, Skill Retrieval has become an important mechanism for LLM Agents to access task-relevant knowledge and operational guidance. Existing retrieval methods primarily rely on functional relevance, focusing on identifying task-supporting Skills. However, functional relevance alone may be insufficient for selecting Skills that support efficient downstream execution: viable Skills for the same task may guide Agents toward different technical approaches, resulting in substantially different downstream execution costs. To investigate this issue, we introduce **Cost-Aware Skill Retrieval** and construct the **Cost-Aware Skill Retrieval Benchmark**. Experiments on this benchmark support this concern, showing that strong recall of task-supporting Skills does not necessarily translate into a consistent preference for more execution-efficient Skills. Motivated by this finding, we develop a lightweight **Cost-Aware Reference Method** that uses a two-stage process to improve cost-aware Skill selection. Experimental results indicate that the method can favor lower-cost Skills and translate this preference into actual downstream savings, reducing Agent execution time by up to 51.55% and context-adjusted token consumption (excluding repeated preloaded-context tokens) by up to 46.37%, while maintaining comparable task performance. These findings provide a useful reference for future research in this direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill Retrieval, Retrieval Benchmark, Agent Execution Efficiency, Cost-Aware Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1h8TOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for cost-aware skill retrieval and a two-stage reference method that constructs candidate routes and selects skills based on anticipated execution costs. The problem is practically relevant, and the experiments demonstrate promising savings. However, benchmark generalizability and the attribution of these gains require stronger evidence.

Critical strengths

The study evaluates actual downstream execution rather than retrieval metrics alone. The paired-success analysis and cardinality-matched Top-M baseline provide useful controls. Reporting full and context-adjusted token consumption alongside selection overhead also improves transparency.

Critical weaknesses
  1. The novelty and independent contribution of cost-aware reasoning need to be more clearly established. The authors should further discuss the connections between the proposed method and existing studies, including:

    • SayCan: Feasibility-aware skill selection;
    • Voyager: Reuse of executable skills;
    • AeroWeaver: Experience-guided skill selection.

    These related studies would help clarify how the proposed cost optimization objective differs from broader notions of skill utility and what additional contributions the proposed method provides.

  2. Stronger experimental evidence is needed to demonstrate the preservation of task performance. In Table 9, the success rates of all three retrievers decrease compared with the Top-10 baseline. For example, the success rate of BM25 drops from 71.43% to 65.71%. Furthermore, conducting only three trials per task is insufficient to establish equivalent task performance across different methods.

  3. The experimental configuration and efficiency claims require further clarification. In the downstream task evaluation, the paper uses the union of three skill selection results, whereas the selection-level experiments are based on individual runs. In addition, the estimated end-to-end latency assumes that skill selection can be performed in parallel, and the calculation formula does not explicitly account for the latency introduced by skill retrieval itself.

Questions for authors
  • Do the gains persist against a relevance-only LLM selector with comparable inference resources?

  • How does a single selection compare with the three-run union in success rate and total cost?

  • Do the final retrieval results retain their cost advantage when comparisons are restricted to tasks successfully completed under both conditions?

6 comments
AuthorsOpenReview ↗

Author Response Summary

We thank the reviewer for the constructive suggestions. Below, we address the three Critical Weaknesses (W) and three Questions for Authors (Q), summarizing the corresponding revisions and key findings.

W1: Related Work and Positioning of Our Contributions

We have expanded the discussion of how our work relates to and differs from existing utility-aware Skill selection approaches. Execution cost is an important dimension of the broader concept of Skill Utility. Our work focuses on how to identify and prioritize lower-cost alternatives when multiple Skill sets can all support task completion.

Our primary contribution is the construction of a dedicated Cost-Aware Skill Retrieval Benchmark, which provides a reproducible evaluation basis for this problem. Meanwhile, through explicit cost-aware decision-making in our Reference Method and downstream execution experiments, we demonstrate the feasibility of this research direction and validate the evaluation value of the benchmark.

W2: Trade-off Between Task Performance and Execution Cost

We have revised the presentation of our results to explicitly characterize them as a Success–Cost Trade-off. Rather than claiming that comparable task performance is maintained, we now report the observed reductions in task success rate alongside the corresponding execution cost savings. We have also added task-level bootstrap confidence intervals for Pass Rate differences, clarified that the current experiments do not establish equivalent task performance, and revised the corresponding conclusions in the manuscript (Appendix G.3).

W3: Experimental Configuration and End-to-End Cost Accounting

The original manuscript did not sufficiently clarify the experimental configuration and cost accounting. First, Multi-Sample Union is an integral component of our Reference Method. Single selection and multi-sample Union are configurations of the same method with different sampling numbers B, designed to mitigate the instability in Skill coverage associated with single selection. We have clarified this mechanism in Section 4.3 and expanded the relevant evaluations in Section 5.4 and Appendix H.3 to better align the experimental protocols.

Regarding cost accounting, we have clarified that multiple Skill Selection runs are executed in parallel, with latency determined by the longest parallel branch rather than the sum of individual execution times. We have also explicitly incorporated initial Retrieval, Selection, and Agent Execution latency into End-to-End Time and revised the corresponding formulas.

Q1: Comparison with a Relevance-Only Selector under Comparable Inference Budgets

We introduced a Utility-Aware Union baseline in Appendix H.2. This variant changes only the Selector objective to select routes based on task suitability and relevance, without explicitly considering execution costs, while keeping the remaining settings consistent with the Cost-Aware method and maintaining comparable inference overhead. The results demonstrate that explicit cost-aware decision-making yields additional efficiency gains beyond task-suitability- and relevance-based selection. Compared with Utility-Aware Union, Cost-Aware Union reduces End-to-End Time by 25%–32% and Full Tokens by 35%–44%, while retaining its cost advantage even on samples where both methods successfully complete the tasks.

Q2: Comparison Between Single Selection and Three-Sample Union

We added Skill coverage analyses for Standard and High-Cost Skills under B = 1, 3, and 5 (Section 5.4), together with downstream ablation experiments comparing single selection with three-sample Union (Appendix H.3). The upstream recall results show that Union recovers some Standard Skills missed by single selection. The downstream execution results show that single selection already provides certain cost benefits but incurs a more noticeable loss in task success rate, whereas three-sample Union further improves task success rate while reducing end-to-end execution costs, achieving a better trade-off between task success and execution cost.

Q3: Cost Advantages on Jointly Successful Samples

We added a paired analysis of jointly successful samples comparing Top-10 with Cost-Aware Union in Appendix G.3. The results show that, when considering only cases where both methods successfully complete the same tasks, End-to-End Time is still reduced by 17.47%–31.79%, and token consumption by 53.39%–64.01% across the three Retrievers. These results demonstrate that the cost advantages persist even after accounting for all selection overhead and excluding the effects of unsuccessful executions.

Detailed experimental settings, results, and explanations for each issue are provided in the corresponding responses below.

AuthorsOpenReview ↗

Response to W1–W3: Related Work, Task Performance, and Experimental Clarifications

We thank the reviewer for the constructive suggestions. We have made corresponding additions and revisions to address the three major concerns, as detailed below.

Summary of Revisions

  1. Related Work: We have added discussions of representative works closely related to our research problem in the revised manuscript, further clarifying the role of execution cost within the broader concept of Skill Utility, as well as the research objectives and contributions of our Cost-Aware Skill Retrieval Benchmark.

  2. Abstract, Section 1, and Discussion: We no longer claim that task performance is maintained and instead explicitly characterize the results as a Success–Cost Trade-off.

  3. We have added a new subsection on Multi-Sample Skill Set Aggregation, clarifying that this mechanism is an integral component of the Reference Method and that different values of B represent different configurations of the same method. We also compare the selection results and overhead under B = 1, 3, and 5.

  4. Appendices G and H: We have added downstream execution experiments for single selection and paired analyses on jointly successful samples, and improved the calculation and reporting of End-to-End Cost.

W1: Related Work and Positioning of the Reference Method

We acknowledge that execution cost is indeed one dimension of the broader concept of Skill Utility. Prior studies have explored related issues from different perspectives, including task feasibility, knowledge expansion, Skill reuse, and execution experience. Building on these studies, our work focuses on a more specific problem: when multiple Skill sets can all support task completion, how can we prioritize those with lower expected downstream execution costs?

Our primary contribution is the construction of a dedicated Cost-Aware Skill Retrieval Benchmark, which provides a reproducible evaluation basis for this problem. The Reference Method serves as an initial exploration, demonstrating the feasibility and practical value of this research direction through explicit cost-aware decision-making and downstream execution experiments, while also illustrating the benchmark's capability to support the study and evaluation of related methods.

W2: Task Performance Degradation and the Number of Repeated Trials

We agree with the reviewer's concern. The original statement "maintaining comparable task performance" was not sufficiently precise, and three repeated trials are insufficient to establish comparable task performance.

In the revised manuscript, we explicitly characterize the results as a Success–Cost Trade-off: compared with Top-10, Cost-Aware Union exhibits an average Pass Rate reduction of 3.17 pp, while reducing End-to-End Time and End-to-End Full Tokens by 14.89% and 51.38%, respectively. We now report the changes in task success rate alongside the execution cost savings and no longer claim that our method maintains comparable task performance.

W3: Experimental Configuration and End-to-End Cost Accounting

Regarding the experimental configuration, the original manuscript did not sufficiently clarify the relationship between single Skill Selection and three-sample Union. In the revised manuscript, Section 4.3 explicitly clarifies that Multi-Sample Skill Set Aggregation is an integral component of the Reference Method, and that single selection and multi-sample Union are configurations of the same Reference Method with different sampling numbers B. When B = 1, Union reduces to the single selection result; when B > 1, it takes the union of multiple independently generated selection results executed in parallel. The overall workflow is illustrated in Figure 4. Section 5.4 now consistently reports the selection results and overhead under different B configurations, while Appendix H.3 further provides downstream execution evaluations of single selection.

Regarding cost accounting, multiple Skill Selection runs are executed in parallel by design, and Selection Latency is determined by the longest parallel branch. We have also explicitly incorporated initial Retrieval Latency, Selection Latency, and Agent Execution Time into End-to-End Time and improved the corresponding cost calculations.

AuthorsOpenReview ↗

Response to Q1–Q2: Additional Experiments and Analyses

To address the reviewer's question, we introduced a Utility-Aware Union baseline in Appendix H.2. This variant uses an LLM to select Skills associated with candidate routes solely based on task suitability, without an explicit objective of optimizing execution costs.

Both methods reuse exactly the same Top-10 Retrieval results and three sets of Planner outputs, with identical LLM call structures, B = 3 Union Aggregation, and downstream execution settings. The only difference lies in the Selector objective, which is changed from minimizing expected execution cost to maximizing task suitability, ensuring comparable inference resource consumption. The actual Selection Overhead is also included in the end-to-end cost. The experimental workflow is illustrated in Figure 5, and the complete prompts are available in our anonymized repository.

Experimental Results.

Compared with Utility-Aware Union, Cost-Aware Union exhibits a clearer preference for lower-cost Skills and achieves better downstream execution efficiency. The following percentages represent cost reductions achieved by Cost-Aware Union relative to Utility-Aware Union.

MetricBM25Qwen
Standard Skill Hits (Utility-Aware → Cost-Aware)51 → 5448 → 53
High-Cost Skill Hits (Utility-Aware → Cost-Aware)46 → 3749 → 33
Pass Rate (Utility-Aware → Cost-Aware)71.43% → 65.71%67.62% → 74.29%
Agent Execution Time Reduction39.25%39.89%
End-to-End Time Reduction32.36%25.04%
End-to-End Full Tokens Reduction43.99%34.87%

1. Skill Selection Preference (Tables 12 and 15). Cost-Aware Union retains more Standard Skills and fewer High-Cost Skills across both retrievers. The Cardinality-Matched Top-M comparison further supports this observation.

2. Downstream Execution Performance (Table 17). Even after accounting for Selection Overhead, Cost-Aware Union substantially reduces end-to-end execution time and token consumption. However, the two methods do not exhibit a consistent advantage in Pass Rate across the two retrievers.

3. Paired Analysis on Successful Cases (Table 18). Among the jointly successful samples (59 for BM25 and 67 for Qwen), Cost-Aware Union still reduces End-to-End Time by 22.58% and 31.30%, respectively, and End-to-End Full Tokens by 34.87% and 49.26%, respectively.

These results demonstrate that, given the same candidate routes and comparable inference resources, explicitly accounting for execution costs provides additional execution efficiency gains beyond selection based solely on task suitability.

Q2: Comparison Between Single Selection and Three-Sample Union

We added a Cost-Aware w/o Union experiment in Appendix H.3, which performs single selection without Union Aggregation. Specifically, this experiment reuses the three independent Cost-Aware Route Skill Selection results generated by the full method at B = 3, before Union Aggregation, and uses each result separately for multiple downstream executions (Figure 6).

Table 19 reports the Pass Rate and full end-to-end costs of w/o Union and B = 3 Union:

RetrieverConfigurationPass Rate ↑E2E Time ↓E2E Full Tokens ↓
BM25Top-1071.43%248.46s833.86K
BM25Top-M51.11%345.51s663.42K
BM25w/o Union59.68%300.28s413.43K
BM25Union (B = 3)65.71%222.06s346.42K
QwenTop-1077.14%269.67s814.16K
QwenTop-M56.83%319.96s864.95K
Qwenw/o Union63.49%281.54s572.46K
QwenUnion (B = 3)74.29%257.72s428.56K

The results demonstrate two key findings:

  • Single selection is already effective. Compared with Top-M using the same number of Skills, w/o Union improves Pass Rate by 8.57 and 6.66 pp on BM25 and Qwen, respectively, while reducing both end-to-end execution time and Full Tokens. However, compared with Top-10, its lower Skill coverage also leads to a more noticeable loss in task success rate.

  • B = 3 improves the success–cost trade-off. Compared with w/o Union, Union improves Pass Rate by 6.03 and 10.80 pp on BM25 and Qwen, respectively, while reducing End-to-End Time by 26.05% and 8.46%, respectively. Full Tokens are also further reduced. Compared with Top-10, B = 3 still exhibits some reduction in task success rate but achieves substantially lower end-to-end execution costs.

Overall, single selection already provides cost-aware benefits but incurs a noticeable loss in task success rate. Multi-Sample Union further improves Skill coverage and task success rate while reducing end-to-end execution costs, achieving a better trade-off between task success and execution cost, albeit with some remaining reduction in task success rate compared with Top-10.

Reviewer_GeecOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cost-aware skill retrieval: selecting skills that support task completion while reducing downstream execution costs. The authors construct a benchmark of 35 tasks with paired standard and high-cost skill sets, alongside distracting skills. To construct such data, they start with existing tasks and construct two reference routes, one with standard cost, one with higher cost. They then construct corresponding skill sets based on these routes. The authors further propose a prompting-based method that first constructs candidate execution routes from retrieved skills and then selects skills with a lower-cost route. Experiments show that this selection can reduce agent execution costs compared with relevance-based retrieval.

Critical strengths
  1. The paper addresses a practical limitation of relevance-based retrieval: functionally suitable skills can lead to substantially different execution costs.
  2. The evaluation includes downstream execution and selection overhead, providing more evidence of practical utility than retrieval metrics alone.
Critical weaknesses
  1. Limited benchmark scale. The benchmark contains only 35 tasks, which is insufficient to establish broad conclusions about cost-aware skill retrieval. Repeated executions do not compensate for limited task diversity.
  2. The reported efficiency gains come with lower task accuracy. Compared with Top-10 retrieval, cost-aware union has lower success rates across all three retrievers, including a decline from 71.43% to 65.71% for BM25. These results do not adequately support the claim that task performance is preserved.
  3. The proposed single-pass cost-aware retrieval is not evaluated end-to-end. Table 4 reports selection overhead and retrieval metrics, while Table 9 evaluates a different variant that unions the selection results of three runs. No table jointly reports selection overhead, downstream task accuracy, and total cost for the proposed method against baselines.
  4. Missing adaptive retrieval baselines. The paper does not compare against agentic search that lets the model write queries, inspect candidates, and determine how many skills to retrieve or retain [1]. Such a comparison, accounting for search overhead, is necessary to determine whether the proposed cost-aware selection offers benefits beyond allowing the agent to search for skills itself.

[1] Liu et al. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings.

Questions for authors

Please see the weakness.

5 comments
AuthorsOpenReview ↗

Author Response Summary

We thank the reviewer for the constructive suggestions. Below, we address the four Critical Weaknesses (W), summarizing the corresponding revisions and key findings.

W1: Benchmark Scale and Generalizability

We acknowledge that the current 35 tasks have limitations in terms of scale and diversity. Nevertheless, for each task, we constructed and validated two technically feasible routes with different execution costs, providing a targeted evaluation basis for Cost-Aware Skill Retrieval. We also conducted experiments with different Agents and retrieval methods, which preliminarily demonstrate the suitability of our benchmark as a dedicated evaluation benchmark. In the revised manuscript, we further clarify the scope of our study, avoid overgeneralization, and explicitly acknowledge the limited task scale and diversity as limitations of the current benchmark.

W2: Trade-off Between Task Success Rate and Execution Cost

We have revised the presentation of our results to explicitly characterize them as a Success–Cost Trade-off. Rather than claiming that comparable task performance is maintained, we now report the observed reductions in task success rate alongside the corresponding execution cost savings. We have also added task-level bootstrap confidence intervals for Pass Rate differences, clarified that the current experiments do not establish comparable task performance across methods, and revised the corresponding conclusions in the manuscript (Appendix G.3).

W3: Consistency Between Method Definition and End-to-End Evaluation

The original manuscript did not sufficiently clarify our method. Multi-sample selection followed by Union Aggregation is an integral component of our method, designed to mitigate insufficient Skill coverage resulting from single selection. Single selection and multi-sample Union are configurations of the same Reference Method with different sampling numbers B. We have clarified this mechanism in §4.3 and added recall analyses with 1, 3, and 5 selections (§5.4), as well as downstream ablation experiments comparing single selection with three-sample Union (Appendix H.3). The results show that single selection already provides certain cost benefits but suffers from insufficient Skill coverage and a more noticeable reduction in task success rate. Three-sample Union improves both Skill coverage and task success rate while further reducing end-to-end execution costs.

W4: Agentic Search and Adaptive Skill Selection Baselines

Regarding the comparison with Agentic Search, we interpret the reviewer's suggestion as potentially involving two distinct aspects: (1) whether Cost-Aware Selection can provide additional benefits when applied after Agentic Search as the initial retrieval baseline; and (2) whether an Agent that autonomously searches for and selects Skills based on task requirements can achieve results comparable to our method. We conducted additional experiments to investigate both aspects.

1. Initial Retrieval: Comparison with the Agentic Search Baseline

We reproduced Agentic Search (Hybrid) and applied our Cost-Aware Reference Method to its retrieval results. The results show that Agentic Search achieves recall performance comparable to existing retrieval methods, without exhibiting a clear preference for retrieving lower-cost Skills. Furthermore, Cost-Aware Selection still demonstrates a similar preference for lower-cost Skills when applied to its retrieval results.

Due to the time constraints of this revision and the substantial time required for downstream Agent execution experiments, we have not completed a full end-to-end evaluation of this comparison or formally included this baseline in the revised manuscript. Nevertheless, our additional experiments show that our method can still effectively distinguish Skills with different execution costs when applied to Agentic Search retrieval results, providing useful preliminary evidence for assessing its effectiveness across different Initial Retrieval methods.

2. Skill Selection: Comparison with Adaptive Selection Based Solely on Task Suitability

We introduced a Utility-Aware Union baseline, which selects Skills solely based on task suitability without considering execution costs, while keeping the remaining experimental settings unchanged. The results show that, compared with Utility-Aware Union, Cost-Aware Union retains more Standard Skills and fewer High-Cost Skills, while reducing downstream execution time and token consumption. These results show that explicit cost awareness provides additional efficiency gains over task-based Utility-Aware selection (Appendix H.2).

Detailed experimental settings, results, and explanations for each issue are provided in the corresponding responses below.

AuthorsOpenReview ↗

Response to W2 and W3: Method Clarification and Success–Cost Trade-off

We thank the reviewer for pointing out issues in the ]method description and presentation of experimental results. We have made the following revisions:

  1. Abstract, Section 1, and Discussion: Revised the claims regarding maintained task performance and explicitly characterized the results as a Success–Cost Trade-off.

  2. Section 4.3: Added a new subsection on Multi-Sample Skill Set Aggregation, clarifying that this mechanism is an integral component of the Cost-Aware Reference Method and that single selection and multi-sample Union are configurations of the same method with different sampling numbers B.

  3. Section 5.4 and Appendix H.3: Extended the selection experiments under B = 1, 3, and 5, and added downstream execution and end-to-end comparisons between single selection and three-sample Union, reporting the corresponding Pass Rates and execution costs.

W2: Efficiency Gains and Degradation in Task Success Rate

We now explicitly characterize the experimental results as a Success–Cost Trade-off: compared with Top-10, Cost-Aware Union exhibits an average Pass Rate reduction of 3.17 pp, while reducing End-to-End Time and End-to-End Full Tokens by 14.89% and 51.38%, respectively. The revised manuscript reports the changes in task success rate alongside the execution cost savings and no longer claims that our method maintains comparable task performance.

W3: Consistency Between Method Configuration and Experimental Evaluation

We thank the reviewer for pointing out that the original manuscript did not sufficiently clarify the correspondence between the method description and experimental configuration. The original manuscript reported results from single Skill Selection but used the Union of three selection results for downstream evaluation, without adequately explaining the relationship between them.

1. Unified Method Definition (Section 4.3, Figure 4)

Multi-Sample Skill Set Aggregation is an integral component of the Reference Method. Single selection and multi-sample Union are configurations of the same Reference Method with different sampling numbers B. When B = 1, Union reduces to the single selection result; when B > 1, it takes the union of multiple independently generated selection results executed in parallel. The overall workflow is illustrated in Figure 4 of the revised manuscript.

2. Additional Evaluations Across Different Configurations (Section 5.4, Appendix H.3)

In the revised manuscript, Section 5.4 systematically compares Skill Recall, selection stability, and Selection Overhead under B = 1, 3, and 5. Appendix H.3 further evaluates Cost-Aware w/o Union (i.e., single selection without Union). Specifically, this experiment reuses the three independent Cost-Aware Route Skill Selection results generated by the full method at B = 3, before Union Aggregation, and uses each result separately for multiple downstream executions (Figure 6).

Table 19 reports the Pass Rate and end-to-end costs of Top-10, Cardinality-Matched Top-M, Cost-Aware w/o Union, and B = 3 Union:

RetrieverConfigurationPass Rate ↑E2E Time ↓E2E Full Tokens ↓
BM25Top-1071.43%248.46s833.86K
BM25Top-M51.11%345.51s663.42K
BM25w/o Union59.68%300.28s413.43K
BM25Union (B = 3)65.71%222.06s346.42K
QwenTop-1077.14%269.67s814.16K
QwenTop-M56.83%319.96s864.95K
Qwenw/o Union63.49%281.54s572.46K
QwenUnion (B = 3)74.29%257.72s428.56K

The experiments demonstrate two key findings:

  • Single selection is already effective. Compared with Top-M using the same number of Skills, w/o Union improves Pass Rate by 8.57 and 6.66 pp on BM25 and Qwen, respectively, while reducing both end-to-end execution time and Full Tokens. However, compared with Top-10, its insufficient Skill coverage also leads to a more noticeable reduction in task success rate.

  • B = 3 improves the success–cost trade-off. Compared with w/o Union, Union improves Pass Rate by 6.03 and 10.80 pp on BM25 and Qwen, respectively, while reducing End-to-End Time by 26.05% and 8.46%, respectively. Full Tokens are also further reduced. Compared with Top-10, B = 3 still exhibits some reduction in task success rate but achieves lower end-to-end execution costs.

Overall, single selection already provides cost-aware benefits but incurs a noticeable loss in task success rate. Multi-Sample Union further improves Skill coverage and task success rate while reducing end-to-end execution costs, achieving a better trade-off between task success and execution cost, albeit with some remaining reduction in task success rate compared with Top-10.

AuthorsOpenReview ↗

Response to W4:Adaptive Retrieval and Skill Selection Baselines

We thank the reviewer for the suggestion regarding the Agentic Search baseline. To clarify the comparison between different methods, we first outline our three-stage experimental framework: Initial Retrieval → Cost-Aware Reference Method → Downstream Agent Execution.

In this framework, Initial Retrieval retrieves a fixed number of Top-K candidate Skills from the full Skill library. The Cost-Aware Reference Method then takes the Top-10 retrieved Skills as input, constructs candidate routes, evaluates their expected execution costs, and selects a Skill set that facilitates efficient downstream Agent execution.

Agentic Search, as suggested by the reviewer, belongs to the Initial Retrieval stage and is at the same comparison level as other retrieval methods. It can serve as a retrieval baseline to examine whether our method provides additional benefits on its retrieval results, but it cannot be directly compared with our method at the same level.

However, the reviewer also suggests comparing with methods that "allow the Agent to autonomously decide how many Skills to retrieve or retain." We interpret this as potentially asking whether an Agent that autonomously selects Skills based on task requirements, without explicitly considering execution costs, can already achieve results comparable to our Cost-Aware Reference Method.

Based on this understanding, we conduct additional experiments at two levels:

1. Initial Retrieval: How does Agentic Search compare with existing retrieval methods in terms of Standard Skill and High-Cost Skill recall? Can the Cost-Aware Reference Method provide additional benefits on top of its retrieval results?

2. Skill Selection: If an Agent autonomously selects which Skills to retain based solely on task requirements, without explicitly considering downstream execution costs, can it achieve execution efficiency comparable to the Cost-Aware Reference Method?

We conduct separate comparative experiments to investigate these two questions, as described below:

Reviewer_GSTgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies skill retrieval when functionally valid skills lead to different execution costs. For 35 tasks, the authors take the original reference route and design a higher-cost alternative, then derive paired skill sets and add background distractors. Conventional retrievers recall both types without consistently favoring cheaper execution. The proposed method uses an LLM to plan candidate routes and select a lower-cost route. Downstream evaluation uses the union of three independent selections and reports execution savings, alongside somewhat lower success rates than Top-10 retrieval.

Critical strengths

The cost difference is validated through actual agent execution with two models. For GPT-5.6-Luna, Table 2 reports average execution times of 130 versus 342 seconds under standard and high-cost skills. Appendix E further checks initial context sizes and restricts the comparison to runs where both conditions succeed; the time gap remains substantial for both models. This is stronger evidence than assigning cost labels from reference solutions alone.

Critical weaknesses

The comparison does not isolate the value of cost awareness.** The method adds task reasoning and route selection as well as a cost objective. A planner that simply chooses a coherent, suitable route might obtain much of the same benefit. Top-M controls skill count, but is a weak substitute for such a baseline: it is slower than Top-10 for all three retrievers (329 vs. 248 seconds for BM25, 312 vs. 263 for Qwen, and 373 vs. 328 for SkillRouter). BM25's success rate also falls to 60%. This suggests that truncating the ranking may remove skills the task needs. Beating Top-M therefore does not establish that cost reasoning is the source of the gains. I would compare against the same route planner selecting for task completion without a cost instruction.

The benchmark difficulty is unclear, and the performance claim is too strong.** High-cost routes are designed by the authors and retained only if they are at least 50% slower. Appendix B explicitly excludes artificial delays and deliberately redundant computation. This does not establish whether the alternatives are plausible engineering choices. Without concrete route pairs in the paper, it is hard to judge whether this benchmark requires cost reasoning or mostly rewards choosing the more sensible approach.

The practical gains should also be presented as a possible success-cost trade-off. In Table 9, success rates decrease relative to Top-10 for all three retrievers, most notably from 71.43% to 65.71% for BM25. After selection overhead, the estimated end-to-end time savings over Top-10 are 10.6%, 4.5%, and 30.3% for BM25, Qwen, and SkillRouter, respectively. "Comparable performance" needs uncertainty estimates, particularly for BM25. Section 5.4 also misidentifies the comparison behind the headline savings: the 51.55% time reduction and 46.37% token reduction in Table 9 are relative to Top-M, not Top-10.

Questions for authors
  1. How does the method compare with the same route planner followed by selection based on task completion alone, without a cost instruction?

  2. What are the execution results for a single selection rather than the union of three? With GPT-5.6-Luna, Standard Final Recall is about 0.55–0.59, compared with 0.88–0.92 for the initial Top-10 retrieval. Does the union restore coverage that a single selection loses?

  3. Could you show a few representative standard/high-cost route pairs and explain when the high-cost approach would be a reasonable engineering choice? Examples where the cheaper route is not obvious would help establish the difficulty of the benchmark.

  4. Can you provide task-level uncertainty estimates for the success-rate differences in Table 9, and check whether the savings persist when both methods succeed? The successful-pair analysis in Appendix E concerns benchmark validation, not these method comparisons.

5 comments
AuthorsOpenReview ↗

Author Response Summary

We thank the reviewer for the constructive suggestions. Based on the relationships among the raised issues, we organize the three major concerns (W, Critical Weaknesses) and four specific questions (Q, Questions for Authors) into the following four groups, summarizing the corresponding revisions and key findings.

W1 & Q1: The Independent Contribution of the Cost-Aware Mechanism

We added a Utility-Aware Union baseline experiment in Appendix H.2. Under otherwise identical conditions, this variant selects routes solely based on task suitability, without considering execution costs. Compared with this variant, our method still reduces end-to-end execution time by 25%–32% and end-to-end token consumption by 34%–43%. Furthermore, our method retains its cost advantage on samples where both methods succeed. These results demonstrate that the efficiency gains do not arise solely from reasonable route planning and task-suitability selection; explicit cost awareness provides additional benefits in execution efficiency.

Q2: Single Selection vs. Multi-Sample Union

Using the union of three selections can indeed recover some Skill coverage lost through single selection, which is also part of our original design. We have clarified this mechanism in §4.3 and added recall analyses with 1, 3, and 5 selections (§5.4), as well as downstream ablation experiments comparing single selection with three-sample Union (Appendix H.3).

The results show that, at the upstream Skill selection level, Union recovers some Standard Skills missed by single selection, thereby improving Skill coverage. At the downstream execution level, single selection already provides certain cost benefits, but incurs a noticeable loss in task success rate. Introducing Union improves the task success rate while further reducing end-to-end execution costs, resulting in a better trade-off between success rate and cost.

W2 & Q3: Benchmark Validity and Difficulty

We added five groups of case studies in Appendix B.3 to illustrate the engineering rationale and applicable scenarios for the corresponding Standard and High-Cost routes, demonstrating that High-Cost routes also have valid engineering justifications.

Meanwhile, we introduced an anonymized SKILL-group cost-discrimination experiment covering all 35 tasks. The three models achieved classification accuracies ranging from 46% to 65%, indicating that identifying low-cost SKILLs is not straightforward and providing further evidence of the benchmark's discriminative difficulty.

W3 & Q4: The Trade-off Between Success Rate and Cost

We have revised the presentation of our results to explicitly characterize them as a Success–Cost Trade-off, rather than claiming that comparable task performance is maintained. We also added uncertainty estimates for the differences in success rates and paired comparisons on samples where both methods succeed. The results confirm that the cost advantages persist under this comparison (Appendices G.3).

In addition, we corrected the erroneous comparison baseline in the original manuscript and standardized the relevant data and descriptions throughout the paper.

Detailed experimental settings, results, and explanations for each issue are provided in the corresponding responses below.

AuthorsOpenReview ↗

Response to W1 and Q1: Independent Contribution of Cost-Aware Route Selection

We thank the reviewer for the constructive suggestions regarding our method design and experimental comparisons. To isolate the contribution of explicit cost awareness from that of general task-suitability-based route selection, we added a Utility-Aware Union baseline experiment in Appendix H.2.

Experimental Setup. The Utility-Aware Selector selects routes solely based on task suitability, without considering explicit execution costs. Both methods use identical Top-10 Retrieval results and the same three sets of Planner outputs. The selection objective is the only difference, while the subsequent Union Aggregation and downstream execution settings remain identical. The complete prompts are available in our anonymized repository, and the experimental workflow is illustrated in Figure 5.

Experimental Results.

Compared with Utility-Aware Union, Cost-Aware Union exhibits a clearer preference for lower-cost Skills and achieves better downstream execution efficiency. Unless otherwise specified, the following results report the percentage reductions achieved by Cost-Aware Union relative to Utility-Aware Union.

MetricBM25Qwen
Standard Skill Hits (Utility-Aware → Cost-Aware)51 → 5448 → 53
High-Cost Skill Hits (Utility-Aware → Cost-Aware)46 → 3749 → 33
Pass Rate (Utility-Aware → Cost-Aware)71.43% → 65.71%67.62% → 74.29%
Agent Execution Time Reduction39.25%39.89%
End-to-End Time Reduction32.36%25.04%
End-to-End Full Tokens Reduction43.99%34.87%

1. Skill Selection Preference (Tables 12 and 15). Cost-Aware Union retains more Standard Skills and fewer High-Cost Skills on both retrievers. The Cardinality-Matched Top-M comparison further supports this observation.

2. Downstream Execution Performance (Table 17). Even after accounting for Selection Overhead, Cost-Aware Union substantially reduces end-to-end execution time and token consumption. However, the two methods do not exhibit a consistent advantage in Pass Rate across the two retrievers.

3. Paired Analysis on Successful Cases (Table 18). Among the instances where both methods succeed (59 for BM25 and 67 for Qwen), Cost-Aware Union still reduces End-to-End Time by 22.58% and 31.30%, respectively, and End-to-End Full Tokens by 34.87% and 49.26%, respectively.

Overall, these results demonstrate that, given the same candidate routes, explicitly accounting for execution costs leads to a more favorable selection preference toward Standard Skills and provides additional execution efficiency gains. These improvements cannot be attributed solely to route planning or general task-suitability-based selection.

AuthorsOpenReview ↗

Response to Q2: Single Selection vs. Multi-Sample Union

We thank the reviewer for pointing out the potential loss of Skill coverage associated with single selection and for questioning the role of Multi-Sample Union.

Multi-Sample Skill Set Aggregation is an integral component of our Reference Method, rather than a separate method. Single selection and multi-sample Union are two configurations of the same method, differing only in the number of selection samples B. When B = 1, Union reduces to the single selection result; when B > 1, it takes the union of the Skill sets obtained from multiple independent selections.

To clarify this mechanism and further evaluate its effectiveness, we made the following revisions:

  1. Section 4.3: Added a new subsection on Multi-Sample Skill Set Aggregation, clarifying its role in the Reference Method and the relationship between single selection and multi-sample Union.
  2. Section 5.4: Extended the Skill-selection evaluation to systematically compare B = 1, 3, and 5 in terms of Skill coverage, selection stability, and additional overhead.
  3. Appendix H.3: Added downstream ablation experiments comparing single selection with three-sample Union, together with the corresponding execution results (Figure 6).

We further examine the effectiveness of this mechanism from two perspectives.

1. Skill Selection Results under Different B Configurations (Section 5.4, Table 4).

We systematically evaluated B = 1, 3, and 5. The Standard/High-Cost Final Recall results obtained with GPT-5.6-Luna are as follows:

RetrieverB = 1 (Standard / High-Cost)B = 3 (Standard / High-Cost)B = 5 (Standard / High-Cost)
BM250.556 / 0.3100.672 / 0.4460.704 / 0.468
Qwen0.551 / 0.3090.671 / 0.4230.670 / 0.502
SkillRouter0.590 / 0.2380.680 / 0.4830.723 / 0.508

The results show that Multi-Sample Union improves Standard Skill coverage over single selection by recovering some Skills missed in individual selections. Across all retrievers and B configurations, Standard Recall remains higher than High-Cost Recall. Moreover, compared with the corresponding Cardinality-Matched Top-M baselines, all configurations retain more Standard Skills and fewer High-Cost Skills (Section 5.4, Table 4).

2. Downstream Execution Results of Single Selection (Appendix H.3, Table 19).

We introduced a Cost-Aware w/o Union ablation variant, which performs single selection without Union Aggregation. Specifically, we reuse the three independent Cost-Aware Route Selection outputs generated by the full method at B = 3, before Union Aggregation, and evaluate each output separately in downstream execution (Figure 6).

The results are as follows:

RetrieverConfigurationPass Rate ↑E2E Time ↓E2E Full Tokens ↓
BM25Top-M51.11%345.51s663.42K
BM25w/o Union59.68%300.28s413.43K
BM25Union (B = 3)65.71%222.06s346.42K
QwenTop-M56.83%319.96s864.95K
Qwenw/o Union63.49%281.54s572.46K
QwenUnion (B = 3)74.29%257.72s428.56K

The results demonstrate two key findings:

  • Single selection is already effective. Compared with the cardinality-matched Top-M baseline, w/o Union improves Pass Rate by 8.57 and 6.66 percentage points on BM25 and Qwen, respectively, while reducing both end-to-end execution time and token consumption. However, compared with Top-10, its lower Skill coverage also leads to a more noticeable loss in task success rate.

  • B = 3 improves the success–cost trade-off. Compared with w/o Union, Union improves Pass Rate by 6.03 and 10.80 percentage points on BM25 and Qwen, respectively, while reducing End-to-End Time by 26.05% and 8.46%, respectively. End-to-End Full Tokens are also further reduced. Although B = 3 still exhibits some success-rate reduction compared with Top-10, it achieves substantially lower end-to-end execution costs.

In summary, single selection already provides cost-aware benefits but incurs a noticeable loss in task success rate. Multi-Sample Union further improves Skill coverage and task success rate while reducing downstream execution costs in our experiments, achieving a more favorable trade-off between task success and execution cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.