acceptodds
Under review as a conference paper at ICLR 2027

What Does an Agent Need? Bundle-Aware Agent Resource Composition

11 views

Abstract

As catalogs of models, tools, and reusable skills expand, determining which resources an agent should use becomes increasingly challenging. Tool-selection knowledge embedded in model parameters can lag behind catalog changes, while semantic top- retrieval reduces context length but may retain noise and exclude complementary resources needed to satisfy implicit task requirements. Beyond retrieving relevant resources or reusing predefined agents, how to compose heterogeneous resources for a new task remains underexplored. We formulate Task-Conditioned Agent Resource Composition (AgentComposition): jointly selecting a backbone model and composing a tool–skill bundle to meet task requirements. We propose Retrieval-Grounded Bundle-Aware Agent Generation, which retrieves resources and historical configurations as compositional evidence, recombines resources through constrained beam search with a fine-tuned generator, and reranks complete configurations using generator likelihood and a bundle-aware critic. Evaluated against reference configurations, our method achieves rank-discounted component recall and precision at 10 (RDCR@10/RDCP@10) of on AgentSelect and on SkillsBench. The results also reveal that leading LLM baselines can select relevant resources but struggle to recover complete configurations. Controlled transfer experiments show that AgentSelect-trained initialization improves adaptation to SkillsBench using only four target-domain queries, while end-to-end evaluation shows that predicted configuration rankings broadly align with execution quality. These results support resource composition as a learnable and transferable capability for task-specific agent construction. Our work shifts the focus from individual resource relevance to joint task suitability. AgentComposition thus offers a path toward more precise and personalized agents that adapt to changing resource ecosystems and continually evolve through accumulated task experience.

Primary area:
generative models
Keywords:
Agent Resource Discovery, Agent Composition, Bundle Recommendation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G2FXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Summary: The paper tackles how to automatically select the best combination of resources like LLMs, tools, and skills for an AI agent to successfully complete a given task. The authors define this problem as Task-Conditioned Agent Resource Composition. Instead of just finding individually useful tools, the goal is to create a complete bundle of resources that work together for a specific need.

Key COntributions: The paper's main contribution is a framework called Retrieval-Grounded Bundle-Aware Agent Generation. This system works in three main steps: 1) Retrieve: It first searches a large catalog to find relevant resources (models, tools) and historical examples of successful resource combinations for similar tasks. 2) Recombine: Using the retrieved items as a guide, a fine-tuned AI generator intelligently creates new, promising combinations (or "bundles") of resources and 3) Rerank: A special "bundle-aware critic" then evaluates these new bundles, scoring how well the components are predicted to work together. It reranks them to find the optimal configuration.

Evals: The aUthors validated their method on two benchmark datasets, AgentSelect and SkillsBench, comparing it against other baselines, including GPT-4

findings:

  • THe optimal selction of the bundle outperforms baselines
  • Showed that LLMs struggle to find good optimal composition of tools
  • The framework can adapt to unseen resource domains.
Critical strengths
  1. High-Impact Problem Formulation: The paper introduces and formally defines AgentComposition (Task-Conditioned Agent Resource Composition). It uniquely shifts the paradigm from simple, individual tool retrieval to the holistic composition of synergistic "bundles" consisting of models, tools, and skills.
  2. Principled Three-Stage Architecture: The proposed Retrieve-Generate-Rerank framework is highly structured. By leveraging in-context historical compositions, constrained beam search, and a dedicated "bundle-aware" critic, it provides a robust and principled approach to generating compatible resource configurations.
  3. Strong Benchmark Performance: The methodology delivers good results on two major benchmarks, AgentSelect and SkillsBench, shows composition-aware modeling is superior to standard LLM selection.
  4. Empirical Execution Alignment: Rather than relying solely on static metric overlaps, the paper includes end-to-end execution validation. It successfully demonstrates a strong positive correlation between the critic's offline rankings and actual runtime success when the composed agents are deployed.
  5. Cross-Domain Generalization: The framework demonstrates transferability and adaptability. When evaluated on unseen resource domains, the model adapts and generalizes much more effectively than standard LLMs.
Critical weaknesses
  1. Post-Hoc Critic Disconnect: The bundle-aware critic is used only as a final reranker because incorporating its feedback directly into the beam search generation process actually degrades performance. This exposes a lack of architectural integration between the generator and the critic, which is a major methodological weakness.
  2. Low-Scale eval: The framework’s actual task-execution validation is limited to a very small sample size of 30 queries, resulting in high statistical variance.
  3. Incremental Architectural Novelty: The pipeline relies on a highly standard "Retrieve-Generate-Rerank" workflow common in recommendation systems. There is no formal mathematical formulation of the unique combinatorial challenges in agent resource composition, this seems more of a systems engineering paper rather than a foundational machine learning contribution.
  4. Training Cap from Teacher Dependency: The critic is trained entirely on preferences synthesized by a teacher LLM rather than actual runtime execution outcomes. This creates a performance ceiling where the system is bound to replicate the teacher's inherent biases and errors, rather than learning to discover novel, optimal tool combinations from real-world trial and error.
Questions for authors
  1. How does the proposed framework scale and perform when the underlying resource catalog is highly dynamic—such as when APIs, models, or skills are frequently updated, deprecated, or added in real-time?
  2. Your ablation studies (Section 5.1) revealed that integrating the Bundle-Aware Critic directly into the beam search generation process (Beam+Critic) actually degraded performance compared to using it as a post-hoc reranker. What is your hypothesis for why real-time guidance failed, and what architectural adjustments might allow the critic to constructively prune search paths during generation?
  3. Currently, the critic is trained using preference feedback synthesized by a "teacher" LLM. Have you explored training the critic directly on actual runtime execution success or error codes (RL from environment feedback) to bypass the performance ceiling imposed by a teacher model?
10 comments
AuthorsOpenReview ↗

rebuttal_G2FX_part1of7(Opening)

Thank you for your careful review and for

  • Recognizing the high-impact problem formulation of task-conditioned agent resource composition, which shifts the focus from individual tool retrieval to composing complete bundles of models, tools, and skills.
  • Finding the retrieve–generate–rerank framework principled and well structured.
  • Acknowledging the strong performance on AgentSelect and SkillsBench, the alignment with end-to-end execution, and the cross-domain generalization of our method.

We sincerely thank you for this insightful review. Your questions went beyond what the paper reported and asked how the method itself could improve: how the critic could guide search, whether it could learn from execution feedback, and how it would cope with a changing resource catalog. These questions shaped much of our revision. The new experiments directly address most of your concerns: the scale of the execution evaluation (W2), learning from execution outcomes (W4, Q3), and dynamic catalogs (Q1).

Two points are more a matter of perspective than of evidence: why the critic reranks complete configurations instead of guiding search (W1, Q2), and how to view our contribution beyond a standard pipeline (W3). We explain our reasoning in\color{#2E7D32}{\text{We explain our reasoning in}} detail and warmly welcome further\color{#2E7D32}{\text{detail and warmly welcome further}} discussion on both.\color{#2E7D32}{\text{discussion on both.}} All revisions, and how they are marked, are summarized in the General Response.

AuthorsOpenReview ↗

rebuttal_G2FX_part2of7(W1,Q2)

W1: Post-Hoc Critic Disconnect: The bundle-aware critic is used only as a final reranker because incorporating its feedback directly into the beam search generation process actually degrades performance. This exposes a lack of architectural integration between the generator and the critic, which is a major methodological weakness.

Q2: Your ablation studies (Section 5.1) revealed that integrating the Bundle-Aware Critic directly into the beam search generation process (Beam+Critic) actually degraded performance compared to using it as a post-hoc reranker. What is your hypothesis for why real-time guidance failed, and what architectural adjustments might allow the critic to constructively prune search paths during generation?

Summarized W1 & Q2: ① The critic only reranks, and guiding search with it lowers performance. ② Why does this guidance fail? (Reviewer BU35 asked the same in W2.) ③ What changes would let the critic prune search usefully?

A1 for W1 and Q2 (Clarification, Explanation, Additional Experiments, Revised Paper): We thank the reviewer for this comment and question on the role of the critic. We apologize that the original submission did not clearly explain the design purpose of the critic. Below, we first clarify why we designed the critic (①), then explain why guiding search with it lowers performance (②), and finally report how it performs inside search (③). The changes are highlighted in magenta\color{#B0407A}{\text{magenta}} (Sections 3.3 and 5.1, Figure 2).

① Design motivation. The critic is a proxy for real execution. Executing candidate configurations at scale is costly and often not possible, especially for a cold-start system. The critic therefore estimates how well a complete configuration will execute without running it. It learns from an LLM teacher (Stage 1) and from counterfactual perturbations judged against references (Stage 2). When execution feedback is available, the new Stage 3 adapts it to that feedback.

See screenshot: Revised Section 3.3

② Why guidance during search fails. The critic is trained to score complete configurations. A partial configuration can score low even when it would grow into a strong one, so applying the critic during search prunes promising paths too early.

③ The critic guided search. Retrieval is the main bottleneck: of the 162 queries whose reference configuration is missing from the candidate pool, 147 miss a backbone or tool that retrieval never returned. Better search can fix at most 15 queries (2.4%), and better ranking up to 41 (6.5%). The critic therefore pays off in reranking, not in guiding search. Larger gains need wider retrieval coverage, or using the critic as a training reward for the generator (see A4). Details follow.

Evidence that guidance adds little. We further train a prefix value model (V) and a critic (Q) on partial configurations. V guides search, and Q reranks the final list.

Setting (200 held-out queries)Top-1 execution score (0–10)
No critic (generator only)7.58
Final reranking, Stage 1+2 critic8.37
Final reranking, + Stage 38.78
Stage 3 prefix value model guiding search and critic reranking8.57
  • Reranking helps, and direct guidance hurts. As a final reranker, the critic lifts the score from 7.58 to 8.78. Guiding search lowers it to 8.57.
  • Prefix-value guidance adds nothing. Against reranking by the same Q, the Top-1 score is unchanged on 198 of 200 queries: guidance swaps a beam slot on 34 queries but changes only 4 Top-1 configurations (2 tied, 2 lower). RDCR@10 is 59.36% vs. 59.43%.

Why search has little room. For each of the 629 AgentSelect test queries, we trace where the reference configuration ends up.

Where the reference configuration isQueries
Already in the Top-10426
In the candidate pool, but ranked outside the Top-1041
Not in the pool: reference backbone missing100
Not in the pool: reference tool combination missing50
Not in the pool: both missing12
  • Better search can recover at most 15 queries (2.4%). It can only fix the 50 tool misses, and in 35 of them retrieval never returned the needed tools.
  • Better ranking can recover up to 41 queries (6.5%). This is where the critic acts.

See screenshot: Revised Figure 2(a)-(b)

AuthorsOpenReview ↗

rebuttal_G2FX_part3of7(W2)

W2: Low-Scale eval: The framework's actual task-execution validation is limited to a very small sample size of 30 queries, resulting in high statistical variance.

Summarized W2: Concern that the execution validation uses only 30 queries, so its results have high variance.

A2 (New Datasets, Additional Experiments, Revised Paper): We thank the reviewer for raising this point, which Reviewer mdAD also raised in W6. We agree that 30 queries are too few for the task-execution validation.

→ In the revision, we evaluate on more queries and add datasets with execution outcomes:

ModeDatasetExecuted byEvaluated byQueriesAgentsExecutionsEvaluationsUsed for (where)
OfflineAgentSelect-Exec (new)UsUs (GPT-5.6-Terra judge)50610 per query5,0605,060Training the critic on execution feedback (Stage 3), structural ablation, and alignment of rankings with execution (Section 5.1, Figure 2, Appendix F)
OfflineBFCL (new)Official releaseOfficial execution check24018 per query4,3204,320Transfer to a new domain with native rewards, with and without target training (Section 5.2, Table 4, Appendix G)
OfflineSWE-Skills-Bench (new)Official releaseOfficial hidden tests9808 per query7,8407,840Same as BFCL, on software-engineering tasks (Section 5.2, Table 4, Appendix G)
OfflineSkillsBench (new)Official releaseOfficial hidden tests2193 in total255255Ranking strong and closely matched agents (Appendix M.4, Figure 6)
OnlineSkillsBenchUsUs (GPT-5.6-Terra judge) and official native verifier5Top 1 agent × 2 (ours + 1 baseline)40 (1 + 3 runs each)80Executing the recommended agents of ours and a baseline under native verifiers (Section 5.3, Appendix M.3)
OnlineAgentSelectUsUs (GPT-5.6-Terra judge)25Top 10 agents (ours) + Top 1 agent × 2 baselines300300Executing the recommended agents of ours and two baselines, and our full top-10 list (Section 5.3, Appendix M.5)

The changes are highlighted in magenta\color{#B0407A}{\text{magenta}}.

See screenshot: Revised Section 4.1 and Table 1

See screenshot: Revised Appendix D.1, execution datasets

See screenshot: Revised Appendix F

See screenshot: Revised Section 5.3, Figure 4

Reviewer_mdADOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates task-conditioned agent resource composition as jointly choosing a backbone model and a bundle of tools and skills, then proposes retrieval-grounded generation with constrained beam search and a bundle-aware critic. Its main recommendation experiments report RDCR at 10 of 60.00 percent on AgentSelect and 68.11 percent on SkillsBench, alongside analyses of search, reranking, and adaptation from four target-domain queries. A separate 30-query execution study finds positive agreement between predicted ranking and judged execution quality, although most APIs are simulated and the five-query SkillsBench subset provides weak evidence about fine-grained ordering.

Critical strengths
  • The paper proposed a useful joint selection problem. The formulation treats model choice and tool or skill choice as one configuration rather than independent retrieval decisions, making complementary and redundant resources explicit. The distinction from retrieving a predefined agent is clear and practically relevant.

  • The system retrieves individual resources together with historical complete configurations, giving the generator evidence about both resource relevance and previously useful combinations. This is a practical way to expose bundle context rather than relying entirely on isolated tool descriptions.

  • Good performance. On AgentSelect, Table 2 reports RDCR@10 of 60.00 versus 56.54 for the best reported retrieval-augmented comparator, Kimi K2.6, a gain of 3.46 percentage points. On SkillsBench, the method reports 68.11 versus 59.97 for Qwen3.8-27B, a gain of 8.14 points.

Critical weaknesses
  • Section 3.4 describes starting from the top retrieved model roots, whereas Algorithm 1 initializes roots using Equation 4, a generator score. Please specify the actual root-selection rule and how the adaptive beam bounds map to the fixed beam width used in the algorithm.

  • Algorithm 1, step 6 introduces an optional “no-tool proposal penalty” without specifying its definition, magnitude, or use in the reported experiments. Please clarify which actions it penalizes and report its setting, since an adjustment during proposal selection could affect the generated candidate pool.

  • A related work is missed in the paper. "Toolken+ Improving LLM Tool Usage with Reranking and a Reject Option" (https://arxiv.org/abs/2410.12004)

  • More baselines are expected to compare or explain why they are not suitable for being baselines. For examples, "ToolGen Unified Tool Retrieval and Calling via Generation", "AnyTool Self-Reflective Hierarchical Agents for Large-Scale API Calls", and "RouteLLM Learning to Route LLMs with Preference Data".

  • Reference recovery is an incomplete measure of agent utility. The main objective rewards overlap with recorded successful configurations rather than task success or cost.

  • Small eval sets. The main SkillsBench test has only 19 queries across 16 tasks, and AgentSelect uses a 200-query subsample of its 629-query test partition.

Questions for authors

NA

7 comments
AuthorsOpenReview ↗

rebuttal_mdAD_part1of4(W1,W2)

Thank you for your careful review and for

  • Recognizing the value of the proposed joint resource composition problem and its practical relevance, compared with the previous problem of retrieving predefined agents.
  • Recognizing that retrieving historical complete agent configurations, rather than isolated tools, is a practical way to guide resource composition.
  • Acknowledging the substantial performance improvements achieved on both AgentSelect and SkillsBench.

Your comments also helped us see where we could improve the clarity (W1, W2), the related work and baselines (W3, W4), and the experimental design and scale (W5, W6). We appreciate you pointing out these limitations, and we have addressed all of them to the best of our ability during this rebuttal. The General Response summarizes all revisions and how they are marked. Below, we respond to each of your concerns.


W1: Section 3.4 describes starting from the top retrieved model roots, whereas Algorithm 1 initializes roots using Equation 4, a generator score. Please specify the actual root-selection rule and how the adaptive beam bounds map to the fixed beam width used in the algorithm.

Summarized W1: ① Suggestion to clarify the root-selection rule. ② Question on how the adaptive beam bounds relate to the fixed beam width in Algorithm 1.

A1 (Clarification, Revised Paper): We apologize for the confusion caused by the ambiguous description in Section 3.4, and we thank the reviewer for the suggestion. The phrase "top retrieved model roots" was inaccurate. We revised Section 3.4, Algorithm 1, Section 4.2, and Appendix D.2 to state the procedure precisely. The changes are highlighted in purple\color{#6B4FA0}{\text{purple}}.

① Root-selection rule. Retrieval defines the available backbone candidates Mr(Q)\mathcal{M}_r(Q) (at most 10). The generator scores each complete backbone identifier by its mean token log-probability (Eq. (4), with an empty prefix). The top P=5P=5 become root proposals and go through the same pruning step as every later depth (see ②). Retrieval rank does not order the roots. It only enters the prompt as evidence.

② Adaptive bounds and the beam width. Algorithm 1 uses one pruning width for all steps, defined by the adaptive bounds. We now state the two quantities that control the search separately:

SymbolMeaningValue
PPmax. proposals kept at each expansion (a backbone at the root step, an unused tool or termination afterwards)5
W(η)W(\eta)paths kept by a global pruning step with η\eta inputsEq. (6)

W(η)=min⁡{η, bmax⁡, max⁡(bmin⁡,⌈rkeep η⌉)}.W(\eta)=\min\{\eta,\ b_{\max},\ \max(b_{\min},\lceil r_{\mathrm{keep}}\,\eta\rceil)\}.

The adaptive bounds (bmin⁡,bmax⁡)(b_{\min},b_{\max}) are the lower and upper limits of this width. We use (bmin⁡,bmax⁡)=(1,2)(b_{\min},b_{\max})=(1,2) and rkeep=0.4r_{\mathrm{keep}}=0.4. For example, if retrieval returns 10 backbones, the generator keeps the top 5 as root proposals, and pruning then keeps W(5)=2W(5)=2 of them.

See screenshot: Revised Section 3.4

See screenshot: Revised Section 3.4, expansion and pruning


W2: Algorithm 1, step 6 introduces an optional "no-tool proposal penalty" without specifying its definition, magnitude, or use in the reported experiments. Please clarify which actions it penalizes and report its setting, since an adjustment during proposal selection could affect the generated candidate pool.

Summarized W2: Suggestion to define the no-tool proposal penalty in Algorithm 1 (which action it penalizes and its magnitude) and to report its setting in the experiments.

A2 (Clarification, Revised Paper): We thank the reviewer for pointing out this omission. We added the definition to Section 3.4, the setting to Appendix D.2, and an explicit reference in Algorithm 1 (line 6). The changes are highlighted in blue\color{#2B6CB0}{\text{blue}}.

The penalty applies only to the empty-tool-set action τ∅\tau_{\emptyset}. It only decides whether a backbone-only configuration enters the candidate pool. All reported experiments on AgentSelect and SkillsBench use λ∅=10\lambda_{\emptyset}=10. λ∅\lambda_{\emptyset} does not affect the termination action τend\tau_{\mathrm{end}} of a nonempty tool set or the path score in Eq. (5), so beam pruning and final reranking are unchanged.

See screenshot: Revised Section 3.4, no-tool penalty

See screenshot: Revised Appendix D.2

AuthorsOpenReview ↗

rebuttal_mdAD_part2of4(W3)

W3: A related work is missed in the paper. "Toolken+ Improving LLM Tool Usage with Reranking and a Reject Option" (https://arxiv.org/abs/2410.12004)

Summarized W3: Suggestion to discuss a missing related work, Toolken+ (Yakovlev et al., 2024).

A3 (Related Work, Revised Paper): We thank the reviewer for the suggestion. We added Toolken+ to the related work in Section 2, Agent Resource Retrieval and Selection (highlighted in teal\color{#1F8A70}{\text{teal}}). Toolken+ improves tool calling during generation. When ToolkenGPT proposes a call, it reranks the top-kk tools, and a learned reject option can skip the call. (Reviewer BU35 asked for related token-based tool selectors, ToolGen and ToolWeaver, in W4.)

See screenshot: Revised Section 2

AuthorsOpenReview ↗

rebuttal_mdAD_part3of4(W4)

W4: More baselines are expected to compare or explain why they are not suitable for being baselines. For examples, "ToolGen Unified Tool Retrieval and Calling via Generation", "AnyTool Self-Reflective Hierarchical Agents for Large-Scale API Calls", and "RouteLLM Learning to Route LLMs with Preference Data".

Summarized W4: Suggestion to compare with ToolGen, AnyTool, and RouteLLM, or explain why they are not suitable baselines.

A4 (Explanation, New Baselines, Revised Paper): We thank the reviewer for the suggestion. Below, we first explain why the original forms of these methods do not apply directly, and then report the results of adapting all three to our task. These results further support our claim: even when adapted to our task, existing methods fall well short of recovering complete configurations. ToolGen's results also reflect its reliance on heavy supervision and its difficulty with cold-start tools. (Reviewer BU35 also requested tool-selection baselines in W4: ToolGen, ToolWeaver, and PORTS.)

① Explanation of inapplicability. The three methods address different parts of agent resource use, and none of them outputs a complete agent configuration. ToolGen performs generative tool retrieval and calling within a model's agent loop and chooses the next tool at each step for a fixed backbone. AnyTool uses hierarchical LLM agents to collect individual APIs into a candidate pool. RouteLLM routes each query between a strong and a weak LLM and involves no tools. Before execution, we recommend a backbone and a tool bundle as one compositional configuration.

② Results after adaptation. Each adaptation keeps the core contribution of the original method (shown in italics). All methods are evaluated on the 629 AgentSelect test queries, and Table 2 reports all eight metrics.

MethodTools fromBackbones fromRDCR@10RDCP@10
Component-Retsemantic retrievalCF ranking20.7211.78
ToolGenToolGen's generative retrieval with tokenized tools, fine-tuned to generate a complete tool sequenceCF ranking30.2831.05
RouteLLMsemantic retrievalRouteLLM's MF router21.9610.83
AnyToolAnyTool's hierarchical API retrievalCF ranking10.9612.14
Ours (with critic)joint generationjoint generation65.4862.71
Ours (generator only)joint generationjoint generation65.8865.28
  • ToolGen. We follow its two training stages: tool-token memorization, then query–tool supervision. Our adaptation uses 47,800 document-to-token examples, one per tool in our catalog, and 5,566 query-to-tool-sequence examples covering 13,668 tool labels. ToolGen must then map each query directly to newly added tool tokens over the full catalog. Our generator instead selects among retrieved resources whose descriptions are in its context, so it can use the pretrained model's language understanding. Most reference tools in the test set are also cold-start, and ToolGen can reach them only through the tool tokens learned during memorization. ToolGen thus relies heavily on supervision and does not generalize easily to new tools.
  • AnyTool. We keep its category–service–API navigation and solvability check as a retriever of multiple APIs.
  • RouteLLM. RouteLLM-MF and our CF retriever share the same basic scoring design. Both encode the query and each candidate backbone separately and score their compatibility from the two vectors, with matrix factorization in RouteLLM and a two-tower model in our CF retriever. Our CF retriever represents each backbone by its capability description, so it also ranks backbones unseen in training. On AgentSelect Part I, it ranks backbones better than RouteLLM-MF:
Backbone rankerP@5R@5nDCG@5P@10R@10nDCG@10
RouteLLM-MF44.6823.550.428838.5639.790.4462
Our CF retriever69.4335.430.632756.2157.070.6244

Our CF retriever is better at every cutoff from 5 to 50.

Retrieval is a replaceable module in our framework, so any of these retrievers can be plugged in. These additional experiments further support the point made in our paper: existing agent resource retrieval methods, even when adapted to our task, fall well short of recovering complete configurations. We added these baselines to Table 2, describe the adaptations in Appendix H.4, and reference them in Section 4.3 (highlighted in olive\color{#5B7F1E}{\text{olive}}).

See screenshot: Revised Section 4.3 and Table 2

See screenshot: Revised Appendix H.4

Reviewer_BU35OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies task-conditioned agent resource composition, jointly selecting a backbone model and a bundle of tools and skills. The authors propose a retrieval-grounded framework that uses resource profiles and historical configurations as context for a fine-tuned generator, which constructs candidate configurations through constrained beam search. A bundle-aware critic provides an optional score for final reranking alongside generator likelihoods. Experiments on AgentSelect and SkillsBench show improvements in reference-component recovery, with additional analyses of cross-domain adaptation and the association between configuration rankings and execution quality.

Critical strengths
  • This paper studies a practical problem of selecting and composing resources for LLM agents.
  • The paper is clearly written and the framework is easy to follow.
Critical weaknesses
  • The critic's supervision and contribution to execution quality lack sufficient validation.
  • Comparisons with relevant tool selectors and evidence of execution gains over existing methods are missing.
Questions for authors

I have the following concerns and questions for the authors:

  • Critic Supervision. Stage 1 uses LLM preferences, while Stage 2 mainly rewards reference-component matching. The paper does not sufficiently establish that these targets reflect execution utility, especially when alternative configurations can also succeed (Appendix E). With backbone comparisons fixed, the authors should compare teacher-only, reference-only, and combined tool-preference supervision against an execution-supervised critic, using held-out task outcomes or native verifier results.

  • Critic Contribution. Figure 2 shows that using the critic during beam search reduces RDCR from 60.0 to 52.9 relative to final reranking. However, the recovery-metric ablations do not establish the critic's benefit to task execution. Please explain this degradation and compare no critic, final reranking, and critic-guided search on execution outcomes under matched search budgets.

  • Execution Evaluation. Section 5.3 compares rank positions within the method's own candidate pool, which also includes terminated pruned prefixes. This correlation does not establish execution gains over existing methods. The authors should compare task execution against strong RAG and Bundle-Ret recommendations.

  • Tool-Selection Baselines. Relevant methods such as ToolGen [R1], ToolWeaver [R2], and PORTS [R3] are missing from Table 2. ToolWeaver explicitly models tool co-usage, making it particularly relevant to the proposed bundle-aware selection. Please compare against at least a representative generative selector and an LLM-aligned retriever.

References:

  • [R1] ToolGen: Unified Tool Retrieval and Calling via Generation. ICLR 2025.
  • [R2] ToolWeaver: Weaving Collaborative Semantics for Scalable Tool Use in Large Language Models. ICLR 2026.
  • [R3] PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models. EMNLP 2025.
10 comments
AuthorsOpenReview ↗

rebuttal_BU35_part1of6(Opening)

Thank you for your careful review and for

  • Recognizing the practical value of studying task-conditioned agent resource composition, which jointly selects a backbone and a bundle of tools and skills.
  • Finding the paper clearly written and the retrieval-grounded framework easy to follow.
  • Acknowledging the improvements in reference-component recovery on AgentSelect and SkillsBench, together with our cross-domain adaptation analysis.

Your review was remarkably thoughtful, and we are truly grateful for it! Each of your comments came with a concrete experiment: a supervision ablation for the critic (W1), a comparison of no critic, final reranking, and critic-guided search on execution outcomes (W2), a direct execution comparison with Bundle-Ret and RAG (W3), and representative tool-selection baselines (W4). We ran all of them, and they shaped many of the most important changes in our revision. The General Response lists every revision and how it is marked, and our point-by-point replies follow.

AuthorsOpenReview ↗

rebuttal_BU35_part2of6(W1)

W1: Critic Supervision. Stage 1 uses LLM preferences, while Stage 2 mainly rewards reference-component matching. The paper does not sufficiently establish that these targets reflect execution utility, especially when alternative configurations can also succeed (Appendix E). With backbone comparisons fixed, the authors should compare teacher-only, reference-only, and combined tool-preference supervision against an execution-supervised critic, using held-out task outcomes or native verifier results.

Summarized W1: ① Concern that the critic's training targets (LLM preference, reference matching) may not reflect execution utility. ② Request for a supervision ablation (teacher-only / reference-only / combined vs. an execution-supervised critic).

A1 (New Dataset, Additional Experiments, Revised Paper): We thank the reviewer for this comment on the critic's supervision. We agree that LLM preferences and reference matching are only proxies for execution utility and are not enough. We therefore built AgentSelect-Exec, an execution-feedback dataset with held-out test queries. We added Stage 3 to train the critic on execution, and we ran the requested ablation. The changes are highlighted in magenta\color{#B0407A}{\text{magenta}} (Sections 3.3, 4.1, and 5.1, Figure 2, Appendix F).

① Execution-supervised critic. AgentSelect-Exec executes the ten top-ranked configurations of 506 AgentSelect queries with 18 callable backbones, giving 5,060 executions.

#BackboneDeveloper
1GPT-6-AstraOpenAI
2GPT-6-LunaOpenAI
3GPT-6.1-SolOpenAI
4GPT-6.1-TerraOpenAI
5DeepSeek-R1DeepSeek
6DeepSeek-V3.2DeepSeek
7DeepSeek-V4-ProDeepSeek
8DeepSeek-V4.1-FlashDeepSeek
9Qwen3-8BAlibaba Qwen
10Qwen3-32BAlibaba Qwen
11Qwen3.6-35B-A3BAlibaba Qwen
12Qwen3.8-27BAlibaba Qwen
13Kimi-K2.6Moonshot AI
14GLM-4.5-AirZhipu AI
15GLM-5.3Zhipu AI
16Seed-OSS-36B-InstructByteDance Seed
17LongCat-2.0Meituan
18Hy4-previewTencent Hunyuan
RoleModelInput tokensOutput tokensTotal tokens
Backbone18 LLMs19,777,5744,413,13224,190,706
Tool environmentGPT-5.6-Luna12,870,4401,622,58914,493,029
JudgeGPT-5.6-Terra10,206,6061,496,89611,703,502
Total42,854,6207,532,61750,387,237

The queries are split into 200 training, 106 validation, and 200 held-out test queries. Stage 3 fine-tunes the critic on the training split. Execution scores every configuration, not only the reference, so successful alternatives are rewarded as well.

See screenshot: Revised Section 3.3, Stage 3

② Supervision ablation. All critics rank the same ten executed configurations for each of the 200 test queries (Figure 2). Top-1 is the execution score (0–10) of each critic's top-ranked configuration. The other metrics compare critic scores with execution scores.

Critic supervisionTop-1 execution score ↑Pairwise agreement (%) ↑Spearman ρ ↑MAE ↓RDCR@10 ↑
None (generator only)7.5843.09−0.143—60.68
Teacher only (Stage 1)7.9956.850.1532.6561.24
Reference only (Stage 2)7.6853.260.0833.5761.26
Teacher + reference (Stages 1–2)8.3755.770.1353.0961.24
Teacher + reference + execution (Stages 1–3)8.7867.960.3902.0861.13

These results confirm your concern: matching reference components is not the same as execution utility.

  • Reference-only supervision recovers the most reference components, but its top choices execute worst among the critics.
  • Teacher supervision transfers better to execution, and combining the two proxies works better than either one.
  • Execution supervision gives the largest gain. Teacher and reference signals remain useful when execution signals are hard to obtain.

See screenshot: Revised Figure 2(c)-(e)

AuthorsOpenReview ↗

rebuttal_BU35_part3of6(W2)

W2: Critic Contribution. Figure 2 shows that using the critic during beam search reduces RDCR from 60.0 to 52.9 relative to final reranking. However, the recovery-metric ablations do not establish the critic's benefit to task execution. Please explain this degradation and compare no critic, final reranking, and critic-guided search on execution outcomes under matched search budgets.

Summarized W2: ① Question why critic-guided search lowers RDCR and what the critic contributes. ② Request for execution-outcome comparison of no critic / final reranking / critic-guided search.

A2 (Clarification, Additional Experiments, Revised Paper): We thank the reviewer for this question about the critic's contribution. We apologize that the original submission did not clearly explain the role of the critic in our method. We have rewritten the paragraph in Section 3.3 to state its design purpose. We also compare the three settings on execution outcomes. The changes are highlighted in magenta\color{#B0407A}{\text{magenta}} (Sections 3.3 and 5.1, Figure 2(a)).

① Why critic-guided search lowers recovery, and what the critic contributes. (1) Why: The critic is trained to judge complete configurations. In beam search, it has to score partial ones instead. A partial configuration can get a low score even when it would grow into a good one, so the search prunes valuable branches too early. We therefore apply the critic only to completed candidates (Section 5.1).

Alternative design: Search can also be guided by a value model trained on partial configurations, while a reranker trained on complete configurations still ranks the final list. Because the value model is trained on partial configurations, it does not prune good branches early, and recovery stays at the level of reranking alone (RDCR@10 59.36 vs. 59.43, RDCP@10 48.76 vs. 48.70). It shows no clear advantage over reranking alone and needs a second model, so we keep the simpler design in the paper.

(2) What: The critic is a proxy for execution. It scores complete configurations, because executing every candidate is costly and often impossible for a cold-start system.

See screenshot: Revised Section 3.3

② No critic vs. final reranking vs. critic-guided search on execution. We compare the three settings with the same search budget:

SettingTop-1 execution score (0–10)
No critic (generator only)7.58
Final reranking, Stage 1+2 critic8.37
Final reranking, + Stage 38.78
Stage 3 critic guiding search and reranking8.57

As a final reranker, the critic raises the top-ranked score by 0.79 to 1.20 over the generator alone. Guiding search lowers it from 8.78 to 8.57. We therefore use the critic only for final reranking.

See screenshot: Revised Figure 2(a)-(b)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All

General Response to All Reviewers

We sincerely thank all three reviewers for the time they devoted to our paper and for their thoughtful, constructive reviews. Each comment was specific and genuinely helpful. Together, they have made this paper much stronger. During the rebuttal, we followed your suggestions with new experiments, new datasets, and new baselines. We believe these additions now answer most of your concerns directly.

We are also encouraged that the reviewers found that:

  • The problem is valuable and practical. Task-conditioned agent resource composition jointly selects a backbone and a bundle of tools and skills, beyond retrieving individual tools or predefined agents (G2FX, BU35, mdAD).
  • The framework is principled and easy to follow, and retrieving complete historical configurations is a practical way to guide composition (G2FX, BU35, mdAD).
  • The results are strong on AgentSelect and SkillsBench, with cross-domain generalization and alignment with end-to-end execution (G2FX, BU35, mdAD).

Common concerns and our responses.

  1. Evaluation scale (G2FX-W2, mdAD-W6). We now evaluate on all 629 AgentSelect test queries and on 78 SkillsBench test queries. We also add three datasets with execution outcomes. We build AgentSelect-Exec with 5,060 executed configurations. BFCL and SWE-Skills-Bench come with official executions. We also use the official SkillsBench runs (Section 4.1, Table 1, Appendices D.1, F, and G). With full test sets and execution outcomes, the evaluation is now sounder, more complete, and closer to real-world use.
  2. Execution-based evaluation (BU35-W1–W3, G2FX-W2, W4, Q3, mdAD-W5). The new Stage 3 adapts the critic to execution feedback. We evaluate rankings by judged execution on AgentSelect-Exec and by native rewards on BFCL, SWE-Skills-Bench, and the official SkillsBench runs. We also run the agents recommended by our method and by the baselines online (Sections 3.3 and 5, Appendix M). Our top-ranked agents execute best in every comparison with RAG-Qwen3.8-27B and Bundle-Ret.
  3. Additional baselines (BU35-W4, mdAD-W3, mdAD-W4). We add five methods to Table 2 (Section 4.3, Appendix H.4):
    • Stronger retrieval and routing: PORTS (LLM-aligned tool retrieval), AnyTool (hierarchical retrieval by LLM agents), and RouteLLM (LLM routing).
    • Token-based tool learning, which learns an ID token or code for each tool: ToolGen and ToolWeaver. Section 2 also discusses Toolken+.
    • They join the bundle methods already compared: Bundle-Ret, Text2Bundle, and DDBC-Adapt.
    • All adapted methods fall well short of ours.
  4. Critic ablation (G2FX-W1, Q2, BU35-W1, W2). We compare critics trained on teacher, reference, and execution signals. We also compare no critic, final reranking, and critic-guided search. All comparisons use execution outcomes (Section 5.1, Figure 2). As a final reranker, the critic lifts the Top-1 execution score from 7.58 to 8.78.

Other revisions. We clarify root selection, the proposal budget, the pruning width, and the no-tool proposal penalty (Section 3.4, Algorithm 1, and Appendix D.2, for mdAD-W1 and W2).

Revised manuscript. We have revised the paper accordingly. In the marked version, the key words of each change are colored by topic, and a box in the right margin names the reviewer and the weakness it addresses. A summary of revisions precedes the appendix.

Summary of revisions.

ColorChangeLocationResponds to
PurpleRoot selection, proposal budget PP, and pruning width W(η)W(\eta)§3.4, Alg. 1, App. B, C, D.2mdAD-W1
BlueNo-tool proposal penalty λ∅\lambda_{\emptyset}§3.4, App. D.2mdAD-W2
OrangeProblem formulation: execution utility and bundle effects§3G2FX-W3
TealRelated work on Toolken+§2mdAD-W3
OliveNew baselines: ToolGen, ToolWeaver, PORTS, AnyTool, RouteLLM§4.3, Table 2, App. H.4BU35-W4, mdAD-W4
AmberEvaluation: full 629-query AgentSelect test set, re-split SkillsBench, reference recovery versus execution quality§4.1, §4.5, Tables 1–3, App. D.1, K, LmdAD-W5, W6
MagentaExecution feedback: Stage 3, AgentSelect-Exec, execution-based ablation and alignment, transfer to BFCL and SWE-Skills-Bench, native execution on SkillsBench, execution comparison with baselines§3.3, §5, App. F, G, MBU35-W1–W3, G2FX-W1, W2, W4, Q1, mdAD-W5

Structural changes (not colored): Details of Sections 4.1–4.2 move to the new Appendix D. SkillsBench results move to Table 3 (Section 5.2). Section 5.3 is now an online study (Appendix M). A new Figure 2 (execution analysis) replaces the former search ablation. The case study moves to Appendix N.

We respond to each reviewer in the individual replies. We hope our responses address your concerns\color{#2E7D32}{\text{We hope our responses address your concerns}} and would welcome further discussion.\color{#2E7D32}{\text{and would welcome further discussion.}}

Sincerely,

The Authors

AuthorsOpenReview ↗

Revised Paper on Anonymous Link

Dear Area Chair and Reviewers,

Thank you again for your time and your constructive reviews.

We have uploaded the revised paper. To make every change easy to find, we also provide a marked version. Both are available at an anonymous link:

What changed. The reviewers made many useful suggestions on the experiments, and we followed them. As a result, the figures and several tables in the experiments have changed. We also reorganized some paragraphs so that the revisions are consistent and well integrated across the whole paper. Related paragraphs in the earlier sections and in the appendix are revised to match.

The screenshots linked in our replies come from the marked version.

We hope the revision shows\color{#2E7D32}{\text{We hope the revision shows}} the effort we made during the rebuttal\color{#2E7D32}{\text{the effort we made during the rebuttal}} and how much the paper has improved.\color{#2E7D32}{\text{and how much the paper has improved.}} We would greatly appreciate it\color{#2E7D32}{\text{We would greatly appreciate it}} if you could take a look\color{#2E7D32}{\text{if you could take a look}} when you have time,\color{#2E7D32}{\text{when you have time,}} and we warmly welcome\color{#2E7D32}{\text{and we warmly welcome}} any further questions.\color{#2E7D32}{\text{any further questions.}}

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.