acceptodds
Under review as a conference paper at ICLR 2027

From One Query to Many Tools: Latent Demand Decomposition for Multi-Tool Retrieval

Abstract

Modern agentic systems depend on selecting relevant tools before a language model can execute a task. As tool registries grow, selecting a small candidate set becomes a retrieval problem. However, single-embedding retrievers can struggle when a query contains multiple semantically distinct demands, as a single query representation may not align closely with every required tool. We introduce LatDec (Latent Demand Decomposer), a retrieval architecture that maps a query to multiple latent demand representations in a single encoder forward pass. Its independent heads are trained using Hungarian matching to align each representation with a distinct target tool. At inference, Confidence-Proportional Head Budgeting (CPHB) converts each head’s confidence into its share of the retrieval budget. As a complementary signal, LatDec constructs a confidence-weighted population vector from all head representations, blends it with the frozen base query embedding, and combines the resulting ranking with the CPHB ranking using reciprocal rank fusion. We evaluate LatDec on two benchmarks: CapCorpus, a 22K-tool MCP retrieval benchmark that we construct and validate through independent human judgment, and the independently developed LiveMCPBench benchmark. On CapCorpus, LatDec substantially outperforms the frozen Sentence Transformer, matched bi-encoder, and multi-vector baselines. On LiveMCPBench, the benchmark’s current sample size does not support a clear distinction between the full LatDec model and the baselines. Within LatDec, however, adding embedding-space fusion increases retrieval performance over CPHB alone by 26.3% relative, corresponding to an absolute Recall@10 improvement of 0.0797 (95% CI [0.0445, 0.1172]). In a quartile analysis, the gain is larger for queries whose confidence-weighted population vector is least aligned with the base query embedding, highlighting the importance of balancing demand-specific specialization at the head level with preservation of the query’s overall semantics at the population level.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multi-tool retrieval, tool retrieval, representation learning, dense retrieval, agentic systems, Model Context Protocol

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_e1pUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LatDec, which maps a frozen query embedding to multiple demand-specific vectors and confidence scores. Hungarian matching supervises head–tool assignments, while confidence-based budget allocation and embedding fusion produce the final ranking. The full method improves Recall@10 on CapCorpus, an author-constructed benchmark with synthetic queries. On the 95-task LiveMCPBench evaluation, fusion improves CPHB, but the full method is not statistically distinguishable from the baselines.

Critical strengths
  1. LatDec learns multiple retrieval directions without an auxiliary LLM decomposition call. Its frozen tool representation also permits new tools to be indexed without retraining tool-side parameters, making the architecture relevant to evolving tool catalogs.

  2. The full method improves CapCorpus Recall@10 from 0.5365 for the strongest baseline to 0.5694. Paired confidence intervals and component ablations provide credible evidence of benefits within this benchmark and reveal the distribution-dependent effect of fusion.

Critical weaknesses
  1. Generalization to queries requiring more tools remains uncertain. H=3 is selected on synthetic validation data, and only H targets receive positive supervision at each training step when |T*|>H. Figure 3 shows gains for queries requiring at least four tools, but this group contains only 156 test queries. Appendix G analyzes head redundancy, but the interaction between head count and tool count remains unreported. The 95-task OOD evaluation does not establish an advantage over baselines, and neither complete tool-set coverage nor end-to-end task success is reported.

  2. Dependence on the base encoder is not evaluated. All heads derive from one frozen all-MiniLM-L6-v2 query embedding. Multiple projections can improve retrieval alignment, but their effectiveness depends on the information preserved by that embedding. No alternative-base-encoder experiment is reported, leaving the scope of the proposed remedy for the single-vector bottleneck unclear.

Questions for authors
  1. How does LatDec encourage coverage of all required tools when their number exceeds H? Beyond Figure 3 and Appendix G, could you compare H=3, 5, 8, and 10 within tool-count groups, reporting Recall@10 and the fraction of queries whose entire annotated tool set is retrieved in the top 10? If available, could you also report end-to-end task success?

  2. Which aspect of the single-vector bottleneck does LatDec address when all heads share one frozen input embedding? Do the gains persist with a different base encoder and corresponding single-vector baselines under matched training and evaluation conditions?

4 comments
AuthorsOpenReview ↗

Evaluation Correction and Updated Results

We identified an implementation mismatch in the submitted CapCorpus combined-method evaluation. The manuscript describes reciprocal rank fusion (RRF) of the CPHB and fused-query rankings, but the reported score of 0.5694 was computed using elementwise maximum fusion with H=3. We apologize for this error.

We re-evaluated the existing checkpoints using the RRF rule specified in the manuscript. Applying the same selection criterion, validation Recall@10, selects H=8: the scores are 0.6364, 0.6650, 0.6823, and 0.6789 for H=3, 5, 8, and 10, respectively. This evaluation correction does not change the training procedure or loss function.

We then used the validation-selected H=8 checkpoint consistently on both benchmarks. Its three inference protocols achieve:

Inference protocolCapCorpus Recall@10LiveMCPBench Recall@10
CPHB + query-fusion RRF0.67940.3745
CPHB0.66820.2629
Query fusion0.53610.4143

The combined RRF protocol has the highest observed recall on CapCorpus; query fusion has the highest observed recall on LiveMCPBench. All three protocols use the same trained checkpoint, selected using CapCorpus validation only.

The updated manuscript incorporates the corresponding model-selection description, results, confidence intervals, and interpretation. Our responses use these updated values.

AuthorsOpenReview ↗

Response to Question 1: Head Count and Tool-Set Coverage

Reviewer Question 1:

How does LatDec encourage coverage of all required tools when their number exceeds H? Could you compare H=3, 5, 8, and 10 within tool-count groups, reporting Recall@10 and complete-set retrieval? If available, could you report end-to-end task success?

Author Response

Head Count and Tool-Set Coverage

When the number of required tools exceeds H, Hungarian matching supervises H head–target pairs in each training step. The remaining targets receive no direct matched-pair positive supervision in that step, and assignments can change during training. Each head can retrieve several tools at inference, so H limits simultaneous training assignments while the final retrieval budget determines how many tools can be returned.

Using the corrected RRF protocol, we evaluated all four checkpoints within each test tool-count group.

Each cell reports Recall@10 / complete-set@10, where complete-set@10 is the fraction of queries whose entire annotated tool set appears in the top ten retrieved candidates.

H & \text{Two tools} & \text{Three tools} & \text{Four tools} \\\\ & n=11{,}839 & n=2{,}639 & n=156 \\\\ \hline 3 & 0.6933\\,/\\,0.4517 & 0.4855\\,/\\,0.0792 & 0.3766\\,/\\,0.0000 \\\\ 5 & 0.7211\\,/\\,0.4968 & 0.5126\\,/\\,0.0890 & 0.4183\\,/\\,0.0064 \\\\ 8 & 0.7335\\,/\\,0.5152 & 0.5376\\,/\\,0.1239 & 0.4279\\,/\\,0.0064 \\\\ 10 & 0.7340\\,/\\,0.5183 & 0.5327\\,/\\,0.1197 & 0.4279\\,/\\,0.0128 \end{array}$$ All queries in the ≥4-tool group require exactly four tools. For this group, H=8 achieves 0.4279 Recall@10 [0.3910, 0.4663], compared with 0.2740 for BiEncoder-128, 0.2452 for ColBERT, and 0.1875 for frozen ST. To distinguish partial from complete coverage, the following table shows how many queries retrieve exactly zero through four required tools among the top ten candidates $$\begin{array}{l|rrrrr|c} \text{Method} & 0 & 1 & 2 & 3 & 4 & \text{Recall@10} \\\\ \hline \text{H3-RRF} & 23 & 53 & 58 & 22 & 0 & 0.3766 \\\\ \text{H5-RRF} & 17 & 47 & 63 & 28 & 1 & 0.4183 \\\\ \text{H8-RRF} & 18 & 45 & 58 & 34 & 1 & 0.4279 \\\\ \text{H10-RRF} & 16 & 46 & 63 & 29 & 2 & 0.4279 \\\\ \text{H8-CPHB} & 17 & 49 & 54 & 33 & 3 & 0.4295 \\\\ \text{BiEncoder-128} & 40 & 69 & 39 & 8 & 0 & 0.2740 \\\\ \text{BiEncoder-384} & 40 & 73 & 36 & 7 & 0 & 0.2660 \\\\ \text{ColBERT} & 33 & 95 & 26 & 2 & 0 & 0.2452 \\\\ \text{Frozen ST} & 65 & 69 & 18 & 4 & 0 & 0.1875 \end{array}$$ H=8 RRF retrieves exactly three required tools for 34 queries and all four for one query, giving 35 of 156 queries with at least three tools retrieved, compared with eight for BiEncoder-128. Increasing H improves partial coverage, while complete-set retrieval remains rare across the evaluated methods. At H=5, 8, and 10, all four targets can receive matched supervision; the remaining coverage gap therefore shows that sufficient matching capacity alone does not ensure complete retrieval. Across the full test set, H=8 achieves 0.4392 complete-set@10. This measures candidate-set coverage. End-to-end task success would additionally require executing the tools and evaluating the resulting answers, which was outside the scope of retrieval evaluation conducted here.
AuthorsOpenReview ↗

Response to Question 2: Single-Vector Bottleneck and Base-Encoder Comparison

Reviewer Question 2:

Which aspect of the single-vector bottleneck does LatDec address when all heads share one frozen input embedding? Do the gains persist with a different base encoder and corresponding single-vector baselines under matched training and evaluation conditions?

Author Response

What LatDec Addresses in the Single-Vector Bottleneck

The reviewer identifies an important distinction between the information preserved by the encoder and how that information is used for retrieval. A single frozen query embedding supplies one cosine-ranking direction over the tool catalog. For a request containing several demands, tools relevant to different parts of the request must compete within that ranking. LatDec learns multiple mappings from the shared embedding into retrieval directions, with Hungarian matching encouraging different heads to align with distinct target tools. Confidence-based allocation then distributes retrieval slots across those directions.

LatDec therefore addresses the constraint of retrieving several demands through one ranking direction. Its effectiveness still depends on the information retained by the frozen encoder, and it cannot recover distinctions that the encoder has discarded. The matched, task-trained single-vector baseline helps assess this contribution empirically. With the original encoder, H=8 CPHB achieves 0.6682 Recall@10 on CapCorpus versus 0.5365 for BiEncoder-128, the strongest evaluated single-vector baseline. This comparison supports the benefit of multiple retrieval directions under the evaluated encoder and training setup.

Evaluation with a different base encoder

We thank the reviewer for this suggestion. To test dependence on the base encoder, we replaced all-MiniLM-L6-v2 (encoder with six transformer layers) with all-MiniLM-L12-v2 (encoder with twelve transformer layers). This increases encoder depth from six to twelve Transformer layers while retaining 384-dimensional embeddings. Hence, in this analysis all retrieval methods are evaluated using the newly encoded L12 representations. We re-encoded the CapCorpus queries and tools and trained LatDec H=8 and BiEncoder-128, the strongest single-vector baseline in our original CapCorpus evaluation, from scratch using the same hyperparameters. On the same 14,634-query test split, LatDec RRF achieves 0.6773 Recall@10 [0.6724, 0.6821], compared with 0.4080 for frozen cosine retrieval and 0.5333 for BiEncoder-128. The paired improvements are +0.270 [0.264, 0.275] and +0.144 [0.139, 0.149], respectively. This is close to the corresponding result using the original six-layer encoder, 0.6794. These results show that LatDec’s retrieval advantage over matched single-vector baselines persists when replacing the six-layer base encoder with a twelve-layer counterpart, supporting that the gains are not specific to the original all-MiniLM-L6-v2 encoder.

MethodSix-layer base encoderTwelve-layer base encoder
Frozen cosine0.40090.4080
BiEncoder-1280.53650.5333
LatDec RRF0.67940.6773
AuthorsOpenReview ↗

Summary of Manuscript Updates

In light of the reviewers' suggestions, we have substantially revised the manuscript and uploaded the updated version.

The principal changes are as follows:

1. Inference protocols and component contributions. Section 3.4 distinguishes CPHB, query fusion, and their RRF combination. Section 5.3 and Appendix F compare learned confidence allocation against uniform allocation and the combined protocol against simpler CPHB + frozen-ST ranking fusion.

2. Head count and tool-set coverage. Appendices G and I report H=3, 5, 8, and 10 within tool-count groups, including Recall@10, complete-set@10, and the number of required tools retrieved for four-tool queries. Appendix G also presents head-subspace overlap and per-head coverage diagnostics.

3. External evaluation and adaptive inference. Section 5.4 reports the three protocols separately on LiveMCPBench. Appendix J explains the query-similarity analysis and tests whether confidence, ranking agreement, and source-query proximity can guide automatic protocol selection.

4. Online LLM-decomposition comparison. Section 5.5 and Appendix K compare prompted decomposition with LatDec on matched tool catalogs and final retrieval budgets, reporting retrieval quality, complete-set retrieval, query-time LLM calls, and measured latency. The text specifies the supervision and measurement conditions.

5. Base-encoder dependence. Appendix L adds an all-MiniLM-L12-v2 evaluation, with queries and tools re-encoded and LatDec and BiEncoder-128 trained from scratch. The comparison tests whether gains over matched single-vector baselines persist when replacing the original six-layer encoder with a twelve-layer counterpart.

6. Related work and presentation. The related-work section explicitly discusses AMER’s precedent and architectural relationship to LatDec. We rewrote the abstract and discussion for clarity, redesigned the results figures, and clarified metric definitions and statistical procedures.

We sincerely thank the reviewer for their thoughtful evaluation and constructive feedback, which have helped us improve the clarity and rigor of the manuscript.

Reviewer_aAvVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the Ghost Centroid Problem in multi-tool retrieval, where a single query embedding fails to capture multiple distinct sub-task demands. To resolve this, the authors propose LatDec (Latent Demand Decomposer), which generates multiple latent demand representations in a single encoder pass using independent projection heads. During training, Hungarian Matching dynamically pairs heads with target tools without requiring extra LLM calls at inference. Additionally, the authors construct CapCorpus, an MCP tool retrieval benchmark with ~22K tools, and test zero-shot generalization on the independent LiveMCPBench. While LatDec outperforms baselines on CapCorpus, its performance gains on LiveMCPBench lack statistical significance. Further ablations show that while embedding fusion aids LiveMCPBench, it degrades performance on CapCorpus, highlighting nuanced trade-offs between component designs.

Critical strengths
  1. LatDec decomposes query demands directly within representation learning via multiple heads and Hungarian matching, eliminating runtime generative sub-query decomposition. Frozen tool-side embeddings ensure compatibility with existing tool indices.

  2. The paper provides extensive ablations on CapCorpus and uses paired bootstrap confidence intervals on LiveMCPBench, explicitly acknowledging the lack of statistically significant gains on the latter without overclaiming zero-shot capability.

Critical weaknesses
  1. On LiveMCPBench, full LatDec achieves 0.3825 Recall@10, failing to outperform the frozen ST Cosine baseline (0.3865). Paired bootstrap confidence intervals confirm no statistically significant difference between LatDec and baselines, indicating that gains may be restricted to the CapCorpus distribution.

  2. Ablation results are contradictory: CPHB alone achieves 0.6257 Recall@10 on CapCorpus, outperforming CPHB+Fusion (0.5694). On LiveMCPBench, CPHB and uniform head budget allocation yield identical performance (0.3028), showing no clear advantage for learned confidence over uniform allocation.

Questions for authors
  1. What primary factor drives the marginal gains on LiveMCPBench: query distribution shift, tool corpus shift, or cross-space alignment failure? Can evaluations be expanded to additional human-authored tool benchmarks?

  2. Given that CPHB and uniform allocation perform identically on LiveMCPBench (Appendix F), what evidence demonstrates the efficacy of learned confidence? Why use fixed RRF over adaptive fusion based on query complexity?

  3. How does the model effectively represent target tools left unmatched during Hungarian matching when H=3H=3? Can the authors evaluate head-count scaling specifically on test queries requiring ≥4\ge 4 tools?

5 comments
AuthorsOpenReview ↗

Evaluation Correction and Updated Results

We identified an implementation mismatch in the submitted CapCorpus combined-method evaluation. The manuscript describes reciprocal rank fusion (RRF) of the CPHB and fused-query rankings, but the reported score of 0.5694 was computed using elementwise maximum fusion with H=3. We apologize for this error.

We re-evaluated the existing checkpoints using the RRF rule specified in the manuscript. Applying the same selection criterion, validation Recall@10, selects H=8: the scores are 0.6364, 0.6650, 0.6823, and 0.6789 for H=3, 5, 8, and 10, respectively. This evaluation correction does not change the training procedure or loss function.

We then used the validation-selected H=8 checkpoint consistently on both benchmarks. Its three inference protocols achieve:

Inference protocolCapCorpus Recall@10LiveMCPBench Recall@10
CPHB + query-fusion RRF0.67940.3745
CPHB0.66820.2629
Query fusion0.53610.4143

The combined RRF protocol has the highest observed recall on CapCorpus; query fusion has the highest observed recall on LiveMCPBench. All three protocols use the same trained checkpoint, selected using CapCorpus validation only.

The updated manuscript incorporates the corresponding model-selection description, results, confidence intervals, and interpretation. Our responses use these updated values.

AuthorsOpenReview ↗

Response to Question 1: External Evaluation and Distribution Shift

Reviewer Question 1:

What primary factor drives the marginal gains on LiveMCPBench: query distribution shift, tool corpus shift, or cross-space alignment failure? Can evaluations be expanded to additional human-authored tool benchmarks?

Author Response

External evaluation and distribution shift

We appreciate the reviewer’s observation that the inference protocols behave differently on LiveMCPBench than on CapCorpus. Using the H=8 checkpoint selected in the corrected RRF validation sweep (described in our correction note), query fusion alone achieves 0.4143 Recall@10 on the external LiveMCPBench evaluation, compared with 0.3865 for frozen ST, an observed 7.2% relative increase (paired difference +0.0279 [−0.0153, 0.0742]). The combined CPHB + query-fusion RRF protocol achieves 0.3745.

These results reveal different uses of the information learned by the same LatDec heads. On LiveMCPBench, query fusion combines their confidence-weighted population representation with the frozen ST query representation and achieves the highest observed Recall@10 among the tested LatDec and raw ST protocols. On CapCorpus, combining that fused-query ranking with confidence-budgeted head retrieval through RRF performs best. Thus, the learned head representations contribute under both evaluations, although the most effective inference protocol differs.

To investigate the different protocol preferences, we measured each query’s maximum cosine similarity to the CapCorpus training queries. The mean is 0.76 for held-out CapCorpus queries and 0.57 for LiveMCPBench queries, showing that the external queries are farther from the training set in the frozen ST space. Yet in the lowest-similarity CapCorpus validation group, combined RRF still exceeds query fusion alone by 0.113 Recall@10 [0.080, 0.149]. Thus, low similarity does not by itself predict a preference for query fusion within CapCorpus. Because the query source and tool catalog change together on LiveMCPBench, these evaluations cannot establish whether query shift, catalog shift, or cross-space alignment is the primary driver. Evaluating another human-authored query set against a fixed tool catalog would test the role of query shift more directly. For now, the protocol-specific LiveMCPBench results show how LatDec behaves under this combined change.

To test whether this protocol-dependent behavior extends beyond LiveMCPBench, MCP-Atlas offers a promising second set of human-authored MCP tasks. Its public tasks provide reference tool-call trajectories, but its native evaluation exposes a restricted tool menu for each task. A comparable fixed-catalog Recall@10 evaluation requires mapping those trajectories to a common tool index and checking whether the reference calls adequately represent required tools. We have therefore kept the present comparison to the verified LiveMCPBench evaluation rather than report a hastily adapted MCP-Atlas score.

AuthorsOpenReview ↗

Response to Question 2: Learned Confidence and Adaptive Fusion

Reviewer Question 2:

Given that CPHB and uniform allocation perform identically on LiveMCPBench (Appendix F), what evidence demonstrates the efficacy of learned confidence? Why use fixed RRF over adaptive fusion based on query complexity?

Author Response

Learned Confidence and Adaptive Fusion

We tested confidence-based allocation against uniform allocation using the H=8 checkpoint selected in the corrected RRF validation sweep. CPHB achieved 0.6682 Recall@10 versus 0.6248 for uniform allocation on CapCorpus (paired difference +0.0434 [0.0408, 0.0459]), and 0.2629 versus 0.2230 on LiveMCPBench (+0.0400 [0.0180, 0.0649]). The uniform control gives each of eight heads one slot and averages over all 28 possible assignments of the two remaining top-10 slots. The Appendix F tie remains valid for H=3. At H=8, the comparison demonstrates that learned confidence helps direct the limited retrieval budget toward productive heads on both evaluations.

We also explored the reviewer’s suggestion of adapting fusion to each request. We fitted logistic-regression routers using head-confidence entropy, maximum normalized confidence, and top-10 overlap between the RRF and query-fusion rankings, with a second router additionally using similarity to training queries. Using separate fit, threshold-tuning, and check subsets of CapCorpus validation, both selected policies routed every query to RRF and matched its check-set Recall@10 of 0.6887. This is consistent with the above mentioned query-similarity diagnostic: even in the CapCorpus validation group least similar to CapCorpus training queries, RRF exceeded query fusion alone by 0.113 Recall@10 [0.080, 0.149]. Together, these results show that the tested features did not identify when switching to query fusion would improve retrieval within CapCorpus. Adaptive fusion therefore requires a predictor of the relative benefit of the retrieval paths that transfers across distributions. Our experiments establish how CPHB, query fusion, and their RRF combination contribute to tool retrieval using the same trained heads, while leaving automatic selection among these inference strategies for future work.

Reviewer_gSzJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LatDec retrieves tools for requests containing multiple demands. It maps a frozen query embedding to several retrieval vectors, trains them, and combines their results using confidence-based allocation and fusion. The method avoids online LLM-based decomposition. Experiments report improved Recall@10 on synthetic CapCorpus, while the comparison on 95 LiveMCPBench tasks remains inconclusive.

Critical strengths
  1. The small query-side model retrieves multiple tools while retaining a shared frozen index and avoiding online LLM-based decomposition. This is a useful approach when additional generative processing is costly.
Critical weaknesses
  1. The paper does not clearly establish what its method adds to existing multi-vector retrieval.
  • Learning multiple query vectors with Hungarian matching has a close precedent in AMER, although LatDec’s small parallel heads provide a useful computational difference.
  • Confidence weighting and rank fusion are also established ideas, so the contribution depends on what this particular design improves. On LiveMCPBench, confidence-based allocation achieves the same Recall@10 as uniform allocation, while the corresponding CapCorpus comparison is missing. Fusion reduces performance on CapCorpus but improves it on LiveMCPBench. This supports an external retrieval benefit, but does not isolate the advantage of the proposed fusion design over a simpler alternative.
  • A comparison with directly combining the CPHB and frozen-retriever rankings would clarify what the proposed fusion adds.
  1. The comparisons are insufficient to establish an advantage over relevant tool-retrieval methods.
  • The task-trained bi-encoders and ColBERT-style baseline are useful controls, but they do not test LatDec against an approach that explicitly decomposes a request or plans retrieval for its different demands. Such a comparison is particularly relevant because LatDec avoids online generative decomposition.
  • A representative alternative should be evaluated on the same tool catalog and final retrieval size, with both retrieval quality and measured inference cost reported and differences in supervision stated.
  • Without this evidence, the paper does not establish whether its lightweight design provides a worthwhile trade-off.
Questions for authors
  1. Could the authors demonstrate what latent decomposition contributes in practice? In particular, how much retrieval quality is retained or improved by avoiding online LLM-based decomposition, and what reduction in query-time latency and cost does this provide compared with a relevant decomposition-based approach?
5 comments
AuthorsOpenReview ↗

Evaluation Correction and Updated Results

We identified an implementation mismatch in the submitted CapCorpus combined-method evaluation. The manuscript describes reciprocal rank fusion (RRF) of the CPHB and fused-query rankings, but the reported score of 0.5694 was computed using elementwise maximum fusion with H=3. We apologize for this error.

We re-evaluated the existing checkpoints using the RRF rule specified in the manuscript. Applying the same selection criterion, validation Recall@10, selects H=8: the scores are 0.6364, 0.6650, 0.6823, and 0.6789 for H=3, 5, 8, and 10, respectively. This evaluation correction does not change the training procedure or loss function.

We then used the validation-selected H=8 checkpoint consistently on both benchmarks. Its three inference protocols achieve:

Inference protocolCapCorpus Recall@10LiveMCPBench Recall@10
CPHB + query-fusion RRF0.67940.3745
CPHB0.66820.2629
Query fusion0.53610.4143

The combined RRF protocol has the highest observed recall on CapCorpus; query fusion has the highest observed recall on LiveMCPBench. All three protocols use the same trained checkpoint, selected using CapCorpus validation only.

The updated manuscript incorporates the corresponding model-selection description, results, confidence intervals, and interpretation. Our responses use these updated values.

AuthorsOpenReview ↗

Response to Question 1 (Part 1): Architectural Contribution and Component Controls

Reviewer Question:

Could the authors demonstrate what latent decomposition contributes in practice? In particular, how much retrieval quality is retained or improved by avoiding online LLM-based decomposition, and what reduction in query-time latency and cost does this provide compared with a relevant decomposition-based approach?

Author Response

We address this question in two parts. This comment covers Part 1, addressing the concerns regarding AMER and the contributions of confidence-based allocation and query fusion.

AMER and LatDec’s Distinct Contribution

We acknowledge the reviewer’s observation that AMER is a close precedent for learning multiple retrieval vectors with Hungarian matching, and this has been now discussed explicitly in the manuscript. The methods differ in both vector generation and how those vectors contribute to the final result. AMER’s query model generates vectors autoregressively, with each subsequent prediction conditioned on preceding vectors, and interleaves their retrieval lists round-robin. LatDec generates its head vectors independently from one frozen query embedding; it also learns a confidence for each head and uses those confidences to allocate a fixed tool-retrieval budget for each request. Its fused-query path combines the head population with the original query representation. Thus, LatDec’s design couples parallel demand representations with query-dependent allocation, while retaining a fixed tool index. The experiments below test what these design choices contribute and address the reviewer’s separate question about online LLM-based decomposition.

Confidence Allocation and Fusion Controls

We tested the two controls requested in your review using the corrected, validation-selected H=8 checkpoint. For head allocation, CPHB achieved 0.6682 Recall@10 versus 0.6248 for uniform allocation on CapCorpus (paired difference +0.0434 [0.0408, 0.0459]), and 0.2629 versus 0.2230 on LiveMCPBench (+0.0400 [0.0180, 0.0649]). [Note - The uniform control gives each of the eight heads one slot, then averages the results over all 28 possible pairs of heads that can receive the two remaining top-10 slots]. The tie reported in Appendix F of the original submission remains valid for H=3; but the additional H=8 evaluation shows a benefit from confidence-based allocation on both benchmarks. We have updated the manuscript with these latest results.

For fusion, combining CPHB with the fused-query ranking achieved higher CapCorpus Recall@10 than combining CPHB with the frozen ST ranking: 0.6794 versus 0.6690 (+0.0104 [0.0084, 0.0124]). On LiveMCPBench, query fusion alone achieved the highest observed Recall@10 among LatDec’s three inference protocols at 0.4143, above the simpler CPHB + frozen-ST fusion at 0.3944. The combined CPHB + query-fusion RRF protocol achieved 0.3745; its paired difference from CPHB + ST was −0.0199 [−0.0576, 0.0165]. Thus, learned confidence allocation contributes under both evaluations, while the relative benefit of combining the retrieval paths changes across them. We will distinguish the three LatDec protocols and their results clearly in the revision. These component results lead to the practical question in your review: whether generating explicit subqueries online improves retrieval enough to justify its query-time cost.

AuthorsOpenReview ↗

Question 1 (Part 2): Retrieval Quality and Query-Time Cost

Reviewer Question:

Could the authors demonstrate what latent decomposition contributes in practice? In particular, how much retrieval quality is retained or improved by avoiding online LLM-based decomposition, and what reduction in query-time latency and cost does this provide compared with a relevant decomposition-based approach?

Author Response

This comment covers Part 2, comparing LatDec with an online LLM-decomposition pipeline in retrieval quality and query-time cost, and explaining its practical role within an agent workflow.

Online Decomposition: Retrieval and Query-Time Cost

To assess the query-time trade-off, we evaluated a prompted GPT-6.1 Sol decomposition pipeline on the same tool catalogs and final top-10 size. The LLM generated one to eight capability-specific subqueries; the frozen ST encoder retrieved tools for each, and RRF merged the lists. On a fixed 512-query CapCorpus test sample, LatDec H=8 RRF achieved 0.7008 Recall@10 and retrieved complete tool sets for 235 queries, compared with 0.4524 and 101 queries for this LLM based decomposition pipeline (paired Recall@10 difference +0.2484 [0.2194, 0.2771]). LatDec was trained on CapCorpus query-tool pairs, while the LLM pipeline used prompting without tool-retrieval training, so this performance gap should be interpreted in light of that supervision difference. On LiveMCPBench, LatDec query fusion alone achieved 0.4143 versus 0.4064 for LLM decomposition (+0.0080 [−0.0451, 0.0615]); LatDec RRF achieved 0.3745. The practical distinction is query-time work: the query decomposition pipeline makes one remote LLM API call per request and took about 3.5 seconds per query end to end, including network round-trip and subsequent retrieval. LatDec makes no query-time LLM call. Across three timing passes on each benchmark, its median local query latency was 8-38 ms with the models already loaded into memory.

LatDec’s Role Within an Agent Workflow

Together, these results highlight the practical role of latent decomposition within a broader agentic system. LatDec serves as a retrieval layer between a user request and a large tool catalog. Using parallel heads and confidence-guided retrieval budgets, it retrieves candidate tools for distinct demands in a single, low-latency pass. An agent can then build a plan grounded in those tools or evaluate them against an existing plan before deciding which to invoke. This division of work lets LatDec handle multi-demand candidate retrieval without a query-time LLM call, while the agent retains responsibility for planning, tool selection, and task execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.