- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes LatDec, which maps a frozen query embedding to multiple demand-specific vectors and confidence scores. Hungarian matching supervises head–tool assignments, while confidence-based budget allocation and embedding fusion produce the final ranking. The full method improves Recall@10 on CapCorpus, an author-constructed benchmark with synthetic queries. On the 95-task LiveMCPBench evaluation, fusion improves CPHB, but the full method is not statistically distinguishable from the baselines.
Critical strengths
-
LatDec learns multiple retrieval directions without an auxiliary LLM decomposition call. Its frozen tool representation also permits new tools to be indexed without retraining tool-side parameters, making the architecture relevant to evolving tool catalogs.
-
The full method improves CapCorpus Recall@10 from 0.5365 for the strongest baseline to 0.5694. Paired confidence intervals and component ablations provide credible evidence of benefits within this benchmark and reveal the distribution-dependent effect of fusion.
Critical weaknesses
-
Generalization to queries requiring more tools remains uncertain. H=3 is selected on synthetic validation data, and only H targets receive positive supervision at each training step when |T*|>H. Figure 3 shows gains for queries requiring at least four tools, but this group contains only 156 test queries. Appendix G analyzes head redundancy, but the interaction between head count and tool count remains unreported. The 95-task OOD evaluation does not establish an advantage over baselines, and neither complete tool-set coverage nor end-to-end task success is reported.
-
Dependence on the base encoder is not evaluated. All heads derive from one frozen all-MiniLM-L6-v2 query embedding. Multiple projections can improve retrieval alignment, but their effectiveness depends on the information preserved by that embedding. No alternative-base-encoder experiment is reported, leaving the scope of the proposed remedy for the single-vector bottleneck unclear.
Questions for authors
-
How does LatDec encourage coverage of all required tools when their number exceeds H? Beyond Figure 3 and Appendix G, could you compare H=3, 5, 8, and 10 within tool-count groups, reporting Recall@10 and the fraction of queries whose entire annotated tool set is retrieved in the top 10? If available, could you also report end-to-end task success?
-
Which aspect of the single-vector bottleneck does LatDec address when all heads share one frozen input embedding? Do the gains persist with a different base encoder and corresponding single-vector baselines under matched training and evaluation conditions?
4 comments
Response to Question 1: Head Count and Tool-Set Coverage
Reviewer Question 1:
How does LatDec encourage coverage of all required tools when their number exceeds H? Could you compare H=3, 5, 8, and 10 within tool-count groups, reporting Recall@10 and complete-set retrieval? If available, could you report end-to-end task success?
Author Response
Head Count and Tool-Set Coverage
When the number of required tools exceeds H, Hungarian matching supervises H head–target pairs in each training step. The remaining targets receive no direct matched-pair positive supervision in that step, and assignments can change during training. Each head can retrieve several tools at inference, so H limits simultaneous training assignments while the final retrieval budget determines how many tools can be returned.
Using the corrected RRF protocol, we evaluated all four checkpoints within each test tool-count group.
Each cell reports Recall@10 / complete-set@10, where complete-set@10 is the fraction of queries whose entire annotated tool set appears in the top ten retrieved candidates.
H & \text{Two tools} & \text{Three tools} & \text{Four tools} \\\\ & n=11{,}839 & n=2{,}639 & n=156 \\\\ \hline 3 & 0.6933\\,/\\,0.4517 & 0.4855\\,/\\,0.0792 & 0.3766\\,/\\,0.0000 \\\\ 5 & 0.7211\\,/\\,0.4968 & 0.5126\\,/\\,0.0890 & 0.4183\\,/\\,0.0064 \\\\ 8 & 0.7335\\,/\\,0.5152 & 0.5376\\,/\\,0.1239 & 0.4279\\,/\\,0.0064 \\\\ 10 & 0.7340\\,/\\,0.5183 & 0.5327\\,/\\,0.1197 & 0.4279\\,/\\,0.0128 \end{array}$$ All queries in the ≥4-tool group require exactly four tools. For this group, H=8 achieves 0.4279 Recall@10 [0.3910, 0.4663], compared with 0.2740 for BiEncoder-128, 0.2452 for ColBERT, and 0.1875 for frozen ST. To distinguish partial from complete coverage, the following table shows how many queries retrieve exactly zero through four required tools among the top ten candidates $$\begin{array}{l|rrrrr|c} \text{Method} & 0 & 1 & 2 & 3 & 4 & \text{Recall@10} \\\\ \hline \text{H3-RRF} & 23 & 53 & 58 & 22 & 0 & 0.3766 \\\\ \text{H5-RRF} & 17 & 47 & 63 & 28 & 1 & 0.4183 \\\\ \text{H8-RRF} & 18 & 45 & 58 & 34 & 1 & 0.4279 \\\\ \text{H10-RRF} & 16 & 46 & 63 & 29 & 2 & 0.4279 \\\\ \text{H8-CPHB} & 17 & 49 & 54 & 33 & 3 & 0.4295 \\\\ \text{BiEncoder-128} & 40 & 69 & 39 & 8 & 0 & 0.2740 \\\\ \text{BiEncoder-384} & 40 & 73 & 36 & 7 & 0 & 0.2660 \\\\ \text{ColBERT} & 33 & 95 & 26 & 2 & 0 & 0.2452 \\\\ \text{Frozen ST} & 65 & 69 & 18 & 4 & 0 & 0.1875 \end{array}$$ H=8 RRF retrieves exactly three required tools for 34 queries and all four for one query, giving 35 of 156 queries with at least three tools retrieved, compared with eight for BiEncoder-128. Increasing H improves partial coverage, while complete-set retrieval remains rare across the evaluated methods. At H=5, 8, and 10, all four targets can receive matched supervision; the remaining coverage gap therefore shows that sufficient matching capacity alone does not ensure complete retrieval. Across the full test set, H=8 achieves 0.4392 complete-set@10. This measures candidate-set coverage. End-to-end task success would additionally require executing the tools and evaluating the resulting answers, which was outside the scope of retrieval evaluation conducted here.Response to Question 2: Single-Vector Bottleneck and Base-Encoder Comparison
Reviewer Question 2:
Which aspect of the single-vector bottleneck does LatDec address when all heads share one frozen input embedding? Do the gains persist with a different base encoder and corresponding single-vector baselines under matched training and evaluation conditions?
Author Response
What LatDec Addresses in the Single-Vector Bottleneck
The reviewer identifies an important distinction between the information preserved by the encoder and how that information is used for retrieval. A single frozen query embedding supplies one cosine-ranking direction over the tool catalog. For a request containing several demands, tools relevant to different parts of the request must compete within that ranking. LatDec learns multiple mappings from the shared embedding into retrieval directions, with Hungarian matching encouraging different heads to align with distinct target tools. Confidence-based allocation then distributes retrieval slots across those directions.
LatDec therefore addresses the constraint of retrieving several demands through one ranking direction. Its effectiveness still depends on the information retained by the frozen encoder, and it cannot recover distinctions that the encoder has discarded. The matched, task-trained single-vector baseline helps assess this contribution empirically. With the original encoder, H=8 CPHB achieves 0.6682 Recall@10 on CapCorpus versus 0.5365 for BiEncoder-128, the strongest evaluated single-vector baseline. This comparison supports the benefit of multiple retrieval directions under the evaluated encoder and training setup.
Evaluation with a different base encoder
We thank the reviewer for this suggestion. To test dependence on the base encoder, we replaced all-MiniLM-L6-v2 (encoder with six transformer layers) with all-MiniLM-L12-v2 (encoder with twelve transformer layers). This increases encoder depth from six to twelve Transformer layers while retaining 384-dimensional embeddings. Hence, in this analysis all retrieval methods are evaluated using the newly encoded L12 representations. We re-encoded the CapCorpus queries and tools and trained LatDec H=8 and BiEncoder-128, the strongest single-vector baseline in our original CapCorpus evaluation, from scratch using the same hyperparameters. On the same 14,634-query test split, LatDec RRF achieves 0.6773 Recall@10 [0.6724, 0.6821], compared with 0.4080 for frozen cosine retrieval and 0.5333 for BiEncoder-128. The paired improvements are +0.270 [0.264, 0.275] and +0.144 [0.139, 0.149], respectively. This is close to the corresponding result using the original six-layer encoder, 0.6794. These results show that LatDec’s retrieval advantage over matched single-vector baselines persists when replacing the six-layer base encoder with a twelve-layer counterpart, supporting that the gains are not specific to the original all-MiniLM-L6-v2 encoder.
| Method | Six-layer base encoder | Twelve-layer base encoder |
|---|---|---|
| Frozen cosine | 0.4009 | 0.4080 |
| BiEncoder-128 | 0.5365 | 0.5333 |
| LatDec RRF | 0.6794 | 0.6773 |
Summary of Manuscript Updates
In light of the reviewers' suggestions, we have substantially revised the manuscript and uploaded the updated version.
The principal changes are as follows:
1. Inference protocols and component contributions. Section 3.4 distinguishes CPHB, query fusion, and their RRF combination. Section 5.3 and Appendix F compare learned confidence allocation against uniform allocation and the combined protocol against simpler CPHB + frozen-ST ranking fusion.
2. Head count and tool-set coverage. Appendices G and I report H=3, 5, 8, and 10 within tool-count groups, including Recall@10, complete-set@10, and the number of required tools retrieved for four-tool queries. Appendix G also presents head-subspace overlap and per-head coverage diagnostics.
3. External evaluation and adaptive inference. Section 5.4 reports the three protocols separately on LiveMCPBench. Appendix J explains the query-similarity analysis and tests whether confidence, ranking agreement, and source-query proximity can guide automatic protocol selection.
4. Online LLM-decomposition comparison. Section 5.5 and Appendix K compare prompted decomposition with LatDec on matched tool catalogs and final retrieval budgets, reporting retrieval quality, complete-set retrieval, query-time LLM calls, and measured latency. The text specifies the supervision and measurement conditions.
5. Base-encoder dependence. Appendix L adds an all-MiniLM-L12-v2 evaluation, with queries and tools re-encoded and LatDec and BiEncoder-128 trained from scratch. The comparison tests whether gains over matched single-vector baselines persist when replacing the original six-layer encoder with a twelve-layer counterpart.
6. Related work and presentation. The related-work section explicitly discusses AMER’s precedent and architectural relationship to LatDec. We rewrote the abstract and discussion for clarity, redesigned the results figures, and clarified metric definitions and statistical procedures.
We sincerely thank the reviewer for their thoughtful evaluation and constructive feedback, which have helped us improve the clarity and rigor of the manuscript.
Evaluation Correction and Updated Results
We identified an implementation mismatch in the submitted CapCorpus combined-method evaluation. The manuscript describes reciprocal rank fusion (RRF) of the CPHB and fused-query rankings, but the reported score of 0.5694 was computed using elementwise maximum fusion with H=3. We apologize for this error.
We re-evaluated the existing checkpoints using the RRF rule specified in the manuscript. Applying the same selection criterion, validation Recall@10, selects H=8: the scores are 0.6364, 0.6650, 0.6823, and 0.6789 for H=3, 5, 8, and 10, respectively. This evaluation correction does not change the training procedure or loss function.
We then used the validation-selected H=8 checkpoint consistently on both benchmarks. Its three inference protocols achieve:
The combined RRF protocol has the highest observed recall on CapCorpus; query fusion has the highest observed recall on LiveMCPBench. All three protocols use the same trained checkpoint, selected using CapCorpus validation only.
The updated manuscript incorporates the corresponding model-selection description, results, confidence intervals, and interpretation. Our responses use these updated values.