- Limited methodological novelty relative to closely related tool-discovery and planning work
The main methodological contribution appears incremental relative to existing approaches to agentic tool discovery and iterative retrieval.
In particular, MCP-Zero (Fei et al., 2025), which the paper cites, already explores proactive tool discovery through descriptions of required tools and retrieval from a large tool repository. More recent work such as ToolQP (Fang and Glass, 2026) also studies iterative tool-query planning and reinforcement learning for improved tool retrieval.
MultiSearch-R similarly generates abstract descriptions of desired tools, maps them to concrete tools using a dense retriever, and iteratively selects tools and executes queries. The multi-granularity reward formulation combines tool-selection accuracy, semantic relevance of retrieved evidence, and final-response quality.
While the application to search sources is distinct, the paper does not sufficiently establish what fundamentally new methodological capability is introduced beyond adapting these existing ideas to a particular tool environment. The reward formulation also seems to combine relatively standard intermediate and outcome-level signals rather than introduce a clearly new optimization principle.
The more distinctive contribution may therefore be the benchmark and its focus on search-source overlap, rather than the proposed planning or RL method.
A direct comparison with adapted tool-discovery and iterative retrieval-planning approaches would be important for establishing the incremental contribution of MultiSearch-R. The comparisons with Search-o1, Search-R1, and WebThinker are useful but do not fully answer this question, since these systems were not necessarily developed for source selection from a large repository of similar tools.
- Benchmark construction raises concerns about realism and what is actually being evaluated
MultiSearch-Bench is constructed by first collecting evidence from selected websites and subsequently generating questions and reference answers from that evidence. This ensures that the questions are answerable from the collected documents, but it introduces two important limitations.
First, the generated questions may reflect the characteristics of the originating sources rather than naturally occurring information needs. Second, the reference tool set identifies the sources used during benchmark construction, not necessarily all sources capable of answering the question correctly. This is particularly relevant in a benchmark explicitly motivated by overlapping source functionality.
The paper does consider category-level and subcategory-level alternatives to exact tool matching, which partly addresses this concern. Nevertheless, the main rewards and tool-selection evaluations continue to rely heavily on the originating reference sources.
There is also an important distinction between the claimed search-tool orchestration setting and its implementation. Appendix A.3 indicates that only ten platforms have custom search wrappers, while the remaining sources are accessed through Google-backed domain-restricted search. Consequently, most of the benchmark evaluates selecting websites or search domains under a largely shared search backend, rather than selecting among genuinely heterogeneous search tools with different native query capabilities.
This is a reasonable controlled setting, but it is narrower than the broader search-tool orchestration problem discussed in the introduction.
Finally, the benchmark contains 800 single-source and 200 multi-source tasks, corresponding to only 20 multi-source examples in the reported 100-example test split. Since multi-source orchestration is central to the motivation, aggregate results alone are insufficient to establish how well the method handles the most relevant cases.
It would be helpful to report separate results for single-source and multi-source tasks, evaluate tasks independently written from real user information needs, and quantify the difficulty introduced by semantically similar but factually unsuitable sources.
- The experiments do not adequately isolate the source of the final-answer improvements
The reported final-answer improvements are substantial, but it is not clear how much of the improvement is attributable specifically to the proposed reasoning-based tool-selection mechanism.
The ablation results in Appendix Table 12 are informative. On the 8B model, removing the planning reward reduces the final-answer score from 0.862 to 0.855; removing the execution reward reduces it to 0.851; using only the sparse outcome reward yields 0.852; and removing requirement analysis yields 0.857.
These ablations show larger differences in reference tool F1 than in final-answer quality. For example, removing requirement analysis changes the final-answer score by only 0.005. Without uncertainty estimates for these particular differences, it is difficult to determine how much evidence they provide for the importance of the individual components to the final outcome.
More importantly, the stage-wise diagnostics in Appendix F suggest that downstream evidence processing is a major source of improvement.
For the 8B model, replacing the naive retrieval baseline's candidate tools with reference tools increases its answer score only from 0.5247 to 0.5390. In another diagnostic configuration using reference URLs, replacing truncated raw evidence with two-stage summaries raises the score from 0.6007 to 0.8343.
These results are valuable, but they suggest that evidence representation and summarization may explain a substantial portion of the observed performance gap. Since the diagnostic configurations vary multiple aspects of the pipeline, they do not provide a clean causal decomposition.
I would like to see a controlled comparison in which a strong baseline uses the same webpage extraction, summarization, answer-generation procedure, and comparable inference budget as MultiSearch-R, differing primarily in its tool-selection and planning strategy.
Without such comparisons, the experiments establish that the integrated system performs well, but provide weaker evidence that reasoning-driven source planning is the principal reason for its advantage.
- Concerns regarding the reliability and independence of final-answer evaluation
This is my most substantial concern regarding the reported quantitative results.
The paper uses GPT-4o-mini extensively for benchmark generation, reference-answer generation, webpage summarization, response rewards, and final-answer evaluation. Using an LLM judge is not inherently problematic, but the degree of overlap between training and evaluation makes independent validation particularly important.
The authors conduct additional human and LLM-judge comparisons in Appendix E. However, Table 10 reports a mean positive bias of 2.242 and a concordance correlation coefficient of only 0.290 for GPT-4o-mini relative to human evaluation. By comparison, Claude Sonnet 4.5 and Gemini 2.5 Flash achieve concordance coefficients of 0.802 and 0.769, respectively.
These findings raise a significant calibration concern for the judge used to produce the principal final-answer scores. Correlation with human judgments is not equivalent to absolute agreement, and a systematically lenient judge may not preserve the magnitude of performance differences between systems.
The reported validation also appears focused on answers generated by MultiSearch-R rather than a balanced evaluation of the competing methods. Consequently, it is not clear whether the primary judge evaluates all methods with comparable reliability.
The key-point-based evaluation further measures agreement with LLM-generated reference answers, which is not necessarily equivalent to independently verified factual correctness or evidence grounding.
I would strongly encourage the authors to reproduce the principal comparisons using independent judges and blinded human evaluations across all major methods. Given the reported calibration results, this is important for interpreting the headline answer-quality gains.
- Several conclusions are stronger than the experimental evidence supports
The paper places considerable emphasis on the “High Recall, Low Quality” phenomenon. The reported observation is useful: for example, at the 8B scale, Multi Search Filter achieves higher tool-selection F1 than MultiSearch-R (0.5810 versus 0.4873), but much lower final-answer quality (0.5247 versus 0.8623).
However, the broader observation that retrieval accuracy does not necessarily translate into downstream task success is already well established in retrieval and tool-use research. The novelty here would lie in explaining the phenomenon specifically for overlapping search sources.
The current experiments do not fully establish that explanation. In fact, Appendix F suggests that evidence acquisition and processing may be at least as important as the source-selection mechanism. I therefore find the empirical observation convincing within the benchmark, but the interpretation that it establishes the necessity of the proposed reasoning-driven planning approach is less convincing.
Similarly, the claims regarding improved efficiency require qualification. While search calls and iteration counts decrease after training at the larger model scales, reported output-token consumption increases across all three scales. The paper also does not provide a comprehensive account of latency or total operational costs, including the summarization and retrieval components.
Finally, the paper motivates its approach in terms of open-ended, multi-source deep research. However, the benchmark consists primarily of questions generated from predefined evidence with corresponding reference answers. This does not fully test research settings involving conflicting evidence, uncertain conclusions, source credibility, or genuinely open-ended synthesis.
I would recommend framing the conclusions more narrowly around the demonstrated source-constrained question-answering environment.
- Presentation and technical clarity
The paper is generally understandable, and the overall pipeline is clearly presented. Nevertheless, several claims could be stated more precisely.
In particular, expressions such as “paradigm shift,” “unexplored question,” and “paving the way” overstate the scope of the contribution relative to existing research.
The discussion of judge reliability also needs revision in view of the poor absolute agreement reported for GPT-4o-mini. There are minor grammatical and editorial issues throughout, and some implementation details would benefit from clearer specification.
These are secondary concerns relative to the experimental and methodological issues above, but improving the precision of the writing would make the actual contribution easier to assess.
Thank you for the response. My W1 remains unaddressed as the authors did not provide experiments beyond its own benchmark and I see no revision of moving more experiments into the main paper (still in Appendix). In recognition of the authors' effort, I will increase the informativeness and soundness score.