- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the post-training of multimodal large language models (MLLMs) for multimodal web search. The authors systematically study three aspects: (1) evaluation protocol sensitivity, demonstrating that evaluation choices significantly affect measured performance; (2) supervised fine-tuning, analyzing how the tool-call distribution in training trajectories shapes student agent behavior; and (3) reinforcement learning, proposing a branching rollout method that enables step-level credit assignment using only terminal rewards. Experiments are conducted on seven multimodal search benchmarks, and the proposed method achieves competitive results compared to direct-inference and agentic-inference baselines.
Critical strengths
- Multimodal web search is an important and actively researched direction. The systematic study of post-training aspects (evaluation, SFT, RL) is timely and provides useful guidance for the community.
- The sensitivity analysis of evaluation components (summarizer, judge model, judge prompt) is valuable and highlights a real problem: lack of standardized evaluation protocols makes method comparison difficult. The recommendation for a shared harness with fixed open-source models is practical and actionable.
- Seven benchmarks provide broad coverage across different types of multimodal search tasks, and models achieves competitive performances.
Critical weaknesses
- The experiments on the evaluation setup lack meaningful insights. They merely demonstrate that different configurations affect the results, but fail to provide any guidance on which setting should be adopted for multimodal search.
- Regarding the impact of SFT on model behavior, while the statistical metrics support the authors' claims, a more fine-grained analysis is needed. Specifically, the authors should clarify whether SFT primarily alters the distribution of tool types, or if it changes sequence lengths and reasoning patterns. Additionally, although Claude 4.6 is used as the teacher model, its performance should be reported in Figure 2 for proper reference.
Questions for authors
- Authors should include comparisons with more relevant models, such as RedSearcher and Point-Seeker.
5 comments
Rebuttal Response
W2: Fine-grained analysis of SFT behavior and teacher reference
We thank the reviewer for this suggestion which makes the manuscript strong. We conducted additional analyses to distinguish the effects of SFT on tool-use distribution, sequence lengths, and reasoning patterns. These findings complement Section 3.3 of main paper by providing a more fine-grained understanding of the behavior transferred from the teacher (Claude 4.6), and we will revise the manuscript accordingly.
First, consistent with Section 3.3, SFT shifts the student's tool-use distribution toward the training corpus. However, some changes in tool frequencies are influenced by the interaction protocol, which requires an image-search operation at the first turn. To isolate learned behavior, we additionally examine decisions where the model has free choice:
| Tool-use metric | Teacher (Claude 4.6) | SFT | Base |
|---|---|---|---|
| Web visits (% of research calls) | 24.3% | 25.5% | 13.2% |
| Queries per text search | 2.58 | 2.20 | 1.27 |
| URLs per web visit | 1.67 | 1.55 | 1.00 |
These results further support our observation that SFT transfers the teacher's tool-use behavior, not only in tool selection but also in how tools are invoked, including the number of queries per search and URLs examined per visit.
Second, SFT does not substantially alter per-turn reasoning length. The median reasoning length remains similar (106 tokens for Base vs. 104 for SFT), as does the 90th percentile (199 vs. 200). However, SFT changes final-answer behavior: median answer length increases from 33 to 157 tokens, and 97% of SFT answers follow the required format, compared with 7% for Base. Furthermore, the ratio of excessively long rollout lengths to the median decreases from 3.7× to 1.2×, indicating more consistent sequence lengths.
Third, SFT also affects reasoning patterns, particularly their surface structure. For example, 92% of SFT first-turn reasoning follows the teacher's opening structure, compared with 10% for Base. However, the transfer of underlying reasoning decisions is more limited: on decisions where the teacher and Base differ, SFT moves 45% of the way toward the teacher. Similarly, the tendency to commit to an answer increases from 3.7% (Base) to 20.6% (SFT), compared with 40.5% for the teacher. Only 0.99% of SFT reasoning sentences match teacher trajectories verbatim, suggesting that these changes are not simply due to memorization.
Overall, these findings reinforce our conclusion in Section 3.3 that SFT distills the tool-use behavior represented in the training corpus, while clarifying that this transfer extends beyond tool-type distributions to tool invocation patterns, answer formatting, and reasoning structure, without substantially changing per-turn reasoning length.
The training corpus consists of Claude 4.6-generated trajectories after quality filtering. Thus, the "Corpus" row in Figure 2(b) already represents the teacher's tool-use behavior. We will relabel it as "Claude 4.6 (Training Corpus)" to make this explicit, include Claude 4.6's benchmark performance as a reference in Figure 2(a), and incorporate the additional behavioral analysis into the revised manuscript.
I appreciate the authors’ response, which has clarified all my concerns. I will maintain my original rating.
Rebuttal Response
We thank the reviewer for the constructive feedback and for recognizing the importance of our systematic study. We address the concerns below and will incorporate the suggested clarifications and comparisons in the revised manuscript.
W1: Actionable insights from the evaluation sensitivity analysis
We agree that identifying evaluation sensitivity is most useful when accompanied by concrete recommendations. Our experiments provide several actionable insights beyond demonstrating that evaluation configurations affect performance.
First, the choice of summarizer affects not only the reported accuracy but also the agent's search trajectory, as it determines which retrieved evidence is presented to the policy. We observe changes of up to 6.00 percentage points when varying the summarizer alone. Second, judge prompts introduce systematic differences: switching from recall-based to stricter semantic-equivalence judging generally reduces measured accuracy by up to 5.52 points, whereas changing the judge model produces smaller and less consistent effects. These findings suggest that evaluation components should not be treated as interchangeable.
Based on these observations, we recommend a standardized evaluation harness with a fixed open-source summarizer and judge, explicitly specified judge prompts, and matched interaction budgets. In particular, Qwen3-235B-A22B provides a reproducible open-source reference configuration, for which we already report additional results in Appendix A. We retain the Claude Sonnet summarizer and GPT-5-mini judge for our headline results to maintain comparability with existing evaluation conventions.
Importantly, our analysis also shows that increasing the interaction budget does not consistently improve performance. Therefore, we recommend reporting both matched-budget and native-budget evaluations, along with tool calls and inference costs, to distinguish improvements in the search policy from differences in the surrounding evaluation system.
We will revise Section 3.2 to make these practical recommendations more explicit and clarify the rationale for the adopted evaluation configuration.
Q1: Comparisons with REDSearcher and POINTS-Seeker
We thank the reviewer for pointing out these relevant works. We agree that REDSearcher (Chu et al., 2026) and POINTS-Seeker (Liu et al., 2026) are important baselines for multimodal agentic search. We provide their published results alongside ours below.
Comparison on multimodal search benchmarks (accuracy, %):
The REDSearcher and POINTS-Seeker numbers are taken from their respective published results; our numbers are from Table 4. A dash indicates that the corresponding result is not reported.
We will update Table 4 to include both REDSearcher and POINTS-Seeker, with the appropriate citations, and expand the related-work discussion to position our approach relative to these methods.