- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes LMM-Searcher, a framework for long-horizon multimodal deep search. The framework stores retrieved or processed images in an external file system, retains only textual identifiers (UIDs) in the context, and loads visual content on demand via tools such as fetch-image to reduce context overhead while maintaining access to historical visual evidence. In addition, the authors design a data synthesis pipeline based on multimodal webpage QA and multi-hop knowledge graph extension. Combining synthesized data with open-source datasets, they collect approximately 12K trajectories via rejection sampling to perform multi-turn SFT on Qwen3-VL-30B-A3B-Thinking, followed by parameter merging with the text-only search model MiroThinker-1.7-mini. Experiments across four multimodal search benchmarks demonstrate that the proposed method substantially improves over the base model and supports scaling up to 100 interaction turns.
Critical strengths
- Offloading visual assets externally and loading them on demand via fetch-image provides a sensible and practical approach to reducing visual encoding overhead.
- The paper is clearly written and easy to follow.
Critical weaknesses
- Limited standalone contribution of the proposed data synthesis pipeline. The newly synthesized dataset contains only 1,589 samples (accounting for just 12.5% of the 12,736 total training trajectories), and its average interaction length (17.26 turns) is shorter than that of the open-source REDSearcher-Text dataset (21.97 turns), which contradicts the claim in the main text that the pipeline produces "significantly longer trajectories." Furthermore, in the data ablation (Table 3), using open-source visual data alone (combined with model merging) raises the average score from 25.6 to 41.1 (accounting for 94% of the total improvement), whereas adding the proposed synthesized data yields an average gain of only 1.2 points (40.9 42.1, while MMSearch+ slightly drops from 33.8 to 32.9 compared to using open-source visual data alone). Thus, the standalone benefit of the proposed data synthesis pipeline appears very limited.
- Multiple numerical inconsistencies across experimental tables. In Table 4, the first row labeled as the Seed-1.8 baseline (9.8 / 14.4 / 16.0 / 62.0 / 25.6) is identical to the scores of Qwen3-VL-30B-A3B-Thinking in Table 1 and Table 3, indicating an obvious table entry error. Moreover, between Table 1 and Table 2, the scores of GPT-5 and Seed-1.8 change substantially on MMBC (23.7 36.5 and 25.5 35.1), yet remain completely identical across the other three benchmarks.
Questions for authors
See Weaknesses.
5 comments
Thanks for the authors' detailed reply. My concerns have been well addressed. I still have three follow-up questions:
-
Are samples that reach the interaction limit without a final answer counted as unsuccessful? When is the additional summarization prompt in Table 6 triggered?
-
Under the 100-turn setting, how does the average input context length per model call, including image tokens, compare between loading all retrieved images and using fetch-image on demand, with the same history-retention policy?
-
What distinguishes the query-synthesis pipeline from prior work such as REDSearcher? A concrete example illustrating why the synthesized query requires images from subsequently retrieved webpages would help clarify the contribution.
Based on the rebuttal and additional experiments, I am raising my rating from 1 to 2.
Official Response to Reviewer jMim
We thank the reviewer for the follow-up questions and for reconsidering the paper. We address each question below.
Follow-up Comment 1: Scoring at the interaction limit and the finalization prompt.
Response 1: When the prompt is used and how answers are scored.
For the benchmark scores, reaching the interaction limit does not automatically make a sample unsuccessful. When the search loop ends, we issue the finalization prompt in Table 6. This happens when the agent reaches the turn limit or stops requesting tools. The prompt asks the model to give a final answer based on the evidence already collected. It does not start another search round. We then evaluate the final answer. If no valid answer is available after this step, or the answer is incorrect, the sample is counted as unsuccessful. This setting aligns with previous works like MiroThinker and Vision-Deep-Research.
Figure 4 uses a completion-within-budget criterion. A trajectory that remains unfinished at a given turn threshold is counted as unsuccessful at that threshold. The curves are computed from the same long rollouts by applying different thresholds. They are not separate evaluation runs at each turn limit.
Follow-up Comment 2: Average input context per model call under the same history-retention policy.
Response 2: Loading all images vs. fetching images on demand.
We fix the history-retention policy to keep-all and compare two image-loading strategies under a 100-turn limit. Load-all automatically loads candidate images returned by the tools. Fetch-image loads an image only when the agent requests it. Both retain all historical tool responses. We use the same recorded MMBC trajectories from LMM-Searcher. Under this comparison, on-demand fetching reduces the average input length by 24.8%, with the history-retention policy held fixed. We also report the Mean Cumulative Context Length in Table 7 of the paper, showing an approximately 26% reduction in token consumption over the entire trajectory.
| Image-loading strategy | Mean input tokens per model call ↓ |
|---|---|
| Load all retrieved images | 68,910 |
| Fetch images on demand | 51,833 |
Follow-up Comment 3: The distinction from REDSearcher and an example of later webpage-image use.
Response 3: Linking the final answer to an image retrieved during search.
Both pipelines use multi-hop graphs and entity fuzzification. The distinction we emphasize is how the visual question is built. We start from an image-rich webpage and create a question about a detail in one of its images. We then extend the entity clues into a multi-hop search question. The final visual question remains tied to the webpage image.
For example, one question starts from a webpage photo of Maksudur Rahman Mostak and asks: "What word is printed in large letters on the front of the man's sweatshirt?" We replace the person's name with multi-hop clues based on relations between entities. The solver first follows these clues to identify the person. It then locates the relevant webpage photo and reads the word on the sweatshirt. The person's identity and the entity relations alone do not specify that word.
In this design, the image needed for the final answer is retrieved during search. An image used to identify an intermediate entity does not replace this final visual step. This is the dependency that our synthesis pipeline is designed to introduce. The tool-call distribution provides supporting observations, but the concrete question shows more directly how the final answer is tied to later visual evidence.
Official Response to Reviewer jMim
We thank the reviewer for the careful assessment and constructive feedback. We appreciate the opportunity to clarify the contribution of our data synthesis pipeline and address the inconsistencies in the reported results.
Comment 1: Limited standalone contribution of the proposed data synthesis pipeline.
Response 1.1: Scope of the trajectory-length claim.
We agree that “significantly longer trajectories” is too broad. Table 5 shows that trajectories collected from our synthesized queries average 17.26 turns, compared with 5.63 for FVQA, 7.16 for LiveVQA, and 13.21 for REDSearcher-MM. We will therefore revise the claim to state that our synthesized queries yield longer average trajectories than the three evaluated open-source multimodal query sources.
Response 1.2: Incremental benefit of the synthesized data.
In the paper, we conduct cumulative data ablations by incrementally adding data sources to the training set, aiming to demonstrate that our synthesized data provide additional gains beyond existing data resources. We agree, however, that the reviewer's request for a fair comparison across data sources is well motivated. We therefore conduct an additional ablation in which we randomly sample 1,500 examples from each data source and train a separate model on each sampled set for five epochs, matching the number of training examples and epochs across conditions.
The results show that the existing text-only search data yield the smallest improvement, as the model learns little about how to search for images online. The open-source multimodal search data improve the average score from 25.6 to 33.3, while our synthesized data improve it from 25.6 to 35.8. We have included this comparison in the main text.
Regarding the MMSearch-Plus decrease in the original cumulative ablation, the score drops from 33.8 to 32.4 when textual queries are added to the visual-query training set. Adding our synthesized queries then improves the score from 32.4 to 32.9. Thus, our synthesized data improve performance relative to the same mixture without them. The separate 1,500-example ablation further indicates that our synthesized data are more effective on MMSearch-Plus than either of the evaluated open-source data groups under matched sample-count and epoch settings.
Comment 2: Numerical inconsistencies across experimental tables.
Response 2: Numerical consistency across tables.
We sincerely thank the reviewer for their careful reading of our manuscript. The first row of Table 4 inadvertently reported the Qwen results. We have replaced them with the Seed-1.8 results, consistent with the Direct Answer row for Seed-1.8 in Table 2. We have also standardized the tool ablation to a maximum of 30 interaction turns and updated the affected MMBC scores, averages, and performance differences accordingly.
Regarding Tables 1 and 2, Section 5.1 specifies a 30-turn evaluation budget, whereas Section 5.2.2 specifies 50 turns for the framework comparison. The scores in Table 2 were evaluated under a 50-turn budget. The MMSearch-Plus, VisBrowse, and MMSearch scores for GPT-5 and Seed-1.8 were mistakenly copied from the 30-turn results in Table 1. We have corrected these entries. Since Qwen completed all tasks within 20 turns, its scores are identical under the 30-turn and 50-turn budgets.