acceptodds
Under review as a conference paper at ICLR 2027

Towards Long-horizon Agentic Multimodal Search

Abstract

Multimodal deep search agents can solve complex tasks by iteratively gathering textual and visual evidence, but long-horizon reasoning is hindered by managing accumulated multimodal contexts while preserving visual evidence for continued use. To address this, we propose LMM-Searcher, a long-horizon multimodal deep search framework built on a file-based visual representation mechanism. By storing visual assets externally and referencing them with lightweight textual identifiers (UIDs), our approach reduces context overhead while preserving multimodal information for future retrieval. We further introduce a data synthesis pipeline for generating complex cross-modal multi-hop reasoning data. Using 12K synthesized trajectories, we fine-tune Qwen3-VL-Thinking-30A3B into a specialized multimodal deep search agent. Experiments on four benchmarks show that LMM-Searcher scales to 100-turn search horizons and achieves state-of-the-art performance among open-source models on challenging benchmarks such as MM-BrowseComp and MMSearch-Plus, while maintaining strong cross-model generalization. Code, data, and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal, large language model, search agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jMimOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes LMM-Searcher, a framework for long-horizon multimodal deep search. The framework stores retrieved or processed images in an external file system, retains only textual identifiers (UIDs) in the context, and loads visual content on demand via tools such as fetch-image to reduce context overhead while maintaining access to historical visual evidence. In addition, the authors design a data synthesis pipeline based on multimodal webpage QA and multi-hop knowledge graph extension. Combining synthesized data with open-source datasets, they collect approximately 12K trajectories via rejection sampling to perform multi-turn SFT on Qwen3-VL-30B-A3B-Thinking, followed by parameter merging with the text-only search model MiroThinker-1.7-mini. Experiments across four multimodal search benchmarks demonstrate that the proposed method substantially improves over the base model and supports scaling up to 100 interaction turns.

Critical strengths
  1. Offloading visual assets externally and loading them on demand via fetch-image provides a sensible and practical approach to reducing visual encoding overhead.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Limited standalone contribution of the proposed data synthesis pipeline. The newly synthesized dataset contains only 1,589 samples (accounting for just 12.5% of the 12,736 total training trajectories), and its average interaction length (17.26 turns) is shorter than that of the open-source REDSearcher-Text dataset (21.97 turns), which contradicts the claim in the main text that the pipeline produces "significantly longer trajectories." Furthermore, in the data ablation (Table 3), using open-source visual data alone (combined with model merging) raises the average score from 25.6 to 41.1 (accounting for 94% of the total improvement), whereas adding the proposed synthesized data yields an average gain of only 1.2 points (40.9 →\to 42.1, while MMSearch+ slightly drops from 33.8 to 32.9 compared to using open-source visual data alone). Thus, the standalone benefit of the proposed data synthesis pipeline appears very limited.
  2. Multiple numerical inconsistencies across experimental tables. In Table 4, the first row labeled as the Seed-1.8 baseline (9.8 / 14.4 / 16.0 / 62.0 / 25.6) is identical to the scores of Qwen3-VL-30B-A3B-Thinking in Table 1 and Table 3, indicating an obvious table entry error. Moreover, between Table 1 and Table 2, the scores of GPT-5 and Seed-1.8 change substantially on MMBC (23.7 →\to 36.5 and 25.5 →\to 35.1), yet remain completely identical across the other three benchmarks.
Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer jMim

We thank the reviewer for the careful assessment and constructive feedback. We appreciate the opportunity to clarify the contribution of our data synthesis pipeline and address the inconsistencies in the reported results.

Comment 1: Limited standalone contribution of the proposed data synthesis pipeline.

Response 1.1: Scope of the trajectory-length claim.

We agree that “significantly longer trajectories” is too broad. Table 5 shows that trajectories collected from our synthesized queries average 17.26 turns, compared with 5.63 for FVQA, 7.16 for LiveVQA, and 13.21 for REDSearcher-MM. We will therefore revise the claim to state that our synthesized queries yield longer average trajectories than the three evaluated open-source multimodal query sources.

Response 1.2: Incremental benefit of the synthesized data.

In the paper, we conduct cumulative data ablations by incrementally adding data sources to the training set, aiming to demonstrate that our synthesized data provide additional gains beyond existing data resources. We agree, however, that the reviewer's request for a fair comparison across data sources is well motivated. We therefore conduct an additional ablation in which we randomly sample 1,500 examples from each data source and train a separate model on each sampled set for five epochs, matching the number of training examples and epochs across conditions.

MMBCMMSearch+VisBrowseMMSearchAvg.
Qwen3-VL-30B-A3B-Thinking9.814.416.062.025.6
Open-source Visual Query (1500)14.322.527.468.933.3
Open-source Textual Query (1500)12.618.422.264.029.3
Our Synthesized Query (1500)18.125.930.069.035.8

The results show that the existing text-only search data yield the smallest improvement, as the model learns little about how to search for images online. The open-source multimodal search data improve the average score from 25.6 to 33.3, while our synthesized data improve it from 25.6 to 35.8. We have included this comparison in the main text.

Regarding the MMSearch-Plus decrease in the original cumulative ablation, the score drops from 33.8 to 32.4 when textual queries are added to the visual-query training set. Adding our synthesized queries then improves the score from 32.4 to 32.9. Thus, our synthesized data improve performance relative to the same mixture without them. The separate 1,500-example ablation further indicates that our synthesized data are more effective on MMSearch-Plus than either of the evaluated open-source data groups under matched sample-count and epoch settings.

Comment 2: Numerical inconsistencies across experimental tables.

Response 2: Numerical consistency across tables.

We sincerely thank the reviewer for their careful reading of our manuscript. The first row of Table 4 inadvertently reported the Qwen results. We have replaced them with the Seed-1.8 results, consistent with the Direct Answer row for Seed-1.8 in Table 2. We have also standardized the tool ablation to a maximum of 30 interaction turns and updated the affected MMBC scores, averages, and performance differences accordingly.

Regarding Tables 1 and 2, Section 5.1 specifies a 30-turn evaluation budget, whereas Section 5.2.2 specifies 50 turns for the framework comparison. The scores in Table 2 were evaluated under a 50-turn budget. The MMSearch-Plus, VisBrowse, and MMSearch scores for GPT-5 and Seed-1.8 were mistakenly copied from the 30-turn results in Table 1. We have corrected these entries. Since Qwen completed all tasks within 20 turns, its scores are identical under the 30-turn and 50-turn budgets.

Thanks for the authors' detailed reply. My concerns have been well addressed. I still have three follow-up questions:

  1. Are samples that reach the interaction limit without a final answer counted as unsuccessful? When is the additional summarization prompt in Table 6 triggered?

  2. Under the 100-turn setting, how does the average input context length per model call, including image tokens, compare between loading all retrieved images and using fetch-image on demand, with the same history-retention policy?

  3. What distinguishes the query-synthesis pipeline from prior work such as REDSearcher? A concrete example illustrating why the synthesized query requires images from subsequently retrieved webpages would help clarify the contribution.

Based on the rebuttal and additional experiments, I am raising my rating from 1 to 2.

AuthorsOpenReview ↗

Official Response to Reviewer jMim

We thank the reviewer for the follow-up questions and for reconsidering the paper. We address each question below.

Follow-up Comment 1: Scoring at the interaction limit and the finalization prompt.

Response 1: When the prompt is used and how answers are scored.

For the benchmark scores, reaching the interaction limit does not automatically make a sample unsuccessful. When the search loop ends, we issue the finalization prompt in Table 6. This happens when the agent reaches the turn limit or stops requesting tools. The prompt asks the model to give a final answer based on the evidence already collected. It does not start another search round. We then evaluate the final answer. If no valid answer is available after this step, or the answer is incorrect, the sample is counted as unsuccessful. This setting aligns with previous works like MiroThinker and Vision-Deep-Research.

Figure 4 uses a completion-within-budget criterion. A trajectory that remains unfinished at a given turn threshold is counted as unsuccessful at that threshold. The curves are computed from the same long rollouts by applying different thresholds. They are not separate evaluation runs at each turn limit.

Follow-up Comment 2: Average input context per model call under the same history-retention policy.

Response 2: Loading all images vs. fetching images on demand.

We fix the history-retention policy to keep-all and compare two image-loading strategies under a 100-turn limit. Load-all automatically loads candidate images returned by the tools. Fetch-image loads an image only when the agent requests it. Both retain all historical tool responses. We use the same recorded MMBC trajectories from LMM-Searcher. Under this comparison, on-demand fetching reduces the average input length by 24.8%, with the history-retention policy held fixed. We also report the Mean Cumulative Context Length in Table 7 of the paper, showing an approximately 26% reduction in token consumption over the entire trajectory.

Image-loading strategyMean input tokens per model call ↓
Load all retrieved images68,910
Fetch images on demand51,833

Follow-up Comment 3: The distinction from REDSearcher and an example of later webpage-image use.

Response 3: Linking the final answer to an image retrieved during search.

Both pipelines use multi-hop graphs and entity fuzzification. The distinction we emphasize is how the visual question is built. We start from an image-rich webpage and create a question about a detail in one of its images. We then extend the entity clues into a multi-hop search question. The final visual question remains tied to the webpage image.

For example, one question starts from a webpage photo of Maksudur Rahman Mostak and asks: "What word is printed in large letters on the front of the man's sweatshirt?" We replace the person's name with multi-hop clues based on relations between entities. The solver first follows these clues to identify the person. It then locates the relevant webpage photo and reads the word on the sweatshirt. The person's identity and the entity relations alone do not specify that word.

In this design, the image needed for the final answer is retrieved during search. An image used to identify an intermediate entity does not replace this final visual step. This is the dependency that our synthesis pipeline is designed to introduce. The tool-call distribution provides supporting observations, but the concrete question shows more directly how the final answer is tied to later visual evidence.

Reviewer_bXayOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents LMM-Searcher, a long-horizon multimodal search agent that stores images outside the context, keeps textual UIDs or URLs as references, and retrieves images through a fetch-image tool when needed. The authors also synthesize multi-hop questions from visually rich webpages and use 12,736 successful teacher trajectories to fine-tune a Qwen3-VL-based model, followed by model merging. The system is evaluated on four search benchmarks, with additional gains reported when the interaction limit is increased to 100 turns.

Critical strengths
  1. The same-backbone framework comparisons in Table 2 are useful. They give evidence that the search workflow can help models other than the fine-tuned LMM-Searcher, rather than relying only on a comparison against older published systems.

  2. The engineering choice is practical for image-heavy browsing. Search results and scraped pages can return far more images than an agent needs to inspect, and the Appendix case illustrates how a retrieved image can be examined only after the search has narrowed down the target.

Critical weaknesses
  1. The new part of the visual-memory mechanism is hard to identify. Visual ChatGPT already keeps images in external files and uses textual filenames to pass them between visual tools. LMM-Searcher uses URL-based handles (Figure 1) and adds fetch-image to a web-search workflow, but Section 3 does not introduce a clear rule for selecting, retaining, or recovering visual evidence. The asserted one-to-one image/UID mapping is also underspecified: are these stable storage URLs or source URLs, and what happens if the source URL changes? The distinction from a prior file-reference interface needs to be more precise.

  2. The experiments do not separate image access from context management. Removing fetch-image in Table 4 also prevents the agent from inspecting retrieved images, so the accuracy drop is not an ablation of UID references alone. Table 2 changes the search and browsing workflow, not just the memory interface. A file-reference agent with the same tools and image access is the natural control. There are also inconsistencies in the reported Seed-1.8 results: the initial row in Table 4 matches the Qwen3-VL agent-workflow row in Table 1, while the Seed-1.8 MMBC result differs between Tables 1 and 2.

  3. The evidence for long-horizon visual memory is limited. In the 100-turn setting only the five most recent tool results are retained (Section 5.1), even though the proposed mechanism depends on keeping textual handles for older images. It is unclear how an old UID remains discoverable after the tool output containing it is discarded. The Appendix case fetches images shortly after they are found, rather than after many intervening turns. The paper lacks delayed-retrieval statistics, context-cost measurements, or a comparison that separates extra turns from the history-truncation policy.

  4. The synthesis pipeline substantially overlaps with REDSearcher, which already uses multi-hop graphs, entity fuzzification, and intermediate visual constraints. Starting from image-rich webpages may create a different dependency, but Section 4 does not show how this changes the resulting tasks. The Table 3 average rises only from 40.9 to 42.1 when the proposed data are added, without a matched-data-budget control. Figure 3 shows tool-call proportions rather than proving that later webpage images are necessary, and Appendix Table 5 shows shorter average trajectories than REDSearcher-Text, despite the broad claim about longer trajectories.

Questions for authors
  1. What specific selection, retention, or re-access behavior is new relative to a filename-based visual agent with the same search and image tools? Are the UIDs stable local identifiers or webpage URLs?

  2. Can the authors compare UID-based loading, ordinary file references, and image retention in context using the same model, tools, visual access, and turn budget? Please include accuracy and cumulative context cost.

  3. With only five recent tool results retained, how are older image UIDs recovered in 100-turn runs? Could the authors report re-fetch success by the age of the original observation and isolate the effect of history truncation?

  4. Which synthesized questions require visual evidence obtained from later webpages, rather than a REDSearcher-style intermediate visual bottleneck? Does the gain persist against the same number of alternative synthetic examples?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer bXay - Part 1

We thank the reviewer for the careful assessment and constructive feedback. Our responses are provided below.

Comment 1: The distinction from existing file-reference interfaces and the handling of image UIDs are unclear.

Response 1.1: An image-as-file harness for autonomous multimodal search.

Our goal is to develop a harness that better supports vision-language models in autonomous search. Visual ChatGPT uses a prompt manager to coordinate interactions between ChatGPT and visual foundation models. Our emphasis is on providing an harness in which a single search agent can decide which visual evidence to inspect, when to inspect it, and which tools to apply. Once the tool interfaces are specified, the agent determines the sequence of search, image retrieval, and image manipulation actions without manual intervention. The contribution we emphasize is the integration of this image-as-file abstraction into an autonomous multimodal search harness. It gives the agent control over image access and manipulation, supporting flexible tool-use sequences and on-demand visual inspection rather than prescribing when images must be loaded.

Response 1.2: Image storage, reference stability, and retrieval.

The underlying system handles image storage and loading. Images returned by tools, including search engines, webpage extraction, and image operations such as zoom-in, are stored in object storage and assigned storage URLs. These URLs serve as the image UIDs. They identify the stored copies, rather than merely referring to the original webpage image locations.

An image and its storage URL remain available throughout the lifetime of the query. The URL does not change during this period, and the associated storage is released after the query is completed. Consequently, a change to the source webpage URL after the image has been stored does not change the handle for that stored copy. When the agent supplies a storage URL to an image-access tool, the system resolves the reference and loads the image for the requested operation; images returned to the agent are inserted into its context.

The agent does not need to manage the storage backend. It only decides which images to inspect and how to use them. Storage persistence and the visibility of a reference in the model's current context are separate issues.

Comment 2: The experiments do not isolate image access from context management, and some reported results are inconsistent.

Response 2.1: Comparing image-loading and context-retention strategies.

In our implementation, UID-based loading is a form of file-reference access: the UID identifies an externally stored image that can be retrieved through a tool. We therefore do not treat a UID and an ordinary file reference as distinct memory mechanisms. The comparison of interest is between automatic image loading and on-demand fetching, under matched context-retention policies.

We compare the following four settings:

SettingAccuracy ↑Mean Cumulative Context Length ↓Max Cumulative Context Length ↓Estimated Mean KV-cache ↓Main-Model Request Time (min/task) ↓
load all + keep all25.06111,015130,21410.16 GiB75.8
load all + keep 527.3228,933102,3722.65 GiB22.2
fetch image + keep all27.8581,950128,0727.50 GiB56.7
fetch image + keep 530.0926,60994,2962.44 GiB19.1

Comparing the first and third rows holds the keep-all policy fixed, while comparing the second and fourth rows holds the keep-5 policy fixed. In both comparisons, on-demand fetching achieves higher accuracy and lower context usage than full loading. These comparisons support the benefit of on-demand image access under both retention policies, rather than a distinction between UIDs and filenames.

Response 2.2: Corrections to the experimental tables.

We thank the reviewer for identifying the numerical inconsistencies. Some entries were inadvertently copied from other experimental settings. We have corrected the affected results in Tables 2 and 4. In particular, the first row of Table 4 now reports the Seed-1.8 results rather than the Qwen results. We also clarify that Table 1 uses a 30-turn evaluation budget, whereas the framework comparison in Table 2 uses 50 turns; the tool ablation in Table 4 is standardized to 30 turns.

AuthorsOpenReview ↗

Official Response to Reviewer bXay - Part 2

Comment 3: It is unclear how older image references remain accessible after history truncation.

Response 3: Image-reference accessibility under different history-retention policies.

We thank the reviewer for highlighting this issue. We clarify that on-demand image access and history retention are separate design choices. Under fetch-image + keep-all, historical tool responses retain the URLs and descriptions of candidate images, including images that were not loaded when first encountered. The agent can therefore revisit these records after intervening turns and selectively fetch a previously uninspected image, without having loaded all candidate images into context in advance.

The keep-5 configuration introduces an additional history-truncation policy to reduce context overhead. It retains earlier assistant messages and tool-call arguments, but may remove candidate URLs appearing exclusively in older tool responses. Accordingly, the reference-retention behavior of keep-all should be distinguished from that of keep-5. Image URLs recorded in earlier tool-call arguments therefore remain available for subsequent retrieval. For example, in an MMBC trajectory, the agent successfully fetched the same image at turns 35 and 64, with its URL preserved in the earlier tool-call arguments after the original tool response had left the keep-5 window.

The comparison between load-all + keep-all and fetch-image + keep-all in Response 2 addresses on-demand image loading while holding history retention fixed. It is therefore the relevant comparison for examining selective access to historical visual candidates without introducing the effects of history truncation. Separately, comparing fetch-image + keep-all with fetch-image + keep-5 at the same 100-turn limit varies history retention while holding the image-loading policy and interaction budget fixed.

Comment 4: The distinction from REDSearcher and the contribution of the synthesized data need stronger evidence.

Response 4.1: The distinction from REDSearcher

The distinction we emphasize is the starting point of question synthesis. We begin with an image-rich webpage and construct a visual question grounded in one of its images, such as “what word is printed in large letters on the front of the man's sweatshirt?” The man in the image is Maksudur Rahman Mostak. We then extend the question through relations to Maksudur Rahman Mostak to form a multi-hop query. The resulting task links the extended entity-identification process to locating the relevant image and answering the original visual question. Thus, the pipeline is built around a webpage-grounded visual question whose reasoning chain is subsequently extended. This gurantees that the agent must browse the image in the webpage to answer the question.

In contrast, REDSearcher introduces visual constraints at selected nodes in its reasoning graph, but this does not necessarily require the agent to access the corresponding images during task execution. As shown in Figure 3, REDSearcher-MM trajectories make almost no use of the fetch-image tool and text-search tool accounts for 75% of all tool calls, suggesting that its generated questions can often be answered without retrieving and inspecting the images embedded in the webpages. In our pipeline, however, the original webpage image is directly tied to the final answer, making visual access an essential part of task completion.

Response 4.2: A controlled comparison at a fixed sample budget.

To evaluate the contribution of our data under matched sample-count and epoch settings, we use a common base of 5,000 training trajectories and construct three additional training sets by adding 1,500 trajectories from open-source visual queries, open-source textual queries, or our synthesized queries. We train all four models for three epochs and compare their performance below:

Training dataTrajectoriesMMBCMMSearch+VisBrowseMMSearchAvg.
Common base5,00015.024.530.369.334.8
Common base + open-source visual queries6,50016.526.832.669.736.4
Common base + open-source textual queries6,50016.925.231.169.235.6
Common base + our synthesized queries6,50018.328.534.469.937.8

Among the three 6,500-trajectory conditions, our synthesized data achieve the highest average score of 37.8, compared with 36.4 for open-source visual queries and 35.6 for open-source textual queries. These results support the benefit of our synthesized data when the common base, total number of training examples, and number of epochs are held fixed. The 5,000-trajectory model serves as a reference.

AuthorsOpenReview ↗

Official Response to Reviewer bXay - Part 3

Response 4.3: Scope of the trajectory-length claim.

We agree that the original statement about longer trajectories was too broad. Consistent with our response to Reviewer 3, we narrow the comparison to the three evaluated open-source multimodal query sources. Our synthesized queries produce trajectories averaging 17.26 turns, compared with 5.63 for FVQA, 7.16 for LiveVQA, and 13.21 for REDSearcher-MM.

Reviewer_5ZRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LMM-Searcher stores visual assets outside the context and references them through textual UIDs, with tools for fetching and processing images on demand. The authors collect 12,736 teacher trajectories from existing and newly synthesized queries, fine-tune Qwen3-VL-30B-A3B-Thinking, and merge its language components with MiroThinker. They evaluate the resulting agent on four multimodal search benchmarks.

Critical strengths
  • Persistent image references and on-demand perception are a useful design for searches that revisit visual evidence. The workflow is clearly illustrated.
  • The framework improves strong backbones at the same stated 50-turn cap. For Seed-1.8, MMBC improves from 21.4 to 35.1 and MMSearch-Plus from 11.0 to 46.7. This provides evidence that the overall framework is useful beyond the trained model.
  • The final model substantially improves over its base model across all four benchmarks. Figure 4b includes training-only, merging-only, and combined variants, providing a useful check on the contribution of model merging.
Critical weaknesses
  1. The main visual-memory mechanism is not isolated. Removing fetch-image in Table 4 also removes access to full images through that tool. The accuracy drop therefore supports the importance of visual access, but does not establish the benefit of external storage over retaining the same visual evidence in context. The paper also reports no context-token occupancy, memory usage, or latency measurements, leaving the claimed reduction in context overhead unquantified.
  2. The strongest long-horizon comparison changes both the turn budget and the context policy. MMBC rises from 22.3 to 30.1 when moving from 30 turns to 100 turns while retaining only the latest five tool responses. Competing trained agents are not reported under the same extended setting. Figure 4 shows later correct completions, but does not resolve how much of the gain comes from the proposed memory design. The claim of SOTA across all four benchmarks also needs narrowing: on MMSearch, Table 1 reports 72.3 versus REDSearcher's 72.9.
  3. Evidence for the new query-synthesis pipeline is modest. Adding its 1,589 trajectories raises the average from 40.9 to 42.1, while the visual-only training variant already reaches 41.1 and scores higher on MMSearch-Plus (33.8 versus 32.9). The incremental ablation changes dataset size and training compute, without a matched-size comparison or uncertainty estimates. Longer teacher trajectories and more visual tool calls do not establish that the intended hops are necessary. The per-attribute irreversibility rule also does not exclude shortcuts through combined clues or search results.
  4. Several reporting details need correction. Section 5.2.1 gives 28.7 on MMSearch-Plus, inconsistent with Table 1. The first row of Table 4 is labeled Seed-1.8 but exactly matches the Qwen base-model results. Section 4.1 refers to synthesis prompts in the appendix, which contains dataset statistics and a case study but no such prompts.
Questions for authors
  1. With identical visual access, tools, and turn budgets, how does UID-based storage compare with keeping images in context? Please report context occupancy and inference cost alongside accuracy.
  2. Can the main competing agent be evaluated at 100 turns with the same context policy? Are Figure 4 curves obtained from shared rollouts or separate runs at each cap? Also, is the extra user finalization prompt in Table 6 used during evaluation, and how is it handled by the autonomous-completion criterion?
  3. Does the synthetic-data gain remain when replacing an equal number of existing trajectories at a fixed training budget?
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer 5ZRs - Part 1

We thank the reviewer for the careful assessment and constructive feedback. Our responses are provided below.

Comment 1: The visual-memory mechanism is not isolated, and its context and inference costs are unquantified.

Response 1: Additional ablation on visual access and context management.

Following the reviewer's suggestion, we conduct additional experiments on MMBC to disentangle the effects of visual access and context management. Specifically, we compare two image-loading strategies (load all vs. fetch image) and two context management strategies (keep all vs. keep 5, retaining only the tool responses from the five most recent turns). The following experiments are conducted on LMM-Searcher. We set the turn limit as 100 and the maximum context length as 128K.

SettingAccuracy ↑Mean Cumulative Context Length ↓Max Cumulative Context Length ↓Estimated Mean KV-cache ↓Main-Model Request Time (min/task) ↓
load all + keep all25.06111,015130,21410.16 GiB75.8
load all + keep 527.3228,933102,3722.65 GiB22.2
fetch image + keep all27.8581,950128,0727.50 GiB56.7
fetch image + keep 530.0926,60994,2962.44 GiB19.1

Context lengths are measured in tokens. The KV-cache column donotes the storage associated with the mean context length, assuming BF16 keys and values, 48 layers, four KV heads, and a head dimension of 128, corresponding to 96 KiB per token. The results lead to two observations:

  • Effect of on-demand image fetching. Comparing load all + keep all with fetch image + keep all, on-demand image fetching reduces the mean context length from 111,015 to 81,950 tokens and improves accuracy from 25.06 to 27.85. This suggests that providing the agent with access to relevant visual information on demand is more effective than indiscriminately including all images in the context.
  • Effect of context management. Further, comparing fetch image + keep all with fetch image + keep 5, retaining only the tool responses from the five most recent turns reduces the mean context length from 81,950 to 26,609 tokens, while improving accuracy from 27.85 to 30.09. This demonstrates the effectiveness of the context management strategy in reducing context overhead while improving task performance.

Besides the accuracy and the context length, we also shows the memory usage and the inference cost in the last two columns following the reviewer's suggestion. We can observe the "fetch image + keep 5" also save memory usage and inference time.

Comment 2: The long-horizon comparison changes both the turn budget and context policy, and the evaluation protocol needs clarification.

Response 2.1: Evaluation of a competing agent under the same extended setting.

Since REDSearcher-MM is not publicly available to us for reproduction, we downloaded and deployed Vision-DeepResearch-30B and evaluated it under the same 100-turn cap and keep-5 context policy. The completed results have been added to Table 1:

ModelMMBCMMSearch+VisBrowseMMSearchAvg.
Vision-DeepResearch-30B29.632.745.271.144.7
LMM-Searcher-30B30.134.848.372.346.4

Under the same interaction budget and context policy, LMM-Searcher outperforms Vision-DeepResearch-30B on all four benchmarks, achieving a 1.7-point improvement in average performance. We note that this comparison evaluates the overall system under the extended setting and does not isolate the contribution of the proposed visual-memory mechanism.

AuthorsOpenReview ↗

Official Response to Reviewer 5ZRs - Part 2

Response 2.2: Clarification of the rollout protocol and finalization prompt.

The curves in Figure 4 are obtained by truncating the same set of long-horizon trajectories at different interaction limits and computing the corresponding performance statistics. Therefore, they are based on shared rollouts rather than independent evaluation runs at each interaction limit. We have clarified this in the "interation scaling" paragraph.

The additional user finalization prompt in Table 6 is used during evaluation to request a final answer after the interaction loop. It is triggered in either of the following cases: (1) the agent reaches the maximum interaction limit, or (2) the agent completes its task without requiring further tool calls. In both cases, the prompt asks the agent to summarize its findings and provide a final answer. Therefore, each trajectory will trigger this finalization prompt. This operation aligns with previous work like MiroThinker and the baseline method, Vision-DeepResearch. We have clarified the triggering conditions and their role in the evaluation protocol in the revised manuscript.

For the claimed of SOTA performance, we have narrowed the claim accordingly and explicitly acknowledge the MMSearch exception in the revised manuscript.

Comment 3: The synthetic-data comparison changes sample count and training compute.

Response 3: Comparing data sources at a fixed sample budget.

To address this concern, we conduct a controlled ablation using Qwen3-VL-30B-A3B-Thinking with a common base of 5,000 training trajectories, randomly sampled from a combination of FVQA, LiveVQA, REDSearcher-MM, and REDSearch-Text. We construct three additional training sets, each containing 6,500 trajectories by adding 1,500 trajectories from one of the following sources: open-source visual queries, open-source textual queries, or our synthesized queries. The additional 1,500 trajectories in each condition are required to be disjoint from the common base. All models are trained for 3 epochs to ensure a consistent training budget.

Training dataTrajectoriesMMBCMMSearch+VisBrowseMMSearchAvg.
Common base5,00015.024.530.369.334.8
Common base + open-source visual queries6,50016.526.832.669.736.4
Common base + open-source textual queries6,50016.925.231.169.235.6
Common base + our synthesized queries6,50018.328.534.469.937.8

The primary comparison is among the three 6,500-trajectory conditions, which share the same 5,000-trajectory base and differ only in the source of the additional 1,500 trajectories. The 5,000-trajectory condition serves as a reference. These results demonstrate that the performance gain from our synthesized data persists when comparing different data sources under the same sample count and training budget. We have added these results to the Appendix.

Regarding the original cumulative ablation, the MMSearch-Plus score decreases from 33.8 to 32.4 when textual queries are added to the visual-query training set, and subsequently increases to 32.9 when our synthesized queries are added. Thus, although the final score remains slightly below 33.8, our synthesized data actually improve performance relative to the preceding training configuration.

In addition, following reviewer jMim's suggestion, we conduct a separate data-source ablation by independently training models on 1,500 queries from each data source. The results, reported in Table 3, further demonstrate the effectiveness of our synthesized data.

Comment 4: Numerical inconsistencies and missing synthesis prompts.

Response 4: Corrections and reporting details.

We thank the reviewer for identifying these issues. We correct the MMSearch-Plus value of 28.7 in Section 5.2.1 to 32.9, consistent with Table 1. The first row of Table 4 inadvertently contained the Qwen results, and we have replaced them with the Seed-1.8 results and standardized the tool ablation to a 30-turn cap, updating the affected scores and comparisons accordingly. Finally, we include the actual query-synthesis prompts in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.