acceptodds
Withdrawn from ICLR 2027

Getting the Evidence Right: Adapting Retrieval and Visual Reading for Multimodal Agentic Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal search agents must retrieve evidence that addresses intermediate questions and extract the facts needed to advance their reasoning. Existing approaches typically train the agent while keeping the retriever fixed. Retrieved candidates differ in how fully they support an intermediate query, and even relevant images can be misread. We introduce SPAR (Structuring and Projecting Evidence for Acquisition and Reading), a framework for adapting retrieval and visual reading. SPAR analyzes what each intermediate query requires and records the facts supported by each candidate. This shared analysis yields graded relevance supervision that prioritizes complete support while retaining useful partial evidence. For visual reading, SPAR reuses the image-grounded facts recorded in the same annotations to teach the agent to extract query-relevant information. We then adapt the agent's search policy on complete trajectories with the adapted retriever held fixed. On MultiModalQA, SPAR achieves 76.03% answer accuracy, exceeding the strongest baseline by 8.43 percentage points. Graded relevance supervision outperforms ungraded supervision both before and after policy adaptation. Under fixed evidence, the SPAR agent extracts visual facts more accurately than the policy-only agent. Without further training, SPAR also transfers to WebQA, MuSiQue, and HotpotQA, reaching 61.89% answer accuracy on HotpotQA. We will release our code upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal agentic search, retriever adaptation, graded relevance supervision, visual reading, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Vk8sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents SPAR, a multimodal agentic search framework designed to strengthen both evidence retrieval and image-based information extraction. The approach begins by analyzing agent-generated search trajectories to produce structured annotations that describe the information required by each query, the extent to which retrieved candidates satisfy those requirements, and the factual content supported by visual evidence. These annotations serve two purposes. First, they enable retriever optimization through relevance-aware training objectives that distinguish fully supportive evidence from partially useful candidates. Second, they provide query-specific supervision for improving the agent's interpretation of retrieved images. After these components are adapted, the search agent undergoes further policy optimization with GRPO while the retriever parameters remain unchanged. On MultiModalQA, the resulting system reaches 76.03% answer accuracy, an improvement of 8.43 percentage points over the strongest competing method. The authors also conduct component-level ablations to investigate the effects of retrieval training, relevance grading, and visual reading enhancement. Evaluations on WebQA, MuSiQue, and HotpotQA additionally suggest that the learned capabilities can transfer across multimodal and text-based multi-hop QA settings without task-specific retraining.

Critical strengths

1. Coordinated optimization of retrieval and visual evidence interpretation.

An important merit of this work is its attempt to improve two closely related aspects of multimodal search within a single training framework. Rather than treating retrieval effectiveness and visual comprehension as entirely separate problems, SPAR derives supervision for both from a common set of structured evidence annotations. This design allows the retriever to learn distinctions between different levels of evidential support while simultaneously teaching the agent to identify query-relevant information in images. The connection between these training objectives is reasonably well motivated, and the reuse of supervision across components offers a potentially useful direction for building more capable multimodal search systems.

2. Substantial performance gains supported by multiple experimental analyses.

The experimental findings indicate that SPAR can deliver meaningful improvements over existing approaches. Its 76.03% accuracy on MultiModalQA exceeds the strongest reported baseline by 8.43 percentage points, representing a considerable gain on the primary benchmark. The ablation results also provide evidence that individual training components contribute to overall performance. For example, optimizing the retriever alone increases accuracy from 65.10% to 69.56% when the agent is held fixed. Meanwhile, the evaluation under fixed evidence conditions offers complementary support for improved visual information extraction. Results on WebQA, MuSiQue, and HotpotQA further suggest that the benefits are not confined to the training benchmark or a single modality. Taken together, these experiments provide encouraging empirical support for the effectiveness and transferability of the proposed approach.

Critical weaknesses

1. The unique methodological value of structured evidence modeling is not sufficiently established.

Although SPAR combines retrieval optimization and visual reading enhancement in a reasonably coherent manner, many of its underlying training procedures are based on established methods, including contrastive learning, relevance-aware ranking, supervised visual adaptation, and GRPO. The primary methodological distinction therefore appears to lie in how the framework organizes and reuses evidence supervision. However, the experiments do not establish whether the proposed requirement decomposition and evidence assessment procedures are essential to achieving the reported improvements. A simpler pipeline that directly obtains relevance judgments and visual fact annotations from the same teacher model might provide comparable supervision without the intermediate structure. Without such a comparison, it is difficult to determine how much of SPAR's effectiveness originates from its distinctive annotation design rather than from the additional supervised training itself. This uncertainty limits the strength of its claimed methodological novelty.

2. The evidence for the claimed retrieval and multi-hop reasoning benefits remains largely indirect.

Most experimental conclusions are drawn from end-to-end question-answering performance, with comparatively little analysis of the retrieval behavior that the proposed method is intended to improve. In particular, the absence of retrieval-oriented measurements makes it difficult to assess whether the learned ranking function consistently identifies fully relevant and partially relevant evidence more effectively. The role of partial-support evidence is especially underexplored. Although preserving such candidates is presented as an important design principle, the paper does not directly show whether agents subsequently exploit these candidates to resolve missing information or complete multi-step reasoning chains. Accordingly, the observed advantage of graded relevance training might result primarily from improved ranking of fully supportive documents, rather than from the intended use of incomplete evidence across successive search steps. This leaves a central explanation for the method's effectiveness insufficiently verified.

Questions for authors

1. Added value of the structured annotation pipeline.

What advantages does the proposed query-requirement decomposition and evidence assessment process offer over directly obtaining relevance judgments and visual fact annotations from the same teacher model? A controlled comparison would help determine whether the intermediate structured representation is necessary for the observed improvements.

2. Role of partially supportive evidence in multi-hop search.

Can the authors provide empirical evidence that partially relevant candidates are actually used during later search or reasoning steps? For instance, an analysis of their frequency and contribution within successful multi-hop trajectories would help distinguish genuine improvements in evidence utilization from better ranking of fully supportive results.

3. Retrieval performance beyond end-to-end accuracy.

Could the authors include dedicated retrieval evaluations using metrics such as Recall@K, MRR, and graded NDCG? It would be particularly informative to examine whether SPAR improves the discovery and ordering of both fully supportive and partially supportive candidates, rather than relying exclusively on final QA accuracy.

4. Quality and robustness of automatically generated supervision.

Since the training process relies extensively on annotations produced by Qwen2.5-VL-72B-Instruct, how reliable are its support-level judgments and extracted visual facts? Quantitative annotation-quality measurements, together with an analysis of how labeling errors influence downstream performance, would strengthen confidence in the proposed supervision strategy.

5. Selection effects caused by the initial retriever.

The training data are constructed from the initial retriever's Top-20 candidates, with queries lacking sufficient positive or negative examples being discarded. How many queries are excluded under this filtering procedure? Could the resulting training distribution disproportionately represent easier queries or those already well served by the original retriever?

6. Separation of visual adaptation and policy optimization effects.

The current ablation results do not fully disentangle the influence of visual reading adaptation from subsequent GRPO training. Could the authors provide results for visual adaptation without policy optimization and evaluate its effectiveness with and without retriever adaptation? These additional configurations would clarify the extent to which each stage independently contributes to the final performance.

7. Compatibility between the adapted retriever and the final search policy.

The retriever is optimized using queries sampled from the original agent, whereas subsequent GRPO updates the agent's search behavior without modifying the retriever. Have the authors investigated whether the resulting policy generates queries with substantially different characteristics? If such a distribution shift occurs, how well does the fixed retriever continue to satisfy the information requirements of the updated agent?

8. Additional supervision costs and comparison fairness.

Could the authors provide a breakdown of the computational resources and annotation effort required for teacher-generated supervision, retrieval optimization, visual fine-tuning, and GRPO? It would also be valuable to clarify whether SPAR maintains a comparable advantage when competing methods are given similar supervision quantities and training budgets.

I would be willing to reconsider my current rating and potentially increase it if the rebuttal provides satisfactory explanations or convincing additional evidence addressing these concerns.

Reviewer_hrmtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPAR, a framework for multimodal agentic search that jointly improves evidence retrieval and visual reading. It converts the agent’s search trajectories into structured annotations describing query requirements, complete/partial evidence support, and grounded facts. These annotations are then used to adapt a Qwen3-VL retriever, improve visual reading, and finally re-train the agent’s search policy.

Critical strengths
  1. The central idea of improving the retrieved evidence and making it relevant to the intent of search before passing it to the model makes sense.
Critical weaknesses
  1. The evaluation is not fair. The authors report the MMQA results on the MMQA development set, but during policy adaptation they evaluate every 20 steps and select the checkpoint with the highest MMQA development accuracy. The results are therefore affected by checkpoint-selection bias. Instead , use a separate held-out test set.
  2. The paper claims of learning to recover evidence, but it basically is just a semantic re-ranker because it doesn't introduces additional evidence but re-ranks retrieved evidence. I agree that having relevant retrieved information is important but the proposed solution is not elegant at all. The agent is a 7B model and the retriever is 8B. The cost of adding the retriever doesn't justify the performance improvement.
  3. Missing comparisons. Harness improvement can solve the problem of re-ranking. a. The retrieved results from the search APIs are generally passed through summarizer models to make the retrieved evidence concise. Having a "goal" parameter in the tool definition and making the agent generate the search query and goal can solve the problem to some extent. b. You can use a multimodal embedding model to re-rank the web-search results before passing it as context to the model. These simple baselines are missing.
  4. There is no human in the pipeline or multiple teachers to ensure accurate teacher generated labels when training retriever.
  5. The overall contribution of the paper seems minimal. The baseline comparisons are missing: Vision-DeepResearch [1], REDSearcher [2], HyperEyes [3], MM-DeepResearch [4]. The benchmark evaluation on multimodal search is also missing. Benchmarks like: SimpleVQA [5], LiveVQA [6], BrowseComp-VL [7], VDRBench [8], etc.

[1] Huang, W., Zeng, Y., Wang, Q., Fang, Z., Cao, S., Chu, Z., Yin, Q., Chen, S., Yin, Z., Chen, L. and Chen, Z., 2026. Vision-deepresearch: Incentivizing deepresearch capability in multimodal large language models. [2] Chu, Z., Wang, X., Hong, J., Fan, H., Huang, Y., Yang, Y., Xu, G., Zhao, C., Xiang, C., Hu, S. and Kuang, D., 2026. Redsearcher: A scalable and cost-efficient framework for long-horizon search agents. [3] Li, G., Chen, J., Xu, Y., Zhang, X. and Lu, Y., 2026. HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents. [4] Yao, H., Yin, Q., Yang, M., Zhao, Z., Wang, Y., Luo, H., Zhang, J. and Huang, J., 2026. Mm-deepresearch: A simple and effective multimodal agentic search baseline. [5] Cheng, X., Zhang, W., Zhang, S., Yang, J., Guan, X., Wu, X., Li, X., Zhang, G., Liu, J., Mai, Y. and Zeng, Y., 2025, October. Simplevqa: Multimodal factuality evaluation for multimodal large language models. [6] Fu, M., Peng, Y., Liu, B., Wan, Y. and Chen, D., 2025. Livevqa: Live visual knowledge seeking. [7] Geng, X., Xia, P., Zhang, Z., Wang, X., Wang, Q., Ding, R., Wang, C., Wu, J., Li, K., Zhao, Y. and Yin, H., 2026, April. Webwatcher: Breaking new frontiers of vision-language deep research agent. [8] Zeng, Y., Huang, W., Fang, Z., Chen, S., Shen, Y., Cai, Y., Wang, X., Yin, Z., Chen, L., Chen, Z. and Huang, S., 2026. Vision-deepresearch benchmark: Rethinking visual and textual search for multimodal large language models.

Questions for authors

Look at critical weaknesses.

Reviewer_hiNYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets two bottlenecks in multimodal search agents: the retriever is kept fixed and does not adapt to the agent’s intermediate information needs, and even when the right image is retrieved the agent may misread the visual detail. SPAR proceeds in four steps. (1) Each executed query from the initial RL agent’s trajectories is replayed to obtain the top-20 candidates; Qwen2.5-VL-72B decomposes the query into targets and atomic requirements, then, per candidate, grounds the subject, judges support per requirement, and records the supporting evidence. (2) An acquisition projection maps these annotations to COMPLETE / PARTIAL / NEGATIVE / ABSTAIN labels and fine-tunes Qwen3-VL-Embedding-8B (LoRA) with a weighted multi-positive contrastive loss (partial-support weight 0.5) plus a set-level ranking loss. (3) A reading projection reuses the recorded image/OCR facts as query-conditioned targets, training only the vision encoder and merger with the language model frozen. (4) A further GRPO stage adapts the agent with the adapted retriever held fixed. MMQA development accuracy reaches 76.03 (initial agent 65.10, Agentic-R† 67.60), with transfer to WebQA, MuSiQue, and HotpotQA without further training.

Critical strengths

1.The two bottlenecks (evidence acquisition and visual reading) are well identified, each with an intuitive example and a matching diagnostic.

2.The “one annotation, two supervision signals” design is clean and removes the need for a separate fact-annotation step. Figure 8 and Appendices E–F make the annotation pipeline and prompts transparent.

3.The ablations are more complete than in most comparable papers: stage-wise (Table 2), supervision variants both with the agent fixed and after policy adaptation (Table 3), objective variants (Table 10), and the explicit negative cap (Figure 4). The “initial retriever + further RL” control (66.90) shows that the main gain is not simply an extra round of training.

4.The fixed-evidence reading diagnostic (Figure 3) includes a paired retention analysis rather than accuracy alone.

5.The gains concentrate on Image and Both questions (+14.0 / +22.3 pp), consistent with the paper’s claims.

6.The appendix is candid about baseline adaptations, inference budgets (Table 5), and a failure case (Figure 7).

Critical weaknesses

1.Checkpoint selection on the reported evaluation split (Appendix A.3, Table 4). All policy-adapted variants are evaluated every 20 steps on the MMQA development split and the best checkpoint is kept, yet the MMQA numbers in Tables 1–3 are reported on that same development split (2,441 questions). SPAR, Agentic-R†, and every adapted row in Tables 2–3 therefore carry an optimistic bias; baselines evaluated at fixed checkpoints (RAG, MMSearch-R1†, Search-Align†) are disadvantaged. And, most importantly, the noise from taking the maximum over roughly nine checkpoints is of the same order as the ablation margins (1–2 pp). A validation subset should be held out from the training split for selection, and final-checkpoint numbers should also be reported.

2.External baselines are weak, and the “strongest baseline” is the authors’ own adaptation of Agentic-R. Untrained multimodal RAG (53.22) beats MMSearch-R1† (41.46), Search-Align† (41.99), and ReAct† (34.33) on MMQA; on WebQA it (61.26) even beats the initial agent and Agentic-R†. Table 5 shows very different budgets (Search-Align retrieves 1/1 with 128 tokens per call. MMSearch-R1 is not trained on MMQA and its image search uses the original question). The gaps to external methods are therefore largely uninterpretable, which the paper concedes only in the appendix (“complete-system comparisons”). Agentic-R† runs a single iteration (the original alternates retriever and agent optimization), and the model used for its local-relevance scoring is not stated; if it is weaker than the 72B teacher used by SPAR, the comparison confounds supervision design with teacher strength. The most natural graded baseline is also missing: distilling the teacher’s graded scores directly (in the style of LLM-Embedder / SCARLet).

3.The Table 2 ablation grid covers only 4 cells. Missing: reading only; retriever + reading without policy adaptation; reading + policy with the initial retriever. The +2.37 pp attributed to reading adaptation comes from a single path; its end-to-end effect without policy adaptation, and whether it depends on the adapted retriever, cannot be assessed.

4.Dependence on the 72B teacher, annotation cost, and annotation quality are not reported. 30.5K occurrences × 20 candidates implies well over 600K Qwen2.5-VL-72B calls; the cost is not given. The label distribution over the four classes, and how many occurrences are discarded by the “≥1 positive and ≥3 negatives” filter, are not reported. There is no human audit of the teacher’s support labels or grounded facts — hallucinated facts would be distilled directly into the vision encoder. λ = 0.5 and β = 1 are not swept.

5.Novelty is moderate. Graded / multi-positive contrastive training, LLM-derived relevance labels, and distilling VLM descriptions into an encoder each have precedents. The contribution lies in requirement-level structured annotation for agent queries and its reuse for reading supervision. The set-level ranking term contributes +0.65 pp (noise level). This is a solid engineering contribution rather than a conceptual advance.

Questions for authors

Q1.With checkpoints selected on a held-out validation subset of the training split, do the numbers and conclusions in Tables 1–3 hold? Q2.Which model scores local relevance in Agentic-R†? Can a “direct distillation of the teacher’s graded scores” baseline be added? Q3.For the reading diagnostic: per-agent abstention and format-failure rates; was scoring blind to agent identity; and what explains the super-additive effect of reading + policy adaptation? Q4.Can the missing Table 2 cells be filled in?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.