- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents SPAR, a multimodal agentic search framework designed to strengthen both evidence retrieval and image-based information extraction. The approach begins by analyzing agent-generated search trajectories to produce structured annotations that describe the information required by each query, the extent to which retrieved candidates satisfy those requirements, and the factual content supported by visual evidence. These annotations serve two purposes. First, they enable retriever optimization through relevance-aware training objectives that distinguish fully supportive evidence from partially useful candidates. Second, they provide query-specific supervision for improving the agent's interpretation of retrieved images. After these components are adapted, the search agent undergoes further policy optimization with GRPO while the retriever parameters remain unchanged. On MultiModalQA, the resulting system reaches 76.03% answer accuracy, an improvement of 8.43 percentage points over the strongest competing method. The authors also conduct component-level ablations to investigate the effects of retrieval training, relevance grading, and visual reading enhancement. Evaluations on WebQA, MuSiQue, and HotpotQA additionally suggest that the learned capabilities can transfer across multimodal and text-based multi-hop QA settings without task-specific retraining.
Critical strengths
1. Coordinated optimization of retrieval and visual evidence interpretation.
An important merit of this work is its attempt to improve two closely related aspects of multimodal search within a single training framework. Rather than treating retrieval effectiveness and visual comprehension as entirely separate problems, SPAR derives supervision for both from a common set of structured evidence annotations. This design allows the retriever to learn distinctions between different levels of evidential support while simultaneously teaching the agent to identify query-relevant information in images. The connection between these training objectives is reasonably well motivated, and the reuse of supervision across components offers a potentially useful direction for building more capable multimodal search systems.
2. Substantial performance gains supported by multiple experimental analyses.
The experimental findings indicate that SPAR can deliver meaningful improvements over existing approaches. Its 76.03% accuracy on MultiModalQA exceeds the strongest reported baseline by 8.43 percentage points, representing a considerable gain on the primary benchmark. The ablation results also provide evidence that individual training components contribute to overall performance. For example, optimizing the retriever alone increases accuracy from 65.10% to 69.56% when the agent is held fixed. Meanwhile, the evaluation under fixed evidence conditions offers complementary support for improved visual information extraction. Results on WebQA, MuSiQue, and HotpotQA further suggest that the benefits are not confined to the training benchmark or a single modality. Taken together, these experiments provide encouraging empirical support for the effectiveness and transferability of the proposed approach.
Critical weaknesses
1. The unique methodological value of structured evidence modeling is not sufficiently established.
Although SPAR combines retrieval optimization and visual reading enhancement in a reasonably coherent manner, many of its underlying training procedures are based on established methods, including contrastive learning, relevance-aware ranking, supervised visual adaptation, and GRPO. The primary methodological distinction therefore appears to lie in how the framework organizes and reuses evidence supervision. However, the experiments do not establish whether the proposed requirement decomposition and evidence assessment procedures are essential to achieving the reported improvements. A simpler pipeline that directly obtains relevance judgments and visual fact annotations from the same teacher model might provide comparable supervision without the intermediate structure. Without such a comparison, it is difficult to determine how much of SPAR's effectiveness originates from its distinctive annotation design rather than from the additional supervised training itself. This uncertainty limits the strength of its claimed methodological novelty.
2. The evidence for the claimed retrieval and multi-hop reasoning benefits remains largely indirect.
Most experimental conclusions are drawn from end-to-end question-answering performance, with comparatively little analysis of the retrieval behavior that the proposed method is intended to improve. In particular, the absence of retrieval-oriented measurements makes it difficult to assess whether the learned ranking function consistently identifies fully relevant and partially relevant evidence more effectively. The role of partial-support evidence is especially underexplored. Although preserving such candidates is presented as an important design principle, the paper does not directly show whether agents subsequently exploit these candidates to resolve missing information or complete multi-step reasoning chains. Accordingly, the observed advantage of graded relevance training might result primarily from improved ranking of fully supportive documents, rather than from the intended use of incomplete evidence across successive search steps. This leaves a central explanation for the method's effectiveness insufficiently verified.
Questions for authors
1. Added value of the structured annotation pipeline.
What advantages does the proposed query-requirement decomposition and evidence assessment process offer over directly obtaining relevance judgments and visual fact annotations from the same teacher model? A controlled comparison would help determine whether the intermediate structured representation is necessary for the observed improvements.
2. Role of partially supportive evidence in multi-hop search.
Can the authors provide empirical evidence that partially relevant candidates are actually used during later search or reasoning steps? For instance, an analysis of their frequency and contribution within successful multi-hop trajectories would help distinguish genuine improvements in evidence utilization from better ranking of fully supportive results.
3. Retrieval performance beyond end-to-end accuracy.
Could the authors include dedicated retrieval evaluations using metrics such as Recall@K, MRR, and graded NDCG? It would be particularly informative to examine whether SPAR improves the discovery and ordering of both fully supportive and partially supportive candidates, rather than relying exclusively on final QA accuracy.
4. Quality and robustness of automatically generated supervision.
Since the training process relies extensively on annotations produced by Qwen2.5-VL-72B-Instruct, how reliable are its support-level judgments and extracted visual facts? Quantitative annotation-quality measurements, together with an analysis of how labeling errors influence downstream performance, would strengthen confidence in the proposed supervision strategy.
5. Selection effects caused by the initial retriever.
The training data are constructed from the initial retriever's Top-20 candidates, with queries lacking sufficient positive or negative examples being discarded. How many queries are excluded under this filtering procedure? Could the resulting training distribution disproportionately represent easier queries or those already well served by the original retriever?
6. Separation of visual adaptation and policy optimization effects.
The current ablation results do not fully disentangle the influence of visual reading adaptation from subsequent GRPO training. Could the authors provide results for visual adaptation without policy optimization and evaluate its effectiveness with and without retriever adaptation? These additional configurations would clarify the extent to which each stage independently contributes to the final performance.
7. Compatibility between the adapted retriever and the final search policy.
The retriever is optimized using queries sampled from the original agent, whereas subsequent GRPO updates the agent's search behavior without modifying the retriever. Have the authors investigated whether the resulting policy generates queries with substantially different characteristics? If such a distribution shift occurs, how well does the fixed retriever continue to satisfy the information requirements of the updated agent?
8. Additional supervision costs and comparison fairness.
Could the authors provide a breakdown of the computational resources and annotation effort required for teacher-generated supervision, retrieval optimization, visual fine-tuning, and GRPO? It would also be valuable to clarify whether SPAR maintains a comparable advantage when competing methods are given similar supervision quantities and training budgets.
I would be willing to reconsider my current rating and potentially increase it if the rebuttal provides satisfactory explanations or convincing additional evidence addressing these concerns.