- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces M3Search, a benchmark for multimodal search agents under visual misinformation. It contains 330 human-audited samples covering three types of image manipulation. The authors evaluate 15 agents and compare results on pristine and manipulated images. Average accuracy is 23.9%, with manipulation causing a 19.2 percentage-point drop. Trajectory analysis shows that misleading cues divert agents’ searches and often go unrecognized.
Critical strengths
- Clear figures and writing make the paper easy to follow.
- The paper studies an important problem: the robustness of multimodal search agents to misinformation.
- The experiments are extensive and thorough.
Critical weaknesses
- Some examples in the paper are puzzling. Given an edited image, the model is asked to retrieve information about the altered entity. This raises ambiguity in the ground truth: should the answer refer to the original entity or the modified one?
- The experiments mainly use a ReAct-style setup. Could the results vary substantially across different agent harnesses?
- The paper provides extensive analysis and evaluation, but the validity of the benchmark itself needs stronger verification. Additional expert review or details of human evaluation would strengthen confidence in the conclusions.
Questions for authors
see weakness
4 comments
Reviewer jC6x — Reply 2/2
We thank the reviewer for recommending evaluation across different agent harnesses and stronger documentation of human validation. Appendix E.7 and the construction and agreement analyses address these concerns.
W2: On robustness across different agent harnesses.
We extend the evaluation to seven harnesses: ReAct, LangGraph, AG2, smolagents, Haystack, MMSearch, and Vision-DeepResearch, using four Qwen backbones and MiniMax-M3. Each controller receives the same 330 processed image-question inputs, the undisclosed system prompt, and the five search and browsing tools. Free-running controllers use a 20-action budget followed by finalization; MMSearch follows its staged retrieval-and-synthesis workflow.
Tables 20–22 report Accuracy, MR, and SCA. Representative results are:
| Model | Harness | Accuracy | MR | SCA |
|---|---|---|---|---|
| Qwen3.8-Max | ReAct | 27.16 | 44.85 | 16.03 |
| Qwen3.8-Max | Vision-DeepResearch | 28.66 | 46.92 | 17.81 |
| Qwen3.7-Flash | ReAct | 14.32 | 56.62 | 1.99 |
| Qwen3.7-Flash | MMSearch | 15.32 | 54.87 | 6.26 |
| Qwen3.7-Flash | Haystack | 9.32 | 57.03 | 4.73 |
Values are percentages; the reported averages equally weight the three levels, as in Appendix E.7.
The overall performance pattern and main conclusions remain consistent across harnesses: answer accuracy remains limited, misleading behavior is frequent, and spontaneous conflict awareness is uncommon. For the four Qwen models, the per-model MR span across the seven harnesses is 2.16–6.21 percentage points. Qwen3.8-Max has accuracy of 24.59%-28.66% and SCA of 15.78%-17.81%, while Qwen3.7-Flash has MR of 54.87%-57.03%. These findings support the stability of the central conclusions across both general-purpose agent frameworks and multimodal-search workflows.
W3: On construction-stage human auditing and benchmark validity.
Human auditing is integrated into benchmark construction. As described in Section 2 and Appendix C.4, two researchers independently screened, reviewed, and audited every sample. Checks cover source licensing, visual realism, scene complexity, factual validity, injection quality, answer-evidence preservation, and residual ambiguity, with level-specific acceptance criteria.
For Level 1, both independent vision-language models must recognize the substituted entity and recover preserved contextual cues. The question must uniquely identify the intended target, and each reasoning step is supported by external evidence. Human reviewers further check the image and task validity. For Level 2, the original question-answer pair is preserved, answer-critical evidence must remain complete and unobstructed, and the inserted cue must be visible and semantically recognizable. Failed candidates are discarded, and ambiguous cases receive human review.
W3: On complementary human validation of the evaluation judge.
Appendix E.5 separately compares the LLM judge with a human expert, a doctoral researcher in the relevant field, on 355 trajectories spanning 15 agents and all three levels:
| Judgment | n | Agreement | Cohen's |
|---|---|---|---|
| Accuracy | 355 | 99.15% | 0.973 |
| MR | 355 | 88.73% | 0.766 |
| SCA | 355 | 94.93% | 0.851 |
| Failure attribution | 192 | 85.94% | 0.752 |
Failure attribution is assessed on the 192 trajectories mutually labeled as misled with a mechanism label from both annotators. This judge-validation study is distinct from the two-researcher construction audit. Together, sample-level auditing, semantic verification, judge-human agreement, and cross-harness evaluation provide complementary support for the benchmark and its findings.
Thank you for the authors' response. I have decided to maintain my original score.
Replying to Reviewer jC6x
Thank you for your positive feedback, as well as for taking the time to evaluate our responses and additional experiments during the review process, and for providing constructive suggestions.
Reviewer jC6x — Reply 1/2
We sincerely thank the reviewer for recognizing the importance of multimodal search robustness, the clear presentation, and the extensive experiments. We appreciate the opportunity to clarify which entity determines the ground truth in Level 1.
W1: On whether the answer should concern the original or edited entity.
In Level 1, the question is constructed after the anchor entity is replaced. It explicitly targets the substituted entity, and the ground-truth answer is defined with respect to that entity, not the original one (Section 2.1).
The construction pipeline first selects a salient entity in an authentic image, replaces it with another entity at the same semantic level and with the same compositional role, and then creates a knowledge-intensive multi-hop question about the replacement. The question uses a short, unique visual description to identify the target without directly naming it or revealing intermediate facts. Each reasoning step is independently supported by web evidence.
For example, if entity A in the original scene is replaced by entity B, the post-edit question asks about B and the reference answer is a fact associated with B. The preserved surrounding scene may still suggest A, creating the intended contextual interference. The agent must ground the entity actually specified by the question rather than infer identity solely from the original event or background.
W1: On the relationship to Level 2 and annotation consistency.
Level 2 addresses the complementary case: the original target, question, and answer remain unchanged, while a false contextual cue is inserted into a separate region. Here, the agent should retain the original target and verify the requested fact instead of following the inserted cue.
Both levels therefore follow the same principle: the answer must concern the question-specified entity. Level 1 tests whether preserved context can distract the agent from a substituted target; Level 2 tests whether an injected contextual cue can distract it from an unchanged target. Sections 2.1–2.2 define this distinction through each question's target.