acceptodds
Under review as a conference paper at ICLR 2027

MSearch: Benchmarking Multimodal Search Agents against Visual Misinformation

Abstract

Multimodal large language models have evolved into deep search agents that orchestrate retrieval tools and browse the open web for complex visual questions. Existing benchmarks, however, evaluate agents only under trustworthy evidence, measuring whether they can find the right answer but not whether they can withstand misinformation. We introduce MSearch, a benchmark that systematically examines multimodal search under AI-injected visual misinformation, constructing three levels—anchor entity replacement, context cue injection, and dual misinformation injection within and across images—via targeted image editing, yielding 330 human-audited samples. We evaluate 15 search agents spanning strong proprietary systems, lightweight models, and open-source multimodal search models. MSearch proves highly challenging: the average accuracy across all agents is merely 23.9%, with the strongest, Gemini-3.1-Pro-preview, reaching only 45.6%. Contrasting runs on pristine images with runs on misinformation-injected images reveals that misinformation substantially degrades performance: replacing only the image reduces accuracy by 19.2 points on average, and by as much as 28.8 points for the most susceptible agent. Trajectory analysis further reveals that injected misinformation steers agents toward erroneous search and analysis paths, with an average mislead rate of 42.5%. Agents also seldom spontaneously recognize conflicts between misleading visual cues and other available evidence: conflict awareness averages only 8.3%, and even the highest-performing agent on this measure, Gemini-3.1-Pro-preview, reaches only 25.0%. These findings underscore that, as image generation and editing models grow increasingly powerful, multimodal search agents must learn to verify visual cues against external evidence rather than comply with them. MSearch thus provides a rigorous testbed for diagnosing and defending against visual misinformation, paving the way toward trustworthy multimodal search.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Search Agent,Visual Misinformation,Deepsearch Agent,Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jC6xOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces M3Search, a benchmark for multimodal search agents under visual misinformation. It contains 330 human-audited samples covering three types of image manipulation. The authors evaluate 15 agents and compare results on pristine and manipulated images. Average accuracy is 23.9%, with manipulation causing a 19.2 percentage-point drop. Trajectory analysis shows that misleading cues divert agents’ searches and often go unrecognized.

Critical strengths
  1. Clear figures and writing make the paper easy to follow.
  2. The paper studies an important problem: the robustness of multimodal search agents to misinformation.
  3. The experiments are extensive and thorough.
Critical weaknesses
  1. Some examples in the paper are puzzling. Given an edited image, the model is asked to retrieve information about the altered entity. This raises ambiguity in the ground truth: should the answer refer to the original entity or the modified one?
  2. The experiments mainly use a ReAct-style setup. Could the results vary substantially across different agent harnesses?
  3. The paper provides extensive analysis and evaluation, but the validity of the benchmark itself needs stronger verification. Additional expert review or details of human evaluation would strengthen confidence in the conclusions.
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Reviewer jC6x — Reply 1/2

We sincerely thank the reviewer for recognizing the importance of multimodal search robustness, the clear presentation, and the extensive experiments. We appreciate the opportunity to clarify which entity determines the ground truth in Level 1.

W1: On whether the answer should concern the original or edited entity.

In Level 1, the question is constructed after the anchor entity is replaced. It explicitly targets the substituted entity, and the ground-truth answer is defined with respect to that entity, not the original one (Section 2.1).

The construction pipeline first selects a salient entity in an authentic image, replaces it with another entity at the same semantic level and with the same compositional role, and then creates a knowledge-intensive multi-hop question about the replacement. The question uses a short, unique visual description to identify the target without directly naming it or revealing intermediate facts. Each reasoning step is independently supported by web evidence.

For example, if entity A in the original scene is replaced by entity B, the post-edit question asks about B and the reference answer is a fact associated with B. The preserved surrounding scene may still suggest A, creating the intended contextual interference. The agent must ground the entity actually specified by the question rather than infer identity solely from the original event or background.

W1: On the relationship to Level 2 and annotation consistency.

Level 2 addresses the complementary case: the original target, question, and answer remain unchanged, while a false contextual cue is inserted into a separate region. Here, the agent should retain the original target and verify the requested fact instead of following the inserted cue.

Both levels therefore follow the same principle: the answer must concern the question-specified entity. Level 1 tests whether preserved context can distract the agent from a substituted target; Level 2 tests whether an injected contextual cue can distract it from an unchanged target. Sections 2.1–2.2 define this distinction through each question's target.

AuthorsOpenReview ↗

Reviewer jC6x — Reply 2/2

We thank the reviewer for recommending evaluation across different agent harnesses and stronger documentation of human validation. Appendix E.7 and the construction and agreement analyses address these concerns.

W2: On robustness across different agent harnesses.

We extend the evaluation to seven harnesses: ReAct, LangGraph, AG2, smolagents, Haystack, MMSearch, and Vision-DeepResearch, using four Qwen backbones and MiniMax-M3. Each controller receives the same 330 processed image-question inputs, the undisclosed system prompt, and the five search and browsing tools. Free-running controllers use a 20-action budget followed by finalization; MMSearch follows its staged retrieval-and-synthesis workflow.

Tables 20–22 report Accuracy, MR, and SCA. Representative results are:

ModelHarnessAccuracy ↑\uparrowMR ↓\downarrowSCA ↑\uparrow
Qwen3.8-MaxReAct27.1644.8516.03
Qwen3.8-MaxVision-DeepResearch28.6646.9217.81
Qwen3.7-FlashReAct14.3256.621.99
Qwen3.7-FlashMMSearch15.3254.876.26
Qwen3.7-FlashHaystack9.3257.034.73

Values are percentages; the reported averages equally weight the three levels, as in Appendix E.7.

The overall performance pattern and main conclusions remain consistent across harnesses: answer accuracy remains limited, misleading behavior is frequent, and spontaneous conflict awareness is uncommon. For the four Qwen models, the per-model MR span across the seven harnesses is 2.16–6.21 percentage points. Qwen3.8-Max has accuracy of 24.59%-28.66% and SCA of 15.78%-17.81%, while Qwen3.7-Flash has MR of 54.87%-57.03%. These findings support the stability of the central conclusions across both general-purpose agent frameworks and multimodal-search workflows.

W3: On construction-stage human auditing and benchmark validity.

Human auditing is integrated into benchmark construction. As described in Section 2 and Appendix C.4, two researchers independently screened, reviewed, and audited every sample. Checks cover source licensing, visual realism, scene complexity, factual validity, injection quality, answer-evidence preservation, and residual ambiguity, with level-specific acceptance criteria.

For Level 1, both independent vision-language models must recognize the substituted entity and recover preserved contextual cues. The question must uniquely identify the intended target, and each reasoning step is supported by external evidence. Human reviewers further check the image and task validity. For Level 2, the original question-answer pair is preserved, answer-critical evidence must remain complete and unobstructed, and the inserted cue must be visible and semantically recognizable. Failed candidates are discarded, and ambiguous cases receive human review.

W3: On complementary human validation of the evaluation judge.

Appendix E.5 separately compares the LLM judge with a human expert, a doctoral researcher in the relevant field, on 355 trajectories spanning 15 agents and all three levels:

JudgmentnAgreementCohen's κ\kappa
Accuracy35599.15%0.973
MR35588.73%0.766
SCA35594.93%0.851
Failure attribution19285.94%0.752

Failure attribution is assessed on the 192 trajectories mutually labeled as misled with a mechanism label from both annotators. This judge-validation study is distinct from the two-researcher construction audit. Together, sample-level auditing, semantic verification, judge-human agreement, and cross-harness evaluation provide complementary support for the benchmark and its findings.

Thank you for the authors' response. I have decided to maintain my original score.

AuthorsOpenReview ↗

Replying to Reviewer jC6x

Thank you for your positive feedback, as well as for taking the time to evaluate our responses and additional experiments during the review process, and for providing constructive suggestions.

Reviewer_3z23OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes M^3 Search, a multimodal search agent benchmark which has 330 human audited samples. The misinformation is simulated in three types: 1. anchor entity replacement, 2. context cue injection, and 3. dual misinformation injection. The authors observe performance degradation in presence of these misinformation.

Critical strengths
  1. The motivation of the benchmark makes sense, but the paper is not comprehensive and doesn't provide actionable insights.
Critical weaknesses
  1. The images of the benchmark are edited using a model which might introduce artifacts. It doesn't cater to true misinformation where the search agent retrieves something from the web, let's say an article and it has wrong information in it.
  2. The benchmark analysis doesn't provide any insights on how to solve the issue of misinformation.
  3. The scope of the benchmark misinformation is limited to the initial question and image. What about misinformation which are retrieved through the web-search tools ? That scenario is not handled.
  4. All analysis is done using Qwen-3.7 max but the paper does not establish how reliably the judge distinguishes misinformation-induced failures from ordinary grounding, retrieval, or reasoning errors.
  5. Ambiguous Definition of Misinformation: Level 1 rewards trusting the edited entity, whereas Level 2 requires disregarding injected visual cues. This inconsistency raises concerns about whether the benchmark measures a coherent notion of misinformation robustness.
  6. The absence of benign AI-edited controls makes it difficult to isolate the impact of misinformation from general editing effects. Additionally, unmatched question difficulty across levels weakens claims about the increasing challenge of dual misinformation.
Questions for authors

Look at weaknesses.

6 comments
AuthorsOpenReview ↗

Reviewer 3z23 — Reply 1/2

We thank the reviewer for recognizing the motivation and for raising important questions about editing quality, misinformation entry points, and task definitions. Our second reply addresses actionable mitigation (W2) and judge reliability (W4).

W1/W3: On misinformation in the input versus retrieved web content.

M$^3$Search studies misinformation embedded in the initial query image, while agents search the open Web through web search, image search, reverse-image search, region-crop search, and webpage visiting. Retrieval is not restricted to a benchmark-provided knowledge base.

We initially considered misinformation at multiple stages of agentic search. However, misinformation placed on retrieved webpages introduces uncertainty about whether an agent retrieves the manipulated source, visits the relevant page, and encounters the misleading content. We therefore focus on the controllable initial-image entry point while allowing its influence to propagate through open-web search.

This setting tests how visual premises affect entity identification, query formulation, evidence selection, and verification. Appendix D.3 shows that an agent can repeatedly retrieve the correct date of a painting but still follow an injected inscription. Thus, correct web evidence does not automatically prevent input-originating misinformation from affecting the answer. We plan to explore misinformation introduced through retrieved content in future work.

W1: On editing artifacts and image quality.

The source materials are authentic images; AI introduces localized edits. Our pipeline rejects collateral changes outside the target region, geometric inconsistencies, visible compositing artifacts, and loss of important scene content. For Level 1, two independent vision-language models must recognize the replacement and recover the preserved contextual cues. For Level 2, the answer-bearing evidence must remain complete, interpretable, and unobstructed, and the injected cue must be visible and semantically recognizable (Sections 2.1–2.2).

Two researchers independently screened, reviewed, and audited every sample. These checks help ensure that the injected misinformation is plausible and effective without substantially degrading the image, reducing concerns that failures simply reflect low-quality visual corruption.

W5: On the consistency of Level 1 and Level 2.

The two levels intervene on complementary dimensions. In Level 1, we replace an anchor entity and then construct a question about the replacement. The correct answer concerns that replacement; preserved scene context may misleadingly suggest the original entity. In Level 2, the target, question, and answer are preserved, while a false contextual cue is inserted elsewhere. The agent should answer about the original target rather than adopt the injected cue.

The common criterion is to ground the question-specified target and verify the requested fact despite competing visual information. Which visual component is misleading follows from the question's target, not a rule to always trust or always reject edited content.

W6(a): On editing effects and the quality of the paired controls.

Our controls are supported by the construction safeguards above. The accepted interventions introduce a recognizable, semantically meaningful false cue while preserving relevant image content and avoiding substantial quality degradation. For Level 2, the planner explicitly separates the injection region from the answer-critical region; final checks compare pristine and edited images and verify that the original answer remains valid.

The paired experiment uses 240 samples with unchanged questions and reference answers (all Level-2 and multi-image Level-3 samples), evaluated under the same agent settings. Replacing the pristine images with their misinformation-edited counterparts yields a mean accuracy decline of 19.2 percentage points across 15 agents (Section 3.3.1 and Appendix E.1). The quality checks, paired comparisons, and trajectory examples together support the validity of these misinformation interventions and help reduce the concern that gross image degradation drives the findings.

W6(b): On question difficulty and the interpretation of Level-3 results.

The levels are defined by where and how many misleading cues are introduced. The empirical observation in Section 3.2 is that the evaluated Level-3 set has the lowest mean tool-augmented accuracy: 15.1%, versus 33.9% for Level 1 and 22.8% for Level 2. Section 3.2 reports these observed accuracy differences across the three misinformation settings.

AuthorsOpenReview ↗

Reviewer 3z23 — Reply 2/2

We thank the reviewer for emphasizing that diagnostic findings should lead to actionable improvements and that trajectory-level judgments require validation. We address these two points below.

W2: On actionable insights and mitigation strategies.

Section 3.4 and Appendix F present M$^3$Search-Agent, a training-free framework motivated by two observed failures: accepting misleading visual premises before establishing the target's identity, and retaining an incorrect answer despite retrieving conflicting evidence.

Public-reference grounding requires reverse-image or crop-based retrieval before the first textual search or answer, supplies up to two public reference images, and uses candidate titles for independent web verification. Evidence-only context reconstruction intercepts the first eligible final-answer draft and rebuilds the context from the original question and public evidence, excluding the initial image, previous reasoning, and draft answer. The same model continues within the remaining action budget, without being told that the image is edited. Figure 4 and Appendix F.1 detail the implementation.

On 240 answer-preserving questions across five Qwen backbones, the results are:

SettingAccuracy ↑\uparrowMR ↓\downarrowSCA ↑\uparrow
Original agent10.9173.348.19
Disclosure23.9549.7843.05
Few-shot23.5846.9346.26
M$^3$Search-Agent24.6347.4416.84

Values are percentages, using the five-backbone means in Table 23. Disclosure supplies an editing prior; Few-shot additionally supplies four negative-trajectory demonstrations and corresponding verification behavior. Both improve accuracy and conflict awareness. M$^3$Search-Agent achieves higher mean accuracy without that prior, which is practically valuable because users will not necessarily warn an agent about manipulated inputs.

Component ablations further connect the improvements to the proposed mechanisms. Removing public-reference grounding reduces accuracy to 18.90%; removing context reconstruction reduces it to 16.95% and increases MR to 57.73% (Table 24). These results provide concrete guidance for improving evidence acquisition and context management, beyond simply reporting performance degradation.

W4: On distinguishing misinformation-induced failures from ordinary errors.

Our MR criterion requires trajectory evidence that the misleading cue is adopted as a premise, propagated through retrieval, or reflected in the answer; it does not classify every incorrect answer as misinformation-induced (Section 3.1). We additionally validate the judge against human assessment in Appendix E.5.

A human expert, a doctoral researcher in the relevant field, evaluates 355 sampled trajectories covering all 15 agents and all three levels. Table 15 compares human and LLM judgments on answer correctness, misleading behavior, conflict awareness, and failure attribution:

JudgmentnAgreementCohen's κ\kappa
Accuracy35599.15%0.973
MR35588.73%0.766
SCA35594.93%0.851
Failure attribution19285.94%0.752

Failure attribution uses the 192 trajectories that both annotators label as misled and for which both provide a mechanism label. It compares DVA, RH, SES, and CAR; the other three judgments use the shared set of 355 trajectories.

The MR agreement directly supports the reliability of identifying misinformation-related behavior, while the conditional failure-attribution agreement supports the subsequent mechanism categorization. These results show close alignment between the judge and human assessment on both answer-level and trajectory-level evaluations.

Dear Reviewer,

Thank you again for your valuable feedback. We would be very grateful if you could kindly take a moment to review our responses when convenient. Please let us know if any concerns remain.

Thank you so much for your time and consideration.

Reviewer_ZibLOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Existing multimodal‑search benchmarks assume authentic input images. This work presents M³Search, a human‑audited benchmark for testing multimodal search agents against AI‑edited visual misinformation, featuring three difficulty levels: anchor‑entity replacement, contextual‑cue injection, and single/multi‑image dual misinformation.

Multiple proprietary and open‑source agents are evaluated under a unified ReAct framework. Image manipulations severely degrade model performance. Agents are frequently misled and rarely spontaneously spot conflicts between visual cues and retrieved web evidence, often stopping search early and trusting deceptive visual inputs. Simply warning agents about potential AI edits brings only limited protection. Four major failure modes are identified. M³Search provides test resources and trajectory‑aware metrics to support the development of more robust multimodal search agents.

Critical strengths
  1. Novel evaluation angle: Most prior multimodal‑search benchmarks trust input visuals; M³Search targets pre‑retrieval visual premise contamination — misinformation resides in the query image before search starts, which is a realistic, under‑studied threat.

  2. Carefully constructed benchmark: Three hierarchical misinformation levels, human auditing, paired pristine/edited samples, multi‑image cases; full construction pipeline documented for reproducibility.

  3. Comprehensive evaluation setup: Unified ReAct harness shared across all 15 agents; fair comparison between closed‑source SOTA and open‑source search agents. Evaluates not just final answer accuracy but trajectory‑level metrics: mislead rate, spontaneous conflict awareness, tool‑call statistics, failure‑mode attribution.

  4. Insightful diagnostic findings: Reveals concrete failure patterns. Misleading visuals often reduce search effort (agents stop verifying). Even telling agents images are edited only brings limited improvement. Categorizes four distinct error mechanisms with real case trajectories.

  5. Full ablation studies: Compares pristine vs edited inputs; ablation on disclosing edit warning; detailed per‑agent / per‑level statistics in appendix.

Critical weaknesses
  1. Synthetic misinformation only: All test cases are AI‑generated controlled edits. Does not cover real‑world naturally‑occurring visual misinformation (real fake photos, out‑of‑context news imagery). Real‑world generalization remains unknown.

  2. Benchmark domain scope: Samples are drawn from nine domains but remain limited scale (330 total). May lack diversity for very niche domains.

Questions for authors
  1. Can we design lightweight agent strategies to detect visual‑input contamination without knowing edits exist? For example: mandatory cross‑verification of key visual entities against web evidence before generating search queries.

  2. How well do findings from synthetic AI‑edited M³Search transfer to real‑world misinformation (out‑of‑context photos, real forgeries)? What gaps will appear?

  3. What agent design choices improve spontaneous conflict awareness? Is it prompt engineering, tool‑use modification, or needs fine‑tuning on misinformation trajectories?

3 comments
AuthorsOpenReview ↗

Reviewer ZibL — Reply 1/2

We thank the reviewer for recognizing our novel evaluation perspective, careful benchmark construction, and trajectory-level diagnostic analysis. We address the concerns about real-world relevance and dataset scale below; our second reply addresses the mitigation questions.

W1/Q2: On synthetic edits and real-world relevance.

Our image materials come from the real world; AI is used only as a tool to inject misleading information into these existing materials. This distinction matters: the task is not recognition of an entirely artificial scene, but reasoning and searching about a real visual subject in the presence of a localized false cue.

Appendix D.3 illustrates this with a painting whose reference answer is 1894. Reverse-image search and crop-based search repeatedly retrieve A Small House in Moscow (1894), but the agent ultimately answers 1905, following the inserted inscription. Thus, relevant external evidence is available and retrieved; the failure arises when the agent gives the misleading visual cue greater weight than corroborating public evidence.

Appendix D.4 shows a complementary case: an inserted "25 AGE" caption conflicts with the reference age of 19 for an existing character image. The agent explicitly recognizes the discrepancy but still returns the injected value. Together, these cases show how realistic edits can both suppress correct evidence and survive explicit conflict recognition.

Because the underlying materials are real and the interventions introduce plausible visual misinformation, these samples reflect, to some extent, real-world scenarios in which misleading visual content contaminates subsequent search. Out-of-context imagery is not the primary focus here; we study the distinct mechanism of misinformation embedded in the initial query image.

W2: On benchmark scale, domain coverage, and statistical stability.

M$^3$Search contains 330 individually audited samples across nine domains: 62 Level-1, 134 Level-2, and 134 Level-3 samples. Figure 2 illustrates coverage including art and heritage, sports, architecture, transportation, technology, and media and pop culture. Two researchers independently screened, reviewed, and audited every sample, with quality and factual-validity checks rather than indiscriminate acceptance of generated edits.

Appendix E.6 further examines sample-size sensitivity across all 15 tool-augmented agents. We perform 20,000 stratified bootstrap replicates at each of six resampling sizes (100, 150, 200, 250, 300, and 330), maintaining the original level proportions and using the same level-wise aggregation.

MetricMean 95% CI width at 300At 330
Accuracy9.989.49
MR9.539.06
SCA5.285.04

Widths are in percentage points; full results are in Tables 16–18.

The intervals narrow as sample size increases, with progressively smaller gains near the full benchmark. For every model and metric, the 300- and 330-sample intervals overlap, and no endpoint shifts by more than 0.41 percentage points. The counts of models satisfying the reported accuracy, MR, and SCA range observations also remain unchanged.

These results provide evidence that the benchmark's main conclusions are stable near the full sample size, while the final 30 samples provide relatively limited marginal gains in statistical precision. Together with the sample-by-sample auditing and nine-domain coverage, this supports the reliability of the reported evaluation.

AuthorsOpenReview ↗

Reviewer ZibL — Reply 2/2

We appreciate the reviewer's questions about lightweight mitigation without advance knowledge of image editing and the design choices that improve conflict awareness. Section 3.4 and Appendix F address both questions.

Q1: On lightweight mitigation without an editing prior.

M$^3$Search-Agent is a training-free framework using a single multimodal model and the same search tools. A programmatic controller enforces two mechanisms (Figure 4 and Appendix F.1).

Public-reference grounding. Before the first textual search or answer submission, the controller requires reverse-image or crop-based search. It extracts up to two public image candidates and supplies successfully retrieved reference images to the model. When candidate titles are available, an independent title-based web query precedes further factual queries. The model compares object structure, parts, and layout and checks candidate identities against public evidence rather than relying exclusively on visible labels.

Evidence-only context reconstruction. At the first proposed final answer, provided that at least two regular action turns remain, the controller intercepts the draft and rebuilds the context from the original system rules, question, public tool observations, and reference images. It removes the initial input image, previous assistant reasoning, and draft answer. The same model then continues searching or answers using the remaining budget. This transition occurs at most once; it is programmatic message reconstruction, not an additional model-based summary.

The agent is never told that edits or misinformation exist.

Q1: On effectiveness and component ablations.

Appendix F.2 evaluates five Qwen backbones on 240 answer-preserving questions: 134 Level-2 and 106 multi-image Level-3 samples. The conditions share the search tools, interaction budget, and evaluation pipeline.

SettingAccuracy ↑\uparrowMR ↓\downarrowSCA ↑\uparrow
Original agent10.9173.348.19
Disclosure23.9549.7843.05
Few-shot23.5846.9346.26
M$^3$Search-Agent24.6347.4416.84

Values are percentages, averaged across five backbones following the level-wise aggregation in Table 23.

Without disclosure, M$^3$Search-Agent improves mean accuracy by 13.72 percentage points and reduces MR by 25.90 points. Its mean accuracy exceeds Disclosure and Few-shot by 0.67 and 1.04 points, respectively. Disclosure and Few-shot receive an editing prior and serve as optimistic references under advance warning.

Appendix F.3 reports component ablations:

ConfigurationAccuracy ↑\uparrowMR ↓\downarrowSCA ↑\uparrow
Original agent (baseline)10.9173.348.19
Full M$^3$Search-Agent24.6347.4416.84
w/o public-reference grounding18.9045.7112.67
w/o context reconstruction16.9557.7318.02

Removing grounding reduces mean accuracy by 5.73 points. Removing context reconstruction reduces it by 7.68 points and increases MR by 10.29 points. Both components therefore contribute to mean answer accuracy. Figure 4 and Tables 23–24 provide the complete workflow and model-level results.

Q3: On prompting, tool-use modifications, and fine-tuning.

Disclosure informs the agent that the image is AI-edited without identifying the changes. Few-shot adds four short negative-trajectory examples and their corresponding verification behavior. These interventions raise mean SCA from 8.19% to 43.05% and 46.26%, respectively, while also improving accuracy and reducing MR. They demonstrate that an editing prior and concrete verification examples can improve conflict awareness.

M$^3$Search-Agent's undisclosed setting is particularly valuable in practice: real-world users will not necessarily warn an agent that an image has been edited. Without this prior knowledge, our framework achieves higher mean accuracy than both informed prompting conditions.

The experiments provide two actionable directions: prompting can encourage conflict recognition, while controller-enforced grounding and context reconstruction improve evidence acquisition and answer reconsideration. Fine-tuning on misinformation trajectories is a promising direction that we plan to explore in future work. We thank the reviewer for motivating this investigation.

Dear Reviewer,

Thank you again for your valuable feedback. We would be very grateful if you could kindly take a moment to review our responses when convenient. Please let us know if any concerns remain.

Thank you so much for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Public Comment

We sincerely thank all three reviewers for their careful evaluations and constructive feedback. We appreciate the recognition of the importance of our research question, the novel evaluation perspective, the human-audited construction, and the extensive experiments and clear presentation. The reviews have helped us strengthen the discussion of benchmark validity, evaluation reliability, and actionable mitigation. Below, we clarify common concerns; the individual replies provide the corresponding experimental evidence.

W1/Q2 (ZibL), W1/W3 (3z23): On real-world relevance and the misinformation entry point.

M$^3$Search starts from authentic, pre-existing visual materials, including public multimodal-search datasets and license-verified images. We use AI as an editing tool to introduce localized misinformation rather than generate entirely synthetic scenes. The surrounding scene and relevant visual content are preserved.

For example, Appendix D.3 presents a painting identified by retrieved references as A Small House in Moscow (1894). An inserted "1905" inscription causes the agent to answer 1905 despite repeatedly retrieving the correct date. Appendix D.4 presents an existing JoJo's Bizarre Adventure visual asset with an inserted "25 AGE" caption that conflicts with the reference age of 19; the agent recognizes the conflict but nevertheless adopts the injected value.

These examples combine real-world visual materials with plausible misleading edits and therefore reflect, to some extent, realistic visual-misinformation scenarios. Our focus is misinformation present in the query image before retrieval: it can affect grounding, query formulation, evidence selection, and the final answer even when correct public evidence is available. Out-of-context imagery involves a different misinformation mechanism and is not the primary focus of this study.

W5 (3z23), W1 (jC6x): On the target entity and the level definitions.

Level 1 replaces an anchor entity and constructs a question targeting the replacement. Its ground truth therefore refers to the edited entity; the preserved surrounding context can misleadingly suggest the original entity. Level 2 instead preserves the original target, question, and answer while inserting a false contextual cue into a non-answer-bearing region. Level 3 introduces two misleading cues within one image or across multiple images. These settings probe complementary interactions between the question-specified target and potentially misleading visual context.

W1/W6 (3z23), W3 (jC6x): On image quality and human auditing.

Our construction pipeline combines edit-quality checks, semantic verification, and human auditing (Section 2 and Appendix C.4). Two researchers independently screened, reviewed, and audited every sample. For Level 1, we reject collateral changes, geometric inconsistencies, visible compositing artifacts, and loss of important scene content. Two vision-language models must independently recognize the replacement and recover the preserved contextual cues. For Level 2, the original answer evidence must remain complete, interpretable, and unobstructed, while the injected cue must be visible and semantically recognizable.

These checks help ensure that the misinformation is plausible and effective without substantially degrading image quality, supporting the quality and validity of the accepted samples.

The individual replies additionally report judge-human agreement (Appendix E.5), sample-size sensitivity (E.6), cross-harness evaluation (E.7), and M$^3$Search-Agent with prompting comparisons and component ablations (Section 3.4 and Appendix F). We thank the reviewers again for their helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.