acceptodds
Under review as a conference paper at ICLR 2027

Dormant Agent: Stealthy Cross-Agent Attacks in VLM-Based Multi-Agent Systems via Emotion Steering

Abstract

Vision-language-model (VLM)-based multi-agent systems let a user interact with a client-side interface, behind which specialized agents coordinate through inter-agent communication. We show that this convenience comes at a price: an attacker who knows of, or plants, an unmodified downstream agent in the system (without malicious fine-tuning), which we call a Dormant Agent, can cause it to exhibit a prescribed behavior. However, doing so effectively and stealthily faces two challenges. First, the attacker cannot directly construct or modify messages delivered to the Dormant Agent, and controls nothing but the image fed to the client-side interface. Second, even if the upstream agent could be made to relay an activation signal, inter-agent messages are inspected, and explicit triggers or malicious instructions would stand out as anomalies. We therefore seek a signal that travels covertly and reliably, and propose Emotion-Steering Visual Attack, which manipulates only the upstream image to steer the VLM toward a target emotional expression mode, inducing natural and task-consistent messages that carry the covert emotional signal. We further instantiate a Dormant Agent by merely planting a crafted item in its knowledge base. The item lies dormant under ordinary communication and is retrieved only when the incoming message carries the emotional signal, upon which it drives the agent to attacker-specified outputs. Experiments across multiple VLMs and multimodal datasets show attack success rates of up to 67% while maintaining high dormancy on benign inputs. Our results expose an overlooked cross-agent vulnerability and call for defenses that reason about agents in composition, not in isolation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent Systems, VLM Security, Cross-Agent Attack, Stealthy Attack

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A6GNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a stealthy cross-agent attack in vision-language model (VLM)-based multi-agent systems, where an attacker indirectly manipulates a downstream agent through an adversarial image presented to an upstream visual agent. The authors introduce the Dormant Agent threat model, in which a downstream agent remains inactive under normal communication but can be triggered by a covert emotional signal. To realize this attack, they propose Emotion-Steering Visual Attack (ESVA), which steers the upstream agent toward a target emotional style without directly modifying its messages. ESVA includes a representation-guided variant (ESVA-R) and a prompt-based variant (ESVA-P) for different attacker access settings. The authors further construct a Dormant Agent by inserting a crafted image-text entry into its retrieval knowledge base, enabling the emotional signal to trigger attacker-specified responses. Experiments on MultiModalQA and WebQA across three VLMs demonstrate attack success rates of up to 67%, with additional evaluations of dormancy, signal propagation, and stealthiness.

Critical strengths
  1. The separation between a benign-looking message and a poisoned downstream entry is a meaningful security finding. Activation through an upstream image makes the attack relevant to settings where direct manipulation of the target agent’s input is unavailable.
  2. ESVA-R steers a response attribute rather than forcing a lexical prefix. Its higher attack success and stronger message-quality scores than FixedWords in Table 1 make this design choice convincing across the three upstream VLMs. Besides, the retrieval analysis provides evidence about the intended mechanism, supporting a role for emotional style beyond exact wording.
Critical weaknesses
  1. Natural emotional expression creates a tension between stealth and selective activation. Since the clean messages in Table 1 have near-zero emotion scores, the reported dormancy does not establish whether ordinary enthusiastic communication would remain safe.
  2. Changing expression without altering factual content is central to the method, but the steering objective constrains only a representation shift. The instruction score in Appendix D measures topical relevance rather than visual correctness, leaving preservation of task evidence insufficiently supported. Meanwhile, the only demonstrated target behavior is an Unknown prefix in a fixed RAG system. Since irrelevant retrieved evidence could itself prompt abstention, the results do not clearly distinguish targeted response control from a retrieval failure.
Questions for authors
  1. How is dormancy expected to hold when legitimate tasks or user preferences naturally produce enthusiastic messages? Is the claim intended only for workflows whose normal communication is predominantly neutral?
  2. What supports the claim that ESVA-R changes expression without materially changing the evidence passed downstream? The relevance and fluency scores do not resolve whether factual content survives the steering intervention.
  3. Does the Unknown output reflect an independently induced target response or ordinary abstention after useful evidence is displaced? Besides, the attack success rate is measured by the target agent producing an "Unknown" response. How can the authors distinguish successful behavioral manipulation from ordinary abstention caused by retrieving irrelevant evidence?
  4. The experiments use a fixed downstream retrieval-augmented agent across all settings. How dependent is the attack on this particular retrieval and reranking architecture? Could the authors clarify whether the proposed mechanism would remain effective under different downstream agent configurations?
3 comments

We sincerely appreciate your expertise and the time you have devoted to reviewing our work.

Response to W1 & Q1. Thank you for this insightful comment! We perform new experiments to evaluate dormancy under ordinary enthusiastic communication where legitimate tasks or user preferences naturally produce enthusiastic messages. To approximate this setting in a controlled experiment, we prepended one of two common positive response prefixes, P1 and P2, to a randomly selected 30% or 50% of the clean upstream messages, while keeping other contents unchanged. P1 denotes “Thanks for asking! Here's what I found:” and P2 denotes “Sure! Here's what I found:”.

As shown in Table R1, DR remains above 90% across both conversational prefixes and coverage levels. Even when a positive conversational prefix is added to 50% of the clean upstream messages, DR remains at 91%, only 5 percentage points below the 96% clean baseline. These results show that the Dormant Agent maintains a high DR even when benign upstream messages contain surface-level expressions of enthusiasm.

Table R1. DR under benign conversational prefixes on MMQA.

PrefixCoverageDR
Clean (Original)0%96%
P130%92%
P150%91%
P230%92%
P250%91%

Response to W2 & Q2 & Q3. Thank you for this interesting point. We perform new experiments.

Preservation of task evidence. To address this concern, we introduce a new metric, Task Evidence Preservation Rate (TEPR), defined as the fraction of clean samples with correct factual content whose factual content is preserved under ESVA-R. A higher TEPR means less destruction of the original factual evidence. We use Qwen3-VL-Plus guided by a prompt to produce binary judgments on paired clean and attacked samples. Each judgment set identifies whether the clean sample holds correct factual content and whether the attacked counterpart retains this factual content. TEPR is calculated from these paired outputs. As shown in Table R2, TEPR ranges from 96.7% to 100.0% across both datasets and all three upstream VLMs. Together with the emotion-steering results in our paper’s Table 1, these findings support our claim that ESVA-R changes the emotional expression of upstream messages while largely preserving the task-relevant evidence passed downstream.

Table R2. TEPR under ESVA-R across two datasets and three upstream VLMs.

DatasetUpstream VLMTEPR
MMQAQwen2.5-VL (7B)99.0%
Qwen3-VL (8B)99.0%
Gemma4-E4B100.0%
WebQAQwen2.5-VL (7B)99.0%
Qwen3-VL (8B)98.0%
Gemma4-E4B96.7%

Targeted response control. To distinguish targeted response control from ordinary abstention, we extended the evaluation from the original “Unknown” target (T0) to two additional attacker-specified outputs: T1 (“Sorry, I can't help with that.”) and T2 (“The answer is unavailable.”). As shown in Table R3, compared with the T0 baselines, the average ASR over T1 and T2 across the three upstream VLMs decreases by only 4.3 percentage points for ESVA-R and increases by 1.3 percentage points for ESVA-P. These comparable results show that our attack’s effectiveness extends beyond the particular prefix and support its ability to induce different attacker-specified response forms. In addition, we examine the generation loss ablation on WebQA with Qwen2.5-VL (7B). Removing the generation loss Lgen\mathcal{L}_{\mathrm{gen}} (Eq. 8) reduces ASR from 67% to 0%, even though the poisoned entry is still retrieved, selected, and passed to the generator in 59% of trials. Therefore, the “Unknown” output reflects an independently induced target response, instead of ordinary abstention triggered when useful evidence is displaced.

Table R3. Target-output generalization on WebQA, measured by end-to-end ASR (%).

Upstream VLMMethodT0T1T2
Qwen2.5-VL (7B)ESVA-P151716
ESVA-R676262
Qwen3-VL (8B)ESVA-P485152
ESVA-R514846
Gemma4-E4BESVA-P393739
ESVA-R545149

Response to Q4. Good question! To address this concern, we conducted additional experiments under four alternative configurations: replacing the retriever with OpenCLIP ViT-L/14, replacing the reranker with Qwen3-VL-8B, removing the reranker, and replacing the generator with Qwen3-VL-8B. For each configuration, we re-optimize the adversarial knowledge-base image and evaluate the attack using the same upstream messages as in our main experiments.

As shown in Table R4, our attack achieves 55–67% ASR and 90–98% DR across these four modified configurations, retaining 82–100% of the original pipeline’s ASR. Reranker and generator replacement preserve near-baseline attack success. These results indicate that our attack is not restricted to the original downstream architecture and remains effective across alternative retrieval, reranking, and generation configurations.

Table R4. Attack effectiveness across alternative downstream agent configurations on WebQA.

SettingRetrieverRerankerGeneratorRt@3 ↑Top-1 Selection ↑ASR ↑DR ↑
Original pipelineCLIPQwen2.5-VL-7BQwen2.5-VL-7B0.670.670.670.93
Retriever replacementOpenCLIPQwen2.5-VL-7BQwen2.5-VL-7B0.550.550.550.93
Reranker replacementCLIPQwen3-VL-8BQwen2.5-VL-7B0.660.660.660.90
Without RerankerCLIPNoneQwen2.5-VL-7B—0.580.570.98
Generator replacementCLIPQwen2.5-VL-7BQwen3-VL-8B0.670.670.670.93
AuthorsOpenReview ↗

Requesting to read our rebuttal

Dear Reviewer,

Thank you again for your feedback. We have now posted our responses. We believe our rebuttal has addressed the points raised in your review.

We understand this is a busy time, but we would appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns. We are happy to answer any additional questions you might have.

--

Authors

Reviewer_d1TJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a stealthy cross‑agent attack surface in VLM‑based multi‑agent systems named Dormant Agent threat, where an adversary can only manipulate client‑side input images and inject one benign multimodal entry into a downstream knowledge‑base without modifying model weights. It proposes ESVA, an emotion‑steering visual attack including a white‑box representation‑level variant ESVA‑R and a query‑free black‑box variant ESVA‑P, to induce covert emotional signals in upstream‑generated inter‑agent messages. Once the emotional signal arrives, the pre‑planted knowledge‑base entry is retrieved and reranked to trigger attacker‑specified outputs. Empirical results across multiple VLMs show high benign‑input dormancy rates, and low auditor detection rates.

Critical strengths

Research on attacks against multimodal multi-agent systems is significant. The paper proposes a novel stealthy cross‑agent threat paradigm, which highlights security risks for multi‑agent systems that are not assessed by per‑component isolated auditing.

Critical weaknesses
  1. The attack assumes that an attacker can freely inject multimodal entries into a knowledge base, failing to account for how real-world content moderation mechanisms might hinder such actions.

  2. The experiments are limited to the CLIP-VLM RAG stack and lack validation across a diverse range of retriever and reranker architectures.

Questions for authors
  1. What is the noise level of sentiment evaluators based on LLMs? Could human annotations or other evaluation metrics be provided to substantiate their results?

  2. In a RAG pipeline, how would the effectiveness of an attack change if the CLIP retriever were replaced or the VLM reranker were removed?

  3. Experiments are limited to two closed-form multihop VQA datasets, and it remains unclear whether the emotion-steering trigger can activate planted dormant entries within open retrieval corpora in realistic multi-agent tool environments such as GAIA or COMMA.

5 comments

We sincerely appreciate your expertise and the time you have devoted to reviewing our work.

Response to W1. Thank you for raising this practical point. In fact, we have explicitly considered stealthiness in the design of our knowledge-base payload. Unlike prior RAG poisoning attacks [1] that inject anomalous or malicious prompts into text entries, our caption c∗c^* is completely benign, natural, and free of any instructions or triggers. Furthermore, the carrier image IkbadvI_{\text{kb}}^{\text{adv}} uses a benign image with bounded imperceptible perturbations (∥δkb∥∞≤0.0625\Vert{}\delta_{\text{kb}}\Vert{}_\infty \le 0.0625) and no overlaid text. Moreover, requiring only a single contributed entry avoids volume-based anomaly detection in open or collaborative knowledge bases.

To empirically verify the stealthiness, we audited our poisoned multimodal entries using open-source content moderators Llama-Guard-3-8B [2] and Llama-Guard-4-12B [3] (supporting image detection), both sourced from HuggingFace. The moderators classified the entry z∗z^* as benign across both visual and textual modalities.

[1] MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Knowledge Poisoning Attacks. (Ha et al., ACL 2026)

[2] https://huggingface.co/meta-llama/Llama-Guard-3-8B

[3] https://huggingface.co/meta-llama/Llama-Guard-4-12B


Response to W2/Q2. Thank you for raising this concern. Our method remains effective across different downstream component configurations, demonstrating generalizability beyond the original CLIP–VLM RAG stack. Specifically, we evaluated four alternative settings: replacing the retriever with OpenCLIP ViT-L/14, replacing the reranker with Qwen3-VL-8B, removing the reranker, and replacing the generator with Qwen3-VL-8B. For each configuration, we re-optimize the adversarial image and then evaluate on the same upstream messages used in our main expriments.

As shown in Table R1, our method achieves 55–67% ASR and 90–98% DR across these four modified configurations, retaining 82–100% of the original pipeline’s ASR. Reranker and generator replacement preserve near-baseline attack success. These results support the generalizability of our method across different downstream agent configurations.

Table R1. Generalization of our method across downstream component configurations on WebQA.

SettingRetrieverRerankerGeneratorRt@3 ↑Top-1 Selection ↑ASR ↑DR ↑
Original pipelineCLIPQwen2.5-VL-7BQwen2.5-VL-7B0.670.670.670.93
Retriever replacementOpenCLIPQwen2.5-VL-7BQwen2.5-VL-7B0.550.550.550.93
Reranker replacementCLIPQwen3-VL-8BQwen2.5-VL-7B0.660.660.660.90
Without RerankerCLIPNoneQwen2.5-VL-7B—0.580.570.98
Generator replacementCLIPQwen2.5-VL-7BQwen3-VL-8B0.670.670.670.93

Response to Q1. Thank you for the insightful question! Following your suggestions, we perform new experiments.

Noise level of sentiment evaluators. To quantify the noise of our LLM-based sentiment evaluator, we used DeepSeek-V4-Flash at temperature 0.7 to independently rate each message three times. For each upstream VLM and dataset, we evaluated 25 ESVA-R messages and computed the mean within-sample standard deviation (MSD) of the 0–4 sentiment scores. As shown in Table R2, MSD ranges from 0 to 0.1155 across all six settings, indicating low sampling variability in sentiment scoring.

Table R2. Noise level of the LLM-based sentiment evaluator.

Upstream VLMJudge MSD (MMQA) ↓Judge MSD (WebQA) ↓
Qwen2.5-VL (7B)0.04620.0000
Qwen3-VL (8B)0.11550.0924
Gemma4-E4B0.11550.0231

Human annotations. Using the same sentiment scoring rubric (Appendix D.3), five human annotators independently evaluated 50 ESVA-R messages generated by Qwen2.5-VL (7B) on WebQA. For each message, we averaged the five human ratings and compared the resulting mean with the corresponding LLM score. The mean absolute error was only 0.20 points on the 0–4 scale.

Overall, the low scoring variability and small average discrepancy between LLM scores and mean human ratings support the reliability of our sentiment evaluation.

Response to Q3. Thank you for this insightful question. Following your suggestion, we conducted supplementary evaluations in the multimodal multi-agent environment COMMA [4], paired with a web-scaled retrieval corpus. Specifically, we constructed the benign corpus by randomly sampling 10,000 distinct cross-domain entries from the original WebQA release, which provides a large collection of open images and text collected from Wikipedia and Wikimedia Commons [5].

COMMA evaluates collaborative puzzle solving between a Solver that observes the puzzle and an Expert that provides guidance based on a solution manual, with the two roles naturally corresponding to our upstream and downstream agents, respectively. In our adaptation, the Expert retains the manual in context and additionally receives content retrieved from the expanded corpus. We used 150/50/30 training/validation/test instances across ten puzzle types (15/5/3 per type). Consistent with our main experiments, training and validation instances were used exclusively for knowledge-base payload optimization during the priming stage, whereas ESVA-R was applied to held-out test images to construct upstream adversarial inputs during the activation stage.

As shown in Table R3, ESVA-R increases the message emotion score from 0.00 to 3.70 while largely preserving the task relevance and fluency of upstream messages. More importantly, our attack achieves 80% ASR (24/30) on attack trials and 100% DR on the corresponding clean trials. These results support our attack’s effectiveness beyond the original VQA setting and suggest its potential applicability to more realistic multi-agent environments.

Table R3. Evaluation results on test instances from COMMA.

ConditionESMsg.ES_{Msg.}ISMsg.IS_{Msg.}FSMsg.FS_{Msg.}ASRDR
Clean0.003.773.37-1.00
Attack3.703.733.730.80-

[4] COMMA: A Communicative Multimodal Multi-Agent Benchmark. (Ossowski et al., TMLR 2025)

[5] WebQA: Multihop and Multimodal QA. (Chang et al., CVPR 2022)

AuthorsOpenReview ↗

Requesting to read our rebuttal

Dear Reviewer,

Thank you again for your feedback. We have now posted our responses. We believe our rebuttal has addressed the points raised in your review.

We understand this is a busy time, but we would appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns. We are happy to answer any additional questions you might have.

--

Authors

Reviewer_xP4wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates stealthy cross-agent attacks in VLM-based multi-agent systems. The authors propose Emotion-Steering Visual Attack (ESVA), which manipulates upstream images to induce emotion-bearing messages that activate a poisoned downstream RAG agent. ESVA includes representation-level steering (ESVA-R) and visual prompt steering (ESVA-P). Experiments on MMQA and WebQA report attack success rates of up to 67% with low detection rates under the evaluated security auditor.

Critical strengths

S1. Interesting threat model. The paper investigates indirect cross-agent activation through upstream visual inputs, without directly manipulating inter-agent messages during deployment.

S2. Coherent technical design. ESVA combines representation-level emotion steering with retrieval-conditioned knowledge-base poisoning, providing a complete end-to-end attack pipeline.

S3. Useful empirical evaluation. Experiments cover three upstream VLMs, two datasets, unseen queries, message paraphrasing, and stealthiness. The attack remains effective after two relay hops.

Critical weaknesses

W1. Insufficient causal evidence for emotion-based activation. Table 2 reports 16% ASR after emotion neutralization, while the poisoned caption in Fig. 9 contains emotion-related terms. Lexical and semantic confounders are not adequately controlled.

W2. Limited knowledge-base scale. Appendix B.2 uses only 298 benign images and one poisoned entry. The stability of Top-3 retrieval and attack success in larger RAG knowledge bases remains untested.

W3. Missing ablations for downstream optimization. Table 2 reports identical Rt@3, Rr@1, and ASR values. The contributions of the reranking loss, generation loss, and PCGrad are not separately evaluated.

W4. Incomplete dormancy evaluation. Table 1 reports DR values of 0.90–0.96. Naturally occurring high-arousal benign queries are not tested, leaving unintended activation risks unclear.

W5. Simplified attack objective. Attack success is defined by the output prefix "Unknown" (Sec. 5.1). It remains unclear whether the method can induce more complex targeted behaviors.

W6. Limited baseline comparisons. End-to-end comparisons mainly use FixedWords, while AgentPoison and ConjunctiveAgents are evaluated through adapted lexical triggers rather than complete attack pipelines.

W7. Limited stealthiness evaluation. Figure 4 relies on a single LLM-based security auditor. The reported 0–2% flag rate does not establish robustness against other detection strategies.

W8. Strong downstream white-box assumption. Section 3.2.1 assumes white-box access to the retriever, reranker, and generator during payload construction. Transferability to unknown or modified downstream components remains unexplored.

W9. Limited statistical and task-fidelity evaluation. Each dataset contains only 100 evaluation queries, with 61 unique upstream images for MMQA (Appendix B.2). Confidence intervals and factual task-accuracy measurements are missing.

Questions for authors

Q1 (W1). Can the authors isolate the effect of emotion from lexical and semantic similarity, particularly given the emotion-related poisoned caption and the 16% neutralized ASR?

Q2 (W2). How do ASR and DR change when the knowledge base grows to 1K or 10K entries, especially with semantically similar distractors?

Q3 (W3). Given the identical Rt@3, Rr@1, and ASR values in Table 2, what are the individual contributions of the reranking loss, generation loss, and PCGrad?

Q4 (W4). How frequently do naturally occurring positive high-arousal benign queries activate the poisoned entry or trigger the target behavior?

Q5 (W5). Can the method induce more complex targeted outputs beyond the "Unknown" prefix?

4 comments

We sincerely appreciate your expertise and the time you have devoted to reviewing our work.

Response to W1 & Q1. Thank you for the insightful observation! To examine the potential confound raised by the emotion-related caption in Fig. 9, we added two controls using the same 100 ESVA-R messages and RAG settings. First, we replaced the caption with a factual, emotion-neutral description (“A portrait of a woman with reddish hair wearing a dark sleeveless top against a gray background.”) and reoptimized the image perturbation from the same carrier image. Its Rt@3, Rr@1, and ASR remained close to the previously reported results (see Table R1). Second, we used the clean carrier image without perturbation while retaining the original caption. All three metrics fell to zero. These controls indicate that the original caption’s emotional wording alone cannot account for the observed attack success, while a reoptimized image remains effective with an emotion-neutral caption. Taken together with the results in the paper’s Table 2, these findings suggest that downstream attack activation is driven primarily by the PH emotion signal, rather than the specific lexical wording.

Table R1. Caption and image controls on WebQA with Qwen2.5-VL-7B.

SettingRt@3Rr@1ASR
Previously reported setting (paper)0.670.670.67
New: Reoptimized image + emotion-neutral caption0.680.680.68
New: Unperturbed carrier image + original caption0.000.000.00

Response to W2 & Q2. Thank you for raising this important concern. To evaluate knowledge-base scalability, we conducted additional experiments, expanding the knowledge base to 1K and 10K entries by randomly sampling additional distinct benign entries from the original WebQA release. As shown in Table R2, ASR decreases by only 5 to 7 percentage points relative to the original setting, while DR increases to 0.96–1.00. Rt@3 and Rr@1 remain at or above 0.60 across both enlarged settings.

Table R2. Knowledge-base scaling results on WebQA with Qwen2.5-VL (7B).

KB ScaleRt@3 ↑Rr@1 ↑ASR ↑DR ↑
Original0.670.670.670.93
1K0.620.620.620.96
10K0.610.600.601.00

What’s more, the original 298 benign images provide 100% coverage of the ground-truth visual evidence for the evaluation queries. Thus, every query competes with at least one query-specific, highly task-relevant benign entry, rather than only unrelated distractors. Combined with the 1K and 10K results, this demonstrates that our attack remains effective as the KB grows while preserving strong semantic competition.


Response to W3 & Q3. Thank you for pointing this out. In Table 2, the identical Rt@3, Rr@1, and ASR values indicate that no additional failures occurred after retrieval with the optimized image payload. This does not imply that the reranking and generation objectives are dispensable. Following your suggestion, we added three ablation experiments, reported in Table R3, to assess the individual contributions of the reranking loss, generation loss, and PCGrad. Removing Lrr\mathcal{L}_{\mathrm{rr}} yields Rt@3 of 0.61 but reduces Rr@1 to 0.19, indicating that many retrieved payloads fail to survive reranking without this objective. Removing Lgen\mathcal{L}_{\mathrm{gen}} yields Rr@1 of 0.59 but zero ASR, showing that the payload can still be selected but fails to induce the target response. These results support the distinct roles of the two losses in payload selection and response manipulation. PCGrad mitigates conflicts among objective gradients during joint optimization. Removing it slightly reduces ASR, suggesting a modest empirical benefit in this setting.

Table R3. Downstream objective ablations on WebQA with Qwen2.5-VL-7B.

SettingRt@3Rr@1ASRDR
Full0.670.670.670.93
w/o Lrr\mathcal L_{\mathrm{rr}}0.610.190.190.98
w/o Lgen\mathcal L_{\mathrm{gen}}0.590.590.000.95
w/o PCGrad0.660.660.660.93

Response to W4 & Q4. Thank you for raising this concern. Following your suggestion, we conducted a new experiment to additionally assess sensitivity to naturally occurring positive conversational cues. We prepended one of two positive response prefixes, P1 and P2, to a randomly selected 30% or 50% of the clean upstream messages, while keeping other contents unchanged. P1 denotes “Thanks for asking! Here's what I found:” and P2 denotes “Sure! Here's what I found:”.

As shown in Table R4, DR remains above 90% across both conversational prefixes and coverage levels. Even when a positive conversational prefix is added to 50% of the clean upstream messages, DR remains at 91%, only 5 percentage points below the 96% clean baseline. These results show that the Dormant Agent maintains a high DR even when benign upstream messages contain surface-level expressions of enthusiasm.

Table R4. DR under positive conversational prefixes on MMQA.

PrefixCoverageDR
Clean (Original)0%96%
P130%92%
50%91%
P230%92%
50%91%

Response to W5 & Q5. Good question! Our method is not limited to the "Unknown" prefix (T0) and can induce more complex targeted outputs. To demonstrate this, we conducted additional experiments with two complete sentence targets: T1 (“Sorry, I can't help with that.”) and T2 (“The answer is unavailable.”). As shown in Table R5, compared with the T0 baselines, the average ASR over T1 and T2 across the three upstream VLMs decreases by only 4.3 percentage points for ESVA-R and increases by 1.3 percentage points for ESVA-P. These results show that attack effectiveness remains largely stable when more complex outputs are targeted.

Table R5. Target-output generalization of ESVA-P and ESVA-R on WebQA, measured by end-to-end ASR (%).

Upstream VLMMethodT0T1T2
Qwen2.5-VL (7B)ESVA-P151716
ESVA-R676262
Qwen3-VL (8B)ESVA-P485152
ESVA-R514846
Gemma4-E4BESVA-P393739
ESVA-R545149

Response to W6. Thank you for raising this point. As mentioned in Sec. 5.1, AgentPoison and ConjunctiveAgents require capabilities excluded by our threat model: during activation, the adversary controls only the upstream image, not textual queries or inter-agent messages. Specifically, AgentPoison inserts an optimized textual trigger into the victim agent’s query, whereas ConjunctiveAgents inserts a trigger key into the user query and an adversarial template into a remote agent. Neither method is designed to make the upstream agent generate a trigger by modifying only its input image. Therefore, we exclude them from complete end-to-end comparison. Nevertheless, to comprehensively evaluate the stealthiness of our trigger, we include their lexical triggers to compare message-level detectability (Fig. 4; Appendix B.5.1). We will clearly articulate this distinction in the final manuscript.


Response to W7. Thank you for highlighting this limitation. We extend the evaluation with Llama-Guard-3-8B [1], a widely used open-weight safety model specifically fine-tuned for content moderation, and a Rule-based Filter. The latter checks for instruction overrides, control transfer, and role markers, system-prompt extraction, anomalous identifiers, hidden Unicode characters, encoded payloads, and excessive word repetition. As shown in Table R6, flag rates for the three reported strategies remain within 0–3% across both datasets. These results strengthen the evidence for the stealthiness of our trigger and support its robustness against multiple detection strategies.

Table R6. Flag rates (%) of ESVA-R-generated messages under different detection strategies.

DetectorMMQAWebQA
Llama-Guard-3-8B0.00.0
Rule-based Filter3.00.0
Security Auditor (Original)2.00.0

[1] https://huggingface.co/meta-llama/Llama-Guard-3-8B

Response to W8. Thank you for this point! We would like to clarify that our threat model requires white-box knowledge, but not control, of the downstream retriever, reranker, and generator solely for offline payload optimization. This setting applies to publicly documented or locally reproducible pipelines, such as those built with open CLIP retrievers and open-weight VLMs. At deployment, the adversary controls only one upstream image and plants a single benign-looking knowledge-base entry (0–3% detector flag rate), consistent with threat models considered in prior multi-agent and RAG-poisoning work [2–5].

Importantly, requiring white-box knowledge does not restrict the attack to a particular downstream stack. To examine this, Table R1 evaluates alternative downstream configurations: across three upstream VLMs (Qwen2.5-VL, Qwen3-VL, Gemma4) and two datasets (MMQA, WebQA), the attack remains effective when the retriever is replaced (OpenCLIP ViT-L/14), the reranker is replaced (Qwen3-VL-8B) or removed, or the generator is replaced (Qwen3-VL-8B), achieving 55–67% ASR (vs. 67% originally) and 90–98% DR after re-optimization. These results show that our attack remains effective across different downstream architectures. Transfer to an unobserved downstream stack without re-optimization represents a distinct threat model beyond our scope. We will clarify this boundary in Sec. 3.2.1 and identify surrogate- and ensemble-based optimization for unknown downstream stacks as future work.

[2] Agents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks (Shahroz et al., ACL 2025)

[3] Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. (Gu et al., ICML 2024)\

[4] One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image. (Shereen et al., TMLR 2026)

[5] AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. (Chen et al., NeurIPS 2024)


Response to W9. Thank you for highlighting these evaluation gaps. We have added two complementary analyses.

Statistical uncertainty. We now report 95% confidence intervals for ASR and DR across all six dataset–model settings using 10,000 cluster-bootstrap resamples. Resampling by upstream image accounts for shared visual inputs in MMQA (61 image clusters; 100 for WebQA), explicitly quantifying uncertainty at the current evaluation size.

Table R7. ASR and DR with 95% confidence intervals.

DatasetSource VLMDRFixedWords ASRESVA-P ASRESVA-R ASR
MMQAQwen2.5-VL (7B)0.96 [0.91, 0.99]0.31 [0.21, 0.41]0.32 [0.20, 0.45]0.64 [0.51, 0.75]
Qwen3-VL (8B)0.90 [0.84, 0.95]0.31 [0.22, 0.42]0.58 [0.47, 0.69]0.34 [0.23, 0.44]
Gemma4-E4B0.92 [0.85, 0.97]0.22 [0.10, 0.35]0.36 [0.23, 0.51]0.43 [0.30, 0.56]
WebQAQwen2.5-VL (7B)0.93 [0.88, 0.98]0.30 [0.21, 0.39]0.15 [0.08, 0.22]0.67 [0.57, 0.76]
Qwen3-VL (8B)0.91 [0.85, 0.96]0.29 [0.20, 0.38]0.48 [0.38, 0.58]0.51 [0.41, 0.61]
Gemma4-E4B0.90 [0.84, 0.95]0.33 [0.24, 0.42]0.39 [0.30, 0.49]0.54 [0.44, 0.64]

Task fidelity. We evaluate end-to-end factual QA on the same 100 WebQA queries with Qwen2.5-VL (7B), using WebQA’s public accuracy approximation, QA-Acc-Approx. As shown in Table R8, the poisoned KB retains 95.5% of the clean KB baseline score under clean messages. With the poisoned KB fixed, attack-induced PH messages produce a 51.9% relative score reduction. This directly demonstrates a small utility cost under benign communication but a substantial degradation of factual task performance under attack.

Table R8. End-to-end factual QA on WebQA with Qwen2.5-VL (7B).

ConditionQA-Acc-Approx
Clean message + Clean KB0.4219
Clean message + Poisoned KB0.4030
Attack-induced PH message + Poisoned KB0.1939
AuthorsOpenReview ↗

Requesting to read our rebuttal

Dear Reviewer,

Thank you again for your feedback. We have now posted our responses. We believe our rebuttal has addressed the points raised in your review.

We understand this is a busy time, but we would appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns. We are happy to answer any additional questions you might have.

--

Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you all again for your time and thoughtful reviews. We appreciate the follow-up feedback received so far, including one reviewer’s updated assessment following our rebuttal.

As the discussion period draws to a close, we would appreciate hearing whether our responses and additional experiments have adequately addressed your concerns. We would be happy to clarify any remaining questions.

--

Authors

AuthorsOpenReview ↗

General Response to Area Chair

Dear Area Chair,

We sincerely thank all reviewers for their constructive suggestions and thoughtful engagement. Below we summarize the recognized strengths, reviewer-specific responses, and evidence added during the rebuttal.

1. Commonly Recognized Strengths

  • S1. Novel cross-agent threat paradigm: a stealthy threat exposing security risks beyond isolated component auditing (d1TJ).

  • S2. A meaningful threat under restricted access (xP4w, A6GN).

  • S2. A coherent technical design (xP4w, A6GN).

  • S3. Useful empirical evidence: cross-model effectiveness and message quality (xP4w, A6GN), plus robustness to unseen queries and paraphrase relays (xP4w).

2. Reviewer-Specific Outcomes and Responses

We summarize our responses and the discussion outcomes below.

  • Reviewer xP4w: Added mechanism controls, downstream ablations, alternative targets, KB scaling, dormancy tests, additional detectors, and statistical/task-fidelity analyses; clarified threat assumptions and baseline comparability (C1-C6, C8-C10).

  • Reviewer d1TJ: Score increase confirmed from 2 to 3

Most of my concerns have been addressed, so I will raise my score to 3: Weak acceptance.

  • Reviewer A6GN: Added factual-evidence preservation measurements, positive-prefix dormancy tests, target-response controls, and alternative downstream configurations (C2-C4, C6).

3. Consolidated Concerns and Actions Taken

CodeConcernRaised byAction
C1Potential confound raised by the emotion-related captionxP4wAdd caption and image controls. The attack remains effective with an emotion-neutral caption after image reoptimization, while the original caption with an unperturbed image fails, weakening a caption-only explanation.
C2Targeted behavior controlxP4w, A6GNAblate downstream objectives and test two additional more complex response targets. The results support distinct roles for payload selection and response manipulation, with comparable attack success across the additional response targets.
C3Dormancy under benign enthusiasmxP4w, A6GNPerform new experiments to evaluate dormancy under ordinary enthusiastic communication. High dormancy (91-92%) is retained, indicating robustness to the surface-level positive cues.
C4Factual evidence and task fidelityxP4w, A6GNAdd VLM-based factual-evidence checks and end-to-end QA evaluations. The results show high preservation of initially correct upstream evidence and limited benign utility loss, while attack messages substantially degrade downstream QA.
C5Knowledge-base scalexP4wExpand the knowledge base to 1K and 10K entries, preserving strong semantic competition. The attack remains effective with high dormancy at both scales.
C6Dependence on the downstream stackxP4w, d1TJ, A6GNTest four alternative downstream configurations and reoptimized the KB image for each. The attack remains effective when changing the retriever or generator, or replacing or removing the reranker.
C7Generality beyond VQAd1TJConduct supplementary evaluations in the multimodal multi-agent environment COMMA with a 10K-entry KB. Results of 80% ASR and 100% DR extend evidence for the attack to collaborative puzzle solving beyond VQA.
C8Message stealthiness and KB moderationxP4w, d1TJAdded new message detectors and separately tested KB-entry moderation. Low message flag rates (0-3%) and benign classifications of the planted entry support the stealthiness of our attack.
C9Statistical and evaluator reliabilityxP4w, d1TJAdded image-cluster bootstrap confidence intervals, repeated sentiment scoring, and human annotations. These quantify uncertainty and support scoring consistency and human-LLM agreement.
C10Threat capabilities and baseline comparabilityxP4wClarified offline white-box knowledge, frozen model parameters, and image-only control during activation. Explained that previous trigger-based methods (including AgentPoison and ConjunctiveAgents) require capabilities outside this activation threat model, so comparisons assess message detectability.

We appreciate your pivotal role in this final assessment and thank you for considering the substantial evidence provided in our rebuttal.

Sincerely,

Authors of Submission 2953

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.