- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a stealthy cross-agent attack in vision-language model (VLM)-based multi-agent systems, where an attacker indirectly manipulates a downstream agent through an adversarial image presented to an upstream visual agent. The authors introduce the Dormant Agent threat model, in which a downstream agent remains inactive under normal communication but can be triggered by a covert emotional signal. To realize this attack, they propose Emotion-Steering Visual Attack (ESVA), which steers the upstream agent toward a target emotional style without directly modifying its messages. ESVA includes a representation-guided variant (ESVA-R) and a prompt-based variant (ESVA-P) for different attacker access settings. The authors further construct a Dormant Agent by inserting a crafted image-text entry into its retrieval knowledge base, enabling the emotional signal to trigger attacker-specified responses. Experiments on MultiModalQA and WebQA across three VLMs demonstrate attack success rates of up to 67%, with additional evaluations of dormancy, signal propagation, and stealthiness.
Critical strengths
- The separation between a benign-looking message and a poisoned downstream entry is a meaningful security finding. Activation through an upstream image makes the attack relevant to settings where direct manipulation of the target agent’s input is unavailable.
- ESVA-R steers a response attribute rather than forcing a lexical prefix. Its higher attack success and stronger message-quality scores than FixedWords in Table 1 make this design choice convincing across the three upstream VLMs. Besides, the retrieval analysis provides evidence about the intended mechanism, supporting a role for emotional style beyond exact wording.
Critical weaknesses
- Natural emotional expression creates a tension between stealth and selective activation. Since the clean messages in Table 1 have near-zero emotion scores, the reported dormancy does not establish whether ordinary enthusiastic communication would remain safe.
- Changing expression without altering factual content is central to the method, but the steering objective constrains only a representation shift. The instruction score in Appendix D measures topical relevance rather than visual correctness, leaving preservation of task evidence insufficiently supported. Meanwhile, the only demonstrated target behavior is an Unknown prefix in a fixed RAG system. Since irrelevant retrieved evidence could itself prompt abstention, the results do not clearly distinguish targeted response control from a retrieval failure.
Questions for authors
- How is dormancy expected to hold when legitimate tasks or user preferences naturally produce enthusiastic messages? Is the claim intended only for workflows whose normal communication is predominantly neutral?
- What supports the claim that ESVA-R changes expression without materially changing the evidence passed downstream? The relevance and fluency scores do not resolve whether factual content survives the steering intervention.
- Does the Unknown output reflect an independently induced target response or ordinary abstention after useful evidence is displaced? Besides, the attack success rate is measured by the target agent producing an "Unknown" response. How can the authors distinguish successful behavioral manipulation from ordinary abstention caused by retrieving irrelevant evidence?
- The experiments use a fixed downstream retrieval-augmented agent across all settings. How dependent is the attack on this particular retrieval and reranking architecture? Could the authors clarify whether the proposed mechanism would remain effective under different downstream agent configurations?
3 comments
Response to Q4. Good question! To address this concern, we conducted additional experiments under four alternative configurations: replacing the retriever with OpenCLIP ViT-L/14, replacing the reranker with Qwen3-VL-8B, removing the reranker, and replacing the generator with Qwen3-VL-8B. For each configuration, we re-optimize the adversarial knowledge-base image and evaluate the attack using the same upstream messages as in our main experiments.
As shown in Table R4, our attack achieves 55–67% ASR and 90–98% DR across these four modified configurations, retaining 82–100% of the original pipeline’s ASR. Reranker and generator replacement preserve near-baseline attack success. These results indicate that our attack is not restricted to the original downstream architecture and remains effective across alternative retrieval, reranking, and generation configurations.
Table R4. Attack effectiveness across alternative downstream agent configurations on WebQA.
| Setting | Retriever | Reranker | Generator | Rt@3 ↑ | Top-1 Selection ↑ | ASR ↑ | DR ↑ |
|---|---|---|---|---|---|---|---|
| Original pipeline | CLIP | Qwen2.5-VL-7B | Qwen2.5-VL-7B | 0.67 | 0.67 | 0.67 | 0.93 |
| Retriever replacement | OpenCLIP | Qwen2.5-VL-7B | Qwen2.5-VL-7B | 0.55 | 0.55 | 0.55 | 0.93 |
| Reranker replacement | CLIP | Qwen3-VL-8B | Qwen2.5-VL-7B | 0.66 | 0.66 | 0.66 | 0.90 |
| Without Reranker | CLIP | None | Qwen2.5-VL-7B | — | 0.58 | 0.57 | 0.98 |
| Generator replacement | CLIP | Qwen2.5-VL-7B | Qwen3-VL-8B | 0.67 | 0.67 | 0.67 | 0.93 |
Requesting to read our rebuttal
Dear Reviewer,
Thank you again for your feedback. We have now posted our responses. We believe our rebuttal has addressed the points raised in your review.
We understand this is a busy time, but we would appreciate it if you could take a moment to review our responses and let us know if they adequately address your concerns. We are happy to answer any additional questions you might have.
--
Authors
We sincerely appreciate your expertise and the time you have devoted to reviewing our work.
Response to W1 & Q1. Thank you for this insightful comment! We perform new experiments to evaluate dormancy under ordinary enthusiastic communication where legitimate tasks or user preferences naturally produce enthusiastic messages. To approximate this setting in a controlled experiment, we prepended one of two common positive response prefixes, P1 and P2, to a randomly selected 30% or 50% of the clean upstream messages, while keeping other contents unchanged. P1 denotes “
Thanks for asking! Here's what I found:” and P2 denotes “Sure! Here's what I found:”.As shown in Table R1, DR remains above 90% across both conversational prefixes and coverage levels. Even when a positive conversational prefix is added to 50% of the clean upstream messages, DR remains at 91%, only 5 percentage points below the 96% clean baseline. These results show that the Dormant Agent maintains a high DR even when benign upstream messages contain surface-level expressions of enthusiasm.
Table R1. DR under benign conversational prefixes on MMQA.
Response to W2 & Q2 & Q3. Thank you for this interesting point. We perform new experiments.
Preservation of task evidence. To address this concern, we introduce a new metric, Task Evidence Preservation Rate (TEPR), defined as the fraction of clean samples with correct factual content whose factual content is preserved under ESVA-R. A higher TEPR means less destruction of the original factual evidence. We use Qwen3-VL-Plus guided by a prompt to produce binary judgments on paired clean and attacked samples. Each judgment set identifies whether the clean sample holds correct factual content and whether the attacked counterpart retains this factual content. TEPR is calculated from these paired outputs. As shown in Table R2, TEPR ranges from 96.7% to 100.0% across both datasets and all three upstream VLMs. Together with the emotion-steering results in our paper’s Table 1, these findings support our claim that ESVA-R changes the emotional expression of upstream messages while largely preserving the task-relevant evidence passed downstream.
Table R2. TEPR under ESVA-R across two datasets and three upstream VLMs.
Targeted response control. To distinguish targeted response control from ordinary abstention, we extended the evaluation from the original “Unknown” target (T0) to two additional attacker-specified outputs: T1 (“Sorry, I can't help with that.”) and T2 (“The answer is unavailable.”). As shown in Table R3, compared with the T0 baselines, the average ASR over T1 and T2 across the three upstream VLMs decreases by only 4.3 percentage points for ESVA-R and increases by 1.3 percentage points for ESVA-P. These comparable results show that our attack’s effectiveness extends beyond the particular prefix and support its ability to induce different attacker-specified response forms. In addition, we examine the generation loss ablation on WebQA with Qwen2.5-VL (7B). Removing the generation loss Lgen (Eq. 8) reduces ASR from 67% to 0%, even though the poisoned entry is still retrieved, selected, and passed to the generator in 59% of trials. Therefore, the “Unknown” output reflects an independently induced target response, instead of ordinary abstention triggered when useful evidence is displaced.
Table R3. Target-output generalization on WebQA, measured by end-to-end ASR (%).