- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates multimodal memory poisoning against web agents. MemoryMischief combines a visual trigger optimized for malicious-memory retrieval with a reusable post-recall component using surrogate-based adversarial perturbations and OCR instructions. A separate memory component specifies the malicious goal. Experiments cover three agent frameworks, four VLM backbones, four sandbox attack categories, and two memory schemas. The paper reports high retrieval and action success, including up to 99.15% end-to-end success, together with component ablations and lightweight defense evaluations.
Critical strengths
-
The study connects poisoned-memory retrieval to executable agent behavior. Separating retrieval success, conditional action success, and end-to-end success provides a useful evaluation framework. The deterministic sandbox checks also offer more concrete evidence than judging malicious textual responses alone.
-
The component ablations show a meaningful multimodal contribution. In Table 4, adding the complete visual component increases end-to-end success from 73.59% for text-only poisoning to 93.97%. The evaluation across several agents and VLM backbones provides additional evidence that the observed vulnerability is not confined to one policy model.
Critical weaknesses
-
The claimed black-box, memory-only threat model is not adequately reconciled with the method. Section 3.1 grants the attacker only memory-level access and excludes access to retriever internals. However, retrieval optimization explicitly uses the retrieval encoder’s embeddings and benign-memory embeddings, while attack activation requires a crafted trigger in the current webpage observation. The paper does not clearly explain how these capabilities are obtained under the stated restrictions. Figure 4 provides surrogate-to-target retrieval results, but does not establish the complete attack under an unknown retriever. This distinction matters because the headline success rates may depend on stronger access than the proposed practical threat model allows.
-
The experimental protocol lacks details needed to assess the headline results and comparisons. The manuscript does not sufficiently specify task counts and splits, the default memory corpus size and retrieval depth, numerical trigger and perturbation settings, or the adaptations and tuning of the baselines. Attack success rates also lack trial counts and uncertainty estimates, although benign utility includes repeated-run statistics. Furthermore, all three agents use the authors’ shared memory interface, so the agent matrix provides limited evidence about independent memory implementations. These gaps make it difficult to verify whether the gains persist under matched poisoning budgets and realistic retrieval conditions.
Questions for authors
-
What exact access is required during retrieval-stage optimization? Are the main results obtained using the victim retriever or a surrogate, and who controls insertion of the trigger into the current webpage?
-
Can you provide the complete evaluation configuration, including task counts, train/test separation, memory size, retrieval depth, attack budgets, baseline adaptations, and confidence intervals for ASR-ra?
-
Does a single injected memory remain effective across later episodes as benign memories accumulate, without reinsertion or re-optimization? This would materially strengthen the claimed persistence beyond reuse in fixed evaluation settings.
Dear Reviewer,
Thank you again for your detailed and technically specific feedback.
We have made considerable efforts to address all three questions you raised. Beyond clarifying the threat model and surrogate-based optimization setting, we provided a more comprehensive experimental configuration, including task splits, memory size, retrieval settings, attack budgets, baseline adaptations, and repeated-run statistics. More importantly, we conducted an additional memory accumulation experiment, demonstrating that a single poisoned memory maintains substantial retrieval effectiveness as the memory corpus grows from 740 to 6418 vectors, without reinjection or trigger re-optimization.
We sincerely hope these additional clarifications and experiments address your concerns. As the rebuttal period is approaching its end, we would greatly appreciate any further feedback you may have.
Thank you very much for your time and consideration.