- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the vulnerability of visual web agents to adversarial perturbations that redirect their actions toward incorrect webpage elements while preserving valid action formats. The authors systematically analyze how web agents utilize visual and textual evidence and evaluate four types of visual attacks, including local noise, global noise, single-patch, and multi-patch across three web-agent frameworks. To mitigate these attacks, they propose Evidence-Grounded Action Repair (EGAR), a training-free defense that separates task understanding, local evidence extraction, and evidence-based action correction. Experimental results demonstrate that visual perturbations can effectively manipulate web-agent decisions, while EGAR substantially improves attack recovery and corrects some errors on clean inputs, achieving a 91.9% recovery rate on successful adversarial attacks.
Critical strengths
-
The paper addresses an important security concern in visual web agents, highlighting how adversarial perturbations can redirect agents toward incorrect targets while preserving valid actions.
-
The authors evaluate four types of visual attacks across multiple web-agent frameworks and model backbones, providing a systematic analysis of their effectiveness.
-
The proposed EGAR framework achieves promising results in recovering correct actions under adversarial attacks, significantly outperforming existing image-based defense baselines.
Critical weaknesses
-
The experiments primarily focus on single-step action selection rather than complete web-agent trajectories. It remains unclear whether the proposed defense can effectively improve task completion rates and maintain robustness in realistic, long-horizon web interactions.
-
The proposed EGAR framework primarily combines existing techniques, including LLM-based requirement extraction, VLM-based attribute recognition, and rule-based verification. While the results are promising, the core methodology is relatively straightforward, and the paper could better distinguish its contributions from existing verification and action-correction approaches.
-
The defense is primarily evaluated against attacks optimized for the original web agents rather than EGAR itself. An adaptive attacker could potentially manipulate the local evidence extraction process, undermining the effectiveness of the proposed verification mechanism.
-
EGAR relies heavily on extracting explicit task requirements and matching them against observable attributes. Its effectiveness on more complex, ambiguous, or subjective user requests remains insufficiently explored.
Questions for authors
N/A