acceptodds
Under review as a conference paper at ICLR 2027

Valid Actions, Wrong Targets: Visual Attacks and Evidence-Based Defense for Web Agents

Abstract

Web agents translate natural-language requests and webpage observations into browser actions, while visual web agents additionally use screenshots to capture information that page text and structure may omit. This visual channel can also be manipulated: an agent may be redirected by screenshot content even when non-visual evidence is sufficient to determine the correct next action. We investigate this risk across three web-agent interfaces using paired tasks and controlled visual interventions, complemented by web observations. Image masking and image–text conflicts show that visual reliance changes with the task, even on the same page. We evaluate local noise, global noise, single-patch, and multi-patch attacks on open-weight models through generated native actions and target-element binding. Motivated by these findings, we propose Evidence-Grounded Action Repair (EGAR), a training-free defense that separates task understanding, local evidence extraction, and evidence-based correction. EGAR bypasses the screenshot when non-visual evidence uniquely determines a supported action. Otherwise, it reads task-relevant attributes from local candidate regions and checks them against the task requirements before selectively revising the target element. Offline evaluations show improved recovery of adversarially redirected actions and correction of some clean-task errors, while preserving most correct clean decisions. Our findings show that controlling visual exposure and explicitly checking task-relevant evidence provide complementary ways to improve the robustness of visual web agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
web agent, visual attack, adversarial noise, training-free defense, web-agent defense

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Wt4uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the vulnerability of visual web agents to adversarial perturbations that redirect their actions toward incorrect webpage elements while preserving valid action formats. The authors systematically analyze how web agents utilize visual and textual evidence and evaluate four types of visual attacks, including local noise, global noise, single-patch, and multi-patch across three web-agent frameworks. To mitigate these attacks, they propose Evidence-Grounded Action Repair (EGAR), a training-free defense that separates task understanding, local evidence extraction, and evidence-based action correction. Experimental results demonstrate that visual perturbations can effectively manipulate web-agent decisions, while EGAR substantially improves attack recovery and corrects some errors on clean inputs, achieving a 91.9% recovery rate on successful adversarial attacks.

Critical strengths
  1. The paper addresses an important security concern in visual web agents, highlighting how adversarial perturbations can redirect agents toward incorrect targets while preserving valid actions.

  2. The authors evaluate four types of visual attacks across multiple web-agent frameworks and model backbones, providing a systematic analysis of their effectiveness.

  3. The proposed EGAR framework achieves promising results in recovering correct actions under adversarial attacks, significantly outperforming existing image-based defense baselines.

Critical weaknesses
  1. The experiments primarily focus on single-step action selection rather than complete web-agent trajectories. It remains unclear whether the proposed defense can effectively improve task completion rates and maintain robustness in realistic, long-horizon web interactions.

  2. The proposed EGAR framework primarily combines existing techniques, including LLM-based requirement extraction, VLM-based attribute recognition, and rule-based verification. While the results are promising, the core methodology is relatively straightforward, and the paper could better distinguish its contributions from existing verification and action-correction approaches.

  3. The defense is primarily evaluated against attacks optimized for the original web agents rather than EGAR itself. An adaptive attacker could potentially manipulate the local evidence extraction process, undermining the effectiveness of the proposed verification mechanism.

  4. EGAR relies heavily on extracting explicit task requirements and matching them against observable attributes. Its effectiveness on more complex, ambiguous, or subjective user requests remains insufficiently explored.

Questions for authors

N/A

Reviewer_3orPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how adversarial visual perturbations can redirect web agents toward incorrect webpage elements while preserving valid browser actions. The authors investigate four attack settings across three web-agent pipelines and propose Evidence-Grounded Action Repair (EGAR), a training-free defense that separates task understanding, local evidence extraction, and action correction. Experiments show that visual attacks can effectively manipulate target selection, while EGAR achieves 91.9% recovery on successful attacks and preserves most correct decisions on clean inputs.

Critical strengths

1 The paper highlights that syntactically valid browser actions can still violate user intent by targeting the wrong element. 2 The four perturbation settings cover different levels of visual control. Evaluating actual generated actions and element bindings is more convincing than using token probabilities alone. 3 The authors investigate cross-backbone performance, attack transfer, clean-action preservation, and inference latency.

Critical weaknesses

1 The evaluated baselines mainly focus on image purification, while EGAR additionally performs task requirement extraction and evidence-based action correction. I wonder how it compares with simpler alternatives, such as directly prompting a VLM to verify the proposed action against the task requirements, or applying explicit requirement matching without the full EGAR pipeline. These comparisons would help clarify whether the proposed framework offers substantial benefits beyond task-grounded verification.

2 EGAR can bypass visual processing when the structured evidence is sufficient, but the paper does not report attack recovery separately for these cases and those requiring visual evidence. I wonder how much the reported improvement comes from avoiding the attacked visual channel, and how much comes from local evidence extraction and action correction. Reporting recovery rates for these two groups would help clarify the contribution of each path.

Questions for authors

Minor Comments and Typos 1 Appendix A.3: The paragraph ends with an incomplete sentence, “However, the”. 2 Section 1: There are minor citation and formatting issues, including “?He et al., 2024”.

Reviewer_dr7TOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The attack here changes nothing but pixels. Action type, input value, and action format all stay valid, so the agent still produces a well-formed action aimed at the wrong element, while page text, structure, and control bindings are left untouched. EGAR responds by splitting task understanding, local evidence extraction, and target correction into separate steps. When nonvisual evidence already pins down a supported action, it skips visual processing entirely. Otherwise it reads task-relevant attributes from candidate regions and switches targets only if explicit requirement matching supports exactly one alternative.

Critical strengths
  • Attacks are evaluated through generated actions, parsing, and binding to real page elements, which measures target misselection far more directly than changes in output wording or target-token likelihood.
  • The paired task, image-swapping, and masking experiments show how task requirements shape visual reliance. Reading local attributes separately from selecting the target also keeps the defense interpretable.
  • Coverage spans several agent interfaces, model backbones, and perturbation types under a stated attack budget, and reporting both damage to correct clean actions and repair of incorrect ones gives a more honest utility picture than a single headline number.
Critical weaknesses
  • Every tested attack was built against the undefended agent and then frozen. That says nothing about an attacker adapting to the full EGAR pipeline, including reconstruction, local reading, and target correction. Reoptimizing attacks for different base models does not close the gap.
  • The individual components are not cleanly isolated. Baselines are mostly image purification methods, and the existing local-reader ablation keeps the base proposal fixed, so it cannot tell us how much comes from bypassing screenshots, reading local regions, compiling requirements, or applying matching rules. Counts for compilation errors, fallback behavior, and valid attacks would also help interpret the recovery numbers.
Questions for authors
  • Can you optimize attacks against the complete EGAR pipeline and report residual attack success alongside correct-action recovery?
  • Can you run component ablations on the same evaluation set to isolate screenshot bypass, local evidence extraction, requirement compilation, and explicit matching?
  • Can you report task-compilation error rates, fallback rates, and both successful and valid attack counts for each condition, and explain how these affect the reported recovery results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.