- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes AHEAD, a lightweight adversarial detector for GUI grounding models. It uses head-wise attention entropy and Mahalanobis distance to detect abnormal screenshots without adversarial training data or additional model inference.
Critical strengths
- I find the problem timely and practically relevant to GUI agent security.
- The experiments are extensive, covering four models, two datasets, and three attack types, with strong detection results and low overhead.
- I appreciate the ablation studies, cross-domain adaptation experiments, and adaptive attack evaluation.
Critical weaknesses
- Attention entropy has been explored for detecting abnormal VLM behavior [1], and related attention-based detection methods already exist [2, 3]. Mahalanobis scoring is also well established. I see the contribution mainly as combining existing techniques, rather than introducing a fundamentally new detection method.
- Although the features are extracted from coordinate-answer tokens, the algorithm does not explicitly model GUI geometry, element relationships, coordinate semantics, or execution behavior.
- Entropy loses spatial information: attention can shift to a completely different GUI element while keeping the same entropy. Visual attention sinks [4] may also influence the statistics without reflecting meaningful grounding behavior. I think these are inherent limitations of the representation.
- The method requires internal attention access, making it unsuitable for black-box APIs. More importantly, zero-shot cross-domain transfer gives 100% FPR in both directions with source-calibrated thresholds. Performance recovers with 50 clean target samples, but this requires domain-specific adaptation. Cross-model transfer is not evaluated, and the method has not been validated in complete multi-step agent workflows.
[1] Backdoor Cleaning without External Guidance in MLLM Fine-tuning. NeurIPS 2025.
[2] Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens. CVPR 2025.
[3] VADE: Visual Attention Guided Hallucination Detection and Elimination. Findings of ACL 2025.
[4] See What You Are Told: Visual Attention Sink in Large Multimodal Models. ICLR 2025.
Questions for authors
- Could the authors compare AHEAD with a layer-wise attention entropy baseline inspired by [1], using the same Mahalanobis scoring and calibration? I would like to understand how much improvement specifically comes from the head-wise representation.
- Have the authors examined whether attention sinks affect detection? Could they report results after excluding sink-dominated heads or tokens, or discuss cases where attention moves spatially without significant entropy changes?
- Does AHEAD generalize to unseen models without rebuilding the clean reference? Could the authors clarify the distinction between zero-shot transfer and the reported 50-sample adaptation?
- The adaptive evaluation assumes that the attacker does not know the exact detector head subset. How would AHEAD perform against an attacker who knows the selected heads? Could the authors further clarify this?
7 comments
Response to Reviewer 9GpV (2/4)
[W2, W3, Q2] Attention sinks, and spatial information
Kang et al. [1] found that masking visual sink tokens had little effect on task performance and that redistributing some of their attention to other visual tokens improved performance. In our grounding experiments, sink tokens also provided a useful detection cue because visual attacks changed how much image attention the model assigned to them. AHEAD also detected attacks using only the attention entropies of sink-dominated heads. These results support keeping such heads in the pool from which AHEAD randomly samples.
We tested UI-Venus-1.5-2B on AMEX and UGround-V1-2B on Mind2Web. We report TPR at 3% FPR for the sink analyses. A dagger (†) marks follow-up analyses.
Sink tokens. To test whether AHEAD relies on sink tokens, we identified them using the criterion in [1] (), removed them, and renormalized attention over the remaining image tokens before computing entropy. We then refit the clean reference and recalibrated the threshold. In each of 20 draws, we used the same 300 heads in both conditions and averaged TPR across the three attacks. The table reports the median across draws.
| Model / dataset | Image tokens marked as sinks | TPR with all image tokens | TPR after removing sinks |
|---|---|---|---|
| UI-Venus / AMEX | 79% | 100.0% | 99.9% |
| UGround / Mind2Web | 29% | 99.8% | 99.3% |
The share of image attention assigned to sinks alone yielded 81.1–99.7% TPR†. Removing sink tokens reduced AHEAD’s TPR by at most 0.5 percentage points. Attacks therefore affect both the share of attention on sinks and how attention is distributed over the remaining tokens.
Sink-dominated heads. To test whether AHEAD relies on heads that attend mostly to sinks, we classified a head as sink-dominated if it assigned at least half of its image attention to sinks on clean fitting screens. We ran AHEAD on each group separately.
| Heads used by AHEAD | UI-Venus / AMEX | UGround / Mind2Web |
|---|---|---|
| Sink-dominated heads only† | 100.0% (363 heads) | 83.7% (85 heads) |
| All other heads | 84.7% (85 heads) | 98.7% (249 heads) |
On UGround, the 85 sink-dominated heads performed within the range of random 85-head subsets†. On UI-Venus, 85 random heads reached 99.7% TPR†, compared with 84.7% for the 85 heads remaining after exclusion. Sink-dominated heads can detect attacks on their own.
Spatial shifts. Entropy measures attention concentration and discards location. To test how often a head shifts focus with little change in entropy, we compared each clean screen with its redirect, disrupt, and patch versions†. We defined a head’s clean focus region as the smallest set of image tokens receiving at least half of its clean image attention, together with their neighbors. We counted a shift when that region retained less than half of its original attention after the attack. We treated an entropy change as small if its magnitude was below one clean standard deviation for that head.
| Spatial-shift measure | Range across settings |
|---|---|
| Share of all screen–head pairs with a shift and a small entropy change | 3.4–10.1% |
| Share of shifted pairs with a small entropy change | 25–42% |
| Median number of heads per screen with both properties | 8–31 of 300 |
| Share of pairs that shifted under redirect or disrupt | 9–39% |
| Share of pairs that shifted under random noise at the same budget () | 0.3–0.4% |
At its decision threshold, AHEAD detected every redirect and disrupt attack in this paired analysis and every UI-Venus patch attack. On UGround, it detected 58% of patch attacks overall. Detection was 67% on screens with the most heads that shifted focus with little entropy change and 55% on the remaining screens (95% CI for the difference, −2 to +25 percentage points). The reviewer’s concern holds for individual heads. AHEAD can still detect an attack when other heads change their attention concentration, and we found no evidence that UGround patches were harder to detect on screens with more such shifts.
[1] See What You Are Told: Visual Attention Sink in Large Multimodal Models. ICLR, 2025.
Response to Reviewer 9GpV (3/4)
[W4, Q3] Unseen models and clean target-domain adaptation
Model access and transfer. AHEAD requires a new clean reference for each grounding model. Each feature measures the entropy of a specific head, and we have not established that heads at the same index behave alike across models. We designed AHEAD for operators who run an open-weight GUI model and can read its attention during inference. Such models are competitive on grounding benchmarks. On ScreenSpot-Pro, UI-Venus-1.5-8B achieved 68.4% accuracy, compared with 64.3% for Seed1.8 (Table 1 in [1]). We thank the reviewer for raising the black-box setting. We plan to study detection for APIs that expose only the model’s output.
Zero-shot transfer and 50-screen adaptation. For a fixed grounding model, we use zero-shot transfer when the source reference and threshold keep the false-alarm rate within the operator’s tolerance. When the visual-token grid changes shape or the agent moves to a new platform, we check the source threshold on clean target screens. If the false-alarm rate exceeds that tolerance, we adapt AHEAD using 50 clean target screens.
| Clean evaluation condition | Visual-token grid (UI-Venus) | Mean FPR at across four models (%) |
|---|---|---|
| Original AMEX screens | 19×42 | 2.3 |
| Lower resolution | 19×42 | 2.3 |
| Simulated dark mode | 19×42 | 2.8 |
| Unseen apps | 19×42 | 4.3 |
| AMEX screens cropped from 9:19 to 9:16 | 21×38 | 26.8 |
| AMEX screens padded to match Mind2Web | 31×26 | 100.0 |
| Mind2Web | 31×26 | 100.0 |
The table shows when the original AMEX thresholds worked across four models. Mean FPR ranged from 2.3% to 4.3% across conditions that preserved the grid shape, including seven unseen apps. Cropping and padding changed the grid shape. Mean FPR rose to 26.8% after cropping and 100% after padding. Appendix E describes how we use the 50 screens to update the reference mean and covariance and set a new threshold from held-out scores using five-fold cross-fitting.
Evaluation on multi-step agent workflows. To measure how often an alarm would interrupt a multi-step agent task, we used an oracle planner to select target elements in AndroidWorld [2] and UI-Venus-1.5-2B to predict where to click. The agent completed 101 of 112 runs. We fit and calibrated AHEAD on other task types, then scored all 675 grounding steps in the 101 completed runs. The table shows how many runs would still have finished if the agent had stopped at the first alarm. Percentages are relative to the 101 runs completed without AHEAD.
| Grounding steps per run | Runs | Task completion at 3% target step FPR | Task completion at 1% target step FPR |
|---|---|---|---|
| 1–9 | 86 | 80/86 (93.0%) | 81/86 (94.2%) |
| 10–19 | 10 | 10/10 (100%) | 10/10 (100%) |
| 20–24 | 5 | 5/5 (100%) | 5/5 (100%) |
| All | 101 | 95/101 (94.1%) | 96/101 (95.0%) |
At a target step FPR of 3%, AHEAD flagged 17 of 675 clean steps (2.5%). Those alarms occurred in six runs (5.9%). All 15 runs with 10–24 grounding steps would have finished at either threshold.
[1] UI-Venus-1.5 Technical Report. arXiv, 2026.
[2] AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. ICLR, 2025.
Response to Reviewer 9GpV (1/4)
We thank the reviewer for highlighting the practical importance of GUI agent security and for suggesting a direct comparison with layer-wise entropy. We also examine attention sinks, shifts in attention that leave entropy nearly unchanged, and attacks that target the exact heads AHEAD uses.
New experiments
[W1, Q1] Contribution of head-wise attention entropy
Head-wise versus layer-wise entropy. To measure what is gained by retaining individual head entropies, we compared AHEAD with a layer-wise baseline based on [1]. Both use the same coordinate-answer tokens, clean data splits, Mahalanobis score, covariance regularization, and threshold calibration. We changed only how we summarized entropy. As the table below shows, keeping 300 individual head entropies raised mean TPR@3%FPR from 88.4% to 99.0%, a gain of 10.6 percentage points.
Across 24 settings, AHEAD outperformed the layer mean in 18 and tied it in six. Under attack, some heads focus their attention on fewer screenshot regions while others spread it more widely. A layer mean can hide these opposing changes. We use the established Mahalanobis score to measure those changes. We will add this comparison to Section 6.5.
[1] Backdoor Cleaning without External Guidance in MLLM Fine-tuning. NeurIPS, 2025.