- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces AmbiSecBench, a benchmark designed to measure how ambiguity in the user query affects attack success rate in otherwise static environments across three domains of mobile GUI, desktop-based GUI, and physical environments. User prompts are sourced from the existing benchmarks AndroidControl, OSWorld, and EB-Alfred. The prompts are decomposed via the ABAC model and ambiguated by selectively removing one to three of the four attributes of the user prompt. Six small-scale and rather outdated VLMs are evaluated in a single step, with the injection embedded into the visual observation. The paper measures ASR and DSR as well as True Intent Support (TIS) and Perceived Intent-Alignment Advantage (PIAA), two quantities introduced that are similar to a logit-based probing and a reward margin, respectively. The paper finds that ambiguity increases the attack success which is mirrored in a lower TIS and higher PIAA. However, I find several shortcomings of the study. The prompts are sourced from simple agentic tasks and the modifications introduced are partially unrealistic for actually occurring ambiguity (especially L3, the strongest setting). The introduced quantities represent standard measures with new names and do not measure what they claim. The injections are weak, static per domain and not actually harmful. The paper falls short of its claims and has limited contributions.
Critical strengths
- The question of ambiguity is relevant. The paired design of varied user prompt combined with static environment is clean in principle.
- Benchmark includes the setting of physical environments for embodied agents.
Critical weaknesses
- User prompts: The prompts are sourced from other benchmarks, new is only the method on how to adapt them for ambiguity. Most L3 variants are not actually senseful user tasks (also for L2 dependent on what elements are dropped). Therefore, it is unclear what even the “user’s true intent” is. The ABAC model of the prompt assumes simple, short user prompts and does not account for complex, long horizon tasks where the prompt can be more complicated. That model is overfit to a certain simple type of user tasks.
- Models and evaluation: The study considers only small models with little reasoning capabilities, no safety-trained frontier models are evaluated, and no mitigations from prior work for reasoning over ambiguity/user intent are checked. The further defense evaluation is limited, only on a single model with a prompt that comes close to a test-set-aware defense. The benchmark claims agentic tasks, but for GUI and web only single step evaluations are considered. The PIAA scales are not calibrated per model and quantities like DSR are not introduced.
- Injections and adversary capabilities: The injections are not diverse - they are static with essentially one template per domain. No adaptive attacks are considered. The injections do not represent actually harm, just clicking a pop-up for confirm is not necessarily a harmful action, real harmfulness is missing. The adversarial payload is tailored for ambiguity yet is assumed in the threat model that the attacker does not have knowledge about the user prompt. The threat model is not clear. For physical environments it is not stated at all. For GUI environments it is not clear, what access the attacker needs for the injection to appear at the considered evaluation step.
Questions for authors
- Could you specify the assumptions that are made within the threat model? What capabilities does an attacker need to plant the injections at the considered places (across all considered domains).
- How is the ambiguity handled for frontier models or at least highly-capable, safety-aligned models?