acceptodds
Withdrawn from ICLR 2027

Aligning with the Wrong Intent: How User-Intent Ambiguity Enables Semantic Attacks on Multimodal Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As large language models evolve into autonomous agents, semantic attacks can redirect their actions away from the user's intended objective. Existing studies mainly analyze specific attack patterns, leaving unclear how incomplete user instructions affect vulnerability to these attacks. We study this problem from the perspective of intent alignment: malicious contextual information may be interpreted as evidence about what the user wants. We introduce AmbiSecBench, which systematically varies instruction completeness while holding environmental injections fixed across variants of the same decision state. Across GUI, Web, and embodied environments, we measure attack outcomes, candidate-based support for the original intent, and the relative preference for attacker-induced actions. Incomplete instructions consistently reduce support for the original intent and increase attacker-relative action preferences across the evaluated model–domain pairs. Paired clean and attacked comparisons further show that amplification of the attack-induced support loss depends on the domain. An intent-auditing prompt reduces attack success, although its effect on clean-task utility remains untested. These findings characterize how instruction completeness affects agent security and motivate defenses that distinguish user authorization from untrusted contextual information.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Autonomous Agents, Semantic Attacks, Intent Alignment, Agent Security

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YiiGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AmbiSecBench, a benchmark designed to measure how ambiguity in the user query affects attack success rate in otherwise static environments across three domains of mobile GUI, desktop-based GUI, and physical environments. User prompts are sourced from the existing benchmarks AndroidControl, OSWorld, and EB-Alfred. The prompts are decomposed via the ABAC model and ambiguated by selectively removing one to three of the four attributes of the user prompt. Six small-scale and rather outdated VLMs are evaluated in a single step, with the injection embedded into the visual observation. The paper measures ASR and DSR as well as True Intent Support (TIS) and Perceived Intent-Alignment Advantage (PIAA), two quantities introduced that are similar to a logit-based probing and a reward margin, respectively. The paper finds that ambiguity increases the attack success which is mirrored in a lower TIS and higher PIAA. However, I find several shortcomings of the study. The prompts are sourced from simple agentic tasks and the modifications introduced are partially unrealistic for actually occurring ambiguity (especially L3, the strongest setting). The introduced quantities represent standard measures with new names and do not measure what they claim. The injections are weak, static per domain and not actually harmful. The paper falls short of its claims and has limited contributions.

Critical strengths
  • The question of ambiguity is relevant. The paired design of varied user prompt combined with static environment is clean in principle.
  • Benchmark includes the setting of physical environments for embodied agents.
Critical weaknesses
  • User prompts: The prompts are sourced from other benchmarks, new is only the method on how to adapt them for ambiguity. Most L3 variants are not actually senseful user tasks (also for L2 dependent on what elements are dropped). Therefore, it is unclear what even the “user’s true intent” is. The ABAC model of the prompt assumes simple, short user prompts and does not account for complex, long horizon tasks where the prompt can be more complicated. That model is overfit to a certain simple type of user tasks.
  • Models and evaluation: The study considers only small models with little reasoning capabilities, no safety-trained frontier models are evaluated, and no mitigations from prior work for reasoning over ambiguity/user intent are checked. The further defense evaluation is limited, only on a single model with a prompt that comes close to a test-set-aware defense. The benchmark claims agentic tasks, but for GUI and web only single step evaluations are considered. The PIAA scales are not calibrated per model and quantities like DSR are not introduced.
  • Injections and adversary capabilities: The injections are not diverse - they are static with essentially one template per domain. No adaptive attacks are considered. The injections do not represent actually harm, just clicking a pop-up for confirm is not necessarily a harmful action, real harmfulness is missing. The adversarial payload is tailored for ambiguity yet is assumed in the threat model that the attacker does not have knowledge about the user prompt. The threat model is not clear. For physical environments it is not stated at all. For GUI environments it is not clear, what access the attacker needs for the injection to appear at the considered evaluation step.
Questions for authors
  • Could you specify the assumptions that are made within the threat model? What capabilities does an attacker need to plant the injections at the considered places (across all considered domains).
  • How is the ambiguity handled for frontier models or at least highly-capable, safety-aligned models?
Reviewer_AE5dOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AmbiSecBench, a benchmark studying how ambiguity in user instructions affects multimodal agents' susceptibility to prompt injection. It systematically removes information from instructions while keeping the injected attack fixed, across GUI, Web, and embodied environments. The authors find that increasing ambiguity generally increases attack success rates and reduces support for the original user intent. They also propose an intent-auditing prompt as a defense.

Critical strengths

None

Critical weaknesses
  1. Questionable premise regarding the correct behavior under ambiguity.

The paper assumes that the original user intent remains the appropriate ground truth even when critical information is removed from the instruction. However, under such ambiguity, the correct behavior of a capable agent is often to ask for clarification rather than infer and execute an unspecified action. Making assumptions about user intent can itself lead to undesirable outcomes.

For example, in Figure 3, the original instruction "I want to open the existing document 2 in Microsoft Word" is reduced to "In Microsoft Word" at L3. The latter contains no actionable request. A capable agent should ask for clarification rather than attempt to open Document 2. This raises a fundamental concern about the evaluation: why should the original reference action remain the ground truth when the modified instruction no longer specifies it?

  1. Limited model evaluation and unrealistic prompt injections.

The evaluation is restricted to relatively small models (2B–8B), predominantly from 2025, without considering stronger recent models or agents hardened against prompt injection.

Furthermore, the attacks are extremely conspicuous. For example, the injected popup explicitly impersonates a user message using <user> tags and instructs the agent to click "Confirm." A security-aware agent should recognize that instructions embedded in a screenshot do not constitute genuine user instructions.

Consequently, the experiments may primarily demonstrate that these models are susceptible to obvious prompt injections, rather than establish a meaningful vulnerability arising from user-intent ambiguity. The authors should evaluate stronger recent models or agents equipped with prompt-injection defenses to establish whether the claimed vulnerability persists.

Questions for authors
  1. Is this primarily measuring susceptibility to prompt injection? How do the authors distinguish failures in user-intent inference from ordinary failures to recognize and reject untrusted instructions embedded in environmental observations?

  2. How are clarification, refusal, and abstention handled in the evaluation? When user instructions are underspecified, asking for clarification may be the most appropriate action. Are such responses considered successful defensive behavior, or are they treated as failures to follow the original reference action?

  3. Does the attack succeed against stronger models? Could the authors evaluate a subset of AmbiSecBench on recent frontier multimodal models and report prompt-injection detection rates and ASR across ambiguity levels? Given the conspicuous nature of the injections, this would help establish whether the vulnerability persists in more capable agents.

  4. How much of the defense improvement comes from basic prompt-injection awareness? The proposed intent-auditing defense explicitly instructs the model not to trust screenshot content. How does this compare with a simple baseline that merely tells the agent to treat screenshot instructions as untrusted?

Reviewer_54DUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AmbiSecBench to investigate how user-intent ambiguity affects multimodal agents' vulnerability to semantic prompt injection. It systematically removes intent attributes across GUI, Web, and embodied environments, and proposes True Intent Support (TIS) and Perceived Intent-Alignment Advantage (PIAA) to analyze intent inference and action preferences. Experiments show that ambiguity generally increases attack susceptibility, while an intent-auditing prompt reduces attack success.

Critical strengths

Interesting problem formulation. The paper studies the interaction between underspecified user instructions and prompt injection, offering a useful perspective beyond attack-specific evaluations.

Systematic evaluation. Controlling ambiguity levels while keeping injected content fixed enables meaningful comparisons. The analysis across behavior, intent support, and action preference provides complementary insights.

Critical weaknesses

Limited attack diversity and realism. The experiments primarily use fixed, manually constructed injection patterns (e.g., fabricated user clarification and embodied action instructions). It is unclear whether the conclusions generalize to diverse, adaptive, or realistic attacks. Moreover, the evaluation focuses on single-step action prediction rather than end-to-end agent execution.

Insufficient evidence isolating the security effect of ambiguity. Reduced intent support under incomplete instructions is expected even without attacks. The paired clean/attacked analysis shows significant additional TIS degradation only in the embodied setting, weakening the claim of a general ambiguity-driven attack amplification mechanism. The proposed defense also lacks a proper clean-task utility evaluation.

Questions for authors

How do the findings generalize across different attack strategies, injection locations, and adaptive attacks?

Can the authors provide a controlled comparison of attack-induced behavioral changes under clean versus attacked observations across ambiguity levels?

How does intent auditing affect legitimate task completion on clean inputs, particularly when instructions are ambiguous?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.