- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies agents built on vision language models that operate a computer by reading screenshots. When the agent takes a value from the screen, say a contact address shown in a card, the image keeps the value but drops the information about where that value came from. The paper calls the resulting attack Visual Data Injection. An attacker places its own value inside visual content it controls, and the agent copies that value into a sensitive argument of a tool call, such as the recipient of a message. The attack uses no hidden instruction and does not change what the user asked for. The agent does exactly what the user requested, it just takes the value from the wrong source.
To measure this cleanly, the authors build matched pairs of cases. Both cases in a pair share the same image file, the same user request, and the same shown value. Only a hidden label differs, marking the source as trusted in one case and attacker controlled in the other. Since the model sees identical input, it cannot behave differently across the pair. Under this setup, Qwen2.5-VL and Qwen3-VL adopt the attacker's value in 98 of 100 cases each, and GPT-4o in 99 of 100. The same pattern holds for deletion targets and command arguments, not only message recipients.
The defense has three parts. First, the system finds the source of incoming data on its own rather than trusting the model, for example by reading the frame address from the live browser. Second, it records where each value came from as the value moves through tools and memory. Third, before a tool call runs, a checker tests each argument against rules saying which sources may fill which role, so the right to run an operation is checked separately from the right to supply its arguments.
Validation comes in several pieces. With source labels supplied by hand, the checker blocks every attack, which shows the checking rules behave as designed. In a live browser test, the system reads real frame origins, keeps all legitimate executions (33 of 50 for Qwen2.5 and 34 of 50 for GPT-4o), and blocks all 50 cross origin attacks for both models, while a security aware prompt and a task alignment baseline fail to separate the two cases. Memory tests show that keeping the original source of a stored value blocks its later misuse, while trusting the memory container lets the attack through. Error tests show that a missing source triggers a confirmation request, while a source wrongly marked as trusted lets every attack through. A separate prototype recovers which source painted which pixels by suppressing one source at a time and comparing renders, though only for sources registered in advance. The authors also report negative results, for example the attack chain does not complete in a live multi turn loop, and they scope their claims to the controlled settings tested.
Critical strengths
- The paper isolates a real and previously blurry failure, and gives a clean way to measure it.
Agents that act from screenshots can pick up attacker values that carry no injected instruction and change nothing the user asked for. Earlier work on prompt injection mixes this case with instruction tricks and visual perception tricks, so it was hard to tell where the failure actually sits. The matched pair design fixes this. Two cases share the same image file, the same user request, and the same shown value, and only a hidden source label differs. A model that only sees the screen cannot act differently in the two cases, and the experiments confirm this exactly, with near identical adoption in both arms. The design also removes cheap shortcuts. The values have identical format, the trusted and attacker assignments are swapped across arms, and the shared image files are hash verified, so the result cannot be explained by the model spotting odd strings. This is consequential for my assessment because it turns a vague worry into a pinned down problem with a reusable measuring tool, and it proves the failure lives in the pipeline rather than in model cleverness, so model side fixes alone cannot close it.
- The evaluation is careful and honest in a way that makes the security claims believable.
The numbers are internally consistent, the statistical tests are the right choice for paired data, and the authors report results that cut against them, including a live multi turn test where the attack never completes and a small model that could not run the execution harness at all. The ablations do real explanatory work. Merging two argument roles into one check quietly restores the attack, which pinpoints where checking granularity matters. Marking a source as unknown leads to confirmation requests, while marking an attacker source as trusted lets every attack through, which shows the whole defense stands or falls on getting the source right. In the live browser test, reading the frame origin from the browser itself keeps all legitimate executions (33 of 50 and 34 of 50) and blocks all 50 attacks for both models, without changing anything the model sees. This is consequential for my assessment because the paper's practical message, that a system should check who may run an operation separately from who may supply its arguments, is backed by controlled evidence and honest negative results rather than by a plausibility argument.
Critical weaknesses
- The main defense numbers are close to guaranteed by the setup, and the checking machinery largely repeats concurrent work.
In the reference origin experiments, the checker is handed the true source label of every value and matches arguments by literal string comparison. Blocking all attacks under these conditions is nearly automatic, and it repeats what PACT already showed, namely that with given provenance, role based checks achieve perfect security on diagnostic suites. The gate design itself closely follows ECA. What remains new is obtaining the source from outside the model, and that part is validated for exactly one source class, iframe origins in a local two server setup, under the assumption that source metadata cannot be forged. The paper's own error test shows how much this assumption carries. Once an attacker source is falsely labeled as trusted, all 36 configurations let the attack through. So the defense reduces to a root of trust that the paper does not build, and the third stated contribution, the defense framework, is mostly an integration of existing mechanisms plus one acquisition result for a narrow case. This is consequential for my assessment because a fair share of the paper's weight sits on the defense claim, and that claim is thinner than the headline zeros suggest.
- The attack is demonstrated only in a forced single shot setting, and end to end harm is never shown.
The main protocol tells the model to return exactly one tool call using the displayed value, so adopting the shown value is close to instructed behavior rather than a discovered failure. The one live multi turn test points the other way. 46 of 50 attack cases end in an explicit stop, 4 make benign calls, and nothing sensitive executes. The authors call this limited integration evidence, which is honest, but it means the practical conversion of the attack into real harm is asserted by construction, not demonstrated. Model coverage adds to the concern. No purpose built computer use model is run through the execution experiments, no Claude computer use, no computer-use-preview class model, no UI-TARS, and Qwen3 appears only in the adoption experiment without being disclosed as adoption only in the limitations. This is consequential for my assessment because the paper frames VDI as an attack, while the evidence shows an information boundary plus a forced choice demonstration, which is a weaker claim and should be stated as such.
Questions for authors
Some points that require further clarification:
-
Table 3. What exactly does the Decision audit column count, and over what denominator. Under the raw no guard condition, Qwen2.5 shows 50 allow while GPT-4o shows 34 allow for the same policy. As printed this reads like an error. If the difference comes from GPT-4o requesting confirmation in the remaining 16 cases, please state this and define the decision classes.
-
Multi-slot suite. Qwen2.5 adopts attack values in 41 of 50 cases, but only 33 execute under no enforcement. Please state whether the 8 missing cases are the planning failures described for the clean arm, and whether the clean and attack planning failures are the same cases.
-
Multi-slot images. Do the clean and attack arms share the same image files, as in the counterbalanced suite, or do the screenshots differ. If they differ, what rules out visual confounds in Tables 6 and 7.
-
Qwen3. Please confirm that all execution results cover only Qwen2.5 and GPT-4o, and explain why Qwen3 was excluded from the execution experiments. The limitations disclose this for InternVL but not for Qwen3.
-
Live loop, Appendix G.1. Which model was used, and why did no attack chain complete. Was the prompt, the environment, or the monitor different from the single shot setting. This matters because the attack evidence rests on forced single call demonstrations, and the live result currently reads as an unexplained contradiction.
Minor questions or suggestions I have:
-
Same origin attacker content, for example user generated content hosted on the trusted site itself, where the origin check gives no signal. A short discussion of whether any acquisition mechanism could cover this case would help bound the approach.
-
Running one purpose built computer use model, for example a Claude computer use or UI-TARS class model, through the browser origin protocol, even at small scale, would show whether the adoption pattern holds for models trained for this setting.
-
Reporting the per step time cost of acquisition and checking in the live loop, and how the 2.94 second cold start scales with the number of registered sources, would clarify deployment practicality.
Thank you for the review.
Q1. What is new relative to ADI, PACT, and ECA if correct labels suffice to reject attacks?
Rejection is expected when the reference-origin checker receives correct labels. Our contribution is to isolate the information missing at the screenshot interface. Each matched-raster pair fixes the screenshot, request, and candidate value while changing only the source's permission to supply it. A screenshot-only agent thus sees identical inputs. The browser-origin experiment tests whether independently acquired frame metadata can supply the missing evidence at execution time. Falsely marking an external source as trusted allows its value through. The appendix “Positioning against concurrent formulations” relates this result to prior work on data injection and argument authorization.
Q2. What do the forced single-call tasks establish beyond instructed adoption?
The reference-origin prompt requires the displayed value and forbids preliminary inspection, so adoption measures behavior under that constraint. The browser-origin prompt permits inspection or confirmation. We replay one proposal per prompting condition in both live origin settings. Without enforcement, the test outbox records sends to the candidate address in 33/50 cases for Qwen2.5 and 34/50 for GPT-4o under each origin. Enforcement preserves all corresponding same-origin sends and reduces cross-origin candidate sends to 0/50 for both models. Q7 reports the separate autonomous test.
Q3. How do Decision audit and execution counts differ?
Decision audit reports selected policy outcomes across 50 cases. Clean and Attack count completed calls using the candidate recipient. In the raw row, Qwen2.5 has 50 allows and 33/50 candidate executions in each origin condition. GPT-4o has 34 allows and 34/50 candidate executions in each. The browser policy also allows address-book contacts and requests confirmation for unmatched addresses. The audit gives only a partial breakdown of decisions.
Q4. Why do 41/50 adoptions yield 33/50 executions? Why did clean tasks fail?
Qwen2.5 placed the candidate in the target argument in 41/50 attack cases (16/17 recipient, 9/17 deletion, 16/16 command). Without enforcement, 33/50 produced the corresponding tool effect. A plan can contain the value even if the executor later stops on a non-allow decision or an adapter failure. All 16 clean failures involved unsupported preliminary reads (nine recipient, seven deletion). Role-specific enforcement preserved clean completion at 34/50 and reduced attack execution from 33/50 to 0/50.
Q5. Are multi-slot pairs pixel-identical?
The verified pixel-identical control applies to the primary value-counterbalanced single-slot suite, where each pair shares an image path and SHA-256 hash. The multi-slot suite reuses the same 50 source pages and generates values deterministically to test recipient, deletion-target, and command roles.
Q6. Which models contributed execution or adoption results?
We report execution for Qwen2.5 and GPT-4o, and single-slot adoption for Qwen3. InternVL contributes adoption results only. Its extra reads and non-canonical payload fields made an execution comparison in the recipient fixture uninformative.
Q7. Did the attack recur in the live loop?
The loop ran for up to four turns. On each turn, the evaluator captured the Playwright page, requested one action, applied the monitor, and returned the result to the agent. Of 50 attack cases, 46 ended in an explicit stop and four in a benign page call. None reached sensitive execution. The successful attack executions reported in the paper came from controlled replay.
Q8. Can origin identify attacker-controlled content on a trusted site?
Browser origin identifies a transport boundary. Same-origin user content needs a distinct content principal and a rule governing which arguments it may supply. Such a rule could also authorize a legitimate cross-origin service. Our prototype tests a configured same-origin/cross-origin boundary.
Q9. Did you test purpose-built computer-use models?
The execution evaluation covers Qwen2.5 and GPT-4o. It does not include a purpose-built computer-use model.