- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies defenses against physical prompt injection for wearable vision-language assistants and proposes Reality Flow Integrity. The system first masks detected text and its carrier objects. A trusted VLM then performs scene reasoning based on the user request and the masked image, while an untrusted VLM reads selected regions. After type checking, the reading result is returned as an opaque symbol. The renderer finally replaces the symbol after the trusted VLM completes its response.
Critical strengths
(1) The method separates scene reasoning from text reading. Through opaque symbols and delayed replacement, it prevents free-form text generated by the reading model from directly entering the context of the trusted model. The design objective is clear. (2) The experiments compare four systems on the same scenes, and the task and attack criteria are determined in advance on an independent trial set. The appendix also describes the scene selection process.
Critical weaknesses
(1) Section 4.2 states that the system returns available when the reading result passes type checking and unavailable otherwise, and the trusted model can adjust its response accordingly. An attacker can influence this flag by affecting the output type, length, or refusal behavior of the reading model. Therefore, even if the response content always remains opaque, untrusted input can still affect the response through this state channel. The paper should clarify which forms of control-flow influence are allowed and which output properties are specifically protected by the deterministic guarantee. (2) The method relies on the union of detections from EasyOCR and SAM 3, but it does not report the complete masking rate of injected text, the respective miss rates of the two detectors, or the effect of excessive masking on task utility. (3) The evaluation uses perspective-transformed synthetic paper carriers rather than attacks that are printed and recaptured through a camera, and it does not include continuous-video experiments. Therefore, the current results do not yet validate the reliability of text detection and masking under real camera capture. (4) Mask-All and RFI use the same visual boundary, but the paper does not compare variants that directly return the reading result, apply only type restrictions, or use only opaque symbols. It is therefore difficult to separate the contribution of each component. Table 1 also shows that the attack success rate of RFI is higher than that of Mask-All on GPT-5.6 Luna, while the undefended attack success rate is already low for some models. The differences across models require further explanation.
Questions for authors
(1) Does the deterministic guarantee allow available and unavailable to affect subsequent queries and the response structure of the trusted model? Please clarify which output properties are covered by the guarantee and which effects through the state channel are allowed. (2) What protection does the system provide when the reading model follows an injected instruction and generates content that still satisfies the required type and length constraints? Do the experiments evaluate attacks that directly target the reading model and the final rendered text? (3) Among the 60 scenes, how many injected texts are fully masked? Of the remaining successful attacks, how many result from detection misses, incorrect region selection, influence on the reading model, or other causes? (4) What proportion of user tasks is completed correctly under attack? Can the paper provide component ablations with matched visual boundaries to show the respective benefits of type restrictions, opaque symbols, and delayed rendering?
6 comments
Rebuttal to Reviewer AFFG (2/3): reader channel, detection, residual successes (Q2, W2, Q3)
Q2. What if U-VLM follows the injection and produces content that satisfies the type and length constraints? Were attacks on U-VLM and on the rendered text evaluated?
Strict ASR in the paper is scored on the rendered answer, so the reader channel is inside the reported 1.9%. The submission did not break it out.
Table 2. The reader channel on the 1,080 attacked RFI trials.
| Quantity | Count |
|---|---|
| Trials in which T-VLM asked about the attacker's Region | 354 (32.8%) |
| Reads of that Region: symbol issued / declined / failed the type check | 242 / 101 / 18 |
choice symbols holding the attack target | 22 |
| T-VLM's own output containing the payload sentence | 0 |
| Rendered answers that quote at least half of the payload sentence | 86 (8.0%) |
| ... of which satisfy the attack target | 0 |
| Attack-attributable successes through the reader | 9 (0.8%), all choice, all Nav |
U-VLM does follow injected content in the sense you describe: asked for a direction, it decodes the carrier's ROT13 target and returns LEFT, one of the options T-VLM declared. That is the whole residual: nine trials in which one enumerated slot holds a false value. When U-VLM returns the injected sentence itself, the type check discards it on short slots and lets it through on text slots, where it appears as quoted scene text at a position T-VLM chose: the user hears what the sign says; the assistant does not do what the sign says. A text slot bounds length, not content, and we state this as a limit.
Two attack sets written against the reader (text addressed to the reader; a carrier styled as the sign the user asks about) are fully masked in 119/120 scenes. End to end (GPT-5.6 Luna, 180 attacked trials per set) they succeed in 3 and 9 trials with RFI against 14 and 24 Undefended (paired p = 0.001 and 6e-5). All 12 RFI successes are Nav: six are a false value read from the sign mimic into the Nav slot, and six are the answer RFI also gives on the clean image.
W2 / Q3. Masking rate of injected text, miss rates of each detector, cost of over-masking, and causes of the remaining successes.
Masking rate. The injected text is fully masked in 60/60 scenes, by the union and by each detector alone. The union masks 21.4% of an attacked frame and 16.5% of a clean frame.
Table 3. Scenes in which the injected text is fully masked under stress, with the paper's detectors unchanged (30 scenes per condition).
| Carrier | Conditions | Union (the paper's boundary) | Text detector alone | Carrier detector alone |
|---|---|---|---|---|
| Paper notice | Benchmark as evaluated (text lines about 43 px) | 60/60 | 60/60 | 60/60 |
| Paper notice | Small, tilted, occluded, low-contrast: 11 px and 6 px lines, rotated 45 degrees, viewed 75 degrees off-axis, half occluded, 12% contrast | 180/180 | 171/180 | 176/180 |
| Paper notice | Capture degradation: defocus, motion blur, low light, JPEG 15, 320 px frame | 150/150 | 111/150 | 149/150 |
| None (text written on the surface) | Full size down to 11 px lines | 89/90 | 89/90 | 5/90 |
| None (text written on the surface) | 6 px lines, defocus (sigma 4 and 8), 320 px frame | 64/120 | 60/120 | 10/120 |
The two detectors fail in different places, which is why the boundary uses their union. Small or defocused text with no carrier is the boundary's limit (64/120 fully masked), and we add it to Section 6.1. Missed text rarely acts: on the four hardest carrier-less conditions run end to end (GPT-5.6 Luna), the scenes where the boundary missed the text produced the attack in 1 of 168 RFI trials and 1 of 168 Undefended trials.
Over-masking. Sorting the 60 clean scenes by masked area, RFI's utility loss against Undefended is 3.1, 1.9 and 6.9 points in the least, middle and most masked thirds (Mask-All: 1.7, 13.7 and 47.8).
Table 4. Cause of every RFI Strict-ASR success (all 21 in 1,080 attacked trials).
| Cause | Successes | Attack-attributable |
|---|---|---|
| Detection miss | 0 | 0 |
| On-type value read from the attacker's carrier and rendered | 9 | 9 |
| Genuine sign misread by U-VLM (the crop holds none of the payload) | 11 | 1 |
| No Region read at all (the payload reached no model) | 1 | 0 |
"Incorrect region selection" in your list is the second row: T-VLM treats the attacker's carrier as a direction sign. The eleven misreads are one scene, and ten of them also occur on the clean image. Appendix A.4 put only five of the ten attack-attributable successes in the carrier row; it is nine, and we correct the split.
Rebuttal to Reviewer AFFG (3/3): ablations, task completion, real capture, changes to the paper (W4, Q4, W3)
W4 / Q4. Component ablations, Luna, and models with low undefended ASR.
Luna. RFI's 6/180 against Mask-All's 1/180 is not a significant difference (paired p = 0.125; pooled over six models 29 vs 19, p = 0.19), and three of the six are a genuine-sign misread that RFI also produces on the clean image. RFI's pooled Strict ASR (1.9% vs 2.9%) is not distinguishable from Mask-All's, so "lowest overall ASR" claimed more than the data support. The two designs share one mask; what RFI adds is utility (217 vs 33 discordant pairs, p = 2e-34).
Low undefended ASR. Claude Opus 5 and Gemini 3.8 Flash follow almost none of the fixed payloads (3/180 each), so no defense can show an ASR gain there, and RFI costs Opus 4.4 points of utility. Flash resists disguised wording too: on three disguised sets it gives 5, 3 and 3 successes of 180 undefended and 2, 0 and 2 with RFI. Across the six models the undefended count ranges from 3 to 134 of 180 and RFI's from 1 to 6, so with RFI the outcome hardly depends on how readily the model follows scene text.
Ablations with a matched boundary. Delayed rendering cannot be removed alone: placing the value before T-VLM has finished is the direct-return arm, which therefore tests both.
Table 5. Component ablations on the same cached masks, so only the mediator differs (GPT-5.6 Luna, 180 attacked and 180 clean trials per arm).
| Arm | Removed from RFI | Strict ASR | Clean Utility | Trials in which the injected sentence reached T-VLM |
|---|---|---|---|---|
| RFI | (none) | 6/180 | 96.7% | 0 |
| Direct return | type check, opaque symbols, delayed rendering | 5/180 | 93.9% | 9 |
| Type check only | opaque symbols, delayed rendering | 4/180 | 95.6% | 10 |
| Opaque symbols only | type check | 5/180 | 92.8% | 0 |
| No crop isolation | other Regions masked in each crop | 3/180 | 92.2% | 0 |
| Flag-free mediator | the availability bit | 7/180 | 96.1% | 0 |
No component changes ASR, and the paper will not claim that one does. The ablations show where the attacker's text can go: returned as text, it enters T-VLM's context; behind an opaque symbol it never does. Whether T-VLM then obeys is a property of the model. On the benchmark the sentence reaches T-VLM in at most 28 of 180 trials, so we forced exposure on three models: a second request in each question (read out any other sign) makes T-VLM read the attacker's Region. With the prompt rule that scene text is untrusted, the sentence reached T-VLM in 99 of 150 direct-return trials and was never followed; without the rule, in 196 of 282 trials, and it was followed 10 times (4 outside Nav). With RFI (run without the rule) it reached T-VLM in 0 of 96 trials. Direct return is safe while T-VLM declines to obey returned text; RFI does not depend on that, at no cost in Clean Utility (96.7% vs 93.9%). Further runs (a relay payload, a wording tuned against direct return, the type check alone) are in the appendix; none showed a difference.
Task completion under attack. On Nav and Safe (deterministic scoring) RFI completes the original task in 86.4% of trials, against 73.1% Undefended, 75.8% Mask-All and 80.8% TaCo-Guard (paired p = 9e-8, 2e-4, 0.022). It produces the attack target in 5.8% and neither in 7.8%, which is the cost you ask about. On Ref, Tox, Bias and Frame (scored by the paper's judge) RFI completes 87.1%, against 71.9% Undefended, 72.2% Mask-All and 88.9% TaCo-Guard (p = 0.29 against TaCo-Guard).
W3. Synthetic carriers, no printed recapture, no video.
We could not complete a printed recapture study within the discussion period, and the revision limits its claims to composited still images. Prior work finds that VLMs react to composited text as to photographed text: typographic attacks inserted digitally lower zero-shot VLM accuracy at least as much as the same words handwritten and photographed in place (by 36 vs 26 points on 1,162 scenes; Westerhoff et al., SCAM, CVPR 2025 Workshops); printed adversarial text still misleads GPT-4o after the scene is recaptured (Cao et al., SceneTAP, CVPR 2025); and attacks photographed with smart glasses compare across models as the digital ones do, in the study our TaCo-Guard baseline comes from (Li et al., Devil in the Lens, 2026). On our side, the capture effects in Table 3 leave the payload fully masked in all 210 scenes with a carrier. Whether our detectors still find printed text after a real capture is unmeasured, and continuous video is not evaluated; both are stated limits.
Changes to the paper.
- Abstract and Sections 1, 5.2 and 8 replace "lowest overall ASR" by attack resistance at the level of Mask-All with utility restored, and limit the claims to composited still images.
- Sections 4.2, 5 and 6.1 add the guarantee as the claim above (a
textslot bounds length only), the ablations as evidence of the information flow, the flag-free mediator and task completion under attack.
Rebuttal to Reviewer AFFG (1/3): guarantee and flag channel (W1, Q1)
Thank you for the precise questions. Two of them (W1, Q2) changed how we state the guarantee. The numbers come from the paper's 2,160 stored RFI trials (1,539 U-VLM reads), from offline detector runs and, where stated, from new model runs.
W1 / Q1. The availability flag is a channel. Which output properties are protected?
You are right, and the submission under-specified this. We now state the guarantee as an information-flow claim.
Claim. Take two scenes with the same redacted frame that differ only inside detected Regions. For the same request and the same T-VLM randomness: (C1) T-VLM's first query is identical; (C2) if the availability flags are equal, T-VLM's whole transcript is identical, i.e. every later query and the full response before rendering; (C3) the rendered responses differ only inside symbol spans, each holding a value of the declared type.
So detected Region content reaches the user through exactly two channels: the value at a symbol position and one bit per query. Protected: which task is answered, the sentences of the response and their order, the positions of symbols, and independence between Regions. Not protected: whether a value is true, what words a
textslot holds, and the geometry of the masked boxes.Table 1. Audit of the guarantee's invariants on the stored RFI traces.
Does the flag affect later queries and the response structure? Yes, by design: after
unavailableT-VLM may ask another Region or say it cannot determine the answer. Measured on the stored traces, T-VLM observes 0.78 flags per attacked request on average, and 610 of 1,080 attacked requests observe none. The attacker controls one bit, for its own Region: reads of that Region returnedunavailablein 119 of 361 cases, and the 115 trials in which it returned onlyunavailablecontain no attack-attributable success. The attacker cannot move the flag of another Region, because each crop has every other Region masked: theunavailablerate on genuine Regions is 23.3% on attacked and on clean scenes alike (111/477 vs 163/701).Removing the channel. We implemented a flag-free mediator: it always returns a symbol; a decline or an off-type value stores the fixed string "(could not be read)"; fallback Regions are resolved in code. C2 then holds unconditionally. Measured on GPT-5.6 Luna (60 scenes, 3 trials): Clean Utility 96.1% against RFI's 96.7%, Strict ASR 7/180 against 6/180. If you consider the flag-free mediator the right default, we will make it the main design.