acceptodds
Withdrawn from ICLR 2027

Defending VLMs Against Typographic Attacks via Query-Irrelevant Semantic Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, typographic attacks, adversarial robustness, black-box defense, semantic editing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gfG3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose QuISE, a training-free black-box defense against typographic attacks on VLMs, aiming to mitigate the influence of misleading text in images on model responses. Specifically, the VLM first recognizes the text content; OCR provides its locations, and the VLM then uses the query to select text regions that may influence the answer. Next, rules are used to select replacement candidates from a fixed word pool, and the VLM verifies whether the candidates are irrelevant to both the query and the image with the selected text removed. Finally, two edited images with different replacement texts are generated. The edited answer is adopted only when both images produce valid and consistent answers. Otherwise, the answer from the original input image is retained.

Critical strengths
  1. The motivation behind the method is clear. The authors first compare appearance modifications with semantic modifications, then analyze query relevance and image relevance, and finally propose Q0 I0 replacements.

  2. QuISE is a black-box method that is more suitable for real-world deployment. It does not require modifying the target model weights, and API access alone is sufficient.

  3. The paper is logically organized, easy to understand, and easy to follow.

Critical weaknesses
  1. The loss in model utility on clean inputs is too large. In Figure 7, the clean text reading retention of several models is around 80%, representing a decrease of approximately 20% compared with the clean model. The loss in recognition performance on clean inputs is substantial.

  2. Using two edits may not be optimal. Table 2 shows that three edits achieve higher accuracy and recovery rates. Could using more edits lead to better results?

  3. QuISE only considers images containing a single text region and does not evaluate complex scenarios with multiple text regions. The consensus rule also cannot rule out shared errors.

  4. The replacement text in QuISE may not need to be relevant to the image, and its benefit is unclear. In Figure 4, LLaVA achieves an accuracy that is 0.1% lower when the replacement text is irrelevant to both the query and the image than when it is irrelevant to the query but relevant to the image.

Questions for authors
  1. What specific criteria were used to select two edits in QuISE? Could the authors report the trade-offs among larger numbers of edits, consensus rules, and actual costs on other models?

  2. Can QuISE perform accurate calculations or question answering in complex scenarios with multiple text regions, and can it be evaluated in scenarios where genuine text and attack text coexist?

  3. Why must the candidate words also be irrelevant to the image, namely I0? The independent benefit of this requirement is unclear.

Reviewer_szpiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Typographic attacks, a new attack inserting misleading text that redirects answers away from visual evidence without changing the queried object, pose a critical threat to VLMs. The authors claim existing defense methods are difficult to generalize to closed-source VLMs. And they observed that query relevance largely determines textual influence, and image relevance substantially affects the answer when the text is query-relevant, but has limited impact once the text becomes query-irrelevant. Therefore, this paper proposes QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. Specifically, QuISE identify text regions via Influence-Aware Text Localization stage. Then, Query-irrelevant Semantic Editing-based Defense creates two edited images using semantically distinct replacements. Experimental results on three attack benchmarks and four open-weight and closed-source VLMs show that QuISE outperforms existing representative defense methods.

Critical strengths
  1. The proposed defense method, QuISE, is a model-agnostic, training-free black-box defense, which can be easily applied and adapted to different VLMs, with no addtional cost.
  2. The authors explored the impact of appearance and semantic modifications and identify which replacement semantics most effectively weaken attack-text influence. And the two observations they got are meaningful to understand the potential mechanism of defense methods.
  3. The motivation of this paper is well-written and clear.
  4. The experiments are conducted on different datasets with multiple VLMs, covering mainstream settings.
Critical weaknesses
  1. The novelty and contributions are limited. The proposed defense use existing text-localization models in the first stage. Then edit the target regions using common image processing operations. The key technical contribution is the editing strategy, which is query-irrelevant semantic. Please clarify the main contribution and what challenges this paper has addressed.
  2. The typographic attack can be transfered into real world. How does the proposed QuISE defense perform in the real-world setting? There are many robustness issues such as lighting, angle, distance, etc. Experiments in the digital space only cannot demonstrate the real performance when we deploy the defense method in real world.
  3. Are there any failed samples when using the proposed method? Figure 6 shows samples are successed only. Discussion about the failed examples is important too and often give us some interesting insights and findings.
Questions for authors
  1. Please clarify the key challenges and technical contributions of this paper.
  2. Please evaluate the effectiveness of the proposed defense method under the real-world settings.
Reviewer_BEVROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the defense against the typographic attack (inserting misleading words into an image). The paper proposed defense method QuISE: identify the misleading text in image, replace it with irrelevant text, and then answer the question. Experiments across four VLMs shows positive defense results.

Critical strengths

1.Training free defense. The proposed defense combines a list of operations: query-aware text localization, semantic editing, and prediction. no model fine-tuning or access to gradients and internal representations is needed.

Critical weaknesses

1.The pre-requirement of the success defense is: it needs to identify the attack/misleading text first, then do defense. if we cannot identify the misleading text, then the method does not work. How to address this?

2.The defense cannot solve traditional attack where the trigger is invisible. Only works on specific typographic attack where the misleading text can be easily identified.

3.Table 2, the definition of different baselines are not explained in paper.

Questions for authors

1.How does QuISE compare with simpler strategies: removing localized text without replacing with any irrelevant text? It may be even simplier while as efficient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.