- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose QuISE, a training-free black-box defense against typographic attacks on VLMs, aiming to mitigate the influence of misleading text in images on model responses. Specifically, the VLM first recognizes the text content; OCR provides its locations, and the VLM then uses the query to select text regions that may influence the answer. Next, rules are used to select replacement candidates from a fixed word pool, and the VLM verifies whether the candidates are irrelevant to both the query and the image with the selected text removed. Finally, two edited images with different replacement texts are generated. The edited answer is adopted only when both images produce valid and consistent answers. Otherwise, the answer from the original input image is retained.
Critical strengths
-
The motivation behind the method is clear. The authors first compare appearance modifications with semantic modifications, then analyze query relevance and image relevance, and finally propose Q0 I0 replacements.
-
QuISE is a black-box method that is more suitable for real-world deployment. It does not require modifying the target model weights, and API access alone is sufficient.
-
The paper is logically organized, easy to understand, and easy to follow.
Critical weaknesses
-
The loss in model utility on clean inputs is too large. In Figure 7, the clean text reading retention of several models is around 80%, representing a decrease of approximately 20% compared with the clean model. The loss in recognition performance on clean inputs is substantial.
-
Using two edits may not be optimal. Table 2 shows that three edits achieve higher accuracy and recovery rates. Could using more edits lead to better results?
-
QuISE only considers images containing a single text region and does not evaluate complex scenarios with multiple text regions. The consensus rule also cannot rule out shared errors.
-
The replacement text in QuISE may not need to be relevant to the image, and its benefit is unclear. In Figure 4, LLaVA achieves an accuracy that is 0.1% lower when the replacement text is irrelevant to both the query and the image than when it is irrelevant to the query but relevant to the image.
Questions for authors
-
What specific criteria were used to select two edits in QuISE? Could the authors report the trade-offs among larger numbers of edits, consensus rules, and actual costs on other models?
-
Can QuISE perform accurate calculations or question answering in complex scenarios with multiple text regions, and can it be evaluated in scenarios where genuine text and attack text coexist?
-
Why must the candidate words also be irrelevant to the image, namely I0? The independent benefit of this requirement is unclear.