acceptodds
Under review as a conference paper at ICLR 2027

From Typographic Distraction to Selective Token Intervention in CLIP

Abstract

Typographic attacks can redirect CLIP's prediction toward a rendered word even when the depicted object remains visible. This creates a selective intervention problem: suppress the distracting cue while retaining the image evidence needed for recognition. We study where this interference enters the visual representation through paired clean and attacked images. Restoring clean patch representations at positions overlapping the rendered text largely reverses the shift toward the rendered label, while inserting attacked representations at the same positions induces it. These findings motivate Patch Token Suppression (PTS), which learns a typographic score for each patch and moves representations with high scores toward a centroid estimated from patches assigned low scores in the same image. On synthetic ImageNet attacks, PTS improves object recovery while largely preserving clean accuracy. Evaluations on attacks in real images and ten additional datasets show consistent reductions in text confusion. Ablations identify spatial localization as the principal contributor to robustness, with smaller effects from the replacement target and interpolation rule.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Typographic Attack, Vision Language Models, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V2WkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Patch Token Suppression, which learns a typographic score for each patch and moves representations with high scores toward a centroid estimated from patches assigned low scores in the same image. On synthetic ImageNet attacks, PTS improves object recovery while largely preserving clean accuracy. Overall, this is a solid paper that leans toward the accept side.

Critical strengths
  1. The paper studies CLIP's robustness under typographic attacks, which has important practical significance;

  2. The baseline methods are all state-of-the-art;

  3. The dataset coverage is comprehensive, including multiple test datasets;

  4. The reproducibility and ethics statements are well regulated. Random seeds, optimization settings, mask construction, and baseline reproduction details are provided.

Critical weaknesses
  1. ImageNet appears multiple times in the main text but is not cited (it is only cited in the supplementary material), which is very unprofessional and shows that the authors' paper writing is not rigorous;

  2. Only two models are evaluated, which is too few and may make it difficult to support the authors' claims;

  3. The authors do not rigorously analyze or experimentally demonstrate the reasons for the drop in clean-sample accuracy;

  4. The ablation experiments for some parameters are incomplete.

Questions for authors

See Weaknesses.

Reviewer_MiheOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals that CLIP's vulnerability to typographic attacks stems not from global model representations, but specifically from the visual tokens corresponding to localized text regions. Leveraging this insight, the authors introduce PTS, a training-free strategy that automatically localizes and suppresses these text-contaminated patch tokens to enhance model robustness against typographic interference without modifying the underlying CLIP backbone. Methodologically, through bidirectional activation patching on paired clean and attacked images, the paper empirically demonstrates that typographic distraction is tightly mediated by patch tokens overlapping the rendered text, whereas non-text control patches exhibit negligible impact. Extensive comparative experiments further validate that the proposed PTS consistently outperforms existing baseline approaches across most evaluated settings.

Critical strengths

Rigorous mechanistic analysis via bidirectional activation patching. A major strength of this work lies in its rigorous activation patching analysis, which uncovers the localized mechanics of typographic interference in CLIP. By conducting bidirectional patching experiments, both restoring clean representations into attacked images and injecting text-contaminated tokens into clean ones, the authors effectively trace the causal propagation of typographic bias. Crucially, by contrasting text-overlapping patches with carefully matched non-text control regions, the experimental design provides compelling causal evidence that localized patch representations at text positions directly drive the downstream classification shift. This insightful mechanistic analysis offers a solid, well-substantiated motivation for the proposed defense.

Critical weaknesses
  1. Confounding typographic semantics with synthetic rendering artifacts. A fundamental limitation of the proposed training framework lies in its heavy reliance on supervision signals derived exclusively from a synthetic renderer. Specifically, the localizer projector is trained on 130,000 paired clean and synthetically attacked images guided by binary text-region masks. Consequently, the training exposure is inextricably tied to specific rendering properties, such as preset font families, stroke widths, color distributions, spatial layouts, and alpha-blending styles. This introduces a critical, unaddressed ambiguity: does the learned projector capture genuine semantic typographic interference, or does it merely overfit to dataset-specific synthetic artifacts? Without out-of-distribution evaluations or cross-renderer testing, it remains unproven whether PTS learns a universal typographic detector or an artifact detector tailored strictly to the synthetic generation pipeline.
  2. Indiscriminate suppression of functional text and multimodal reasoning trade-offs. The evaluation is strictly confined to standard image classification scenarios where text acts as a malicious distractor, entirely overlooking real-world environments where text carries essential semantic information (e.g., street signs, product packaging, OCR tasks, or document understanding). A robust vision-language defense mechanism should not indiscriminately dampen all text-aligned visual representations. In practical applications, suppressing text regions risks erasing legitimate visual-textual cues required for fine-grained recognition or multimodal reasoning. Because the paper evaluates performance exclusively on image classification without analyzing the trade-off on functional text comprehension (such as OCR or DocVQA benchmarks), the generalizability and practical safety of PTS in complex, text-rich real-world environments remain highly questionable.
Questions for authors
  1. Generalization to Unseen Typography and Reliance on Synthetic Masks. A core component of the proposed framework is the training procedure, which relies heavily on binary text-region masks derived from synthetic renderer metadata to supervise the localizer projector. This raises a critical generalization concern: in real-world scenarios, adversarial typography often exhibits non-standard fonts, artistic styling, or layout arrangements entirely absent from the training distribution. If the projector encounters such out-of-distribution typographic layouts (e.g., curved, stylized, or hand-crafted text), how gracefully does the localization performance degrade?
  2. Mechanistic Breakdown of Performance Degradation on Fine-Grained Benchmarks. Section 5.4 and the Appendix indicate that while PTS successfully reduces the Text Confusion Rate (TCR) on fine-grained datasets (e.g., Stanford Cars and FGVC Aircraft), it leads to a non-trivial degradation in Object Classification Accuracy (OCA)—for instance, OCA dropping to 33.02% on Stanford Cars. Could the authors provide a deeper analysis of the underlying mechanism driving this performance drop? Specifically, is the suppression of fine-grained discriminative visual cues (e.g., subtle body contours or engine details) primarily caused by false positives from the typographic projector, or does it reflect an inherent limitation of shifting feature representations toward an image-wide centroid?
Reviewer_K4CEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

When using CLIP to do zero shot classification, if rendered words or typography are present in an image, this can often lead to the zero shot prediction matching the typography, instead of the actual object. The authors try to tackle the challenge of suppressing the distracting text signal without removing the visual information needed for object recognition. To do this, the authors use activation patching to locate the interference, and then train a patch localizer on paired clean and attacked images. Their Patch Token Suppression (PTS) method keeps CLIP frozen and moves high-scoring patch representations toward a centroid formed from low-scoring patches. The experiments provide empirical evidence of improved object recovery, although clean accuracy sometimes decreases.

Critical strengths

CS1. The bidirectional activation-patching experiments convincingly identify the intermediate patch representations responsible for typographic interference. I think this is a useful contribution of the paper.

CS2. The method achieves notable improvements over contemporary defenses across several benchmarks, while the ablations also help establish spatial localization as the primary contributor.

Critical weaknesses

Critical Weakness 1: Narrow Task Applicability and Method Utility

The actual underlying task is zero-shot CLIP classification, and the paper proposes a defense against distracting text that leads to misclassification. We can think of some “common sense” defenses easily: the text can be detected with an off-the-shelf OCR, or even a segmentation model like Segment-Anything. Once located, the text can be blurred (like the authors do in Appendix C), or even masked (and optionally inpainted with some diffusion model). Based on the result shown in Tab. 5, most of the performance of PTS can be recovered by the extremely simple OCR+blur baseline (normal 42.72, PTS 66.36, OCR blur 65.26). Table 2 also suggests spatial localization accounts for most of the robustness improvement, while hard thresholding and zero replacement perform comparably to the proposed soft interpolation and adaptive centroid.

If detecting text and suppressing its corresponding patches already solves most of the problem, what is the substantial technical contribution of PTS?

Critical Weakness 2: Identifying harmful vs useful text.

The paper broadly treats typography as a distracting signal regardless of its relevance to the downstream task. It is highly possible, that visually rendered text is an important characteristic of the recognition task, and it is not clear how PTS handles such situations.

For example, if we take the ImageNet dataset, we have classes like comic book (917), cross word (918), street sign (919), etc. that can have text as actual part of the identification object. Can PTS distinguish these useful typography from adversarial typography, or does it simply learn to suppress text-like features?

Questions for authors
  1. Could the authors investigate or explain the degradation in Stanford Cars? For instance, could there be false text detections, suppression of useful text, or removal of visual details within selected patches? Clearly identifying the reason could improve the paper.

  2. Does PTS remain effective against defense-aware typographic attacks that select text placement, appearance, or coverage specifically to evade the learned localizer?

  3. How does PTS behave when text helps identify the object, or when helpful and distracting text occur in the same image? Can the authors explore this in a controlled setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.