acceptodds
Withdrawn from ICLR 2027

TypoShift: Open-Set Retrieval Hijacking and Attention Pathway Surgery for Text-to-Image Person Re-Identification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Backdoor attacks are typically built on a closed-set premise: a trigger redirects an input toward a fixed target. This premise breaks in text-to-image person re-identification (TIReID), where test identities are unseen, but the vulnerability shifts to cross-modal correspondence. We expose this surface through TypoShift, an identity-agnostic attack that exchanges captions across training identities and renders words from the exchanged captions on paired images. With 3% caption-annotation poisoning, TypoShift changes clean Rank-1 by at most 0.45 points yet reduces triggered-gallery Rank-1 to 3.17%, 3.97%, and 5.55% across three benchmarks under full-gallery digital triggering. Paired analysis identifies concentrated class-token attention shifts in a few visual heads. We then introduce Attention Pathway Surgery (APS), which uses validation pairs to select these heads and removes only their class-token contributions. Without retraining or additional parameters, APS reduces the attack success rate at Rank-1 (ASR@1) from 95.65%/93.74%/90.62% to 16.11%/8.65%/6.59%, with at most 0.56-point clean Rank-1 loss. TypoShift reframes open-set backdoors as correspondence hijacking, while APS identifies a compact intervention for this known trigger family.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Image Person Re-Identification, Backdoor Attacks, Open-Set Retrieval, Cross-Modal Retrieval, Typographic Triggers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vLQHOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies backdoor attacks in open-set text-to-image person re-identification (TIReID). The authors argue that the conventional trigger-to-fixed-class formulation is not well suited to TIReID because training and test identities are disjoint, and instead formulate the problem as open-set “correspondence hijacking.” TypoShift implements this idea by swapping captions across training identities and rendering nouns/adjectives extracted from the exchanged captions onto the counterpart images, while keeping identity labels and the original training objectives unchanged. The paper further proposes Attention Pathway Surgery (APS), which identifies visual attention heads with large clean-to-triggered class-token attention shifts and removes only their contributions to the class token, without retraining or adding parameters.

Across CUHK-PEDES, ICFG-PEDES, and RSTPReid, 3% nominal caption-annotation poisoning leaves clean Rank-1 almost unchanged while reducing triggered Rank-1 to 3.17%, 3.97%, and 5.55%, respectively. APS restores triggered Rank-1 to 60.66%, 57.30%, and 56.00% with limited clean-performance degradation. The paper also includes poison-budget studies, rendering-transfer experiments, an MLM ablation, attack transfer to CFine, factorial ablations of caption exchange and typography, and random-head controls for APS.

Critical strengths

A major strength is the formulation of backdoors for the open-set TIReID setting. Because test identities are unseen during training, a conventional fixed target class is indeed an awkward attack objective. Recasting the threat as disruption of image-text correspondence rather than redirection to a predetermined identity is conceptually well motivated and is one of the strongest aspects of the paper.

The experimental design also contains several useful controls. The authors evaluate the same triggered galleries with a benign model to separate the native effect of typography from poisoning-induced amplification, and the factorial ablation shows that neither caption exchange alone nor typography alone reproduces the full TypoShift effect. The paper additionally evaluates multiple poison budgets, rendering variations, and attack transfer to CFine. For APS, validation-only selection, freezing the selected intervention before testing, and matched random-head controls make the proposed pathway interpretation more convincing than an analysis based on attention visualization alone.

Critical weaknesses

My main concern is that the primary attack evaluation does not directly match the operational scenario used to motivate the threat. Most headline results are obtained by triggering the entire test gallery, whereas the realistic scenario described by the paper is that an individual adversary modifies only their own appearance or gallery image. The authors explicitly acknowledge that the full-gallery protocol is a population-level stress test and that the conditional evasion rate for a self-triggered target requires separate measurement. The partial-coverage experiment still randomly triggers a fraction of the gallery and reports aggregate ASR, so it does not answer whether one specific person can successfully evade retrieval when only that person's images are modified. This distinction matters because full-gallery triggering introduces typography on many negative candidates, potentially amplifying ranking collapse through query-related textual cues. I believe a target-only/self-triggered evaluation is necessary to establish the practical security significance of the attack. It would also be important to test truly query-independent natural text and random strings, rather than only person-description-derived phrases.

My second concern is that the generality and comparative evaluation of APS remain limited. APS is a trigger-aware white-box repair method that requires representative clean/triggered validation pairs. Its repair performance has not been evaluated on CFine or on previously unseen trigger families, which the authors explicitly acknowledge. As a result, it remains unclear how well the proposed intervention generalizes beyond the specific model and trigger conditions used for calibration. The current comparisons against clean fine-tuning and random head removal are useful controls, but they are not strong defense baselines. Comparisons against typography-oriented defenses such as Dyslexify or Defense-Prefix, simple OCR/text masking, and applicable multimodal backdoor-repair methods would substantially strengthen the defense claims.

Questions for authors
  1. In the operational self-triggered setting, what is the conditional evasion rate when only one target identity's gallery image(s) contain the trigger and all other gallery entries remain clean? Does the attack remain effective when the rendered text is completely unrelated to both the query and the person's attributes, including arbitrary natural words or random strings?

  2. Does APS repair TypoShift on CFine and on trigger styles or semantic trigger families not used during calibration? Under a comparable information budget, how does APS compare with Dyslexify, Defense-Prefix, OCR/text removal, or existing multimodal backdoor-repair methods?

Reviewer_ckobOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the backdoor attack in open text-to-image person re-identification, and proposes TypoShift, which is an unidentity poisoning strategy that exchanges subtitles between training identities and renders the keywords derived from subtitles to the corresponding images. The generated model retains clean retrieval performance, and is highly susceptible to typesetting-based triggers. This paper further proposes attention pathway surgery (APS), which uses paired cleaning/trigger verification samples to identify sensitive visual attention heads, and only inhibits its class token contribution without further training. Experiments on CGDA-PEDES, ICFG-PEDES and RSTPReid showed that under the proposed trigger conditions, the degradation was strong and APS recovered significantly.

Critical strengths
  1. The paper moves beyond conventional fixed target-label backdoors and formulates correspondence hijacking for open-set cross-modal retrieval, which is well motivated for TIReID.
  2. TypoShift is a data-only poisoning strategy that does not modify the model architecture, training objective, or optimization pipeline, while maintaining clean retrieval performance and producing strong triggered degradation.
  3. APS localizes a small set of trigger-sensitive attention heads and substantially restores triggered retrieval through a parameter-free CLS-only intervention, with limited clean-performance loss.
Critical weaknesses
  1. The main evaluation does not closely match the stated operational threat model. The headline attack results are obtained by triggering the entire gallery, whereas the motivating deployment scenario involves a user modifying only their own gallery image. The appendix further shows that with only 3% random gallery coverage, Rank-1 degradation is just 1.47%, making the practical single-target evasion effectiveness unclear.
  2. The reported ASR is not a conventional per-query attack success rate. Eq. (4) measures the relative decrease in aggregate Rank-K performance rather than the fraction of originally successful queries that become unsuccessful after triggering. Thus, values such as 95.65% should be interpreted as relative retrieval degradation rather than direct attack-success probability.
  3. The empirical results rely on a single random seed. All experiments use seed 42, while attack strength is noticeably non-monotonic across poisoning budgets. This makes it difficult to assess the robustness of the reported low-budget attack effectiveness and the stability of the identified attention pathways.
  4. Attack and defense baselines are limited. Although the paper discusses several multimodal backdoor attacks and specialized defenses such as BadCLIP, CleanCLIP, InverTune, and Dyslexify, the experiments do not provide direct comparisons with adapted attack baselines or strong typography-specific repair methods.
  5. The poisoning budget may understate the actual scope of corruption. The reported poisoning rate is defined over caption annotations rather than images or identities, and the construction preferentially selects identity pairs with fewer captions. Reporting the realized annotation, image, and identity corruption ratios would make the attack budget more interpretable.
  6. The causal interpretation of APS remains incomplete. Only three random four-head interventions are used as controls, and the paper does not establish whether the selected heads are specific to TypoShift poisoning or correspond to pre-existing typography-sensitive circuits in CLIP. APS is also not evaluated on unseen trigger variants or the CFine model without recalibration.
Questions for authors
  1. The main attack results use full-gallery triggering, whereas the stated deployment scenario involves modifying only the target person’s own gallery image. Could the authors report or clarify the single-target conditional evasion rate under this operational setting?
  2. Eq. (4) defines ASR as the relative decrease in aggregate Rank-K performance. Could the authors additionally report the per-query conditional attack success rate, i.e., the fraction of queries that are correct on the clean gallery but become incorrect after triggering?
  3. Since all experiments use a single random seed and the attack strength is non-monotonic across poisoning budgets, how sensitive are the reported results to the poisoned identity selection, trigger sampling, and training randomness? Can the authors provide multi-seed statistics?
  4. Are the APS-selected attention heads specific to the TypoShift-induced backdoor, or do similar typography-sensitive heads already exist in the benign CLIP-based model? A benign-model comparison would materially affect the interpretation of APS as backdoor-pathway localization.
Reviewer_X5yXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies backdoor attacks on text-to-image person re-identification, where training and test identities are disjoint. TypoShift swaps captions between training identities and overlays words from the swapped captions on their images. The resulting model performs normally on clean images but suffers a large drop when gallery images contain similar text overlays. The paper also proposes APS, which identifies trigger-sensitive attention heads using paired validation images and suppresses their contributions to the visual CLS token without retraining. Experiments are conducted on three datasets, primarily with IRRA, with an additional attack experiment on CFine.

Critical strengths
  1. The controls help establish that the attack comes from poisoned training rather than text overlays alone. Evaluating the unpoisoned model on the same triggered gallery is useful. Table. 5 also shows that neither caption swapping nor text overlays alone produce the large degradation observed with their combination.

  2. The attack has a strong effect on all three datasets while largely preserving clean performance. At 3% poisoning, triggered Rank-1 drops to 3.17%, 3.97%, and 5.55%, while clean Rank-1 changes by no more than 0.45 percentage points.

  3. The APS results go beyond visualizing attention changes. Suppressing the selected heads substantially recovers retrieval performance, whereas the three random four-head sets in Table. 7 do not. The paper also clearly states that head selection uses validation data rather than test results.

Critical weaknesses
  1. There is a gap between the main evaluation and the motivating evasion scenario. Most results apply text overlays to the entire gallery. The partial-coverage experiment measures the effect on overall retrieval, but does not tell us how often a particular person escapes retrieval when only that person’s images are triggered. This leaves the individual-evasion claim insufficiently tested.

  2. The defense comparison is limited. APS is closely related to prior typographic attention-circuit interventions discussed in the paper, but the reported evidence does not establish its advantage over those specialized interventions under matched calibration access and clean-utility constraints. Clean fine-tuning alone is an incomplete reference.

  3. The experiments use a single training seed. This is particularly relevant to the nonmonotonic results across poisoning rates on CUHK-PEDES. It is unclear how much these results depend on the sampled identity pairs and phrases, and whether APS consistently finds a similarly effective set of heads across runs.

Questions for authors
  1. Could the authors evaluate evasion when the trigger is applied only to images of a single identity, or to a few images of that identity? Reporting retrieval performance specifically for the affected identity would help distinguish target evasion from the promotion of unrelated distractors.

  2. Could the authors compare APS with the closest existing typographic attention-circuit intervention and with full-head ablation, using the same validation data and clean-performance constraint? This would help clarify the benefits of the proposed head-ranking rule and CLS-only intervention.

  3. How much do attack performance and APS recovery vary across independent training seeds? It would also be useful to examine whether the selected head sets are consistent across training runs and different draws of calibration samples.

  4. The paper evaluates attack transfer across changes in text rendering. Does the APS head mask generalize to these changes without recalibration? For example, if the heads are selected using centered red text, does the same mask remain effective when the font, position, color, or phrase changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.