- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies backdoor attacks in open-set text-to-image person re-identification (TIReID). The authors argue that the conventional trigger-to-fixed-class formulation is not well suited to TIReID because training and test identities are disjoint, and instead formulate the problem as open-set “correspondence hijacking.” TypoShift implements this idea by swapping captions across training identities and rendering nouns/adjectives extracted from the exchanged captions onto the counterpart images, while keeping identity labels and the original training objectives unchanged. The paper further proposes Attention Pathway Surgery (APS), which identifies visual attention heads with large clean-to-triggered class-token attention shifts and removes only their contributions to the class token, without retraining or adding parameters.
Across CUHK-PEDES, ICFG-PEDES, and RSTPReid, 3% nominal caption-annotation poisoning leaves clean Rank-1 almost unchanged while reducing triggered Rank-1 to 3.17%, 3.97%, and 5.55%, respectively. APS restores triggered Rank-1 to 60.66%, 57.30%, and 56.00% with limited clean-performance degradation. The paper also includes poison-budget studies, rendering-transfer experiments, an MLM ablation, attack transfer to CFine, factorial ablations of caption exchange and typography, and random-head controls for APS.
Critical strengths
A major strength is the formulation of backdoors for the open-set TIReID setting. Because test identities are unseen during training, a conventional fixed target class is indeed an awkward attack objective. Recasting the threat as disruption of image-text correspondence rather than redirection to a predetermined identity is conceptually well motivated and is one of the strongest aspects of the paper.
The experimental design also contains several useful controls. The authors evaluate the same triggered galleries with a benign model to separate the native effect of typography from poisoning-induced amplification, and the factorial ablation shows that neither caption exchange alone nor typography alone reproduces the full TypoShift effect. The paper additionally evaluates multiple poison budgets, rendering variations, and attack transfer to CFine. For APS, validation-only selection, freezing the selected intervention before testing, and matched random-head controls make the proposed pathway interpretation more convincing than an analysis based on attention visualization alone.
Critical weaknesses
My main concern is that the primary attack evaluation does not directly match the operational scenario used to motivate the threat. Most headline results are obtained by triggering the entire test gallery, whereas the realistic scenario described by the paper is that an individual adversary modifies only their own appearance or gallery image. The authors explicitly acknowledge that the full-gallery protocol is a population-level stress test and that the conditional evasion rate for a self-triggered target requires separate measurement. The partial-coverage experiment still randomly triggers a fraction of the gallery and reports aggregate ASR, so it does not answer whether one specific person can successfully evade retrieval when only that person's images are modified. This distinction matters because full-gallery triggering introduces typography on many negative candidates, potentially amplifying ranking collapse through query-related textual cues. I believe a target-only/self-triggered evaluation is necessary to establish the practical security significance of the attack. It would also be important to test truly query-independent natural text and random strings, rather than only person-description-derived phrases.
My second concern is that the generality and comparative evaluation of APS remain limited. APS is a trigger-aware white-box repair method that requires representative clean/triggered validation pairs. Its repair performance has not been evaluated on CFine or on previously unseen trigger families, which the authors explicitly acknowledge. As a result, it remains unclear how well the proposed intervention generalizes beyond the specific model and trigger conditions used for calibration. The current comparisons against clean fine-tuning and random head removal are useful controls, but they are not strong defense baselines. Comparisons against typography-oriented defenses such as Dyslexify or Defense-Prefix, simple OCR/text masking, and applicable multimodal backdoor-repair methods would substantially strengthen the defense claims.
Questions for authors
-
In the operational self-triggered setting, what is the conditional evasion rate when only one target identity's gallery image(s) contain the trigger and all other gallery entries remain clean? Does the attack remain effective when the rendered text is completely unrelated to both the query and the person's attributes, including arbitrary natural words or random strings?
-
Does APS repair TypoShift on CFine and on trigger styles or semantic trigger families not used during calibration? Under a comparable information budget, how does APS compare with Dyslexify, Defense-Prefix, OCR/text removal, or existing multimodal backdoor-repair methods?