MARP: Task-Generalized Meta-Attack for Promotion in CLIP-Based Text-to-Image Retrieval

CLIP-based text-to-image retrieval is vulnerable to ranking manipulation: an attacker can promote a target image under selected text queries. We study this threat from a task-level perspective: an attacker may need to solve many promotion tasks with different target images and query sets. Existing iterative attacks typically solve these tasks independently, optimizing a new perturbation for each test task without explicitly exploiting shared structure across tasks. This raises a natural question: can an attack strategy learn across such tasks and generalize to unseen ones? We formulate this problem as a task-generalized meta-attack and propose MARP (Meta Attack for Retrieval Promotion), an amortized generator that produces an ${\ell }_{\infty }$-bounded task-specific perturbation in a single forward pass. MARP jointly conditions the generator on the target image and query set using frozen CLIP representations, and trains it with a bank-aware ranking surrogate that directly compares the adversarial target with competing images in the retrieval bank. On held-out attack tasks constructed from MSCOCO and Flickr30K, MARP achieves the best performance among the evaluated amortized or generator-based attacks while substantially reducing online attack-generation cost relative to iterative PGD. Task-level surrogate improvement is consistently associated with retrieval improvement, while further analyses characterize generalization across query-set regimes, task difficulty, and perturbation budgets.