acceptodds
Under review as a conference paper at ICLR 2027

MARP: Task-Generalized Meta-Attack for Promotion in CLIP-Based Text-to-Image Retrieval

Abstract

CLIP-based text-to-image retrieval is vulnerable to ranking manipulation: an attacker can promote a target image under selected text queries. We study this threat from a task-level perspective: an attacker may need to solve many promotion tasks with different target images and query sets. Existing iterative attacks typically solve these tasks independently, optimizing a new perturbation for each test task without explicitly exploiting shared structure across tasks. This raises a natural question: can an attack strategy learn across such tasks and generalize to unseen ones? We formulate this problem as a task-generalized meta-attack and propose MARP (Meta Attack for Retrieval Promotion), an amortized generator that produces an -bounded task-specific perturbation in a single forward pass. MARP jointly conditions the generator on the target image and query set using frozen CLIP representations, and trains it with a bank-aware ranking surrogate that directly compares the adversarial target with competing images in the retrieval bank. On held-out attack tasks constructed from MSCOCO and Flickr30K, MARP achieves the best performance among the evaluated amortized or generator-based attacks while substantially reducing online attack-generation cost relative to iterative PGD. Task-level surrogate improvement is consistently associated with retrieval improvement, while further analyses characterize generalization across query-set regimes, task difficulty, and perturbation budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attack, Text-to-Image Retrieval, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wn9EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduced MARP, a task-generalized meta-attack that generates task-conditioned promotion perturbations without per-task iterative optimization.

Critical strengths

The paper redefines retrieval-ranking-enhancement attacks from a task-oriented perspective and significantly reduces the cost of such attacks during the testing phase through the use of an amortized generator. The experimental section and appendix are comprehensive, incorporating various ablation studies, analyses of task difficulty, and statistical tests; overall, the work demonstrates a high level of reproducibility.

Critical weaknesses

W1. The distinction between the concept of "Task-Generalized Meta-Attack" (MARP) and conventional conditional generators or amortized optimization remains insufficiently clear. MARP lacks an inner-loop adaptation phase or a typical meta-learning process; in essence, it trains a conditional perturbation generator over a fixed task distribution. Consequently, the terms "meta-attack" and "task-generalized" are somewhat stronger than what the current evidence supports.

W2. The current task is manually constructed using neighboring captions and random target images in the CLIP space, and is still far from real content promotion or malicious exposure scenarios. Random targets are often semantically irrelevant to the query set, which may cause the task to degenerate into "pushing image features toward the query center". It is recommended to add more realistic task settings, such as evaluating semantically relevant goals, semantically irrelevant goals, query sets with different topic spans, and tasks derived from natural query distributions.

W3. Representing a set of queries solely through average pooling leads to the loss of internal multimodal structure, outlier queries, and the varying importance of individual queries. When a query set comprises multiple semantic sub-clusters, an average vector may fail to accurately characterize the intended attack target.

W4. Based on the ablation results, Q-Only has achieved most of the performance gains, and the gains after adding the target image condition are relatively limited. The improvement of bank-aware loss relative to mean-query similarity is also very small, especially on Flickr30K, where HR@1 only increased from 44.61% to 44.66%. These findings are insufficient to conclude that joint conditioning and supervision from the full retrieval bank are the critical factors driving the method's success.

W5. All task construction, training, and evaluation rely on the same CLIP ViT-B/32 model; the current results only demonstrate in-distribution generalization for a fixed white-box model. It is recommended to include at least one additional retrieval model, such as OpenCLIP, SigLIP, or CLIP ViT-L. Additionally, one could incorporate cross-dataset experiments, where models are trained on one dataset and tested on another. The speed comparison accounts only for online generation time and does not include the offline training cost of MARP.

Questions for authors

Please refer to Weakness.

3 comments
AuthorsOpenReview ↗

We clarify task generalization and report additional experimental results.

Dear Reviewer, thank you for your patience and for your careful review and constructive suggestions.

W1. Task generalization and terminology

In this paper, we define an attack task as improving the retrieval rank of a target image xx for a query set TT, denoted by τ=(x,T)\tau=(x,T). Different τ\tau correspond to different attack tasks. We train a perturbation generator to learn what these tasks have in common and generalize its attack capability to unseen tasks. This aligns with the cross-task generalization studied in meta-learning. Our current work considers different tasks within one task type, namely retrieval promotion attack; training a more general attacker across different task types is a promising direction for future work.

W2. Target relevance and constructed tasks

To test whether MARP can promote targets with and without shared object categories, we conduct an experiment on MSCOCO. Here, “category overlap” means that the target and seed images share at least one annotated object category. For each seed image, we hold its query set fixed and compare the two target groups. Across 19,808 tasks, MARP achieves:

Target conditionClean HR@1 (%)Attacked HR@1 (%)Clean MRAttacked MR ↓
Category overlap0.25458.781,62642.56
No category overlap0.00358.622,71855.37

MARP promotes targets in both groups. To examine query-set diversity, we also evaluate independently constructed TF-IDF and two-topic query sets, as described in W3.

W3. Mean pooling and heterogeneous query sets

We added two ways to construct query sets: (1) selecting neighboring queries by TF-IDF similarity between captions; and (2) constructing a two-topic set by selecting 16 TF-IDF neighbors from each of two lexically distinct seed-image neighborhoods:

DatasetOriginal TopK-32 HR@1 (%)TF-IDF HR@1 (%)Two-topic HR@1 (%)
Flickr30K44.6635.2622.59
MSCOCO56.7850.8315.56

MARP, trained on the original TopK-32 query sets, can be applied to both newly constructed query sets without retraining. This indicates some ability to generalize across query-set construction methods.

To examine query-set aggregation, we trained a MARP variant with an attention-based query-set encoder. The encoder uses the target image embedding to weight the queries. We kept each dataset's training objective unchanged:

DatasetTopK-32: mean poolingTopK-32: attentionTwo-topic: mean poolingTwo-topic: attention
Flickr30K HR@1 (%)44.6644.5622.5923.22
MSCOCO HR@1 (%)56.7855.6915.5618.00

The attention variant improves overall HR@1 on the two-topic sets but performs slightly worse than mean pooling on the original TopK-32 sets.

W4. Contribution of image conditioning and the training objective

Our experiments show that query conditioning accounts for most of the performance gain. In the main table, Q-Only achieves 51.32% HR@1 on MSCOCO; adding target-image conditioning raises it to 56.78%, a gain of 5.46 percentage points. On the same test tasks, we also compared the MARP checkpoint task by task with checkpoints trained using mean-query similarity and 16 hard negatives, respectively. The 95% confidence intervals for the HR@1 differences, obtained by bootstrap resampling clustered by seed image, are:

DatasetMARP − mean-query similarity (pp; 95% CI)MARP − 16 hard negatives (pp; 95% CI)
MSCOCO+1.38 [0.98, 1.78]+2.20 [1.77, 2.64]

W5. Retrieval models, dataset transfer, and offline cost

(1) Cross-dataset transfer. With the victim model fixed to CLIP ViT-B/32, we evaluated MARP across datasets:

Training datasetTest datasetTransferred generator HR@1 (%)
Flickr30KMSCOCO32.41
MSCOCOFlickr30K25.46

MARP shows some ability to transfer across datasets.

(2) Other retrieval models. We also retrained MARP for other retrieval models:

Victim modelTraining and test datasetClean HR@1 (%)Attacked HR@1 (%)
CLIP ViT-B/16Flickr30K0.09237.13
CLIP ViT-B/16MSCOCO0.02566.72
ALBEFFlickr30K0.08943.66

(3) Offline training cost. On Flickr30K, MARP takes an average of 19.38 minutes per training epoch, or approximately 8.02 ms per task per epoch (PGD-10 takes 65.95 ms to attack one task). The main table uses the checkpoint selected at epoch 30 on the validation set; reaching that epoch takes 9.69 hours of training. Once trained, the perturbation generator can be applied directly to new tasks under the same attack setting without task-specific retraining. It thus serves as a reusable attacker.

Thank you again for your suggestions.

Thank you for your response. I still have the following questions:

  1. Currently, the authors only report HR@1 results for a single checkpoint on MSCOCO. Furthermore, the performance gain of this loss function—relative to the mean-query similarity baseline—is negligible on the Flickr30K dataset.

  2. Under the "dual-topic query" setting, the HR@1 for MARP drops significantly, and its advantage over the "Q-Only" baseline becomes very small. How can we determine whether this performance decline is due to the increased difficulty of the task itself, or because the current mean-pooled representations and the generator fail to generalize to heterogeneous query targets?

If the authors can provide a reasonable response to these questions, I would be willing to consider increasing my score.

AuthorsOpenReview ↗

Supplementary Experiments: Model Comparison and Two-Topic Query Performance Analysis

Dear Reviewer, thank you for your follow-up. We respond to your two questions below.

Q1. Objective comparison and checkpoint stability

With CLIP ViT-B/32 as the victim model, the reported MARP checkpoints show some advantages over mean-query similarity.

DatasetMetricMARPMean-query similarityDifference (95% CI)
Flickr30KHR@1 (%) ↑44.6644.61+0.05 pp [−0.69, 0.80]
Flickr30KHR@5 (%) ↑67.9167.74+0.18 pp [−0.61, 0.95]
Flickr30KHR@10 (%) ↑76.2075.54+0.66 pp [−0.07, 1.41]
Flickr30KMean rank ↓20.4521.86−1.41 [−2.72, −0.11]
MSCOCOHR@1 (%) ↑56.7855.40+1.38 pp [0.98, 1.78]
MSCOCOHR@5 (%) ↑68.2467.11+1.13 pp [0.71, 1.55]
MSCOCOHR@10 (%) ↑72.9771.95+1.02 pp [0.60, 1.43]
MSCOCOMean rank ↓59.8763.53−3.66 [−6.37, −0.98]

We also trained and evaluated MARP with ALBEF as the victim model on Flickr30K and MSCOCO, and the results demonstrate consistent improvements. (These results measure ALBEF's first-stage ranking based on dual-encoder cosine similarity.)

DatasetMetricMARPMean-query similarityDifference (95% CI)
Flickr30KHR@1 (%) ↑43.6642.86+0.80 pp [0.58, 1.02]
Flickr30KHR@5 (%) ↑62.2561.89+0.36 pp [0.17, 0.54]
Flickr30KHR@10 (%) ↑69.0568.85+0.20 pp [0.03, 0.38]
Flickr30KMean rank ↓36.2736.73−0.46 [−0.73, −0.19]
MSCOCOHR@1 (%) ↑54.9453.75+1.19 pp [1.10, 1.28]
MSCOCOHR@5 (%) ↑64.6063.58+1.02 pp [0.92, 1.12]
MSCOCOHR@10 (%) ↑69.2368.34+0.89 pp [0.80, 0.98]
MSCOCOMean rank ↓70.8073.55−2.75 [−3.18, −2.33]

Q2. Why does performance drop on two-topic query sets?

We conducted additional experiments with two-topic query sets:

DatasetMethodOriginal TopK-32 HR@1 (%)Two-topic HR@1 (%)
Flickr30KMARP, mean pooling44.6622.59
Flickr30KMARP, attention pooling44.5623.22
Flickr30KPGD-1062.3633.76
MSCOCOMARP, mean pooling56.7815.56
MSCOCOMARP, attention pooling55.6918.00
MSCOCOPGD-1068.0724.21

The drop is not unique to MARP:

(1) MARP vs. PGD-10. PGD-10, which directly optimizes each test task, also drops by 28.60 and 43.86 percentage points on Flickr30K and MSCOCO, respectively. Clean mean rank is similar for the original and two-topic query sets (494.03 versus 496.76 on Flickr30K; 2533.02 versus 2528.55 on MSCOCO). Thus, the HR@1 drop cannot simply be attributed to a worse initial target rank.

(2) Mean pooling vs. attention pooling. Keeping each dataset's original training tasks and objective, we replaced mean pooling with attention module guided by the target-image representation. Two-topic HR@1 rises from 22.59% to 23.22% on Flickr30K (+0.63 pp; paired 95% CI [0.11, 1.14]) and from 15.56% to 18.00% on MSCOCO (+2.44 pp; [2.12, 2.75]). This suggests that mean pooling may limit performance on two-topic promotion tasks, although attention recovers only part of the drop. However, on the original TopK-32 setting, attention pooling underperforms mean pooling.

Taken together, we regard the increased difficulty of two-topic tasks as the main explanation for the performance drop. We believe this would be an excellent idea for our next paper.

Thank you again for your constructive questions.

Reviewer_zmQ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MARP, a task-conditioned amortized attack for promoting target images in CLIP-based text-to-image retrieval. It formulates a promotion task as a target image and a set of text queries, and trains a perturbation generator conditioned on their frozen CLIP representations. The generator is optimized using a bank-aware pairwise ranking loss that compares the adversarial target against competing images in the retrieval database. Experiments on MSCOCO and Flickr30K demonstrate that MARP outperforms the evaluated generator-based attacks while achieving substantially lower online attack-generation latency than iterative PGD. The paper also investigates task conditioning, ranking objectives, negative-bank construction, query-set generalization, and the relationship between surrogate improvement and retrieval performance.

Critical strengths

+. Clear problem formulation and motivation. The paper provides a well-defined task-level formulation of retrieval promotion and distinguishes amortized attack generation from conventional instance-wise optimization. The proposed approach addresses a meaningful efficiency concern when attacks must be generated for multiple target–query-set combinations.

+. Reasonable task-oriented method design. Joint target-image and query-set conditioning is aligned with the attack objective. The bank-aware ranking surrogate explicitly accounts for retrieval competitors rather than relying solely on image–text similarity. The theoretical connection to empirical rank is mathematically reasonable, although the resulting bound is loose.

+. Relatively comprehensive empirical analysis. The experiments include multiple iterative and generator-based baselines, conditioning and objective ablations, negative-bank comparisons, perturbation-budget analysis, query-set-regime generalization, and online latency measurements. The additional statistical analysis strengthens the empirical evaluation.

Critical weaknesses

-. The incremental contribution of the bank-aware ranking objective is limited.

Although the bank-aware ranking loss is presented as an important methodological component, the objective ablation in Appendix C.2 (Table 6) shows that simpler alternatives already achieve highly competitive performance. On Flickr30K, mean-query similarity achieves 44.61% HR@1, compared with 44.66% for the proposed objective. On MSCOCO, the corresponding results are 55.40% and 56.78%.

Similarly, Appendix C.3 (Table 7) shows that using only 16 hard negatives achieves 44.31% HR@1 on Flickr30K, close to the full-bank result of 44.66%.

These findings suggest that much of the attack effectiveness may come from query-conditioned amortized generation, while the additional benefit of full-bank ranking supervision is relatively modest in some settings. Although the combined approach is reasonable, the experiments do not fully establish whether the bank-aware objective provides a sufficiently substantial and consistent improvement to constitute a major methodological contribution.

-. The generator-based baseline comparison does not fully establish the superiority of the proposed approach.

The comparison with AnyAttack is informative but not fully task-aligned. As described in Appendix A.2, AnyAttack uses the seed image as a semantic reference rather than directly conditioning on the query set, and its released pretrained checkpoints are evaluated without task-specific fine-tuning.

Consequently, the large performance gap between MARP and AnyAttack may partly reflect differences in conditioning information and optimization objectives rather than the intrinsic effectiveness of the proposed generator.

The I-Only and Q-Only ablations help identify the role of conditioning, but they do not replace comparisons against alternative task-adapted generator architectures or training strategies. Stronger task-matched comparisons would make the claimed methodological advantage more convincing.

-. The evidence for task-level generalization remains restricted to a relatively narrow setting.

The experiments use a single frozen CLIP ViT-B/32 backbone and two image-caption datasets. Moreover, the attack query sets are constructed from caption neighborhoods defined in the same CLIP representation space.

The paper does evaluate different query-set sizes and radius-based constructions, which is valuable. However, these evaluations remain closely tied to the same semantic construction procedure and do not establish generalization to independently collected, heterogeneous, or dynamically changing query sets.

Furthermore, mean pooling is the only evaluated query-set aggregation strategy. Comparisons against alternative permutation-invariant or attention-based encoders could help determine whether the current representation adequately captures diverse query-set objectives.

These limitations do not invalidate the reported results, but they restrict the empirical scope of the task-generalization claim.

-. Computational efficiency and experimental stability require further clarification.

The reported 9–10× online speedup over PGD-10 is meaningful under the stated timing protocol. However, MARP requires offline training for 100 epochs, including full-bank ranking supervision. Training time and the number of attack tasks needed to amortize this cost are not reported, making the overall computational trade-off difficult to assess.

In addition, Appendix A.2 indicates that the reported checkpoints use a single training random seed. Although the paper provides extensive task-level bootstrap analyses, these do not capture variability across independently trained generators. This is particularly relevant when interpreting the relatively small improvements observed in the objective ablations.

Questions for authors

Q1. Contribution of the ranking objective: Given the relatively small gains over mean-query similarity in Table 6 and over limited hard-negative supervision in Table 7, can the authors provide paired statistical comparisons or multiple independent training runs to establish the stability and practical significance of these improvements?

Q2. Comparison fairness: Can the authors evaluate a more task-aligned generator-based baseline, for example by adapting an existing conditional attack generator using the same training tasks and ranking supervision? This would help separate the benefits of task-specific training from those of the proposed method design.

Q3. Generalization and efficiency: How does MARP perform on independently constructed or more heterogeneous query sets? Additionally, what are the total offline training costs and the approximate break-even number of attack tasks relative to iterative optimization?

3 comments
AuthorsOpenReview ↗

We provide additional experiments on the training objective, task-adapted AnyAttack, query-set construction, and training cost.

Dear Reviewer, thank you for reading our paper and for your key comments.

A1. Contribution of the ranking objective

We compared the MARP checkpoint with the mean-query-similarity and 16-hard-negative checkpoints on the same test tasks, pairing the results task by task. Specifically, we resampled seed-image clusters 10,000 times to estimate 95% confidence intervals for the HR@1 differences:

MSCOCO comparisonComparator HR@1 (%)MARP minus comparator (percentage points)Paired 95% CI
Mean-query similarity55.40+1.38[0.98, 1.78]
16 hard negatives54.58+2.20[1.77, 2.64]

On MSCOCO, MARP outperforms these two controls in HR@1 by 1.38 and 2.20 percentage points, respectively. Both paired confidence intervals exclude zero.

A2. Comparison with a task-adapted generator

We fine-tuned AnyAttack and obtained the following results:

DatasetMethodHR@1 (%) ↑HR@5 (%) ↑HR@10 (%) ↑MR ↓
Flickr30KAA-bi35.9859.4568.7231.56
Flickr30KMARP44.6667.9176.2020.45
MSCOCOAA-bi43.2756.9163.0180.45
MSCOCOMARP56.7868.2472.9759.87

MARP performs better across all four metrics on both datasets.

A3. Generalization and efficiency

(1) We added two ways to construct query sets. The first selects 32 neighboring queries based on TF-IDF similarity between captions. The second uses TF-IDF to identify a second seed image whose caption is lexically distinct, then selects 16 neighboring queries around each seed image to form a two-topic set. The results are as follows:

DatasetMethodOriginal TopK-32 HR@1 (%)TF-IDF HR@1 (%)Two-topic HR@1 (%)
Flickr30KI-Only12.8410.209.60
Flickr30KUAP16.1213.6312.44
Flickr30KQ-Only42.8833.9222.20
Flickr30KMARP44.6635.2622.59
MSCOCOI-Only5.975.924.87
MSCOCOUAP11.4610.518.96
MSCOCOQ-Only51.3245.4913.64
MSCOCOMARP56.7850.8315.56

Under both query-set constructions, MARP achieves a higher HR@1 than UAP, I-Only, and Q-Only.

(2) We trained a target-guided attention query-set encoder while keeping the training objective the same. For each dataset, both test settings use the same checkpoint selected on the validation set:

DatasetOriginal TopK-32: mean / attention HR@1 (%)Two-topic: mean / attention HR@1 (%)
Flickr30K44.66 / 44.5622.59 / 23.22
MSCOCO56.78 / 55.6915.56 / 18.00

Attention improves HR@1 on the two-topic tasks but slightly lowers it on the original TopK-32 tasks.

(3) Offline training cost.

On Flickr30K, MARP takes an average of 19.38 minutes per training epoch, or approximately 8.02 ms per task per epoch (PGD-10 takes 65.95 ms to attack one task). The main table uses the checkpoint selected at epoch 30 on the validation set, reaching that epoch takes 9.69 hours of training. Once trained, the perturbation generator can be applied directly to new tasks under the same attack setting without task-specific retraining. It thus serves as a "reusable attacker".

Thank you for the detailed response and additional experiments. The authors have clarified the distinction from existing instance-wise promotion attacks, and the task-adapted AnyAttack comparison supports the value of learning a reusable task-conditioned generator. The TF-IDF, two-topic, attention-pooling, and additional victim-model experiments also substantially strengthen the query-set generalization evaluation. I therefore consider these two concerns largely addressed.

My first remaining concern is the restrictive threat model. Although full-bank features are mainly required during offline training, MARP still assumes white-box model access and complete knowledge of the target query set. Its effectiveness under partial or dynamically changing user queries, or after changes to the victim model and retrieval bank, remains unclear.

Second, the 9–10× speedup applies only to online generation. MARP remains weaker than PGD-10 and requires approximately 9.69 hours of offline training, so the current results do not establish superior efficiency at comparable attack strength or under equal total computation. A quality-matched or compute-matched comparison would better characterize this trade-off.

Finally, retraining MARP for CLIP ViT-B/16 and ALBEF does not demonstrate black-box transfer. The paper also does not evaluate JPEG compression, resizing, cropping, re-encoding, or defenses. Consequently, its practical robustness in a realistic retrieval pipeline remains uncertain.

The rebuttal has strengthened the paper, but these remaining concerns continue to affect my assessment.

AuthorsOpenReview ↗

Clarifying Threat Assumptions, Total Cost, and Robustness Limits

Dear Reviewer, thank you for your follow-up and for recognizing the additional comparisons and generalization experiments. We address the remaining concerns below.

1. Threat model and changing queries

We apologize for not clearly explaining where the target query set TT comes from. In our setting, the attacker first chooses a semantic theme under which to promote a target image and then constructs the corresponding query set TT. For example, to promote a specified image under road-construction queries, the attacker could include “workers paving a road” and “construction workers in safety vests” in TT. To simulate this behavior in our benchmark, we use a seed image's captions as semantic anchors and select the 32 nearest captions in the frozen CLIP text space to form a TopK-32 query set. We fix ∣T∣=32|T|=32 in training and main-table testing so that all tasks have the same number of queries. The retrieval system does not provide the attacker with a list of future user queries. Thus, knowing TT is part of the promotion objective chosen by the attacker. During MARP training, the attacker needs access to the victim model's gradients and image-bank features; at test time, only the target-image and query representations are used, without access to gradients or the full image-bank features.

Appendix E.1 evaluates the same frozen checkpoint with ∣T∣∈{1,4,8,16,32,64,128}|T|\in\{1,4,8,16,32,64,128\}. For K<32K<32, the queries are a prefix of the original TopK-32 set; for K>32K>32, additional neighboring queries are appended. For example, at K=16/32/64K=16/32/64, MARP obtains 48.98/44.66/39.01% HR@1 on Flickr30K and 56.92/56.78/54.70% on MSCOCO. The TF-IDF and two-topic experiments further change query selection and composition. These experiments show how the frozen generator performs under the evaluated changes to TT.

2. Online latency versus total cost

We agree that the reported speedup concerns online attack generation, not attack quality at equal total computation. The main-table HR@1 and the measured online times are:

DatasetMethodHR@1 (%) ↑Online time (ms/task) ↓
Flickr30KMARP44.667.25
Flickr30KPGD-1062.3665.95
MSCOCOMARP56.787.20
MSCOCOPGD-1068.0772.23

3. Effects of Image Transformations on Attack Performance

We added experiments applying transformations to the target image after perturbation generation. For each transformation, we process both the clean and adversarial target images, while keeping the query set and pre-indexed retrieval bank fixed. We use previously reported frozen checkpoints, including task-adapted AA-bi and COCO Q-Only. The tables report attacked HR@1 (%).

Flickr30K methodNoneJPEG 90JPEG 75PNG re-encodingResize 224→160→224
MARP44.6620.1112.0344.6614.05
AA-bi (adapted)35.9816.9310.0135.9512.59
Q-Only42.8818.2010.7542.8813.12
I-Only12.843.471.7412.832.03
UAP16.121.250.0916.122.28
MSCOCO methodNoneJPEG 90JPEG 75PNG re-encodingResize 224→160→224
MARP56.7833.5418.9256.7723.53
AA-bi (adapted)43.2723.4414.8243.2815.77
Q-Only51.3229.7917.7651.3121.75
I-Only5.971.960.465.970.92
UAP11.461.360.4711.462.89

Across all methods and reported transformations, clean HR@1 remains below 0.11% on Flickr30K and 0.02% on MSCOCO. MARP achieves the highest HR@1 under every reported condition on both datasets. PNG re-encoding leaves attacked HR@1 nearly unchanged, while JPEG compression and resizing cause substantial declines for all methods.

Thank you for your comments. We wish you continued success in your research.

Reviewer_6HRoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MARP as an new attack for multi-target image rank promotions. Authors compare MARP with several attacks on 2 datasets (COCO and Flickr) and 1 model, showing slightly subpar performance against PGD counterparts. The paper also provides qualitative analysis on the impact of the number of chosen queries, but the coclusions are generally intuitive but superfacial.

Critical strengths
  1. The idea of this papr is generally intuitive.
  2. Illustration for the pipeline of MARP is also clear. The paper is easy-to-follow.
  3. A proposition is also provided for theoretical foundation to connect the raining loss with resulted ranks.
Critical weaknesses

W1. Research Contirbution. As mentioned in the paper, Promotion Attacks have been well-established in THA (text-to-image) and ViPro (text-to-video). The former proposes the first vision-text retrieval targeted promotion attack, while the latter expand the domain from image to videos. Specifically, ViPro conduct in-depth analysis on the promotion-like attack upper and lower bounds, validated on several models and datasets. Unfortunately, I did not catch significant contribuition upon these two works, regarding both threat models, methods or analysis.

Although a proposition is provided to strengthen the theoretical contribution, that proposition alone only qualifies that sufficiently small loss can transfer to higher ranks, while how much loss drop can be transfered into visible rank promotion, which is more crucial, is left undicussed.

W2. Insufficient Experiments. It is bizzare that the paper did not clearly define the attack scenarios, which acording to my understanding, is completely white-box, since the attacker gets full access to the CLIP model during attack generation. Despite the lack of black-box attacks, the paper only tests their results on CLIP-ViT, which is not a competent model for image-text-retrieval compared to ALBEF/BLIP, etc. Beyond backbones, the paper fails to compare with existing retrieval focused attacks such as SGA(set-level guidance attack)/TRA, etc, substanially demaging the validity of the experiements, not to mention the uncompetitive results of MARP on the 2 tested datasets.

There also lacks specific justifications for the experiment details. For example, the use of soft plus in the optimization loss, how the attackers acquire the targeted queries, and the victim model that is used (is it also CLIP-ViT or a different model?).

Questions for authors

See weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary

We thank each of you for carefully reading our paper and offering constructive suggestions. Your comments have helped us clarify MARP's research question, threat model, and empirical evidence. We summarize our main responses below.

Additional experiments and analyses

  • Task-adapted generator baseline [Reviewers 6HRo, zmQ4]. We fine-tuned AnyAttack for our promotion tasks. MARP achieves better HR@1, HR@5, HR@10, and mean rank than the adapted AA-bi baseline on both Flickr30K and MSCOCO.
  • Query-set generalization [Reviewers wn9E, zmQ4]. We evaluated frozen generators on query sets constructed using caption-text TF-IDF and on heterogeneous two-topic sets. MARP can be applied to these query sets without retraining. On the more difficult two-topic tasks, both MARP and PGD-10 show substantial performance drops. We also designed a target-image-guided attention variant of MARP that improves HR@1 on two-topic sets but performs slightly worse on the original TopK-32 sets. This trade-off also suggests a clear direction for future work. We thank the reviewers for raising this question.
  • Target relevance and retrieval models [Reviewer wn9E]. We evaluated targets with and without overlapping MSCOCO object categories, tested transfer between Flickr30K and MSCOCO, and retrained MARP for CLIP ViT-B/16 and ALBEF. These experiments show that MARP can promote targets across the tested settings.
  • Training objective and efficiency [Reviewers 6HRo, wn9E, zmQ4]. We report paired comparisons with mean-query similarity and limited hard-negative supervision. MARP shows a small HR@1 gain over mean-query similarity on Flickr30K and a larger gain on MSCOCO. We also report offline training cost and distinguish it from the online cost of generating an attack for a new task.

Clarifications for the paper

  • Contribution and threat model [Reviewers 6HRo, wn9E]. Our focus is on the cross-task generalization of a perturbation generator's attack capability. Given a new target image and query set, it generates a perturbation in one forward pass without per-task iterative optimization. We clarify the model access available during training and inference, and that the attacker may replace only the specified target image.

As the rebuttal period draws to a close, we thank the reviewers once again for their thoughtful feedback. We wish you continued success in your research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.