- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduced MARP, a task-generalized meta-attack that generates task-conditioned promotion perturbations without per-task iterative optimization.
Critical strengths
The paper redefines retrieval-ranking-enhancement attacks from a task-oriented perspective and significantly reduces the cost of such attacks during the testing phase through the use of an amortized generator. The experimental section and appendix are comprehensive, incorporating various ablation studies, analyses of task difficulty, and statistical tests; overall, the work demonstrates a high level of reproducibility.
Critical weaknesses
W1. The distinction between the concept of "Task-Generalized Meta-Attack" (MARP) and conventional conditional generators or amortized optimization remains insufficiently clear. MARP lacks an inner-loop adaptation phase or a typical meta-learning process; in essence, it trains a conditional perturbation generator over a fixed task distribution. Consequently, the terms "meta-attack" and "task-generalized" are somewhat stronger than what the current evidence supports.
W2. The current task is manually constructed using neighboring captions and random target images in the CLIP space, and is still far from real content promotion or malicious exposure scenarios. Random targets are often semantically irrelevant to the query set, which may cause the task to degenerate into "pushing image features toward the query center". It is recommended to add more realistic task settings, such as evaluating semantically relevant goals, semantically irrelevant goals, query sets with different topic spans, and tasks derived from natural query distributions.
W3. Representing a set of queries solely through average pooling leads to the loss of internal multimodal structure, outlier queries, and the varying importance of individual queries. When a query set comprises multiple semantic sub-clusters, an average vector may fail to accurately characterize the intended attack target.
W4. Based on the ablation results, Q-Only has achieved most of the performance gains, and the gains after adding the target image condition are relatively limited. The improvement of bank-aware loss relative to mean-query similarity is also very small, especially on Flickr30K, where HR@1 only increased from 44.61% to 44.66%. These findings are insufficient to conclude that joint conditioning and supervision from the full retrieval bank are the critical factors driving the method's success.
W5. All task construction, training, and evaluation rely on the same CLIP ViT-B/32 model; the current results only demonstrate in-distribution generalization for a fixed white-box model. It is recommended to include at least one additional retrieval model, such as OpenCLIP, SigLIP, or CLIP ViT-L. Additionally, one could incorporate cross-dataset experiments, where models are trained on one dataset and tested on another. The speed comparison accounts only for online generation time and does not include the offline training cost of MARP.
Questions for authors
Please refer to Weakness.
3 comments
Thank you for your response. I still have the following questions:
-
Currently, the authors only report HR@1 results for a single checkpoint on MSCOCO. Furthermore, the performance gain of this loss function—relative to the mean-query similarity baseline—is negligible on the Flickr30K dataset.
-
Under the "dual-topic query" setting, the HR@1 for MARP drops significantly, and its advantage over the "Q-Only" baseline becomes very small. How can we determine whether this performance decline is due to the increased difficulty of the task itself, or because the current mean-pooled representations and the generator fail to generalize to heterogeneous query targets?
If the authors can provide a reasonable response to these questions, I would be willing to consider increasing my score.
Supplementary Experiments: Model Comparison and Two-Topic Query Performance Analysis
Dear Reviewer, thank you for your follow-up. We respond to your two questions below.
Q1. Objective comparison and checkpoint stability
With CLIP ViT-B/32 as the victim model, the reported MARP checkpoints show some advantages over mean-query similarity.
| Dataset | Metric | MARP | Mean-query similarity | Difference (95% CI) |
|---|---|---|---|---|
| Flickr30K | HR@1 (%) ↑ | 44.66 | 44.61 | +0.05 pp [−0.69, 0.80] |
| Flickr30K | HR@5 (%) ↑ | 67.91 | 67.74 | +0.18 pp [−0.61, 0.95] |
| Flickr30K | HR@10 (%) ↑ | 76.20 | 75.54 | +0.66 pp [−0.07, 1.41] |
| Flickr30K | Mean rank ↓ | 20.45 | 21.86 | −1.41 [−2.72, −0.11] |
| MSCOCO | HR@1 (%) ↑ | 56.78 | 55.40 | +1.38 pp [0.98, 1.78] |
| MSCOCO | HR@5 (%) ↑ | 68.24 | 67.11 | +1.13 pp [0.71, 1.55] |
| MSCOCO | HR@10 (%) ↑ | 72.97 | 71.95 | +1.02 pp [0.60, 1.43] |
| MSCOCO | Mean rank ↓ | 59.87 | 63.53 | −3.66 [−6.37, −0.98] |
We also trained and evaluated MARP with ALBEF as the victim model on Flickr30K and MSCOCO, and the results demonstrate consistent improvements. (These results measure ALBEF's first-stage ranking based on dual-encoder cosine similarity.)
| Dataset | Metric | MARP | Mean-query similarity | Difference (95% CI) |
|---|---|---|---|---|
| Flickr30K | HR@1 (%) ↑ | 43.66 | 42.86 | +0.80 pp [0.58, 1.02] |
| Flickr30K | HR@5 (%) ↑ | 62.25 | 61.89 | +0.36 pp [0.17, 0.54] |
| Flickr30K | HR@10 (%) ↑ | 69.05 | 68.85 | +0.20 pp [0.03, 0.38] |
| Flickr30K | Mean rank ↓ | 36.27 | 36.73 | −0.46 [−0.73, −0.19] |
| MSCOCO | HR@1 (%) ↑ | 54.94 | 53.75 | +1.19 pp [1.10, 1.28] |
| MSCOCO | HR@5 (%) ↑ | 64.60 | 63.58 | +1.02 pp [0.92, 1.12] |
| MSCOCO | HR@10 (%) ↑ | 69.23 | 68.34 | +0.89 pp [0.80, 0.98] |
| MSCOCO | Mean rank ↓ | 70.80 | 73.55 | −2.75 [−3.18, −2.33] |
Q2. Why does performance drop on two-topic query sets?
We conducted additional experiments with two-topic query sets:
| Dataset | Method | Original TopK-32 HR@1 (%) | Two-topic HR@1 (%) |
|---|---|---|---|
| Flickr30K | MARP, mean pooling | 44.66 | 22.59 |
| Flickr30K | MARP, attention pooling | 44.56 | 23.22 |
| Flickr30K | PGD-10 | 62.36 | 33.76 |
| MSCOCO | MARP, mean pooling | 56.78 | 15.56 |
| MSCOCO | MARP, attention pooling | 55.69 | 18.00 |
| MSCOCO | PGD-10 | 68.07 | 24.21 |
The drop is not unique to MARP:
(1) MARP vs. PGD-10. PGD-10, which directly optimizes each test task, also drops by 28.60 and 43.86 percentage points on Flickr30K and MSCOCO, respectively. Clean mean rank is similar for the original and two-topic query sets (494.03 versus 496.76 on Flickr30K; 2533.02 versus 2528.55 on MSCOCO). Thus, the HR@1 drop cannot simply be attributed to a worse initial target rank.
(2) Mean pooling vs. attention pooling. Keeping each dataset's original training tasks and objective, we replaced mean pooling with attention module guided by the target-image representation. Two-topic HR@1 rises from 22.59% to 23.22% on Flickr30K (+0.63 pp; paired 95% CI [0.11, 1.14]) and from 15.56% to 18.00% on MSCOCO (+2.44 pp; [2.12, 2.75]). This suggests that mean pooling may limit performance on two-topic promotion tasks, although attention recovers only part of the drop. However, on the original TopK-32 setting, attention pooling underperforms mean pooling.
Taken together, we regard the increased difficulty of two-topic tasks as the main explanation for the performance drop. We believe this would be an excellent idea for our next paper.
Thank you again for your constructive questions.
We clarify task generalization and report additional experimental results.
Dear Reviewer, thank you for your patience and for your careful review and constructive suggestions.
W1. Task generalization and terminology
In this paper, we define an attack task as improving the retrieval rank of a target image x for a query set T, denoted by τ=(x,T). Different τ correspond to different attack tasks. We train a perturbation generator to learn what these tasks have in common and generalize its attack capability to unseen tasks. This aligns with the cross-task generalization studied in meta-learning. Our current work considers different tasks within one task type, namely retrieval promotion attack; training a more general attacker across different task types is a promising direction for future work.
W2. Target relevance and constructed tasks
To test whether MARP can promote targets with and without shared object categories, we conduct an experiment on MSCOCO. Here, “category overlap” means that the target and seed images share at least one annotated object category. For each seed image, we hold its query set fixed and compare the two target groups. Across 19,808 tasks, MARP achieves:
MARP promotes targets in both groups. To examine query-set diversity, we also evaluate independently constructed TF-IDF and two-topic query sets, as described in W3.
W3. Mean pooling and heterogeneous query sets
We added two ways to construct query sets: (1) selecting neighboring queries by TF-IDF similarity between captions; and (2) constructing a two-topic set by selecting 16 TF-IDF neighbors from each of two lexically distinct seed-image neighborhoods:
MARP, trained on the original TopK-32 query sets, can be applied to both newly constructed query sets without retraining. This indicates some ability to generalize across query-set construction methods.
To examine query-set aggregation, we trained a MARP variant with an attention-based query-set encoder. The encoder uses the target image embedding to weight the queries. We kept each dataset's training objective unchanged:
The attention variant improves overall HR@1 on the two-topic sets but performs slightly worse than mean pooling on the original TopK-32 sets.
W4. Contribution of image conditioning and the training objective
Our experiments show that query conditioning accounts for most of the performance gain. In the main table, Q-Only achieves 51.32% HR@1 on MSCOCO; adding target-image conditioning raises it to 56.78%, a gain of 5.46 percentage points. On the same test tasks, we also compared the MARP checkpoint task by task with checkpoints trained using mean-query similarity and 16 hard negatives, respectively. The 95% confidence intervals for the HR@1 differences, obtained by bootstrap resampling clustered by seed image, are:
W5. Retrieval models, dataset transfer, and offline cost
(1) Cross-dataset transfer. With the victim model fixed to CLIP ViT-B/32, we evaluated MARP across datasets:
MARP shows some ability to transfer across datasets.
(2) Other retrieval models. We also retrained MARP for other retrieval models:
(3) Offline training cost. On Flickr30K, MARP takes an average of 19.38 minutes per training epoch, or approximately 8.02 ms per task per epoch (PGD-10 takes 65.95 ms to attack one task). The main table uses the checkpoint selected at epoch 30 on the validation set; reaching that epoch takes 9.69 hours of training. Once trained, the perturbation generator can be applied directly to new tasks under the same attack setting without task-specific retraining. It thus serves as a reusable attacker.
Thank you again for your suggestions.