acceptodds
Under review as a conference paper at ICLR 2027

Top-1 Irretrievability of Dense Retrieval

Abstract

Dense image retrieval under Maximum Inner Product Search (MIPS) embeds a query and each retrievable item into a common vector space, and returns the item with the largest inner product. Each target is therefore associated with a retrieval cone of query directions from which it is uniquely top-1 ranked. Our work identifies that, when its embedding lies inside the convex hull of the other items’ embeddings, the cone is empty, and no query can top-1 retrieve the target. We quantify the extreme rarity of this occurrence in natural data, yet we show such irretrievability can be engineered with near-perfect success rates. When embeddings are normalised, exact irretrievability is impossible. We therefore introduce ϵ-retrievability, relaxing the criterion to a retrieval cone of small volume, and develop methods to make the target difficult to retrieve. We demonstrate these methods on a wide variety of embedding models for both unimodal and cross-modal retrieval tasks often with near-perfect success rate.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Retrieval, Embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WCpeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the geometric conditions under which an item in dense retrieval becomes impossible or difficult to retrieve as the top-1 result. It establishes that an item is exactly irretrievable under maximum inner product search (MIPS) if its embedding lies inside the convex hull of the other indexed embeddings. For normalized embeddings, where exact irretrievability is impossible, the authors introduce ε-retrievability based on the normalized volume of the retrieval cone. They propose two decoy-insertion methods that manipulate embedding geometry while keeping the encoder fixed and the target unchanged. Experiments across nine retrieval models and multiple retrieval settings demonstrate high hiding rates, although the results for discrete text decoys are considerably weaker.

Critical strengths
  • Proposition 1 establishes the equivalence between convex-hull containment, an empty retrieval cone, and zero cone volume (Eq. 3). This provides a precise and interpretable foundation for analyzing top-1 irretrievability.
  • Figure 3 systematically examines decoy count, perturbation budget, index size, and embedding dimension, providing useful insights into the method's operating conditions.
Critical weaknesses
  • The main experiments use 48 Flickr30K targets, while several ablations use only 24 targets (Sections 3.3–4.2). Most principal tables report point estimates without uncertainty intervals, making it difficult to assess the variability and generalizability of the reported success rates.
  • Appendix A acknowledges the absence of direct comparisons with existing methods. Although related approaches are discussed in Section 5, the paper does not establish comparative effectiveness under matched perturbation budgets and computational constraints.
  • The cone-volume definition assumes uniformly sampled query directions (Eq. 2), whereas practical queries follow model-dependent distributions.
Questions for authors
  • For the results in Tables 3 and 5 that use embedding-level tangent-direction correction, can the authors demonstrate that the corrected embeddings can be realized by actual inputs and that the reported performance persists after re-encoding those inputs?

  • Can the authors evaluate whether the observed relationship between cone volume and retrieval success holds across alternative query distributions, rather than relying primarily on uniformly sampled directions or a particular query-search procedure?

5 comments

W1: The main experiments use 48 Flickr30K targets, while several ablations use only 24 targets (Sections 3.3–4.2). Most principal tables report point estimates without uncertainty intervals, making it difficult to assess the variability and generalizability of the reported success rates.

The limited size of target set in ablations is mainly due to our limited resources required to accommodate the extensive amount of ablations in our study. With Wilson intervals for the success rates, on 48 targets the interval follows: 100% gives [92.6%, 100%], 97.9% [89.1%, 99.6%], 95.8% [86.0%, 98.8%] and 0% [0%, 7.4%]. Almost every entry of the decoy-insertion row of Table 1 and of the C=64C = 64 column of Tables 3 and 4 is one of the first three, so each has a lower bound at 95% confidence of at least 86.0%. In Table 2 the comparison is paired, and every one of the 48 targets is ranked lower with decoys on every encoder (by 10 to 58 places at C=64C = 64).

Table 2 with 95% intervals for the median caption rank:

EncoderNo decoysC=8C = 8C=64C = 64
OpenCLIP ViT-B/323 [2, 13]7 [6, 16]33 [31, 42]
CLIP RN1015 [2, 24]7 [5, 27]26.5 [22, 42]
SigLIP 2 B/162 [1, 4]6 [6, 8]35.5 [34, 39]
MobileCLIP2 L/142 [1, 3]7 [5, 8]37.5 [33, 42]
MetaCLIP 2 H/143 [1, 5]8 [7, 9]45.5 [42, 48]
SigLIP 2 SO400M1 [1, 2]6 [5, 7]35 [34, 37]
Qwen3-VL-Embedding-2B2 [2, 3]6 [5, 7]27 [25, 28]

W2: Appendix A acknowledges the absence of direct comparisons with existing methods. Although related approaches are discussed in Section 5, the paper does not establish comparative effectiveness under matched perturbation budgets and computational constraints.

We understand the reviewer's concern. We note that the problem Section 3 aims to tackle is unique in the context of retrieval. To our best knowledge, for constructions detailed in Section 3, there is no directly comparable algorithm. For Section 4, our discussion is a strict generalisation (e.g. α=0\alpha = 0) to the existing algorithms.

For this response, we ported the objectives of the two algorithms closest to our Section 3 from data poisoning, Convex Polytope and Bullseye Polytope. We tested these on ViT-B/32 under the same experimental setting. None achieved hull enclosure (guranteed irretrievability).

ObjectiveIrretrievable targets, 1,500 stepsIrretrievable targets, 3,000 steps
Convex Polytope0% [0%, 7.4%]0% [0%, 7.4%]
Bullseye Polytope0% [0%, 7.4%]0% [0%, 7.4%]
Ours100% [92.6%, 100%]100% [92.6%, 100%]

For Section 4, the closest method is FoggySight, which optimises the decoys' embeddings onto the target's. We ran its objective (which is exactly equivalent to α=0\alpha = 0 in our construction) in our pipeline:

DecoysHidden from the text searchMedian caption rankCaptions with the target in the top 10
None12.5%365.4%
α=0\alpha = 0 (FoggySight's objective), C=8C = 843.8%955.4%
α=5∘\alpha = 5^\circ (as in Tables 2 and 3), C=8C = 866.7%759.6%
α=0\alpha = 0 (FoggySight's objective), C=64C = 6475.0%522.9%
α=5∘\alpha = 5^\circ (as in Tables 2 and 3), C=64C = 64100%330%

At α=0\alpha = 0 all decoys are optimised towards the same point, so they end up on one side of the target due to optimisation noise and its cone stays far larger. With 64 decoys, the search still retrieves a quarter of the targets at α=0\alpha = 0 and none at α=5∘\alpha = 5^\circ.

W3: The cone-volume definition assumes uniformly sampled query directions (Eq. 2), whereas practical queries follow model-dependent distributions.

The uniform measure in Eq. (2) is only used to define the cone size, but this uniformality is never used in our evaluation. Table 2 uses human captions and Table 3 uses text query search. We have not identified any particular problem with the way the cone size is defined in our study and experiments.

Q1: For the results in Tables 3 and 5 that use embedding-level tangent-direction correction, can the authors demonstrate that the corrected embeddings can be realized by actual inputs and that the reported performance persists after re-encoding those inputs?

Yes. The correction of App. D.7 removes an optimisation error that all decoys of a target share. We traced this error to the cause described in Section 3.2, where in some directions the embedding responds only weakly to the pixels. Section 3 handles this with Eq. (18) in its loss, and we use it in the same way to build these decoys as actual images, with no correction of their embeddings. On MobileCLIP2, a longer run of the paper's own loss is already enough without the correction. We do not observe any significant changes that can affect the conclusion of our study.

EntryWith the correctionActual images only
Table 3, MobileCLIP2 L/14, C=64C = 64100%100%
Table 5, DINOv2-B/14, α=0.5∘\alpha = 0.5^\circ100%95.8%
Table 5, DINOv2-B/14, α=0.75∘\alpha = 0.75^\circ85.4%81.3%
Table 5, DINOv2-B/14, α=1∘\alpha = 1^\circ35.4%14.6%
Table 5, DINOv3-L/16, α=0.5∘\alpha = 0.5^\circ100%95.8%
Table 5, DINOv3-L/16, α=0.75∘\alpha = 0.75^\circ41.7%100%
Table 5, DINOv3-L/16, α=1∘\alpha = 1^\circ2.1%16.7%

These only affect a small subset of our results. We will update Tables 3 and 5 as well as the appendices to reflect these changes, so that no result in the paper relies on the correction.

Q2: Can the authors evaluate whether the observed relationship between cone volume and retrieval success holds across alternative query distributions, rather than relying primarily on uniformly sampled directions or a particular query-search procedure?

We do not use uniformly sampled directions during evaluation. On ViT-B/32, for each of the 44 decoy configurations (number of decoys CC and angle α\alpha) of Fig. 4, we measured how often the target is retrieved under six query distributions and correlated the result with the cone volume with Spearman correlation:

Query distributionMeasure of retrievalCorrelation with cone volume
Adaptive text search (as in Table 3)target retrieved0.91
The target's own captions (as in Table 2)target in the top 100.96
Corpus captions nearest to the target's owntarget in the top 100.96
All 155,070 captions of the corpustarget in the top 100.97
All other images of the index, used as queriestarget in the top 100.77
Random queries at 5∘5^\circ / 10∘10^\circ from the targettarget ranked first0.16 / 0.15

We observe that a smaller cone makes retrieval of the target harder under any query distribution / scenarios tested.

I hope these can address the reviewer's concerns, and help them adjust their scores accordingly

I do not see the relevant changes highlighted in the main text. The author may highlight the changes made with colors.

Reviewer_2KLNOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how decoy insertion can make an unchanged target irretrievable in dense retrieval. It develops convex-hull-based constructions for exact, query-independent top-1 irretrievability and cone-volume reduction for normalized embeddings. Experiments span nine retrievers and unimodal/cross-modal tasks, with detailed tests achieving 95.8%–100% exact irretrievability across six encoders, supported by LP verification and parameter ablations.

Critical strengths

(1)The decoy-insertion method leaves the target unchanged while achieving query-independent top-1 irretrievability, extending beyond attacks on selected queries.

(2)The geometric construction, realization-error analysis, and LP verification establish a clear connection between theory and implementation.

(3)High success rates across six inner-product encoders, supported by parameter ablations, demonstrate effectiveness under the evaluated conditions.

Critical weaknesses

(1)The convex-hull criterion is established, so novelty depends on demonstrating a clear distinction from existing retrieval-poisoning and decoy-insertion methods.

(2)Evaluation on only 48 targets, with up to 96 optimized decoys and thousands of optimization steps, provides limited evidence of scalability and practical feasibility.

(3)Small cone volume under uniform query directions does not establish suppression of realistic semantic queries, and unique top-1 failure does not imply exclusion from top-k results.

Questions for authors

1.How does the proposed construction differ from prior retrieval-poisoning and decoy-insertion attacks beyond applying the established convex-hull criterion?

2.Do the normalized-embedding attacks suppress realistic semantic queries and top-k retrieval, rather than only reducing cone volume under uniformly sampled directions?

3.What are the computational cost and required model-access assumptions, and how many decoys are necessary to achieve reliable suppression?

4.How sensitive are the results to target selection, random initialization, and larger-scale evaluation beyond the 48 primary targets?

2 comments

W1: The convex-hull criterion is established, so novelty depends on demonstrating a clear distinction from existing retrieval-poisoning and decoy-insertion methods.

We credit the hull criterion in Sec. 2.2 and 5, and our work is the first that discusses the convex-hull criterion in the context of a fixed retriever model. Our contribution demonstrates the feasibility of achieving this irretrievability without modifying the target. Comparing against retrieval-poisoning and decoy-insertion:

  • Retrieval and corpus poisoning attacks (e.g. Zhong et al., 2023, Li et al., 2025) and FoggySight (Evtimov et al., 2021) provides no irretrievability guarantee and no measure of how retrievable the target remains. Our approach guarantees irretrievability for every query. Where a guarantee is impossible (Corollary 1), we measure the cone volume and derive a closed-form placement of the realisable decoys (Sec. 4).
  • Data-poisoning approaches such as Convex Polytope and Bullseye Polytope necessitate finetuning of the model, and again without any guarantee on irretrievability. We therefore aim the decoys' weighted average at a point beyond the target, which gives a known error tolerance (Prop. 2), and optimise in rescaled coordinates (Eq. 7).

Achieving guaranteed irretrievability is not easy. For this response we additionally adapted the objectives of the Convex Polytope and Bullseye Polytope objectives. The hiding success rate is 0%, against our 100% on ViT. Against 64 FoggySight-style decoys, text query search can still top-1 retrieve the target first in 25% of the targets, against 0% for ours.

W2: Evaluation on only 48 targets, with up to 96 optimized decoys and thousands of optimization steps, provides limited evidence of scalability and practical feasibility.

The limited size of target set is mainly due to limitations in our resources required to accommodate the extensive amount of ablations in our study, rather than a scalability / feasibility problem.

For demonstrating scalability, App. E.4 (Table 8) repeats the main experiments on larger corpora with ImageNet-1k (1.28 million) and Open Images V7 (1.91 million). In that ablation, decoy insertion results in an empty retrieval cone for 99.5% of the 192 targets, and 100% of them irretrievable under the text search. The success rate does not e.g. fall as the corpus size grows from about 2,000 images to 1.9 million.

For this response we also ran another 48 Flickr30K targets. All 48 have an empty cone, which brings Section 3's total success rate to 99.6% amongst the 240 runs.

W3: Small cone volume under uniform query directions does not establish suppression of realistic semantic queries, and unique top-1 failure does not imply exclusion from top-k results.

Section 4 already evaluates realistic queries, where Table 2 uses the targets' own human-written captions and a text query search in Table 3. Uniform directions are used only to define the cone volume (Eq. 2) but never used in any formal evaluation.

For top-kk, we note that is outside the scope of our current study, as stated in the title of the paper. We identify the top-kk irretrievability and their corresponding hull construction as a potential future direction.

We extended the caption evaluation of Table 2 for this response (ViT-B/32, R@kk is the ranking of the target in the top kk):

DecoysR@1R@5R@10R@20R@50
None30.8%55.4%65.4%74.6%83.3%
C=8C = 80%32.5%59.6%72.5%82.9%
C=64C = 640%0%0%1.7%74.2%

Q1: How does the proposed construction differ from prior retrieval-poisoning and decoy-insertion attacks beyond applying the established convex-hull criterion?

Please see our response to W1.

Q2: Do the normalized-embedding attacks suppress realistic semantic queries and top-k retrieval, rather than only reducing cone volume under uniformly sampled directions?

Yes for realistic queries. The attacks under normalised embeddings (Section 4) are evaluated with the targets' own human-written captions (Table 2) and a text query search (Table 3), and Fig. 4 (d) further shows the caption rank rising with the number of decoys. Uniform directions are never used as queries in our evaluation. The uniform measure only defines the cone volume (Eq. 2).

For top-kk, we note that it is outside the scope of our current study, as stated in the title of the paper. We identify the top-kk irretrievability and their corresponding hull construction as a potential future direction.

We extended the caption evaluation of Table 2 for this response (ViT-B/32, R@kk is the ranking of the target in the top kk):

DecoysR@1R@5R@10R@20R@50
None30.8%55.4%65.4%74.6%83.3%
C=8C = 80%32.5%59.6%72.5%82.9%
C=64C = 640%0%0%1.7%74.2%

Q3: What are the computational cost and required model-access assumptions, and how many decoys are necessary to achieve reliable suppression?

App. D.9 reports the cost on a single GPU. One target takes 8 to 97 minutes in Section 3 at 96 decoys. The runtime is heavily dependent on the encoder. In Section 4 with 64 decoys, it takes 3.5 to 28 minutes on the same six encoders and 57 minutes on Qwen3-VL-Embedding-2B. The algorithms can be accelerated significantly with fewer decoys. Both constructions need gradient access to the multimodal encoder.

We ablated the number of decoys needed in both Section 3 and 4. For hull containment, for encoders with fewer dimensions, one decoy already suffices. For Section 4, 8 decoys can often achieve decent success rate under text search as well.

Q4: How sensitive are the results to target selection, random initialization, and larger-scale evaluation beyond the 48 primary targets?

All targets used in the evaluation were drawn uniformly at random from Flickr30K. As mentioned in W2, we extended our evaluation with additional 48 target during this rebuttal, achieving 99.6% success rate for the 240 runs.

Section 3 has no random initialisation. The starting images and weights are initialised deterministically. Section 4 draws the starting photographs and the orientation of the decoys at random. For Section 4, we repeated the experiment on the same targets with three further random seeds:

SeedHidden rate, C=64C = 64Median caption rank, C=64C = 64Hidden rate, C=8C = 8Median caption rank, C=8C = 8
0 (paper)100%3366.7%7
1100%3352.1%7
2100%3547.9%7
3100%3458.3%7

With 64 decoys, every seed consistently hides all targets from the text search. With 8 decoys, the hidden rate differs by up to 18.8 points between seeds. These results demonstrate strong robustness against random initialisations

I hope these can address the reviewer's concerns, and help them adjust their scores accordingly

Reviewer_reYYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether an item in a dense retrieval corpus can become impossible to retrieve as the unique top-ranked result. It distinguishes two settings. Under unnormalized inner-product scoring, the authors construct adversarial decoys intended to place an unchanged target embedding inside the convex hull of competing items, eliminating its Top-1 retrieval region. Under normalized scoring, distinct embeddings remain theoretically retrievable, so the authors instead seek to reduce the solid angle of the target’s retrieval cone and make successful queries difficult to find. The paper addresses an interesting question and offers a constructive connection between retrieval geometry and adversarial corpus manipulation. Its strongest potential contribution is the realization of carefully designed decoy configurations through bounded input perturbations with a fixed encoder. However, the current evidence leaves substantial concerns about numerical certification, consistency with the stated threat model, comparisons with existing methods, and the interpretation of retrieval-cone volume.

Critical strengths
  1. The proposed insertion attack leaves both the target item and the retrieval model unchanged. Combining existing corpus embeddings with newly constructed decoys to obtain hull containment is a meaningful direction, particularly when the desired geometry must be realized through bounded changes to actual images.
  2. The presentation of this paper is clear and easy to follow.
  3. Experiments cover multiple vision-language encoders, image retrieval models, and larger corpora. The reported changes in caption ranks and query-search success suggest that decoy insertion can substantially affect retrieval behavior across different representations.
Critical weaknesses
  1. The results in Section 3.3 and Table 1 rely on the verification procedure in Appendix D.1. Specifically, Equation (15) restricts the query coordinates to ([-100,100]) and requires a minimum feasible radius of (10^{-8}). The paper interprets infeasibility of this LP as evidence that no query can retrieve the target as the unique Top-1 result. However, a nonempty but sufficiently narrow retrieval cone may also fail this test because its maximum feasible radius within the bounded domain falls below the threshold. The procedure therefore cannot reliably distinguish an empty cone from an extremely narrow one. Since irretrievability for every query is a central contribution, the paper should provide stronger numerical certificates, report solver tolerances, and examine whether the results remain stable across different bounds and thresholds.
  2. Section 4.2 evaluates attacks that keep the target and encoder fixed while inserting images optimized under a bounded pixel perturbation. However, Appendix D.7 states that some results additionally apply a correction directly to the decoy embeddings, and that these corrected embeddings have not been realized as images. This affects the MobileCLIP2 result with (C=64) in Table 3 and all decoy rows in Table 5. These entries therefore do not establish that the reported hiding rates can be achieved solely by inserting actual perturbed images. The affected results should be clearly identified in the main text and tables, and the corresponding results without embedding correction should be reported to establish performance under the stated attacker capabilities.
  3. The evaluation does not sufficiently establish the advantage over existing methods or separate attack effectiveness from query-search limitations. Section 5 discusses closely related methods, including FoggySight, Convex Polytope, and Bullseye Polytope, but Sections 3.3 and 4.2 provide no explicit comparisons against adapted decoy-insertion baselines. Comparing against target-only perturbation does not show whether the proposed geometric construction outperforms simpler ways of generating competing items. Appendix A’s claim that FoggySight is implicitly evaluated as an (\alpha=0) special case also does not replace a controlled comparison under matched budgets. Furthermore, Table 3 reports ARCA hidden rates of 62.5% and 64.6% for the two SigLIP 2 encoders even without decoys. Although Appendix A acknowledges this limitation, these high baseline failure rates make it difficult to determine how much of the reported hiding reflects the attack rather than weaknesses of the query-search algorithm. Matched insertion baselines, stronger query search, and results conditioned on targets successfully retrieved before the attack would provide more convincing evidence.
Questions for authors
  1. Given the fixed query bound and positive radius threshold in Appendix D.1, Equation (15), how do you distinguish a truly empty retrieval cone from a nonempty cone that is too narrow to pass the numerical test, and how sensitive are the results in Table 1 to these settings?
  2. Can the MobileCLIP2 result with C=64C=64 in Table 3 and the decoy results in Table 5 be reproduced using only actual perturbed images, without the direct embedding correction described in Appendix D.7?
  3. How does the proposed method compare with adapted FoggySight, Convex Polytope, and Bullseye Polytope baselines under matched attack budgets, particularly when using stronger query search and evaluating targets that were successfully retrievable before the attack?
2 comments

W1: The procedure therefore cannot reliably distinguish an empty cone from an extremely narrow one. [...] the paper should provide stronger numerical certificates, report solver tolerances, and examine whether the results remain stable across different bounds and thresholds.

We thank the reviewer for raising this point. Below we provide evidence that the examples are irretrievable not because of numerical issues but because the volume of the cone is exactly zero. That being said, the tolerance of our LP solver is 10−710^{-7}. Therefore, even if our solver was to fail to detect a cone because its radius is below this tolerance, this region would be so small as to be irretrievable in practice, because the neural network encoder would also run into the same numerical issues: it would not be precise enough to produce an activation in this cone.

As a stronger certificate, we solved a second LP for these targets, which searches directly for the weights of a convex combination. It gives explicit weights, strictly positive on every other image, that reproduce the target as a weighted average of the other images and the decoys up to a numerical error of about 10−1410^{-14}, so the target lies strictly inside the convex hull and its cone is empty rather than narrow (Prop. 1). We will add the solver settings and test results to App. D.1.

W2: [...] Appendix D.7 states that some results additionally apply a correction directly to the decoy embeddings, and that these corrected embeddings have not been realized as images. This affects the MobileCLIP2 result with (C=64C=64) in Table 3 and all decoy rows in Table 5. [...]

The correction of App. D.7 removes an optimisation error that all decoys of a target share. We traced this error to the cause described in Section 3.2, where in some directions the embedding responds only weakly to the pixels. Section 3 handles this with Eq. (18) in its loss, which we can similarly use to build realisable images rather than corrections. On MobileCLIP2, a longer run of the paper's own loss without correction. We will report the image-only results in Tables 3 and 5, which affects only a small subset of the paper's results, and so that no result in the paper relies on the correction. We do not observe any significant changes that affect the conclusion of the study.

EntryWith the correction of App. D.7Actual images only
Table 3, MobileCLIP2 L/14, C=64C = 64100%100%
Table 5, DINOv2-B/14, α=0.5∘\alpha = 0.5^\circ100%95.8%
Table 5, DINOv2-B/14, α=0.75∘\alpha = 0.75^\circ85.4%81.3%
Table 5, DINOv2-B/14, α=1∘\alpha = 1^\circ35.4%14.6%
Table 5, DINOv3-L/16, α=0.5∘\alpha = 0.5^\circ100%95.8%
Table 5, DINOv3-L/16, α=0.75∘\alpha = 0.75^\circ41.7%100%
Table 5, DINOv3-L/16, α=1∘\alpha = 1^\circ2.1%16.7%

W3 (first part): Section 5 discusses closely related methods, including FoggySight, Convex Polytope, and Bullseye Polytope, but Sections 3.3 and 4.2 provide no explicit comparisons against adapted decoy-insertion baselines. [...] Appendix A's claim that FoggySight is implicitly evaluated as an (α=0\alpha=0) special case also does not replace a controlled comparison under matched budgets.

For Section 3, Convex Polytope and Bullseye Polytope are data-poisoning attacks. They need the model to be fine-tuned, whereas our encoder stays fixed and the target must be enclosed exactly. They therefore cannot be run as published in our setting, and we ported their objective to the experimental setting of Table 1. Neither objective makes any target irretrievable:

ObjectiveIrretrievable targets, 1,500 stepsIrretrievable targets, 3,000 steps
Convex Polytope0%0%
Bullseye Polytope0%0%
Ours100%100%

For Section 4, FoggySight optimises the decoys' embeddings onto the target's, which is our construction with the angle α\alpha set to 0.

DecoysHidden from the text searchMedian caption rankCaptions with the target in the top 10
None12.5%365.4%
α=0\alpha = 0 (FoggySight's objective), C=8C = 843.8%955.4%
α=5∘\alpha = 5^\circ (as in Tables 2 and 3), C=8C = 866.7%759.6%
α=0\alpha = 0 (FoggySight's objective), C=64C = 6475.0%522.9%
α=5∘\alpha = 5^\circ (as in Tables 2 and 3), C=64C = 64100%330%

At α=0\alpha = 0 all decoys are optimised towards the same point, so they end up on one side of the target due to optimisation noise and its cone stays far larger. With 64 decoys, the search still retrieves a quarter of the targets at α=0\alpha = 0 and none at α=5∘\alpha = 5^\circ.

We will add both comparisons to the paper and revise App. A accordingly.

W3 (second part): Table 3 reports ARCA hidden rates of 62.5% and 64.6% for the two SigLIP 2 encoders even without decoys. [...] Matched insertion baselines, stronger query search, and results conditioned on targets successfully retrieved before the attack would provide more convincing evidence.

The no-decoy rates do not come from a weak search. Without decoys, the search is always able to find a query that ranks the target first for all targets on every encoder. Table 3 then counts a target as retrieved only if this query still ranks it first after being decoded to text then re-tokenised again (App. D.8), so that only queries a user can input are counted. This step often changes the token sequence, and the changed query may no longer retrieve the target:

EncoderSearch finds a query that ranks the target firstQuery no longer ranks the target first after being decoded to text and tokenised again (the no-decoy hidden rate in Tables 3 and 4)
OpenCLIP ViT-B/32100%12.5%
CLIP RN101100%43.8%
SigLIP 2 B/16100%62.5%
MobileCLIP2 L/14100%10.4%
MetaCLIP 2 H/14100%8.3%
SigLIP 2 SO400M100%64.6%
Qwen3-VL-Embedding-2B100%10.4%

For this response we also ran a stronger search on the two SigLIP 2 encoder, where the search stops only when its query still ranks the target first after decoded and retokenised again.

EncoderDecoysHidden rateHidden rate, stronger search
SigLIP 2 B/16None62.5%0%
SigLIP 2 B/16C=8C = 877.1%22.9%
SigLIP 2 B/16C=64C = 64100%100%
SigLIP 2 SO400MNone64.6%0%
SigLIP 2 SO400MC=8C = 870.8%37.5%
SigLIP 2 SO400MC=64C = 64100%100%

Without decoys, the stronger search retrieves every target on both encoders. With 64 decoys it still retrieves none, demonstrating the consistent performance of the algorithm.

We will add these results and clarify in the appendices.

Q1: Given the fixed query bound and positive radius threshold in Appendix D.1, Equation (15), how do you distinguish a truly empty retrieval cone from a nonempty cone that is too narrow to pass the numerical test, and how sensitive are the results in Table 1 to these settings?

Please see our response to W1.

Q2: Can the MobileCLIP2 result with (C=64C=64) in Table 3 and the decoy results in Table 5 be reproduced using only actual perturbed images, without the direct embedding correction described in Appendix D.7?

Please see our response to W2.

Q3: How does the proposed method compare with adapted FoggySight, Convex Polytope, and Bullseye Polytope baselines under matched attack budgets, particularly when using stronger query search and evaluating targets that were successfully retrievable before the attack?

Please see our response to W3.

I hope these can address the reviewer's concerns, and help them adjust their scores accordingly

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.