acceptodds
Withdrawn from ICLR 2027

Toward Robust Multimodal Retrieval for MLLMs with Transport Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal retrieval ranks image-text candidates according to semantic relevance and supports applications such as image-to-text retrieval, text-to-image search, and composed retrieval. However, existing multimodal retrieval models are highly sensitive to adversarial perturbations on image queries. Even when the perturbation is almost invisible, it can move the query representation away from the original cross-modal semantic structure, distort matching between the query and candidates, and sharply degrade ranking performance. To address this problem, we propose ROTA, a training-free framework for robust multimodal retrieval with transport alignment. ROTA constructs transformed views of the attacked query and uses retrieval entropy to select reliable views, rather than relying on a single corrupted query representation. It further enriches candidate texts with descriptive views and aligns image-view and text-view distributions using optimal transport. The resulting retrieval score combines reconstructed query evidence with distribution-level image–text alignment, reducing reliance on brittle single-view matching. Furthermore, we analyze how perturbations in the OT cost matrix affect transport matching, which clarifies why distribution-level alignment can better tolerate small representation changes. Experiments across retrieval benchmarks, adversarial attacks, and retrieval backbones show that ROTA consistently improves performance over direct retrieval with attacked queries.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Retrieval, Robustness, LLM, vLLM, mLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WEkYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ROTA, a training-free framework for multimodal retrieval under adversarial image perturbations. The method selects transformed image views using retrieval entropy, reconstructs the query representation, augments candidate texts with descriptive views, and applies entropy-weighted optimal transport for retrieval scoring. Experiments cover standard and composed retrieval benchmarks, multiple embedding backbones, and several attack methods. The paper also provides a conditional stability analysis for the OT score.

Critical strengths

A practical framework with substantial reported improvements. ROTA operates without updating model parameters and substantially improves attacked retrieval performance over direct retrieval and the evaluated aggregation baselines. The results across different datasets and backbones suggest that the approach has potential beyond a single retrieval configuration.

Useful component and efficiency analyses. The ablations provide evidence that both query reconstruction and transport matching contribute beyond basic view aggregation. The runtime comparison and large-gallery reranking experiment also help assess the practical trade-off between retrieval performance and computational cost.

Critical weaknesses

Robustness against attacks targeting the complete defense is not sufficiently established. The main attacks optimize the backbone retrieval objective rather than explicitly targeting ROTA’s final ranking. The supplementary EOT/BPDA experiment is a useful addition, but its description emphasizes attacks on original and augmented representations without clearly establishing whether the complete selection, reconstruction, weighting, and OT scoring pipeline is attacked. The limited attack configuration and lack of convergence evidence leave uncertainty about the strength of the evaluation. Since adversarial robustness is the central claim, this distinction materially affects the interpretation of the reported gains.

The distinctive contribution over closely related view-based methods needs stronger validation. The paper builds on established augmentation, confidence weighting, text enrichment, and transport matching ideas, and explicitly identifies AWT as closely related. However, a direct comparison with AWT or a carefully adapted counterpart is missing. Although the component ablations are informative, comparisons with simpler confidence-weighted similarity aggregation using identical views would better establish the specific value of transport alignment. The theoretical result concerns fixed views and transport marginals, so it provides limited support for the stability of the adaptive pipeline itself.

Questions for authors
  1. Does the adaptive attack directly optimize ROTA’s final retrieval scores, including its view selection, reconstruction, entropy weighting, and transport matching? Please clarify which components are differentiated through or approximated.
  2. Do the improvements persist under stronger adaptive optimization with additional iterations, restarts, and stochastic samples? Please report the worst performance across the evaluated attacks rather than considering each attack independently.
  3. How does ROTA compare with an adapted AWT baseline and confidence-weighted similarity aggregation when all methods use the same image views, text descriptions, and computational budget?
  4. For composed retrieval, are image-conditioned text descriptions generated from the attacked visual input? Please clarify the information available to the defense and whether query-dependent description generation is included in the reported computational cost.
Reviewer_PyZaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ROTA, a training-free framework for multimodal retrieval under adversarial image perturbations. ROTA selects low-entropy augmented image views, reconstructs the query using their directional coherence, and ranks candidates through entropy-weighted optimal transport between image and descriptive text views. Experiments on Flickr30K, COCO, CIRR, and FashionIQ report substantial improvements over direct retrieval, view ensembling, and adaptation baselines.

Critical strengths

S1. The reported improvements are substantial and supported by broad evaluation. ROTA improves robust retrieval across multiple datasets, backbones, and attacks, including composed retrieval. Table 4 demonstrates complementary benefits from reconstruction and OT, while the adaptive-attack and large-gallery experiments provide useful additional evidence of practical effectiveness.

Critical weaknesses

W1. The methodological novelty is insufficiently established relative to directly relevant prior work. Section 3.2 closely follows DBD [X1]: low-entropy view selection, mean-displacement direction estimation, directional coherence measured by average cosine similarity, and threshold-dependent reconstruction through either amplified displacement or selected-view averaging. In particular, ROTA’s Eq. (6) has the similar functional form as DBD’s reconstruction rule after substituting notation. Section 3.3 also closely follows AWT [X2], which already combines image/text augmentation, entropy-based weighting, and cross-modal OT. Although AWT is cited, DBD is not discussed in the provided manuscript. The paper therefore needs to establish what is methodologically new beyond combining these existing mechanisms and applying them to adversarial retrieval. The change of task alone does not establish the claimed methodological contribution.

W2. The comparisons do not isolate a contribution beyond these existing mechanisms. The principal baselines are Vanilla, Ensemble, and MTA; neither AWT adapted to retrieval nor DBD-based retrieval is evaluated. Table 4 establishes that reconstruction and OT improve performance, but these are precisely the components with close precedents. Consequently, the results do not distinguish the proposed design from a straightforward DBD+AWT combination. Comparisons using the same backbone, transformations, text descriptions, and computation budget are essential to determine whether ROTA provides an advance beyond transferring and combining prior methods. The large gains over direct retrieval cannot resolve this attribution gap.

[X1] Liu et al. Adversarial Attacks Already Tell the Answer: Directional Bias-guided Test-Time Defense for Vision-Language Models. ICLR 2026.

[X2] Zhu et al. AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation. NeurIPS 2024.

Questions for authors
  1. What precisely distinguishes ROTA from DBD [X1] and AWT [X2]? Please provide a component-level comparison covering low-entropy view selection, directional reconstruction, entropy-based weighting, and distribution-level OT alignment. In particular, how do Eqs. (4)–(6) differ from DBD’s direction estimation, directional-bias score, and two-stream reconstruction? How does Section 3.3 differ from AWT’s augmentation–weighting–transport framework?
  2. What technical obstacles arise specifically when transferring these methods to retrieval, and how does ROTA resolve them? Applying existing methods to a new task can be valuable, but the paper should explain why straightforward transfer is insufficient. Do large candidate galleries, multiple valid captions, or composed queries introduce difficulties for confidence estimation, reconstruction, or transport matching? Please identify the corresponding design changes and provide analyses demonstrating that they address these difficulties.
  3. Can the authors provide the following evidence to establish an advance beyond combining known components? (i) AWT adapted to the same retrieval setting; (ii) DBD reconstruction followed by cosine-similarity retrieval; (iii) a straightforward DBD+AWT combination, with matched backbone, transformations, text descriptions, and computation budget; and (iv) a targeted analysis isolating a retrieval-specific failure of these transferred methods and demonstrating how ROTA resolves it. These comparisons are central to my assessment: Table 4 establishes that reconstruction and OT are useful, but does not establish what is new beyond their existing formulations or whether ROTA outperforms their straightforward combination.
Reviewer_svdEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ROTA, a training-free framework for multimodal retrieval under adversarial image perturbations. ROTA generates transformed image views, selects views using retrieval entropy, and reconstructs the query representation according to the directional coherence of feature displacements. It also generates descriptive text views and ranks candidates through entropy-weighted optimal transport between visual and textual view distributions. The evaluation covers standard and composed retrieval on Flickr30K, COCO, CIRR, and FashionIQ with multiple backbones and attacks, supplemented by component ablations, adaptive EOT/BPDA evaluation, and large-gallery experiments. The theoretical contribution establishes an OT retrieval-margin stability bound under bounded cost perturbations with fixed view indices and transport marginals.

Critical strengths
  1. Substantial empirical gains supported by complementary component contributions. Table 1 improves Flickr30K I2T R@1 with LLM2CLIP under PGD from 34.8 for Ensemble to 64.2 for ROTA. Table 4 further shows that adding query reconstruction to image/text views with OT improves PGD/MI-FGSM R@1 from 59.9/50.3 to 64.2/56.1. These results support the contribution of the reconstruction module and constitute the strongest empirical evidence supporting acceptance.
  2. Evaluation across tasks and backbones supports broader research relevance. The main paper and appendix cover four principal datasets, four backbones, and both standard and composed retrieval, while reporting performance under clean and attacked conditions. This breadth suggests that the observed benefits are not confined to a single benchmark or model and provides useful evidence for studying multi-view information and cross-modal matching in robust retrieval.
Critical weaknesses
  1. Adversarial robustness of the complete defended retrieval rule remains insufficiently established. Appendix C.1 reports EOT/BPDA evaluation, but does not explicitly define whether the attack loss targets the final OT ranking scores in Eq. (12), or how adaptive view selection, reconstruction, and input-dependent weights are incorporated. This leaves uncertainty about how much of the reported improvement persists when an adversary directly optimizes against the complete defended ranking decision. Since adversarial robustness is the central claim, this evidential gap limits the persuasiveness of the main conclusion and is my principal concern.
  2. The independent contribution of coherence-guided adaptive reconstruction is insufficiently established. Table 4 supports reconstruction as a whole, and the λ = 1 setting in Figure 3 includes mean reconstruction. However, these results do not sufficiently isolate the benefit of switching reconstruction branches according to directional coherence. The experiments do not directly compare adaptive switching with applying the same directional reconstruction branch to every query while holding the selected views and reconstruction strength fixed. Consequently, the respective contributions of adaptive judgment and generic feature extrapolation remain unclear. This limits the evidence for the methodological advance specifically attributable to the proposed adaptive mechanism.
Questions for authors
  1. Clarification—attack objective: What exact attack loss is used in Appendix C.1? Does it directly target the final OT ranking scores in Eq. (12), accounting for input-dependent view selection, reconstruction, and changing weights? If a surrogate loss is used, please explain why it effectively captures ranking failure of the complete defended system.
  2. Clarification—evaluation evidence: Does the attack-step experiment in Figure 4 use the same adaptive objective and complete defense pipeline as Table 12? What convergence or attack-strength diagnostics support the adequacy of the configuration in Table 12 for evaluating robustness under that objective?
  3. Clarification—mechanism contribution: With the selected views and reconstruction strength held fixed, which results establish that coherence-based branch switching improves over applying the same directional reconstruction branch to every query?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.