acceptodds
Under review as a conference paper at ICLR 2027

Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning

Abstract

Contrastive learning has become a standard approach for unsupervised learning from paired data, as demonstrated by CLIP for image-text matching. However, many domains involve more than two modalities and require objectives that capture higher-order dependencies beyond pairwise alignment. Symile extends CLIP to this setting by replacing the dot product with the multilinear inner product (MIP) over modality embeddings. In this work, we show that there is a fragility which is hidden in the multiplicative interaction: a single weakly informative, misaligned, or missing modality can propagate through the objective and distort cross-modal retrieval scores. We propose Gated Symile, a contrastive gating mechanism that adapts modality contributions on an attention-based, per-candidate basis. The gate suppresses unreliable inputs by interpolating embeddings toward learnable neutral directions with an explicit NULL option when reliable cross-modal alignment is unlikely. Across a controlled synthetic benchmark that uncovers this fragility and three real-world trimodal datasets, Gated Symile achieves higher top-1 retrieval accuracy than well-tuned state-of-the-art (sota) baselines. More broadly, our results highlight gating as a step toward robust multimodal contrastive learning beyond two modalities in the presence of noise, misalignment, or missing inputs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal contrastive learning, modality misalignment, robust representation learning, higher-order interactions, multilinear inner product, modality gating, missing modalities, cross-modal retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TQE2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Gated Symile to improve multimodal contrastive learning under modality misalignment and missing inputs. Its candidate-conditioned gate selectively suppresses unreliable modalities through interpolation toward learnable neutral embeddings. Validation on synthetic and biomedical datasets demonstrates improved retrieval accuracy, particularly under misalignment and missing-modality conditions.

Critical strengths

The paper combines a clear robustness motivation with substantial empirical validation. Controlled misalignment experiments demonstrate the targeted failure mode, while improvements on biomedical retrieval datasets and extensive ablations support the practical value of adaptive modality gating.

Critical weaknesses
  • Limited theoretical support: The robustness derivation assumes fixed gate weights and excludes the final normalization, although the proposed gate depends on the perturbed embedding. Consequently, the bound does not establish robustness for the implemented model, weakening a central theoretical claim.
  • Insufficiently controlled comparisons: The gate introduces 44.1M additional parameters and candidate-conditioned scoring, but no genuinely parameter-matched baseline is provided. It therefore remains unclear whether the gains arise from reliability-aware suppression or increased capacity and scoring expressiveness.
  • Limited validation of real-world robustness: The strongest misalignment evidence comes from synthetic data and deliberately permuted biomedical inputs. While useful controlled tests, these do not establish robustness to naturally occurring noise or conflicting evidence, limiting support for the paper’s broader robustness claims.
Questions for authors
  1. Theoretical clarification: The robustness derivation holds gate weights fixed and excludes final normalization. Can the analysis account for both input-dependent gate variation and normalization, or should the robustness claim be restricted to the fixed-gate, pre-normalized setting?

  2. Controlled comparison: Can you provide a parameter-matched, candidate-conditioned baseline without reliability-aware suppression? This would help distinguish the benefit of the proposed gating mechanism from additional capacity and scoring expressiveness.

  3. Robustness clarification: Beyond deliberately permuted inputs, what evidence supports robustness to naturally occurring modality unreliability? Can the reported real-world gains be separated from improvements attributable to explicit missingness indicators?

1 comment

With the rebuttal phase coming to an end, we thank the reviewer again for their time and feedback. Overall, the new results support our central findings.

We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.

Reviewer_bjfSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the vulnerability of multimodal contrastive learning to unreliable or misaligned modalities, particularly in Symile’s multiplicative interactions. The authors propose Gated Symile, which uses candidate-dependent attention weights, learnable neutral directions, and a NULL option to suppress unreliable modality contributions. Experiments on a synthetic benchmark and three real-world tri-modal retrieval settings demonstrate improvements in top-1 retrieval accuracy over existing baselines. Additional ablation studies and gate behavior analyses further support the effectiveness of the proposed mechanism.

Critical strengths

1.The authors focus on modality-level unreliability and its impact on higher-order multiplicative interactions, rather than simply improving overall cross-modal alignment. Synthetic-XNOR provides a targeted diagnostic setting by preserving modality marginal distributions while breaking cross-modal alignment through sample swapping. This establishes a clear research problem and connects the proposed method to a specific failure mode. 2.Gated Symile achieves the highest reported top-1 retrieval accuracy across all evaluated settings. Importantly, the authors go beyond retrieval performance by analyzing gate weights, embedding geometries relative to neutral directions, and ablation studies. These results show that the model exhibits the expected behavior under controlled modality misalignment. Even compared with Symile using pair sampling, the performance gains on the synthetic dataset remain substantial.

Critical weaknesses

1.The authors derive a perturbation bound in Equations (11)–(12) and Appendix B under the assumption that the gate weights remain fixed. However, the proposed gating mechanism is input-dependent, meaning that perturbations to a non-target modality can also change its gate weights. The derivation holds under the stated assumptions, but it does not fully establish the perturbation robustness of the complete Gated Symile model. In addition, the normalization step may change or even amplify the sensitivity to perturbations. The paper does not provide sufficient conditions or a complete bound to account for these effects. 2.The experimental configurations used in the main comparisons require further clarification. In Table 2, Symile achieves an accuracy of 0.3310 on Synthetic-XNOR, whereas Symile with pair sampling reaches 0.6466 in Appendix Table 10, indicating that the negative-sampling strategy has a substantial impact on performance. Although Gated Symile still clearly outperforms Symile with pair sampling in the corresponding ablation study, the main results do not sufficiently clarify the differences in sampling strategies and experimental settings. This may lead readers to overestimate the performance gains from the gating mechanism itself.

Questions for authors

1.Could the authors clarify how the theoretical perturbation bound applies to the input-dependent gate weights and the final L2 normalization? Specifically, does Equation (12) still hold when \beta changes with the perturbed input? 2.Could the authors clarify the sampling strategies used for each method in the main Synthetic-XNOR results? In Table 2, Symile achieves an accuracy of 0.3310, whereas in Table 10, Symile with n-sampling and pair sampling achieves 0.3171 and 0.6466, respectively. Could the authors present or reference directly comparable results under consistent experimental settings in the main results?

Reviewer_1AiCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds candidate-dependent gates to Symile to reduce the influence of unreliable modalities, blending their features with learned default vectors before scoring. It also introduces a synthetic benchmark with controlled modality misalignment. Experiments on this benchmark report improved retrieval accuracy, with ablations and gate analyses supporting the proposed design.

Critical strengths
  1. The candidate-dependent gate provides a sensible way to handle mismatched inputs. Interpolating toward learned prototypes lets the model reduce an unreliable modality’s influence without zeroing the multiplicative score.
  2. The method shows gains on real data, including a 2.2 percentage-point improvement over Symile with modality dropout on UKB-Union. Cross-validation and retuned ablations provide useful evidence that the approach has practical value.
Critical weaknesses
  1. The paper starts with a broad problem in multimodal contrastive learning, but the main contribution is a fix for Symile’s multiplicative score. I can see why this modification is useful. What is less clear is what we learn beyond how to improve Symile, which leaves me unconvinced about the novelty.

  2. It is hard to tell whether the gains come from handling unreliable inputs or simply adding a stronger scoring network. The component ablations do not settle this. A comparison with a similarly sized candidate-dependent scorer would help show what the gate and prototype replacement actually add.

Questions for authors
  1. Since the gate relies on pairwise similarities, how does it avoid suppressing modalities that are uninformative individually but useful jointly?
  2. Since the gates use pairwise similarities, could they suppress modalities that are informative only jointly? Can the authors test this on a task where neither input alone provides information about the target?
1 comment

With the rebuttal phase coming to an end, we thank the reviewer again for their time and feedback. Overall, the new results support our central findings.

We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.