acceptodds
Under review as a conference paper at ICLR 2027

Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

Abstract

Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimodal alignment can introduce optimization and representation challenges. We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning: poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal alignment. We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization and demonstrating that simple modifications like LeakyReLU activations and residual paths recover these geometric benefits. Across a synthetic benchmark and four real-world datasets including missing modalities, improving Jacobian conditioning boosts retrieval and linear probe performance across multiple contrastive objectives, whereas expressive objectives yield little benefit in linear probes. More broadly, our results show that multimodal contrastive learning depends not only on objective expressivity, but also on the geometric and optimization properties of the underlying encoders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal contrastive learning, geometry-preserving encoders, Jacobian conditioning, representation learning, residual connections, LeakyReLU, cross-modal retrieval, multimodal alignment, encoder architecture, downstream transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_igNLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the influence of encoders' geometric properties on trimodal contrastive learning and identifies encoder Jacobian conditioning as a key factor. Mathematical analysis is provided. Based on this finding, the paper proposes geometry-preserving encoders (GPEs). GPEs introduce simple modifications like LeakyReLU activations and residual paths, yet yield improvements in most retrieval tasks and some downstream tasks. Experiments are conducted on trimodal contrastive learning tasks using a synthetic benchmark and healthcare datasets.

Critical strengths
  1. The paper highlights the importance of encoders' geometry for trimodal contrastive learning, and links encoder trainability to Jacobian conditioning and dynamical isometry. The authors use theoretical analysis and empirical experiments to substantiate their claims.
  2. The proposed practical interventions based on residual paths and LeakyReLU activations are easy to integrate into common encoders and simple to implement.
Critical weaknesses
  1. The title and abstract indicate that the paper contributes to multimodal contrastive learning, but the evidence for the generalizability of the proposed method is limited. Although the proposed method and theoretical analysis in this paper don't depend on healthcare-specific properties, all real-world experiments are carried out on healthcare datasets, despite the availability of non-healthcare trimodal datasets like Symile-M3, CMU-MOSI, MOSEI, etc.. The applicability to settings beyond healthcare remains unclear, and the claims about the method’s generalizability should be qualified to reflect the empirical scope.
Questions for authors
  1. The motivation of this work focuses on proposing a different perspective for multimodal representation learning, as much of the previous work focuses on objective design and representation geometry. While cross-objective experiments show that several multimodal contrastive objectives benefit from GPEs, could the authors also discuss whether GPEs can still provide additional gains when combined with methods that jointly model embedding norms and directions? In particular, to what extent do these approaches alleviate overlapping optimization difficulties, and are their benefits expected to be complementary?
Reviewer_tsvcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies that poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal learning. Based on this observation, the paper proposes to use LeakyRELU and residual connections to recover the geometry.

Critical strengths

The paper uses Jacobian condition analysis for encoder design.

Critical weaknesses
  1. Insufficient motivation for the focus on trimodal learning. I am not convinced that the proposed “geometry” problem is sufficiently motivated in the context of trimodal contrastive learning. Poor Jacobian conditioning is a general issue in neural network optimization and can arise across many architectures and tasks. The paper does not sufficiently explain whether, or why, interactions between modalities or higher-order contrastive objectives introduce or exacerbate this problem. Consequently, the specific insight gained by studying it in the trimodal setting remains unclear.

  2. Limited novelty in the analysis. The relationship between Jacobian conditioning, signal propagation, and network trainability has been extensively studied. The analysis primarily draws on familiar properties of activation derivatives, products of weight matrices, and standard singular-value perturbation bounds. While applying these ideas to multimodal learning can be valuable, the paper does not sufficiently establish a new theoretical or mechanistic insight beyond these general principles.

  3. Established architectural modifications and unclear applicability to other activations. The proposed GPE combines residual connections and LeakyReLU, both of which are established architectural choices. Its contribution therefore depends on demonstrating benefits beyond standard encoder designs. It remains unclear whether comparable gains would persist with GELU, SiLU, or SwiGLU-based feed-forward layers, particularly when residual connections and normalization are already present. Controlled comparisons across these choices are needed to establish the method’s practical relevance and the scope of its explanation.

  4. Insufficiently strong architectural baselines. Although the paper compares several contrastive objectives, the encoder baselines limit the interpretation of the reported gains.

Questions for authors

See my weakness.

1 comment

With the rebuttal phase coming to an end, we thank the reviewer again for their time and feedback. Overall, the new results support our central findings.

We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.

Reviewer_nbX8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper investigates how encoder geometry affects performance in multimodal contrastive learning with three or more modalities. It links poor encoder Jacobian conditioning, characterized by a large ratio between the maximum and minimum singular values, to degraded performance and investigates Jacobian-vector product (JVP)-based regularization as an intervention. It also empirically demonstrates that residual transport and LeakyReLU activations provide effective architectural alternatives. Experiments on synthetic and healthcare datasets show that geometry-preserving encoders (GPEs) improve retrieval performance and yield gains in linear probing in several settings. These findings highlight the importance of accounting for encoder geometry when training multimodal models.

Critical strengths

This paper achieved performance improvements through the proposed method in various trimodal settings.

Critical weaknesses

W1. Clarity of presentation. Several parts of the paper are difficult to follow. Some notation is used without clear definitions, and the baseline methods are insufficiently explained. Extensive experimental details in the main text also make it difficult to identify the purpose and main findings of each experiment.

W2. Limited methodological novelty and theoretical depth. Random-projection Jacobian regularization and random-direction gain constraints have substantial precedent (Hoffman et al., 2019; Odena et al., 2018). Residual mechanisms for improving Jacobian conditioning and LeakyReLU activations are also established (Tarnowski et al., 2019; Maas et al., 2013). The contribution therefore rests primarily on applying these ideas to multimodal contrastive learning. However, the theoretical analysis mainly applies standard singular-value perturbation bounds, without demonstrating that the required conditions hold in the trained models. This provides limited theoretical support for the proposed explanation of the empirical improvements.

W3. Limited experimental support for broader generalization. The evaluation focuses primarily on healthcare datasets and MLP-based encoders or projection heads, while downstream evaluation is restricted to logistic-regression linear probes. The experiments do not systematically establish whether the proposed geometric mechanism generalizes across architectures and application domains.

Questions for authors

Please see the Weaknesses section above.

Reviewer_mAV7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why trimodal contrastive learning (objectives: CLIP, Triangle, Gram, Symile) often trains poorly when modality encoders are plain MLPs — a common situation in healthcare multimodal learning (tabular modalities, foundation-model embeddings with MLP projection heads). The authors' hypothesis is that the bottleneck is not objective expressivity but encoder Jacobian conditioning: during contrastive training, plain ReLU MLP encoders develop collapsing minimum singular values, exploding maximum singular values, and condition numbers up to 10^8–10^10, coinciding with stagnating retrieval accuracy.

The contributions are: (i) a cross-objective, cross-dataset diagnosis of this degeneration (Sections 3.2, 4.3); (ii) a stochastic JVP-based "directional gain band" Jacobian regularizer used as a causal intervention (Section 3.3, Figure 4); (iii) geometry-preserving encoders (GPEs) — residual paths E_i(x) = P_i x + h_i(x) plus LeakyReLU activations — motivated by Weyl-type singular-value perturbation bounds (Section 3.4, Appendix A); and (iv) experiments on Synthetic-XNOR, MIMIC-Symile, MIMIC-IV, UKB-Intersection, and UKB-Union (missing modalities), with a 100-trial Bayesian hyperparameter search per method, 5-fold cross-validation × 3 seeds, measuring top-1 retrieval accuracy and mortality linear probes (AUROC). GPEs yield large retrieval gains (e.g., UKB Gram 0.1498 → 0.6293, CLIP 0.3860 → 0.6944) and small, mixed probe gains. Appendices add a linear-encoder ablation, a Transformer-component study, and full hyperparameter listings; code is attached anonymously, and AI-assisted writing is disclosed.

Critical strengths
  1. Equal 100-trial Bayesian tuning budgets per arm (50 for the 486k-sample UKB-Union), 5-fold cross-validation with mutually exclusive sample identifiers × 3 seeds, mean ± SE with a non-overlap bolding convention, a unified framework, and gradient clipping chosen explicitly to "distinguish Jacobian-induced optimization difficulties from instabilities caused by large gradients." Combined with breadth (4 objectives × 5 datasets, up to 486k samples with missing modalities) and disclosed code, the central empirical claim — plain encoders degenerate during contrastive training and GPEs broadly help retrieval — is established more convincingly than is typical in this subfield. This is the main reason I lean positive.
  2. The paper triangulates the mechanism with three distinct manipulations — a direct intervention (the JVP Jacobian regularizer, Figure 4), a dose–response manipulation (the LeakyReLU negative-slope sweep, Figure 5), and a capacity probe (linear encoders, Table 5) — all consistent with the proposed mechanism. The honest disclaimer about what the regularizer estimates, and the in-text disclosure of exceptions ("Symile on the UKB constitutes a notable exception"; "on MIMIC-IV, Gram represents an outlier and does not benefit from GPEs"), further increase trust in the reported results.
Critical weaknesses
  1. There are no theorems; the analysis consists of textbook inequalities whose key premise is never verified and is partially circular, and whose analyzed architecture (single outer residual) does not match the implemented per-layer residual MLPs; nothing formally links κ(J) to contrastive learning outcomes. The causal claim on real data rests on a single synthetic, single-objective intervention using a regularizer that (by the authors' own note) does not target the extremal singular values it is meant to condition, with no generic-regularization control. Since the paper's framing leans heavily on "theoretically and empirically connecting" conditioning to contrastive learning, this gap is material to the contribution as advertised.
  2. GPE bundles two components that are never isolated in the headline tables (with α = 0 inside the GPE search space and selected values unreported); there is no parameter-matched plain-MLP control; the two self-identified closest related works and classical conditioning baselines are absent; and the MIMIC-Symile non-GPE baseline is a skip-connection-free ResNet, a known-bad configuration that inflates that dataset's gains. Meanwhile, the practically primary metric by the paper's own framing (linear probes, "often more relevant than retrieval in life science applications") moves only marginally and sometimes negatively. These gaps do not overturn the qualitative conclusion, but they leave the magnitude of the GPE effect and its superiority over simpler remedies unestablished.
Questions for authors
  • Why are Gated Symile and PiCME — described in Section 2 as the closest related work — not compared experimentally? How do GPEs compare against classical conditioning interventions (orthogonal/Δ-orthogonal initialization, LayerNorm, spectral normalization, ReZero)?
  • The contribution bullet claims GPEs "consistently improve retrieval and downstream linear probes," yet Gram on MIMIC-IV degrades on both metrics and several probe cells are negative. Would you temper the abstract's "boosts retrieval and linear probe performance across multiple contrastive objectives"?
1 comment

We thank the reviewer again for their time and feedback. Overall, the new results support our central findings.

We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.