acceptodds
Under review as a conference paper at ICLR 2027

Pairing Graph Identifiability of Multimodal Representations

Abstract

ImageBind style models do not observe a joint tuple of all modalities. They train on pairwise collections sampled independently across pairs, for instance an image and text dataset together with an image and audio dataset. Existing identifiability theorems assume the opposite observation, in which every view of a sample is present at once. They therefore do not name the latent factors that independent pairs identify. In this paper, we prove that a latent block is consistently block identifiable from pairwise datasets if and only if the modalities that carry it induce a connected subgraph of the pairing graph. Modalities are its vertices and observed pairs are its edges. On a star, this criterion identifies the factors that both spokes share with the hub, and that intersection is the content of emergent alignment. It does not identify sharing that lives only on the unobserved spoke pair. After each edge has estimated a shared representation, residual encoder disagreement equals the effective resistance of the pairing network. Among trees, a star minimises the worst pair error when the hub carries shared signal. Once that signal vanishes, spoke to spoke error returns to the prior. The identity is tight for linear Gaussian estimators, and it remains an upper bound with a nonnegative remainder for nonlinear mixing functions. Unpaired optimal transport mixes private coordinates, whereas paired barycentres do not. A content encoder of shared factor dimension leaks no private information given that factor, while an invertible embedding of a view leaks its private coordinates. Projection heads trained by edgewise InfoNCE on frozen CLIP and Qwen3-VL features recover the ranking of star, path, and disconnected graphs, and a nonlinear mixing model recovers the spoke factor only after the missing edge is added.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal representation learning, Contrastive learning, Identifiability, Pairing graph

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DJFBOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal representation learning when the training data consist of independently collected pairwise datasets rather than jointly observed multimodal tuples. The authors model the available pairings as a graph whose vertices are modalities and whose edges are observed paired datasets. They derive a connectivity-based criterion for consistent identification of shared latent factors, and further relate finite-sample alignment error to effective resistance and the algebraic connectivity of the pairing graph. The analysis suggests that star-like pairing structures can be favorable under a fixed edge budget when the hub carries sufficient shared signal. The paper also discusses private factors and unpaired transport, and provides experiments on synthetic settings and projection heads trained on frozen CLIP and Qwen3-VL features.

Critical strengths

S1. Clear and relevant problem formulation. The paper identifies a meaningful gap between existing multimodal identifiability results based on jointly observed views and ImageBind-style training based on independently collected pairwise datasets.

S2. Intuitive pairing-graph perspective. Modeling modalities as vertices and observed paired datasets as edges provides a clean way to reason about when information can propagate across unobserved modality pairs, and the connection to effective resistance gives the framework a useful data-design interpretation.

Critical weaknesses

W1. The empirical validation is somewhat removed from the motivating multimodal setting. The paper is motivated by ImageBind-style training over genuinely heterogeneous modalities, but the main real-feature experiments use frozen CLIP/Qwen3-VL representations, with one image view and several templated text views treated as different modalities, and only projection heads are trained. This makes it difficult to assess how directly the conclusions transfer to actual image-text-audio style training.

W2. The practical design implications are only lightly validated. The paper derives recommendations about pairing topology, including the advantage of star-like structures under certain conditions, but the real-feature experiments are relatively small and the star-over-path trend is not uniformly reflected across the Qwen3-VL results. A more realistic heterogeneous-modality experiment would make the practical significance of these theoretical results more convincing.

Questions for authors

Q1. Have the main pairing-graph predictions been tested on a genuinely heterogeneous multimodal setting, such as image-text-audio data, rather than multiple text views derived from the same image set? Such an experiment would help clarify how well the theory transfers to the ImageBind-style setting that motivates the paper.

Q2. How sensitive are the real-feature results to the relatively small dataset and evaluation size? In particular, would the relative behavior of star and path structures remain similar with substantially more paired data and held-out evaluation tuples?

Reviewer_XrzpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies multimodal representation learning from independently collected pairwise datasets, where complete multimodal tuples are unavailable during training. It models the observed modality pairs as a graph and proposes connectivity conditions for consistent identification of shared latent factors. It further relates finite-sample alignment errors to effective resistance and discusses hub selection, private factors, and paired versus unpaired learning. Experiments on synthetic data and projection heads trained on frozen CLIP and Qwen3-VL features examine how pairing structure affects alignment and retrieval.

Critical strengths

The distinction between independently collected pairs and jointly observed views is practically meaningful. The graph formulation provides an intuitive account of how pairing structure can support or limit consistent alignment across modalities.

Critical weaknesses
  1. The scope of Theorem 2 needs clarification. Restricting the model to a subset of modalities does not make other shared blocks private; for example, a globally shared factor remains shared within that subset. Consequently, connectivity and matching the encoder dimension to the target block do not, by themselves, establish recovery of that specific block. The paper should distinguish identification of the full common content from identification of an individual subset-specific block, and state the additional conditions needed for the latter.
  2. The experimental description lacks clarity, and the evaluation is limited in scale. The main text does not clarify the distinction between "spoke to spoke" and "path ends" under the star configuration in Figure 2, including which modality pairs each group evaluates and how their scores are aggregated. This makes the reported differences difficult to interpret. Moreover, the main frozen-feature experiments use only 100 images, providing limited evidence for the robustness and broader applicability of the findings.
Questions for authors
  1. What additional conditions ensure that connectivity identifies the intended latent block when multiple shared blocks are available to the same modalities?
  2. Could the authors clarify the distinction between "path ends" and "spoke to spoke" under the star configuration in Figure 2? Specifically, which modality pairs does each group evaluate, and how are their scores aggregated? This distinction is not clarified in the main text.
Reviewer_FCbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides identifiability results for contrastive learning in a multimodal setting in which independent pairs of modalities are observed during training (e.g., we learn image-text-audio embeddings by only observe image-text or image-audio observations pairs). The setup considered is formalized as a graph where vertices are modalities and edges connect modalities that are observed together (pairs).

The findings are 4 fold:

  • theorem 1: latent factors shared by all vertices in a connected component of the graph are block-identifiable.
  • theorem 2: latent factors shared by a subset of modalities A are block-identified if and only if the modalities in A are observed together.
  • theorem 3: provides a upper bound of the error on the true representation when training the encoder with a finite sample size. This upper bound is inversely proportional to the graph connectivity. A star graph minimizes that error.
  • theorem 6: no privacy guarantee from alignment, meaning that the latents that are specific to each modality are leaked into the representation if this representation is invertible.

On the experimental side, authors train two-layer projection heads on frozen CLIP and Qwen3-VL features, plus synthetic experiments with known latents. Connected graphs reach about 0.65 to 0.77 recall, while disjoint and shuffled graphs sit at chance level. In the synthetic worlds, a star graph recovers the shared factors but not the spoke-only factor until the missing edge is added. The evaluation is small, the metrics are partly undefined, and most of the theory checks for section 5.3 simulate the model the theorems already assume.

Critical strengths
  • Scope of the research: Identifiability results help the community formalize what contrastive learning, a cornerstone of representation learning, actually learns in a well-defined setup. The setup considered by the authors is relevant because used in practice for multi-modal training for which we often don't have paired observations across all modalities we wish to learn from.

  • Some of the results are interesting: Theorem 1 and 2 are interesting results which clarify the identifiability of the latents shared across vertices of the graph and help formalize the multi-modal training.

Critical weaknesses
  • Form:

    • Lack of intuition conveyed throughout the paper: I found most of sections 1, 2, 3, 4.1, 4.2 to be readable and for the intuition to be relatively well conveyed behind each contribution/theorem. Starting from section 4.3, I find myself to struggle a lot with understanding the premises and intuition behind each theorem/corollary. I think the manuscript would greatly benefit from a rephrasing of each theorem/corollary.
    • Missing information and formalism: I find the readability to be impeded by missing information for example, in theorem 3, what is rr, λ\lambda, ReffR_eff, sigmasigma? For the empirical section, in section 5.1, it is unclear what the modalities are in this setup. Also what is a path graph and what does the disjoint graph look like? which modality is used as the hub/spoke? what is the InfoNCE residual, what are the Hungarian scores? This information is central enough to be added to the main text.
  • Content:

    • Theoretical contributions are incremental: theorems 1 and 2 are limited extensions of Daunhawer et al. and Theorem 6 seems to be stating something trivial (see my question below).

    • Empirical validation is narrow :

      • In addition to missing a lot of information, $5 seems to have a strong cofounder (authors use frozen CLIP and Qwen embeddings), results are also noisy (100 samples, recall for shared information is far from 1).
      • I think the paper is missing some more thorough synthetic experiments to confirm the theoretical result on a controllable setup (on which more ablations on the graph structure and data generating process can be conducted) rather than jump directly to more realistic setups ($5.1).
      • For section 5.3, no plots, no tables, results are listed in the text with very limited effort put into helping the reader understand the empirical setup tested and conclusions drawn from numerical results which seems to be simulations using the setup used by the theory.
    • Identifiability results come with limitations, they make assumptions about the data generating process typically and the sample size. While the sample size is discussed, I could not find any discussion about the limitations coming from the DGP or from the empirical setup which was tested.

Questions for authors

Could the authors clarify the contribution in Theorem 6, it sounds to me like the authors are stating something relatively trivial which is that an invertible embedding preserves the mutual information and a population content encoder of dimension dsd_s satisfies I(zp,m∣zs)=0I(z_{p,m}|z_s)=0 (given Theorem 1&2). For theorems 5, 7-8, could you share the intuition behind those contributions?

Remark: there is a problem with the Gabriel & Cuturi reference at line 90, 128.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.