Statistical guarantees for multimodal contrastive learning via pointwise mutual information estimation

Multimodal contrastive learning has emerged as a powerful approach for learning representations from paired multimodal data. In this work, we develop a theoretical framework for analyzing the finite-sample behavior of contrastive learning with the InfoNCE objective. Our approach is based on viewing the learned similarity score as an estimator of the pointwise mutual information (PMI) function, which characterizes the population-optimal similarity score up to an additive constant. By exploiting the curvature of the InfoNCE objective together with a localized analysis around the PMI score, we derive fast convergence rates for the PMI estimation error despite the dependence among negative pairs in InfoNCE. When PMI is Hölder smooth, we further show that InfoNCE achieves the minimax-optimal rate up to logarithmic factors for both joint-input neural networks and CLIP-style dual encoders. For downstream learning, we show that accurate recovery of PMI implies approximate sufficiency of the learned representations and yields a bound on the excess risk of zero-shot classification. We provide numerical experiments to support our theoretical results.