SigMI: Sigmoid Contrastive Models Estimate Pointwise Mutual Information

Histogram-based approximations of mutual information (MI) have long been used as multimodal similarity measures for medical image registration. Representation learning emerged as an alternative approach, but its connection to mutual information is lost when applied to image registration. We connect both lines of work by showing that sigmoid contrastive models like SigLIP estimate pointwise mutual information (PMI): being trained to distinguish samples from a joint distribution from deranged pairs approximating the product of its marginals, their optimal logits recover PMI up to a correction for the sampling class prior. These sigmoidal mutual information estimators (SigMIs) can estimate calibrated PMI, independent of the training prior. Their estimation error can be described in terms of excess classification risk, and they provide information-theoretic interpretations of SigLIP's temperature and bias parameters. Applying our findings to medical image registration, we propose fully convolutional SigMIs that learn dense image representations whose dot products estimate local PMI. We train our models on domain-randomized synthetic images generated from just five brain segmentation maps. Across three registration algorithms and four datasets, SigMIs outperform other similarities in 8 of 12 settings.