Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning

Joint-embedding self-supervised learning aligns representations of augmented views while regularizing their distribution to prevent collapse. Most joint-embedding methods estimate latent regularization from the active minibatch, even though its purpose is to shape the broader representation distribution. Batch size therefore affects not only optimization and memory use, but also the computation of the self-supervised objective itself. We propose out-of-batch regularization, which maintains a dataset-level state in CPU memory and uses globally constructed per-instance anchors during minibatch training. An analysis on natural-image and biomedical datasets shows that these anchors approximate the full-population regularization gradient more faithfully than conventional minibatch estimation at small batch sizes. We apply this principle to contrastive and non-contrastive learning across 2D and 3D data. The resulting methods consistently improve over their corresponding in-batch objectives in image classification, minority-class performance under controlled imbalance, instance retrieval, and continued self-supervised pretraining, with the largest gains at small batch sizes. Our results show that the minibatch does not need to define the statistical scope of anti-collapse regularization.