On the Limitations of Identifiability Theory for Contrastive Learning

Recent identifiability results for contrastive learning connect successful representation learning to recovery of latent factors underlying an invertible data-generating process. We study a limitation of using such recovery as an explanation of downstream performance: latent recoverability, task sufficiency, and stability are distinct properties. Focusing on classification, we first introduce class-conditional structure into standard latent-variable models and show how concentration simultaneously reduces class ambiguity and shrinks the scale of within-class variation. Thus, even exact access to the latent state does not by itself guarantee discriminability, while resolving individual instances at fixed effective precision becomes harder as within-class variation contracts. We then derive pairwise lower bounds linking recovery error, classification error, representation geometry, and probe Lipschitz constants. These bounds expose a recovery–prediction–stability trilemma: when a learned representation compresses latent distinctions that a downstream probe must preserve, accuracy can be maintained only at the cost of increased sensitivity. We further characterize a directional mechanism by which variations in distinct latent factors can interfere through a shared representation. Controlled simulations and experiments on four disentanglement benchmarks exhibit the predicted accuracy–recovery tensions and coupled degradation under perturbations. Our results suggest that identifiability guarantees should be complemented by explicit assumptions on task structure, representation geometry, and stability when they are used to explain downstream performance.