When the Teacher Lags: Contrastive Pre-training under Distribution Drift

Contrastive pre-training typically assumes that training data follow a stable distribution. When the distribution changes, however, a momentum teacher may retain information from earlier data and provide targets that are poorly aligned with the current samples. We study how this lag affects representation learning and use a structural causal model to characterize the resulting source of bias. Guided by this analysis, we propose Resilient Contrastive Pre-training (RCP), which introduces an intervention-inspired contrastive objective to reduce the influence of drift-induced bias during training. RCP integrates a query–key–value module into a momentum-based student–teacher framework and learns from samples within a drift adaptation window. We evaluate the resulting representations through long-tailed classification, domain-shifted classification, and out-of-distribution detection. Across these evaluations, RCP improves upon standard contrastive pre-training baselines, with particularly consistent gains for underrepresented classes. These findings suggest that accounting for drift during pre-training can improve the robustness and transferability of learned visual representations. Codes are available at https://anonymous.4open.science/r/ResilientCL/.