Understanding and Mitigating Catastrophic Inheritance via Discriminative Spectrum Drift

Pre-training on large-scale data has become a standard paradigm for visual transfer learning, but noisy supervision in pre-training data can be inherited by downstream models and degrade their generalization. While prior studies have shown that noisy pre-training affects downstream transfer, its structural effect on downstream representations remains insufficiently understood. In this work, we recast catastrophic inheritance as discriminative spectrum drift, and study how noisy pre-training reshapes class-relevant discriminative information in spectral space. Across multiple commonly used cross-domain visual datasets and controlled noisy pretrained checkpoints, we show that the inherited effect of noisy pre-training is not merely reflected in performance variation, but manifests as a consistent drift of the downstream discriminative spectrum. To characterize this phenomenon, we introduce two continuous descriptors that capture the overall deviation from the clean reference and the average spectral position of discriminative information. Motivated by these findings, we further propose a lightweight spectral adaptation framework for the realistic setting where no clean reference is available and the pretrained backbone remains frozen. Experiments show that our method mitigates discriminative spectrum drift and improves downstream adaptation performance.