What Does Post-Training Preserve in Self-Supervised Representations?

Self-supervised learning produces transferable visual representations, yet it remains unclear which properties are preserved during post-training. We study how full fine-tuning and LoRA reshape self-supervised vision models across network depth and within their final embedding spaces. Our evaluation covers 39 model variants and more than 64,000 images across seven widely used benchmarks: CIFAR-10, CIFAR-100, DTD, Flowers-102, Food-101, OxfordPets, and Tiny-ImageNet, yielding over two million embedding vectors. Our analysis is threefold. First, we use layerwise CKA to trace representational changes throughout the network. Second, we use embedding-level CKA and Gromov-Wasserstein distance to analyze final-embedding alignment and relational geometry. Third, we assess local neighborhood preservation through nearest-neighbor consistency. Our results reveal recurring stability patterns across the tested model configurations and adaptation targets, while alignment and geometric displacement do not always evolve together. In our rank sweep, the highest tested rank produces substantially earlier layerwise divergence than the lowest tested rank, suggesting that rank can affect the depth-wise localization of adaptation. Overall, our findings indicate that different notions of representational preservation do not always evolve together under post-training. These structured forms of specialization are shaped by model family and adaptation capacity, motivating future work on controlling where and how pretrained representations are reorganized. The code used for all our experiments is released in the supplementary material.