Preserving Pretrained Intra-Class Structure in Fine-Grained Visual Adaptation

Vision foundation models often organize samples from the same category into distinct structures in feature space, reflecting variations in viewpoint, pose, appearance, and local details. Modeling such structure is crucial for fine-grained recognition, where subtle inter-class differences coexist with large intra-class variation. Yet downstream fine-tuning is typically driven by category-level supervision and does not explicitly account for this pretrained intra-class organization. We further observe that such structure is category-dependent: some categories form a single compact mode, while others contain several distinct modes. This motivates us to exploit category-specific structure already present in pretrained features. We propose Adaptive Proxy Modeling (APM), which models pretrained intra-class structure with a category-specific set of proxies. For each category, APM estimates the number of visual modes from the spectral structure of its DINOv2 feature graph and constructs corresponding proxies through expectation-maximization. During fine-tuning, these proxies are dynamically updated with the evolving representation and provide structural guidance for downstream learning. Unlike conventional proxy-based methods using a fixed number of freely learned proxies, APM adapts both proxy number and representation to each category without introducing additional trainable parameters. Across fine-grained classification, person re-identification, and vehicle re-identification, APM consistently improves downstream performance without introducing additional trainable parameters. Beyond accuracy, APM also better preserves pretrained representations after fine-tuning, suggesting that effective adaptation can benefit from retaining the category-specific structure already learned by foundation models.