Beyond Source-Domain Accuracy: Controlled Study on Adaptation Granularity for Imaging Sonar Representation Transfer

Transferring pre-trained vision foundation models to imaging sonar tasks has become a prevalent solution for underwater perception with limited data. Existing studies generally follow the empirical assumption that higher source-domain accuracy yields better cross-domain transfer performance, and thus focus primarily on improving source-domain capabilities. However, this common belief lacks systematic controlled validation, and how adaptation granularity governs sonar transfer performance remains unclear. To address this gap, this paper constructs a controlled experimental framework based on a 576M-parameter visual encoder. We systematically evaluate six feature adaptation schemes covering token-level scalar adjustment, channel-level transformation, and global affine mapping, from both source-domain classification accuracy and sonar cross-domain transfer perspectives. Experimental results show that fine-grained adaptation with stronger representation capacity improves source-domain accuracy yet degrades sonar transfer performance in the focal comparison. In contrast, coarse-grained scalar adaptation achieves a transfer accuracy of 96.5%, outperforming source-domain fine-tuning (95.1%) with stable and reproducible gains. A matrix of three backbones and three source domains, all evaluated on the same sonar target, further shows that the preferred granularity changes with the setting. Ablation studies verify that such benefits arise from content-dependent adaptation rather than simple feature aggregation with frozen encoders. Under identical training budgets, comprehensive comparisons with sonar masked autoencoders, LoRA, and drift-anchored fine-tuning further clarify the applicability of our findings. This paper demonstrates that adaptation granularity is a long-overlooked critical dimension in sonar cross-modal transfer. Relying solely on source-domain accuracy leads to misleading evaluation of adaptation strategies. Our controlled experiments clearly characterize the potential and limitations of vision-to-sonar transfer, providing practical guidance for underwater cross-modal perception adaptation design.