Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Foundation features from self-supervised vision models and text-to-image diffusion models transfer well to semantic correspondence, but often confuse symmetric sides and repeated object parts. We introduce 3D-SC, a post-training framework that uses instance-specific 3D geometry to improve both correspondence proposals and their verification. Starting from an image and its category label, we reconstruct an object mesh with SAM3D, refine its image alignment, and canonicalize its orientation. We then render PartField descriptors into the image plane to complement 2D features when proposing matches. A bidirectional geodesic consistency check on the reconstructed meshes filters these proposals into pseudo-labels for training lightweight adapters. The resulting image-based matcher requires no 3D reconstruction at inference and no manual pose or keypoint labels for training. On SPair-71k and its Geo-Aware subset, 3D-SC improves [email protected] by 2.8 and 7.3 points over the strongest weakly supervised method, which additionally relies on manual pose labels. On SPair-U, which evaluates generalization to unseen keypoints, our method outperforms keypoint-supervised methods. These results support using 3D foundation models as geometric teachers for semantic correspondence, with the largest gains on geometrically ambiguous matches.