acceptodds
Under review as a conference paper at ICLR 2027

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Abstract

Foundation features from self-supervised vision models and text-to-image diffusion models transfer well to semantic correspondence, but often confuse symmetric sides and repeated object parts. We introduce 3D-SC, a post-training framework that uses instance-specific 3D geometry to improve both correspondence proposals and their verification. Starting from an image and its category label, we reconstruct an object mesh with SAM3D, refine its image alignment, and canonicalize its orientation. We then render PartField descriptors into the image plane to complement 2D features when proposing matches. A bidirectional geodesic consistency check on the reconstructed meshes filters these proposals into pseudo-labels for training lightweight adapters. The resulting image-based matcher requires no 3D reconstruction at inference and no manual pose or keypoint labels for training. On SPair-71k and its Geo-Aware subset, 3D-SC improves [email protected] by 2.8 and 7.3 points over the strongest weakly supervised method, which additionally relies on manual pose labels. On SPair-U, which evaluates generalization to unseen keypoints, our method outperforms keypoint-supervised methods. These results support using 3D foundation models as geometric teachers for semantic correspondence, with the largest gains on geometrically ambiguous matches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semantic Correspondence, 3D, Geodesic, SAM3D

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vgc9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

3D-SC reconstructs an instance mesh per image with SAM3D, renders PartField descriptors into the image plane to propose matches alongside DINOv2 and Stable Diffusion features, and verifies candidates by bidirectional geodesic consistency on the meshes to select training pseudo-labels. The 3D pipeline runs only offline, so inference uses adapted DINOv2 features alone and training needs no pose or keypoint annotations. It reaches 74.4 and 74.8 [email protected] on SPair-71k and its Geo-Aware subset, exceeding pose-supervised DIY-SC by 2.8 and 7.3 points, and transfers best to unseen keypoints on SPair-U.

Critical strengths
  • Adding rendered PartField descriptors to the proposal features yields gains on the symmetry-focused SPair-Geo-Aware subset. This is consistent with PartField separating visually similar but geometrically distinct parts.
  • Fusing the three complementary streams, DINOv2 semantics, Stable Diffusion appearance, and rendered PartField part descriptors, more than doubles the pseudo-label coverage of the geodesic filter.
Critical weaknesses
  • The geodesic verifier judges each candidate by asking PartField for the cross-mesh counterpart, yet PartField is also one of the three proposal features, so the filter can only reject candidates on which the 2D features and the geometric descriptor disagree, and passes errors both share, such as a left-right swap when PartField gives symmetric parts nearly identical descriptors. In other words, the verification step cannot catch the left-right and repeated-part swaps it was designed to remove whenever PartField itself is blind to them.
Questions for authors
  • Approaches that exploit 3D assets for correspondence matching are not new, e.g. [1] . Apart from employing SAM3D to generate the 3D assets, what constitutes the most significant methodological contribution of this paper?

  • Does the geodesic-distance verification criterion remain reliable when the two instances belong to the same category but carry different part configurations, such as matching an armchair against an armless chair?

[1] You, Yang, et al. "Understanding pixel-level 2d image semantics with 3d keypoint knowledge engine." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.9 (2021): 5780-5795.

Reviewer_NSWKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes 3D-SC, a framework that leverages reconstructed and canonicalized 3D meshes to generate pseudo-labels for semantic correspondence learning. Rendered PartField descriptors complement 2D foundation features, while bidirectional geodesic verification filters unreliable matches. The resulting pseudo-labels supervise an image-based matcher that requires no 3D reconstruction at inference. Experiments cover SPair-71k, Geo-Aware, SPair-U, AP-10K, and SOCO.

Critical strengths

The paper addresses an important challenge in semantic correspondence by incorporating geometric priors into pseudo-label generation. The improvements on Geo-Aware and SPair-U are particularly relevant to the paper's motivation. Moreover, the final matcher does not require 3D reconstruction at inference. The AdaptFormer ablation and comparison with DIY-SC using estimated orientations also strengthen the evaluation.

Critical weaknesses

1. Insufficient evidence for the reliability of dense pseudo-labels.

Table 2 evaluates the false discovery rate (FDR) only at annotated keypoints, whereas the retention statistics refer to dense pseudo-labels. Therefore, the reported 1.78% FDR cannot be directly generalized to all retained dense correspondences (median 234 per pair). Furthermore, different filtering strategies retain substantially different numbers of correspondences, making direct FDR comparisons potentially misleading. An evaluation at matched retention rates would be more convincing. Consequently, the current evidence does not fully support the claim that reconstruction failures primarily produce missing rather than incorrect supervision.

2. Unclear contribution of geodesic distance and inconsistencies in the ablation study.

The geodesic verification relies on PartField-based cross-mesh correspondences, while PartField is also used to generate candidate matches. This introduces a potential dependency between proposal generation and verification, allowing consistent but incorrect matches to survive. Although Table 2 compares against PartField feature similarity, the retention rates differ substantially. A controlled comparison using Euclidean distance and geodesic distance under the same proposals and matched retention would better isolate the contribution of surface geometry.

Additionally, Table 3 appears to report identical configurations with different PCK values (71.6 and 72.4). This discrepancy should be clarified, as it makes the contribution of individual components difficult to interpret.

3. Limited robustness to reconstruction failures and challenging object geometries.

For image pairs involving failed reconstructions, the median number of retained pseudo-labels drops to only 2, compared with 234 for correctly reconstructed pairs. Although filtering may effectively reject unreliable correspondences, it also removes most of the supervision for these difficult examples. This is particularly concerning for heavily occluded, articulated, or geometrically complex objects. The paper would benefit from a more detailed analysis of downstream correspondence performance as a function of reconstruction quality, particularly for challenging animal categories.

Questions for authors
  1. Does the advantage of geodesic filtering persist when different methods are compared at matched pseudo-label retention rates? How reliable are the dense pseudo-labels beyond the annotated keypoints used to compute FDR?

  2. How does geodesic filtering compare with Euclidean-distance filtering using identical PartField correspondences? Also, what distinguishes the apparently identical configurations reporting 71.6 and 72.4 PCK in Table 3?

  3. How does reconstruction quality affect the final matcher's performance on heavily occluded or articulated objects, where most pseudo-labels may be discarded?

Reviewer_faLzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents 3D-SC, a framework for learning semantic correspondence by combining 2D and 3D foundation models. The proposed method uses instance-specific 3D reconstruction and geometric features to generate correspondence proposals, followed by geodesic verification to filter pseudo-labels for training a DINOv2-based correspondence model. Experiments across multiple benchmarks demonstrate improved performance over existing weakly supervised methods, particularly on geometrically ambiguous correspondences.

Critical strengths
  1. Clear problem motivation. Semantic correspondence benefits from both 2D visual features and 3D geometric structure. The paper addresses the limitations of purely 2D approaches by combining DINOv2 and Stable Diffusion features with 3D PartField descriptors and geometric verification. The motivation is clear, and the proposed approach is well aligned with the identified challenges.
  2. Clear and convincing ablation studies. The ablation experiments demonstrate the contributions of the proposed components, including PartField features, geodesic filtering, and feature adaptation. The results support the effectiveness of the proposed pipeline in improving semantic correspondence performance.
Critical weaknesses
  1. Dependence on pretrained foundation models and limited robustness analysis. The proposed method combines several existing pretrained models, so the quality of the generated pseudo-labels largely depends on the accuracy and failure modes of these individual components. For example, incorrect object reconstructions may negatively affect pseudo-label generation and downstream training. Although Section 4.3 provides some analysis of reconstruction failures and shows that they primarily reduce pseudo-label coverage, the paper lacks a more comprehensive evaluation of the pipeline's robustness under challenging conditions and the impact of reconstruction quality on the final trained model.

  2. Minor: Missing downstream ablation for different filtering mechanisms. Although Table 2 compares different filtering mechanisms in terms of pseudo-label quality, there is no corresponding experiment evaluating the performance of the final model trained using these different filtering methods. Such a comparison, particularly between PartField feature-similarity filtering and the proposed geodesic filtering, would help clarify the contribution of the geodesic verification mechanism.

Questions for authors
  1. How robust is the proposed pipeline under varying conditions, such as severe articulation, occlusion, or inaccurate 3D reconstruction? What are the main failure modes or significant performance degradations observed? In particular, how does the quality of the pretrained 3D models affect the final correspondence model?

  2. Could the authors provide an ablation comparing the downstream performance of models trained with pseudo-labels generated by different filtering mechanisms, particularly PartField feature-similarity filtering versus geodesic filtering?

1 comment

Thanks for the response and additional experiments. Most of my concerns are resolved, so I decided to maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion period summary

We thank all reviewers again for their constructive feedback and for engaging with our responses. We are glad that Reviewers faLz and NSWK found that the responses and additional experiments addressed most of their concerns. Neither follow-up raises a specific remaining issue, so to our understanding no point is left open. Should any concern remain, also on the side of Reviewer vgc9, we would be happy to address it in the remaining discussion period.

We have uploaded a revised manuscript that incorporates all experiments and clarifications from the rebuttal. Changes are highlighted in purple; line numbers refer to the revised PDF.

Main paper

  1. Downstream comparison of filters (faLz W2/Q2; NSWK W1/Q1, W2/Q2). Tab. 2 (p. 9) now reports the [email protected] of matchers trained on the labels of each filter, and has two new rows: the spherical filter at matched retention (threshold 0.067) and the Euclidean-distance variant of our filter. A new paragraph discusses these results (L452–464), and the introduction summarises the finding (L85–87).
  2. Robustness to pose and reconstruction errors (faLz W1/Q1; NSWK W3/Q3). Sec. 4.3 now includes the ground-truth-pose and controlled pose-noise experiments (L481–485) and the reconstruction-quality quartile analysis (L495–498).
  3. Occlusion and truncation (faLz Q1; NSWK Q3). New Tab. 4 (L486–493) and a new paragraph (L499–504).
  4. Shared use of PartField in proposal and verification (vgc9 W1; NSWK W2/Q2a). New paragraph with the left–right swap analysis (L505–511), referenced from the method section (L310–311).
  5. Differing part configurations (vgc9 Q2). Sec. 3.2 now discusses pairs in which a part is missing from one instance, e.g. armchair vs. armless chair (L313–315).
  6. Relation to You et al. (2022) (vgc9 Q1). Added to the related work (L124–126).
  7. Typo in Tab. 3 (NSWK W2). We removed the erroneous checkmark in the "cap" column of the 71.6 row, which now differs from the 72.4 row in this component.
  8. Limitations (NSWK W1, W3; vgc9 W1). Sec. 5 now states that SPair-71k has no annotation that isolates articulation, that errors shared by PartField and the 2D features can survive verification, and that label quality can only be measured confidently at annotated keypoints (L518–522).

Supplementary material

  1. Sec. A, Tab. A1 (L843–858). The reconstruction-quality analysis now also reports downstream [email protected] after training on each silhouette-IoU quartile separately.
  2. New Sec. D, "Robustness and error analyses" (L1050–1168):
    • D.1 (L1056–1079, L1128–1129): ground-truth poses, controlled pose noise (new Tab. D1), reconstruction quality, and occlusion/truncation.
    • D.2 (L1131–1139): protocol and results of the left–right swap analysis.
    • D.3 (L1141–1151): protocol of the downstream filter comparison, including the matched-retention and Euclidean variants.
    • D.4 (L1153–1168): per-category label FDR and gain over DIY-SC+OriAny (new Tab. D2, Pearson r = −0.78).
  3. The supplement overview and table of contents are updated accordingly (L764, L776).

We thank the reviewers again for their time; their suggestions have made the paper stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.