- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
3D-SC reconstructs an instance mesh per image with SAM3D, renders PartField descriptors into the image plane to propose matches alongside DINOv2 and Stable Diffusion features, and verifies candidates by bidirectional geodesic consistency on the meshes to select training pseudo-labels. The 3D pipeline runs only offline, so inference uses adapted DINOv2 features alone and training needs no pose or keypoint annotations. It reaches 74.4 and 74.8 [email protected] on SPair-71k and its Geo-Aware subset, exceeding pose-supervised DIY-SC by 2.8 and 7.3 points, and transfers best to unseen keypoints on SPair-U.
Critical strengths
- Adding rendered PartField descriptors to the proposal features yields gains on the symmetry-focused SPair-Geo-Aware subset. This is consistent with PartField separating visually similar but geometrically distinct parts.
- Fusing the three complementary streams, DINOv2 semantics, Stable Diffusion appearance, and rendered PartField part descriptors, more than doubles the pseudo-label coverage of the geodesic filter.
Critical weaknesses
- The geodesic verifier judges each candidate by asking PartField for the cross-mesh counterpart, yet PartField is also one of the three proposal features, so the filter can only reject candidates on which the 2D features and the geometric descriptor disagree, and passes errors both share, such as a left-right swap when PartField gives symmetric parts nearly identical descriptors. In other words, the verification step cannot catch the left-right and repeated-part swaps it was designed to remove whenever PartField itself is blind to them.
Questions for authors
-
Approaches that exploit 3D assets for correspondence matching are not new, e.g. [1] . Apart from employing SAM3D to generate the 3D assets, what constitutes the most significant methodological contribution of this paper?
-
Does the geodesic-distance verification criterion remain reliable when the two instances belong to the same category but carry different part configurations, such as matching an armchair against an armless chair?
[1] You, Yang, et al. "Understanding pixel-level 2d image semantics with 3d keypoint knowledge engine." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.9 (2021): 5780-5795.
Thanks for the response and additional experiments. Most of my concerns are resolved, so I decided to maintain my score.