acceptodds
Under review as a conference paper at ICLR 2027

Semantic Correspondence is a Dense Prediction Task: it Should Be Evaluated as One

Abstract

Semantic correspondence establishes matches between semantically equivalent locations on different object instances. The desired output is a correspondence field that can be queried anywhere on an object, yet the task remains evaluated at sparse annotated landmarks. Current metrics (e.g., PCK) leave predictions between those landmarks unmeasured, leaving the spatial generality of the predicted correspondence field unassessed. We introduce 2D-3D Correspondence Consistency (C3D), an evaluation protocol that probes a fixed correspondence map at independently sampled foreground locations. C3D lifts sampled queries and their predicted matches into 3D via monocular point maps and measures how much the correspondence distorts normalized intra-object distances, either among the samples themselves (annotation-free) or with respect to annotated anchors (annotation-assisted). We evaluate each method’s fixed correspondence map to probe its representation quality, without post-processing or test-time optimization. Computed on SPair-71k, C3D predicts how methods rank on the unseen keypoints of SPair-U better than SPair-71k PCK does, showing that it is a good generalization indicator. Finally, we turn the annotation-assisted structural residual into supervision: we train K2D using pseudo-correspondences obtained by fused semantic–structural transport against ground-truth anchors. This dense supervision improves transfer to unseen keypoint definitions, providing a concrete setting in which dense supervision and dense evaluation can be studied together.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semantic correspondences, dense correspondences, foundation model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MvXZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper exposes the limitations in existing semantic correspondence metrics that models can receive perfect performance on sparse keypoints but behave differently everywhere else. Considering the missing evaluation of denser correspondence fields, 2D-3D Correspondence Consistency (C3D) is proposed to measure the structural consistency of dense correspondence fields, by lifting foreground queries to 3D pointmaps. To further investigate the usefulness of dense correspondence field, the authors build dense pseudo ground truths and train a dense-aware correspondence model K2D. Extensive experiments on multiple benchmarks demonstrate that C3D can better predict model rankings on unseen landmarks, and the proposed dense supervision improves correspondence performance.

Critical strengths
  • The evaluation problem is clearly motivated and identifies a real limitation within existing metrics.
  • The paper comprehensively evaluates across eleven benchmarks of both PCK and the proposed C3D.
  • K3D consistently outperforms previous state of the art.
Critical weaknesses
  • The extent of how geometric consistency reflects semantic correctness is insufficiently evaluated. Geometric consistency can be affected by different shapes or articulations. Though local weighting can partially reduce the effects, the unweighted annotation-assisted distance brings the sensitivity to global shape and articulation.
  • The robustness to the accuracy of geometric prior is not evaluated. Both pseudo label generation and C3D evaluation depend on VGGT generated geometry, but how geometry errors affect them are not analyzed.
Questions for authors
  • In pseudo ground truth generation, the cost matrices of semantic and geometric distances are blended with the weights 0.7 and 0.3, respectively. Could the authors show the magnitude analysis of these two distances after weighting and how each term affects the selected correspondences? An ablation of using different weights can provide more transparency. This will also provide more insights on how geometric consistency can influence the model during training.
  • Geometric foundation models like VGGT can introduce artifacts to either or both of the images during evaluation. Have the authors checked the quality of generated single-frame geometry on difficult scenarios such as substantial occlusion or low-light settings where applicable? Would substituting VGGT with an alternative model such as VGGT-Omega largely influence the results or rankings?
  • One related work [1] that also exploits geometric prior for correspondence is missing in discussion.

[1] Qian, Qiyang, et al. "Bridging viewpoint gaps: Geometric reasoning boosts semantic correspondence." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Reviewer_U4yKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to revisit the metric adopted for Semantic Correspondence benchmarks, which are typically evaluated by PCK (fraction of correctly matched keypoints at a threshold) only at the annotated location, which are typically sparse. The paper proposes an heuristic-based, metric dubbed C3D that evaluates how much a given correspondence field preserves relative distances between points. Additionally a method to densify supervision by relying on VGGT-predicted point maps is proposed

Critical strengths
  • Motivation. The argument that traditional evaluations of sem. Corr. benchmarks are limited in generality due to the sparse set of annotated landmarks is valid, and continues a trend in recent literature (Jamais-Vu, MARCO), providing a new perspective
  • Method. The proposed method is simple, and allows to scale training to larger data collection without overfitting to the specific annotated landmarks
  • Performance and evaluation. The paper reports extensive evaluation across benchmarks, showing good generalization outside of the training domain.
Critical weaknesses
  • Fairness of comparison. I have some concerns regarding the experimental setup. First, the method utilizes extra supervision from the VGGT point maps. This is ok, but it should be explicitly noted! Some methods use object masks, some use only keypoints. This method uses keypoints, mask, pointmaps. Also, I feel strongly against the ‘ORACLE’ version of the method. Apparently this is trained on the test set. This is a very weird baseline to have, and doesn’t provide any meaningful new information. I also don’t see the point of the ‘FULL’ and ‘DIVERSE’. These are trained on all the benchmarks altogether. Given that the point of the paper is to test generalization to unseen keypoints, and no other prior work extends the training set beyond SPair, having these lines defeats the purpose of the paper and the +10 gains on the additional benchmarks do not provide any insight, merely due to including everything in the train set. It can be useful for practical reason if the authors want to release a model trained on larger data; and this can be mentioned briefly towards the end of the paper, but having these lines in the main table reduces the focus on the only version that is relevant for comparison with other methods, i.e. the ‘SPAIR’ ones. That line is the only one that matters when evaluating generalization.
  • Clarity. In the experimental section the ‘DIVERSE’, ‘FULL’, ‘ORACLE’ variants don’t have a clear explanation and it is mentioned what they are only in scattered paragraph. There should be a headline paragraph in the beginning of Sec.4 saying why we have this and what they are. Also, we don’t know what is the dagger in Tab.1.
  • Adoption of the metric. While interesting and potentially useful, the proposed C3D metric feels exploratory, not ready to be adopted by the community. While I appreciate and welcome the purpose of the metric, establishing a dense correspondence benchmark, the metric is presented in 4 different variants, some of which don’t even take into account the available annotations. While it is a good practice to discuss and ablate the formulation of the metric, in order to encourage adoption from the community there should be a more focused presentation. The 4 values seem very correlated in Tab.1, raising the question of whether it might be better to just pick 1. The version not using annotations for example, while it might be useful to evaluate alignment on unlabeled data sets, feels irrelevant in the scope. Also the discussion does not highlight relevant differences among them. I encourage having a section, perhaps in the supp, validating and ablating the validity of the metric, but then the main evaluation should be streamlined to 1 or 2 values at most.
  • Questions. (1) How does the metric behave under heavy differences in scale in the 2 objects? What about under occlusion? (2) The paper computes the dense pseudo-labels offline with frozen features. It is known that DINOv2 provides only partially smooth correspondence fields. Why not refine the pseudo-labels in a self-distillation framework as done in MARCO? Are there gains left on the table?
Questions for authors

The paper is straightforward, in a good way. It highlights a clear gap in the literature, and proposes a useful metric to evaluate dense correspondence. The method is simple (also in a good way) and puts together the pseudo label mining of Shape-of-You, although simplified by exploiting labels as anchors, and integrates the simple architecture of MARCO. The results are nice and show improved generalization across wide benchmarks. I believe the paper to be useful for the community, and I urge the authors to address my concerns by streamlining the presentation of the C3D metrics, and improving the comparison in the main table as discussed in the weaknesses.

Reviewer_gRNdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces 2D-3D Correspondence Consistency (C3D), a complementary evaluation protocol that probes dense semantic correspondence beyond annotated landmarks using normalized 3D distance preservation. It also proposes K2D, which generates dense pseudo-correspondences through transport with appearance and anchor-based structural costs. C3D better predicts method rankings on unseen SPair-U landmarks than SPair-71k PCK, while K2D improves generalization beyond training landmarks.

Critical strengths
  1. Useful evaluation beyond sparse landmarks. C3D probes foreground locations without dense correspondence annotations. Across eight external baselines, C3D-Free better predicts SPair-U rankings than SPair-71k PCK does.
  2. Dense supervision improves unseen-landmark generalization. K2D-SPAIR exceeds MARCO on SPair-U with SPair-only training. A fixed-model ablation also favors its labels over keypoint-only and MARCO-style dense supervision.
Critical weaknesses
  1. Reliability across geometric conditions remains unclear. Monocular reconstruction errors and genuine shape differences can affect C3D independently of correspondence accuracy. Existing setting and corruption sweeps do not test whether C3D remains semantically reliable under these conditions.
  2. Downstream gains do not isolate the structural term. Table A2 isolates geometry at the label-quality level, but Table A3 compares MARCO-style mutual nearest neighbours with structural transport. This also changes the matching algorithm, so the downstream gain cannot be attributed to the structural term alone.
Questions for authors
  1. Please report category-wise agreement between method rankings from C3D on SPair-71k and PCK on SPair-U, using their shared categories.
  2. Please compare downstream training with appearance-only and fused unbalanced transport labels (fusion weights 0 and 0.3), matching training pairs, usable pseudo-label counts, architecture, and training settings. Report SPair-U PCK.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.