acceptodds
Withdrawn from ICLR 2027

One-Shot Relational Adaptation of Vision Foundation Models for 3D Correspondence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision foundation models (VFMs) have demonstrated generalization across diverse 2D vision tasks, yet adapting them for 3D correspondence tasks, such as multi-view matching and pose estimation, remains challenging due to the lack of large-scale 3D correspondence annotations. Existing adaptation strategies typically rely on extensive training data or task-specific supervision, limiting their applicability in data-scarce scenarios. In this work, we propose a one-shot adaptation framework that enhances the 3D correspondence capability of VFMs using only a single overlapping image pair with cross-view correspondences. Our key insight is that a correspondence-annotated image pair provides not only point-level supervision but also rich relational structures among matched points. To exploit this property, we introduce a two-stage adaptation framework. First, an attention-gated convolutional adapter is designed to improve local cross-view feature consistency while introducing minimal modification to the pretrained VFM. Second, a relational adaptation module explicitly models structural relationships among corresponding points and enforces cross-view relational consistency, enabling the adapted representation to preserve relational structures across viewpoints. Experiments on multiple 3D correspondence benchmarks show that our approach generally improves pretrained VFMs and outperforms recent VFM adaptation methods in many settings, while requiring only a single image pair for adaptation. These results highlight the effectiveness of exploiting intrinsic correspondence and relational information for efficient 3D-aware VFM adaptation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Foundation Models, Few-shot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_J1oHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper adapts a frozen VFM for cross-view correspondence using a single overlapping image pair with correspondences derived from dataset depth/pose annotations, via a lightweight two-stage adapter: an attention-gated 3×3 convolution trained with patch-level contrastive loss, followed by a shared linear projection that aligns the two views' within-view pairwise distance matrices under a LogDet regularizer.

At test time the adapter is frozen and applied per-image with the unchanged cosine-NN matching pipeline, and since each run trains on one pair, the authors repeat on 10 sampled pairs and report the mean.

Critical strengths

None.

Critical weaknesses
  1. The "3D correspondence" framing overclaims: no depth, camera parameters or geometric constraint ever enters the model, and the paper's own Table 2 shows an adapter trained on object pairs improves PF-PASCAL cross-instance semantic correspondence, which has no geometry at all.

  2. The closest literature is missing — the training-free VFM/diffusion semantic/geometric correspondence line, such as A Tale of Two Features[1], DIFT[2], Telling Left from Right[3], DIY[4], VGGT Semantic[5], Motcha[6], MASt3R[7] and RoMa[8] is uncited. This reduces the technical reliability of this paper and is the biggest reason that I reject this paper.

  3. A plain 3×3 convolution nearly matches the full method, and since DVT already showed that ViT features suffer from high-frequency artifacts that smoothing alone alleviates — and the paper's own Fig. 4 highlights "fewer artifacts" — the decisive control is missing: an untrained randomly-initialized convolution, or plain Gaussian blur; if those also help, the necessity of both the contrastive objective and the attention gate has to be re-examined.

  4. Selecting adaptation pairs requires camera intrinsics and relative pose (0.4–0.6× FOV), Fig. 5 shows curves with no numeric values and no variance, and the paper never discusses how this geometric prior would be obtained at deployment.

  5. In actual deployment, it is difficult to find image pairs suitable for one-shot learning.

[1] Zhang J, Herrmann C, Hur J, et al. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence[J]. Advances in Neural Information Processing Systems, 2023, 36: 45533-45547.

[2] Tang L, Jia M, Wang Q, et al. Emergent correspondence from image diffusion[J]. Advances in neural information processing systems, 2023, 36: 1363-1389.

[3] Zhang J, Herrmann C, Hur J, et al. Telling left from right: Identifying geometry-aware semantic correspondence[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 3076-3085.

[4] Dünkel O, Wimmer T, Theobalt C, et al. Do it yourself: Learning semantic correspondence from pseudo-labels[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025: 5834-5844.

[5] Yang S, Wei T, Lan Y, et al. Towards Geometry-Grounded Dense Semantic Matching with VGGT Priors[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 586-605.

[6] Xue F, Elflein S, Leal-Taixé L, et al. MATCHA: Towards matching anything[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 27081-27091.

[7] Leroy V, Cabon Y, Revaud J. Grounding image matching in 3d with mast3r[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 71-91.

[8] Edstedt J, Sun Q, Bökman G, et al. Roma: Robust dense feature matching[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 19790-19800.

Questions for authors
  1. Why is the training-free VFM/diffusion correspondence line (especially, Telling Left from Right and MASt3R) neither cited nor compared, when it targets cross-instance semantic matching with no adaptation pair at all?

  2. What is your concrete advantage over those training-free methods?

  3. Your evidence is entirely sparse (R@5px, PCK at sampled keypoints via cosine-NN) and the adapter has no dense-specific design — no cost volume, no cost aggregation, no smoothness or multi-scale prior — so how do you verify that dense correspondence actually improves rather than just nearest-neighbor feature discriminability?

Reviewer_R3EfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a lightweight two-stage adapter to improve the 3D correspondence and geometric reasoning of frozen vision foundation models (VFMs) using only a single image pair with ground-truth correspondences. In Stage I, an attention-gated 3×33\times3 convolution refines local token features using a bidirectional contrastive loss. In Stage II, a LogDet-regularized linear projection aligns mean-normalized pairwise distance matrices across the two views to enforce structural consistency. Tested across five VFMs, the method consistently improves multi-view correspondence and pose estimation, while showing strong zero-shot transfer to video tracking and semantic correspondence on DINOv2 (with mixed results on weaker backbones).

Critical strengths

S1. The method adapts a frozen backbone in just two minutes with minimal inference overhead while delivering strong gains (e.g., boosting DINOv2-B on ScanNet from 17.42 to 31.57 R@5px and from 11.04 to 17.73 on 5∘5^\circ pose recall). Table 9 reports a large margin over one-shot 3DCorrEnhance on every metric (e.g., 31.57 vs. 25.08 R@5px on ScanNet), provided the baseline is properly configured (see Q2).

S2. The results also show that the adapter learns a generalizable correction rather than merely overfitting to the training pair. Adapting on a single external CO3Dv2 pair improves all downstream benchmarks zero-shot (Table 6, e.g., ScanNet R@5px from 17.42 to 29.14), and the NAVI-adapted DINOv2 model transfers well to video tracking and semantic correspondence.

Critical weaknesses

W1. Sections 1 and 2.3 state that 3D adaptation of VFMs under extremely limited supervision "remains underexplored," but the main baseline, 3DCorrEnhance (ICLR 2025), already showed that finetuning on a single pair for one iteration brings large gains. Appendix E actually reproduces this under the paper's own protocol (raising ScanNet R@5px from 17.42 to 25.08; Table 9). On the method side, the two-stage recipe mostly puts together existing techniques with little attribution. In Stage I, the 3×33\times3 conv adapter comes from 3DCorrEnhance, the spatial gating (Eq. 9-12) looks very similar to neighborhood attention (CVPR 2023; cited only as a Table 7 baseline rather than as the source of Eq. 9-10) and DCNv3 (CVPR 2023), and refining frozen DINOv2 on one sample with symmetric InfoNCE (τ=0.1\tau=0.1) closely mirrors DINO-Tracker (ECCV 2024). In Stage II, matching mean-normalized pairwise distance matrices (Eq. 4-5) is essentially the distance-wise loss from Relational Knowledge Distillation (CVPR 2019), just using squared error instead of Huber loss.

W2. The ablations also do not separate the proposed components from the effect of basic finetuning and extra capacity. In Table 7, even a simple Residual MLP with no spatial interaction jumps from 17.42 to 30.36 R@5px. This means nearly all of the ~14-point gain comes simply from training a small adapter on one pair, whereas all the Stage I architectural variants fall within a 1.21-point window (30.36 to 31.57) and the attention gate adds only +0.34 over a plain 3×33\times3 conv. Furthermore, the Stage II ablation is confounded: adding Stage II introduces new parameters (LL), up to 300 extra steps (Appendix A.3, with no stopping rule specified), and the relational loss (LrelL_{rel}) all at once, without controlling for capacity and compute (for example, training LL for the same steps with the contrastive loss LconL_{con} instead of LrelL_{rel}, or running Stage I for the same total steps).

Questions for authors

Q1. In Stage II, if you freeze Stage I and train LL for the same number of steps using the Stage I contrastive loss instead of the relational loss, does it already match the 31.57 R@5px in Table 5? If so, what does the relational objective actually contribute beyond adding extra parameters and training steps?

Reviewer_yP3oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a one-shot relational adaptation approach for adapting vision foundation models trained on 2D images to 3D correspondence tasks using a single overlapping image pair with annotated cross-view correspondences. The framework consists of two stages.

  • First, an attention-gated convolutional module is trained with a contrastive objective that pulls matched patch features closer and pushes unmatched patch features apart.
  • Second, a shared projection is learned to encourage consistency of relative pairwise feature distances across viewpoints.

The approach is evaluated on multiple downstream tasks, including multi-view correspondence estimation, pose estimation, video tracking, and semantic correspondence. The reported results show notable improvements in most evaluated settings.

Critical strengths

The approach is straightforward, and achieves consistent, notable improvements across most evaluated downstream settings.

Critical weaknesses
  • 3DCorrEnhance already reports gains from a single multiview image pair and one update, so the novelty lies primarily in the adapter and training objectives. Although Tables 5 and 7 support the proposed components, their additional benefits over 3DCorrEnhance's architecture and objectives are not clearly isolated.
  • Table 4 compares different training budgets, while Appendix E uses the same image pairs but does not specify the baseline update count or one-shot adaptation time. This leaves the accuracy and runtime advantage over one-shot 3DCorrEnhance under comparable optimization budgets unclear.
Questions for authors
  1. What advantages and trade-offs do the proposed architecture and objectives offer relative to 3DCorrEnhance's LoRA-based adaptation, beyond the benefits already demonstrated for one-pair training?

  2. Could the authors specify the one-shot 3DCorrEnhance optimization settings and compare both accuracy and adaptation time using the same image pairs and comparable optimization budgets?

  3. What might explain the method's stronger performance than FiT3D and DVT despite using much less adaptation data and time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.