acceptodds
Under review as a conference paper at ICLR 2027

MatchVLM: Synergizing Vision-Language Models and Local Feature Matching for Cross-View Point Correspondence

Abstract

Cross-View Point Correspondence (CVPC) aims to identify the same physical point across different viewpoints, providing precise spatial guidance for embodied interaction. Despite recent progress, vision-language models (VLMs) can confuse similar object instances under viewpoint changes, revealing the difficulty of inferring fine-grained correspondences from global visual context alone. To address this limitation, we propose MatchVLM, a collaborative framework that integrates feature matching with a compact VLM, leveraging local correspondence cues to support precise cross-view point prediction. We develop this capability through two training stages. Reference-Oriented Spatial Perception Learning jointly supervises reference reliability assessment and answer prediction through supervised fine-tuning. Distance-Aware Point Refinement employs Group Relative Policy Optimization (GRPO) with a point-level distance reward that directly reflects prediction error, providing spatial feedback beyond token-level supervision. A curriculum within this stage emphasizes cases that remain challenging after supervised fine-tuning. We further introduce CVPC-Bench, comprising 500 manually annotated cross-view point pairs across eight outdoor and two indoor scenes. Experiments demonstrate that MatchVLM-3B achieves the highest overall Score on CrossPoint-Bench among the evaluated models and outperforms both CroPond-3B and CroPond-7B on CVPC-Bench under zero-shot evaluation, supporting its effectiveness in cross-view point correspondence and generalization to diverse scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-view point correspondence, Vision-language models, Local feature matching, Spatial understanding, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5Vi1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of Cross-View Point Correspondence, aiming to identify the same pysical point across viewpoints. The authors propose MatchVLM as a solution. MatchVLM is trained in 2 training stages. In the first, Reference-Oriented Spatial Perception Learning, the model is supervised through supervised-fine-tuning on a known dataset, CrossPoint-378K, modified with additional evidence from the surroundings produced by an external matcher. In order to improve localization accuracy, in the second training stage, Distance-Aware Point Refinement, the authors employ RL with localization-based reward, on samples that were classified as hard based on performance on the SFT-trained checkpoint. The authors introduce CVPC-Bench to evaluate zero-shot performance, and show higher overall performance.

Critical strengths
  1. The results seem impressive: the proposed method mostly outperforms pre-trained VLMs in much larger scales, in addition to downstream-tuned baselines with matching (3B) and even larger-scaled (7B) VLMs.
Critical weaknesses

The design-choices and usage of RL is not clear to me.

  1. The motivation to use RL was stated: "However, token-level cross-entropy measures agreement with the target sequence rather than the spatial distance between predicted and ground-truth points. Smaller coordinate errors therefore do not necessarily incur lower training losses.". This by itself shouldn't be enough to apply RL, as the localization objective is present in the SFT training stage. Did you try to supervise using spatial distance between predicted and GT points in SFT? Supervising using the available GT coordinates seems to be easier than using the suggested reward using GRPO (which is generally more challenging to stabilize).
  2. A more plausible usage of RL would be to eliminate the need of the external matcher. Currently the usage of the external matcher is not measured in trainable params nor in the inference complexity. Could you provide inference times analysis/comparison between MatchVLM and CroPond-3B/7B?
Questions for authors

Please see the 2 questions from the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for recognizing the strong empirical performance of MatchVLM against both much larger VLMs and downstream-tuned baselines. We appreciate the constructive feedback on our RL design and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1: Why RL instead of directly supervising spatial distance in SFT?

Thank you for this insightful question. Supervising the distance of the predicted point is not differentiable in SFT and naturally leads to RL; we nevertheless tried a state-of-the-art distance-aware SFT loss, which brings only marginal gains, far below those of GRPO. The results and corresponding analysis have been added to Appendix B.5 in the main manuscript.

(1) Directly supervising the distance of the decoded point is not differentiable in SFT. The most direct way is to decode the output tokens (sampling or argmax), parse them into a coordinate, and penalize its distance to the ground truth. However, the VLM outputs coordinates as discrete text tokens, and both decoding and parsing are discrete steps, so the resulting distance is a scalar with zero gradient with respect to the model parameters. Moreover, SFT is trained with teacher forcing: each token is predicted from the ground-truth prefix, and the model never generates its own complete coordinates during training, so there is no predicted point to measure. Using the distance of decoded predictions as a training signal therefore requires treating it as a reward and optimizing it with policy gradients, which is exactly what our GRPO stage does. Within SFT, distance can only be approximated indirectly on the token distribution, which we evaluate next.

(2) Even with a dedicated distance-aware loss, supervising distance in SFT brings only marginal gains. To work around the limitation in (1), recent work injects distance into SFT through specially designed token-level losses. We evaluate the recently proposed PDCE [1], which makes cross-entropy (CE) on digit tokens distance-aware: each digit's one-hot target is replaced by a Gaussian centered on the ground-truth digit, and more significant digits are weighted more. Unlike (1), PDCE never decodes a point: it reshapes the per-digit target distribution under teacher forcing, so it remains differentiable but only approximates the distance digit by digit. We apply it to the coordinate digits and keep CE elsewhere; data, initialization, and hyperparameters are identical to our SFT stage.

SFT lossGRPOScoreFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-PointCVPC-Bench EPE (px) ↓
CE✗76.6059.0181.3687.8276.6727.90
PDCE (distance-aware SFT)✗76.8060.2580.4587.1877.3227.22
CE (Ours)✓77.9061.4981.8290.3877.5422.40

PDCE yields only marginal gains on coordinate-output tasks and even degrades the others, leaving the overall performance nearly unchanged. This is because the digit-wise proxy is only loosely aligned with the true point-level distance, and it cannot supervise the option answers of Corr.-Judge at all. In contrast, GRPO consistently improves all tasks and substantially reduces the localization error.

(3) GRPO directly optimizes the localization error of the model's own predictions. GRPO does not need to differentiate through the distance: it samples complete responses, scores each decoded point with Eq. 10, and uses the reward only to weight the gradients of their log-probabilities. This brings three advantages over distance-aware SFT:

  • Exact point-level distance. The reward uses the Euclidean distance d(a) = ‖Φ(a) − p*‖₂ between the decoded point and the ground truth, rather than a digit-wise approximation that ignores carries (199 and 200 are 1 pixel apart, yet every digit differs) and treats the two axes separately.
  • On-policy optimization. SFT supervises each digit under the ground-truth prefix (teacher forcing), whereas GRPO evaluates the points the model actually generates. Together with the curriculum (Eq. 12), it concentrates updates on cases that remain wrong after SFT.
  • A shared reward for all localization answers. Option letters in Correspondence-Judgement admit no digit-level loss, but our reward maps each selected option to its candidate coordinates via Φ(a) (Section 4.3), so a wrong option closer to the ground truth is rewarded more than a distant one. Accordingly, Corr.-Judge shows the largest gain from GRPO.

[1] Zhang et al. SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models. ICML, 2025.

AuthorsOpenReview ↗

Response by Authors (2)

Weakness 2: Inference cost, and using RL to replace the external matcher.

Thank you for this suggestion.

(1) Including the matcher, MatchVLM-3B is more accurate than CroPond-7B at a lower inference cost. We have provided the inference-time comparison with CroPond-3B and CroPond-7B in Appendix C. The reported latency and peak GPU memory of MatchVLM cover the full pipeline, including the matcher and reference construction.

ModelTotal ParamsTrainable ParamsCrossPoint-Bench Score ↑CVPC-Bench EPE (px) ↓Latency (ms) ↓Peak GPU Mem. (GiB) ↓
CroPond-3B3B3B71.6036.65197.67.405
CroPond-7B7B7B76.6027.59253.015.867
MatchVLM-3B (Ours)3B (VLM) + 15M (matcher)3B77.9022.40222.38.441

Best results are in bold.

The matcher adds 15M parameters in total (about 0.5% of the VLM), but it stays frozen throughout training and adds no trainable parameters. MatchVLM-3B incurs only a modest overhead over CroPond-3B, and outperforms CroPond-7B on both benchmarks with lower latency and about half the peak memory.

(2) We are exploring RL-based policies that invoke the matching expert only when needed. We agree that reducing the reliance on the external matcher is a valuable direction. Under our current training setting, matcher-free GRPO ("Original + GRPO" in Table 3) does not yet reach the level of the full system and remains below MatchVLM on all three cross-view tasks, since the distance reward tells the model how far a prediction is but not which local region corresponds across views. Building on the reliability assessment that MatchVLM already learns, we are exploring policy optimization that invokes the matching expert only when needed, aiming at high-precision cross-view understanding with low computational cost.

I thank the authors for the detailed rebuttal and the additional experiments.

Why RL instead of distance supervision in SFT

I thank the authors for clarifying that the distance of a decoded point cannot be optimized directly in SFT, since decoding is discrete and training uses teacher forcing, and for the comparison with a distance-aware SFT loss (PDCE, Table 10). PDCE yields only marginal gains (Score 76.80 vs. 76.60, CVPC EPE 27.22 vs. 27.90), whereas GRPO improves both (77.90 and 22.40). This addresses my concern.

Inference cost and matcher-free RL

I thank the authors for the inference-cost comparison (Appendix C, Table 12), which includes the matcher and reference construction. MatchVLM-3B adds a modest overhead over CroPond-3B (222.3 vs. 197.6 ms) and is both faster and more accurate than CroPond-7B. I also appreciate the comparison with matcher-free GRPO (Table 3), which shows that RL alone does not yet replace the matcher (Score 75.10 vs. 77.90), and the discussion of invoking the matcher only when needed as future work. This addresses my concern.

Overall

The rebuttal addresses both of my concerns, and the additional experiments strengthen the paper. I am therefore raising my score to 4.

Reviewer_tEg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MatchVLM, a framework that integrates local feature matching with a compact VLM for cross-view point correspondence. It employs Efficient LoFTR to construct local correspondence references and adopts a two-stage training strategy combining reference-oriented SFT with distance-aware GRPO to improve point localization. Experiments on CrossPoint-Bench and the newly introduced CVPC-Bench demonstrate improved performance and lower zero-shot localization error compared with CroPond baselines.

Critical strengths
  1. Practical Relevance. The proposed method have clear potential applications in VR/AR, robotics.

  2. Strong Empirical Performance. MatchVLM-3B achieves the highest overall Score on CrossPoint-Bench among the evaluated models and outperforms both CroPond-3B and CroPond-7B on CVPC-Bench under zero-shot evaluation.

Critical weaknesses
  1. The related work discussion is substantially underdeveloped. The Related Work section consists of only two short paragraphs, with limited coverage of several research directions closely related to the proposed method, including local feature matching, cross-view geometric correspondence, and reinforcement learning for spatial localization. Given that the framework builds upon several established techniques, the lack of a thorough discussion makes it difficult to assess the novelty of the proposed approach and its distinctions from prior work.

  2. The necessity of introducing a VLM into the correspondence pipeline is not sufficiently justified. Efficient LoFTR already establishes cross-view feature correspondences, while the subsequent query-neighborhood filtering and DBSCAN clustering further narrow down the candidate region in the target image. Much of the information needed for point localization is therefore already available before the VLM is invoked. It remains unclear what additional capability the VLM provides that cannot be achieved through simpler geometric estimation based on these correspondences.

  3. The framework heavily depends on the quality of a pretrained feature matcher. The proposed pipeline is not trained end-to-end, as Efficient LoFTR remains frozen throughout training. Errors in the initial matching results may propagate through the subsequent filtering and clustering stages, potentially producing misleading reference patches for the VLM. Although the paper introduces reference reliability assessment, it does not sufficiently investigate how robust the overall framework is to inaccurate or missing feature correspondences.

  4. The ablation studies and baseline comparisons are insufficient. While Table 3 examines the effects of reference augmentation and GRPO, it does not isolate the contributions of local visual references and explicit reliability supervision. More importantly, the paper lacks direct localization baselines using traditional feature matching methods, such as nearest-neighbor correspondence transfer or local geometric interpolation. Since Efficient LoFTR already provides candidate cross-view correspondences, it remains unclear whether the VLM substantially improves localization accuracy beyond what the matcher alone can achieve, or primarily converts existing matching results into coordinate predictions. This weakens the justification for the claimed benefits of matcher–VLM collaboration.

  5. The performance advantage on the main benchmark is relatively modest. On CrossPoint-Bench, MatchVLM-3B outperforms CroPond-7B by only 1.30 percentage points in overall Score, while performing slightly worse on the Correspondence-Pointing subtask. Moreover, the paper does not report results across multiple random seeds. Given the limited size of the benchmark, it is difficult to assess whether the reported improvement is sufficiently stable and significant.

Questions for authors
  1. Could the authors clarify how MatchVLM differs from the most closely related approaches in local feature matching, cross-view correspondence, and VLM-based spatial reasoning? In particular, what are the key technical contributions beyond integrating existing matching and VLM techniques?

  2. What specific capabilities does the VLM contribute beyond the correspondences and candidate regions already provided by Efficient LoFTR and DBSCAN clustering? Could the authors provide examples or quantitative analysis showing cases where the VLM successfully resolves ambiguities that cannot be addressed by the matcher alone?

  3. How sensitive is MatchVLM to the quality of the pretrained matcher? Could the authors report performance separately for samples with reliable, unreliable, or missing local references, and investigate how matching errors affect the final localization accuracy?

  4. Could the authors provide additional ablations that isolate the contributions of local visual references and reliability supervision? Moreover, how does MatchVLM compare against direct point localization using Efficient LoFTR correspondences, such as nearest-neighbor transfer or local geometric interpolation, on the same evaluation benchmarks?

  5. Could the authors report results across multiple random seeds or provide confidence intervals for the main benchmark? Given the relatively small improvement over CroPond-7B, additional evidence of statistical significance would help establish the robustness of the reported gains.

4 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for recognizing the practical relevance of our work and its strong empirical performance. We appreciate the detailed feedback and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1 & Question 1: Related work and key technical contributions.

Thank you for this suggestion. We have expanded the Related Work section to discuss the three directions raised by the reviewer, i.e., local feature matching, cross-view correspondence in VLMs, and reinforcement learning for spatial localization. Each paragraph reviews representative methods and ends with the gap that MatchVLM addresses. The expanded Related Work section has been added to the main manuscript.

(1) Key technical contributions beyond integrating existing components.

MatchVLM indeed builds on existing components, namely a pre-trained matcher and a VLM trained with SFT and GRPO, as do most recent works in this area. For example, Visual-RFT and VLM-R1 apply the GRPO recipe to detection and grounding with an IoU reward, RoboRefer and Embodied-R1 apply it to embodied pointing, and CroPond applies SFT on newly constructed data. However, the contribution of these works lies not in new optimizers but in identifying what the task needs and designing a targeted formulation that supplies it, which is why each of them achieves strong gains on its target problem. Likewise, the novelty of MatchVLM lies in the understanding of why VLMs fail at CVPC and in the formulation built on that understanding.

  • (i) VLMs fail at CVPC by confusing similar instances under viewpoint change. CVPC asks for the same physical point across views. VLM-only models infer correspondence from global visual context and therefore confuse similar instances under viewpoint change (Figure 1(a)), and more data or a larger VLM alleviates this only gradually (Table 1).
  • (ii) Reliability-conditioned prediction, rather than simply feeding matcher outputs to the VLM. Local feature matching provides the instance-level discrimination that global context lacks, but its evidence can be wrong or missing. MatchVLM therefore does not simply feed the matcher outputs to the VLM. It first predicts a reliability decision on the references and then generates the answer conditioned on that decision (Eq. 2), so that reliable and unreliable evidence are handled differently. As a result, MatchVLM benefits from reliable references and is not misled by unreliable ones, whereas a variant that receives the same references without reliability supervision is misled. The detailed analysis is provided in our response to Weakness 3.
  • (iii) A small expert unlocking a compact VLM, rather than scaling the VLM. Under this formulation, a 15M-parameter matcher lets a 3B VLM surpass the 7B VLM-only baseline on both benchmarks, and the RL stage further sharpens localization with a reward that reflects point distance rather than token agreement.
AuthorsOpenReview ↗

Response by Authors (2)

Weakness 1 & Question 1 (continued): Related work and key technical contributions.

(2) How MatchVLM differs from the most closely related work.

  • (i) vs. local feature matching: matchers provide only geometric correspondences, not instruction-conditioned answers. Sparse, detector-free, dense, and 3D-aware matchers establish precise pixel-level correspondences, but they operate purely on pixels and take no instruction, i.e., they cannot locate a language-specified target, and they cannot tell when their own matches are wrong or missing. MatchVLM does not compete with them as a coordinate solver. It can determine the query point from a semantic instruction (e.g., "the red chair") and then establish its cross-view correspondence, thereby serving as a general-purpose, instruction-conditioned solver for cross-view correspondence.
  • (ii) vs. cross-view VLMs: from VLM-only prediction to a small expert unlocking a compact VLM. Existing approaches, whether through multi-frame training data (Multi-SpatialMLLM, XVR, CroPond's CrossPoint-378K) or cross-view feature alignment and distillation, are all VLM-only, i.e., correspondence is learned into the VLM's weights and must be inferred by the VLM alone at inference. As a result, similar instances remain easy to confuse (Figure 1(a)), and further improvement relies on scaling the VLM (Table 1). MatchVLM instead lets the VLM assess matcher-derived local references and answer conditioned on that assessment (Eq. 2), and with only 3B parameters it surpasses the 7B VLM-only baseline on both benchmarks.
  • (iii) vs. RL for spatial localization: a bounded, signed distance reward for point refinement. The reward design is central to GRPO, as it defines the optimization objective of the policy. Existing RL methods for spatial localization, such as Visual-RFT and VLM-R1, adopt overlap-based rewards such as box IoU. We instead propose a bounded, signed point-level distance reward (Eq. 10), which encourages predictions within a distance tolerance and penalizes those beyond it, and consistently improves over SFT (Table 3). In addition, curriculum learning focuses the RL budget on cases that remain unresolved after SFT.

Weakness 2 & Question 2: What the VLM contributes beyond the matcher and DBSCAN.

Thank you for this important question. The matcher and DBSCAN provide only a candidate local region, not the answer. The VLM supplies three capabilities that geometric estimation cannot, i.e., judging whether the references are trustworthy, answering when the references are wrong or missing, and solving instruction-conditioned tasks.

(1) Geometric estimation from the same correspondences falls far short of MatchVLM. Directly estimating the target point from Efficient LoFTR correspondences alone, by nearest-neighbor transfer or local geometric interpolation, is substantially worse than MatchVLM on both Corr.-Point and CVPC-Bench, and cannot directly handle the other three tasks. The information available before the VLM is therefore far from sufficient for localization. The detailed results are provided in our response to Weakness 4(2).

(2) The ambiguities that the matcher cannot resolve but the VLM can. When the references are unreliable, geometric estimation inherits the matching error and collapses, whereas MatchVLM still answers most of them correctly. When the references are missing, the matcher produces no prediction at all, whereas MatchVLM performs on par with CroPond-3B. This is exactly the ambiguity that the matcher alone cannot resolve. Even when the references are reliable, MatchVLM remains more accurate than geometric estimation, showing that it does not simply transcribe the matching result into coordinates. The detailed results are provided in our response to Weakness 3.

(3) Instruction-conditioned tasks. The VLM enables language-specified tasks that geometric estimation cannot handle, such as grounding or locating the cross-view correspondence of an object referred to only by an instruction (e.g., "the red chair"). MatchVLM is thus a general spatial-understanding model that jointly understands language and images, rather than a specialized solver for a single geometric task.

AuthorsOpenReview ↗

Response by Authors (3)

Weakness 3 & Question 3: Sensitivity to matcher quality.

Thank you for this suggestion. We have conducted an additional analysis that groups the Corr.-Point and Corr.-Judge samples of CrossPoint-Bench by the status of the matcher references, i.e., reliable, unreliable, and missing, and reports the accuracy (%) of each group. MatchVLM is compared with the VLM-only CroPond and with direct localization from the matcher alone, i.e., nearest-neighbor transfer (Matcher-NN) and local geometric interpolation (Matcher-Interp.). The results and corresponding analysis have been added to Appendix B.4 in the main manuscript.

Reference statusCroPond-3BCroPond-7BMatcher-NNMatcher-Interp.MatchVLM-3B (Ours)
Reliable (486)75.5182.7277.3781.2883.13
Unreliable (122)73.7769.6718.0318.8574.59
Missing (11)5/118/110/11*0/11*5/11

* Without references, the matcher produces no prediction and is counted as incorrect. The Missing group contains only 11 samples, so we report the number of correct samples.

MatchVLM is robust to the quality of the pre-trained matcher, i.e., it achieves the best accuracy with both reliable and unreliable references. With reliable references, MatchVLM outperforms all other methods. With unreliable references, geometric estimation inherits the matching error and collapses, whereas MatchVLM still outperforms both CroPond-3B of the same scale and the larger CroPond-7B, which use no matcher at all. Missing references are rare, and MatchVLM then performs on par with CroPond-3B.

We further test MatchVLM under synthetic misleading references. On 486 samples with reliable references, we replace the target reference patch with a Random, Displaced, or Distractor patch that excludes the ground truth. All variants are SFT models that differ only in how references are used (accuracy, %). The results and corresponding analysis have been added to Appendix B.3 in the main manuscript.

VariantCleanRandomDisplacedDistractor
No reference75.5175.5175.5175.51
References w/o reliability supervision80.0471.4071.8167.90
References + reliability supervision (Ours, SFT)80.4577.1677.3776.13

MatchVLM is not misled by wrong references, i.e., its accuracy stays above the no-reference baseline under all three perturbations. Without reliability supervision, misleading references pull the accuracy below that baseline (CroPond-3B), especially for distractors.

AuthorsOpenReview ↗

Response by Authors (4)

Weakness 4 & Question 4: Separating local references from reliability supervision, and matcher-based localization baselines.

Thank you for this suggestion. We have conducted two additional experiments, i.e., an ablation that keeps the paired local references but removes reliability supervision, and direct localization baselines using Efficient LoFTR correspondences alone. The results and corresponding analysis have been added to Table 3 at Page 9 and Appendix D in the main manuscript.

(1) Local references and reliability supervision each contribute. We retrain SFT with the same paired patches but without reliability supervision, separating the two components. Both variants are evaluated after SFT (before GRPO).

SFT variantScoreFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-Point
Patches, w/o reliability supervision75.8061.4979.0983.9776.46
Patches + reliability supervision (Ours, SFT)76.6059.0181.3687.8276.67

Reliability supervision improves all cross-view tasks that use local references. Together with Table 3, the results show that paired local references and reliability assessment each contribute to the final performance.

(2) MatchVLM does not merely convert matching results into coordinates. We use Efficient LoFTR correspondences alone to map the source query point to the target view by nearest-neighbor transfer (NN) or local geometric interpolation (Interp.).

MethodFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-PointCVPC EPE (px) ↓
Efficient LoFTR, NN✗✗✗59.8328.74
Efficient LoFTR, Interp.✗✗✗63.0725.47
MatchVLM-3B (Ours)61.4981.8290.3877.5422.40

✗: the task cannot be directly handled by a matcher.

With exactly the correspondences available before the VLM is invoked, both baselines are substantially worse than MatchVLM on both Corr.-Point and CVPC-Bench, and neither can directly handle the other three tasks. MatchVLM therefore provides substantial gains on top of the same matching information, rather than merely converting matching results into coordinates.

Weakness 5 & Question 5: Magnitude and robustness of the improvement.

Thank you for this suggestion. We have added 95% confidence intervals on both benchmarks. The improvement of MatchVLM-3B is statistically significant over both CroPond-3B and CroPond-7B, and it is achieved with less than half the parameters of CroPond-7B. The results and corresponding analysis have been added to Appendix B.6 in the main manuscript.

(1) Confidence intervals on both benchmarks. We report 95% confidence intervals from a paired bootstrap over test samples. All models are evaluated on the same samples, so for each sample we take the difference between MatchVLM-3B and the baseline, i.e., correctness (1 or 0) on CrossPoint-Bench and EPE in pixels on CVPC-Bench. We resample the test set with replacement 20,000 times, recompute the mean difference each time, and report the 2.5th and 97.5th percentiles. An interval that excludes zero indicates a statistically significant difference. The intervals quantify the uncertainty from sampling the test set. Inference uses greedy decoding, so repeated evaluation gives identical results. This does not replace an analysis across training seeds.

BenchmarkBaselineBaseline resultMatchVLM-3B (Ours)Δ95% CI
CrossPoint-Bench (Score ↑)CroPond-3B71.6077.90+6.30[2.50, 10.10]
CrossPoint-Bench (Score ↑)CroPond-7B76.6077.90+1.30[0.10, 2.50]
CVPC-Bench (Mean EPE ↓)CroPond-3B36.6522.40−14.25[−20.34, −8.77]
CVPC-Bench (Mean EPE ↓)CroPond-7B27.5922.40−5.19[−11.89, −1.43]

All four intervals exclude zero, i.e., the advantage of MatchVLM-3B over both baselines is statistically significant on both the main benchmark and the zero-shot CVPC-Bench. Against CroPond-3B, which shares the same Qwen2.5-VL-3B base, the improvement is large on both benchmarks. Against CroPond-7B, MatchVLM-3B is still significantly more accurate with less than half the parameters, lower latency, and about half the peak memory (Appendix C).

(2) Corr.-Point compared with CroPond-7B. The gap on Corr.-Point is a single sample out of 463, and MatchVLM-3B outperforms CroPond-7B on the other three tasks (Table 1). The advantage of MatchVLM-3B lies in overall performance, zero-shot generalization, and efficiency.

Reviewer_SkJBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MatchVLM, which combines a local feature matcher with a vision-language model for cross-view point correspondence. It uses two-stage training, icluding Reference-Oriented Spatial Perception Learning through supervised fine-tuning, followed by Distance-Aware Point Refinement through GRPO with a distance-based reward. MatchVLM-3B outperforms CroPond-7B in overall score on CrossPoint-Bench and localization error on the newly introduced CVPC-Bench.

Critical strengths
  • The combination of local matching evidence and VLM reasoning is well motivated, with ablations supporting the benefits of reference augmentation and distance-aware refinement.

  • MatchVLM-3B outperforms CroPond-7B in overall benchmark performance while requiring less inference time and memory.

  • The paper is clearly written and easy to follow.

Critical weaknesses
  • Table 1 omits recent flagship VLMs such as GPT-6 Astra, while Table 2 compares MatchVLM-3B only with CroPond-3B and CroPond-7B on CVPC-Bench. Neither benchmark includes standalone matching experts (without VLMs) such as MASt3R or RoMa v2, leaving the added value of the VLM unclear.

  • The ablation does not separate paired local references from reference reliability assessment, since reference-augmented SFT introduces both components simultaneously.

  • The robustness analysis only considers naturally unreliable references produced by Efficient LoFTR, without testing random, displaced, or visually similar misleading references against no-reference and no-assessment variants.

Questions for authors

Please refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for recognizing our motivation, efficiency, and clear writing. We appreciate the constructive feedback and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1: Recent flagship VLMs and standalone matching experts.

Thank you for this suggestion. We have conducted additional experiments comparing MatchVLM with recent flagship VLMs and standalone matching experts on CrossPoint-Bench and CVPC-Bench.

(1) With only 3B parameters, MatchVLM is competitive with recent flagship VLMs. We evaluate three recent flagship models on CrossPoint-Bench and on our proposed CVPC-Bench for zero-shot evaluation, using the same prompts and metrics as Tables 1 and 2. The results and corresponding analysis have been added to Tables 1 and 2 at Page 8 in the main manuscript.

CrossPoint-Bench (accuracy, %)

ModelScoreFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-Point
GPT-6 Astra82.7075.1687.7394.8778.83
Qwen3.8-Max77.1078.2681.3679.4973.87
Claude Fable 5.171.7068.3280.4580.7765.66
MatchVLM-3B (Ours)77.9061.4981.8290.3877.54

CVPC-Bench, zero-shot (Mean EPE, px ↓)

ModelOutdoorIndoorOverall
GPT-6 Astra27.5234.6729.52
Qwen3.8-Max26.0824.3625.60
Claude Fable 5.157.6254.4656.74
MatchVLM-3B (Ours)16.8936.5922.40

Following scaling laws, recent flagship VLMs pursue stronger capability by jointly scaling model parameters and training data, leading to enormous model sizes and training budgets. For example, Qwen3.8-Max has 2.4T total parameters (95B active) [1], about 800× more than MatchVLM-3B, while GPT-6 Astra and Claude Fable 5.1 are proprietary frontier models that do not disclose their sizes. Despite this gap, MatchVLM outperforms Qwen3.8-Max and Claude Fable 5.1 on CrossPoint-Bench and achieves the lowest overall EPE on CVPC-Bench. GPT-6 Astra obtains the highest CrossPoint-Bench Score, consistent with its strong spatial and embodied capability verified in the recent SenseNova technical report [2]. Even so, MatchVLM reaches 94.2% of its CrossPoint-Bench Score, achieves state-of-the-art performance among open-source models, and surpasses it on CVPC-Bench by 7.12 px in EPE. This demonstrates the effectiveness of our approach.

[1] Qwen Team. Qwen3.8-Max: A New Bar for Coding and Cowork. https://qwen.ai/blog?id=qwen3.8, 2026.

[2] SenseNova Joint Research Team. Has GPT-6-Astra Achieved Embodied Intelligence? arXiv:2610.22471, 2026.

AuthorsOpenReview ↗

Response by Authors (2)

Weakness 1 (continued): Recent flagship VLMs and standalone matching experts.

(2) The VLM brings substantial value beyond its own matcher. We use each matcher alone to map the source query point to the target view by nearest-neighbor transfer. Latency and memory for the matchers cover matching only. For MatchVLM, they cover the full pipeline (matcher + reference construction + VLM). The results and corresponding analysis have been added to Appendix D in the main manuscript.

MethodFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-PointCVPC EPE ↓Latency (ms) ↓Peak Mem. (GiB) ↓
Efficient LoFTR✗✗✗59.8328.7457.31.036
RoMa v2✗✗✗86.3915.54412.07.497
MASt3R✗✗✗85.5314.63378.03.247
MatchVLM-3B (Ours)61.4981.8290.3877.5422.40222.38.441

✗: the task cannot be directly handled by a matcher.

MatchVLM clearly outperforms Efficient LoFTR alone (the matcher used inside it) on both Corr.-Point and CVPC-Bench, demonstrating that the VLM brings substantial value beyond the matcher itself. Heavier dense matchers (RoMa v2 and MASt3R) achieve higher geometric precision, which is their sole objective. However, MatchVLM's full pipeline still runs with lower latency than their matching alone, and it handles all four tasks, three of which matchers cannot directly address, including language-specified tasks such as grounding or locating the cross-view correspondence of an object referred to only by an instruction (e.g., "the red chair"). We use Efficient LoFTR rather than these matchers for training for two reasons:

  • Efficiency. References must be built for all of CrossPoint-378K, and Efficient LoFTR requires far less time and memory per pair.
  • Reliability supervision needs negatives. RoMa v2 and MASt3R yield few naturally occurring unreliable references (reliability rate >97% vs. 78.5% for Efficient LoFTR), providing too few negatives for learning to discriminate reference reliability.

(3) MatchVLM is designed as a general spatial-understanding model, not the best cross-view coordinate solver. Our goal is a model that serves multi-agent collaboration and jointly understands language and images, rather than a specialized solver for a single geometric task. Cross-view collaboration requires several atomic capabilities beyond geometric matching: locating the object referred to by an instruction (Fine-grained Ground), judging whether it is visible in the partner's view (Visibility Reason), and establishing its cross-view correspondence (Corr.-Judge and Corr.-Point). MatchVLM unifies all of these capabilities in one model. Under this goal, our key insight is to use a small expert, a matcher with only 15M parameters, to lift a general VLM toward expert-level spatial precision.

AuthorsOpenReview ↗

Response by Authors (3)

Weakness 2: Separating paired local references from reliability assessment.

Thank you for this suggestion. We have conducted an additional ablation that keeps the paired local references but removes reliability supervision, separating the contributions of the two components. Both variants are evaluated after SFT (before GRPO). The results and corresponding analysis have been added to Table 3 at Page 9 in the main manuscript.

SFT variantScoreFine-grained GroundVisibility ReasonCorr.-JudgeCorr.-PointCross-view tasks (839 samples)
Patches, w/o reliability supervision75.8061.4979.0983.9776.4678.55
Patches + reliability supervision (Ours, SFT)76.6059.0181.3687.8276.6779.98

Reliability supervision consistently improves all cross-view tasks that use local references. Together with Table 3, the results demonstrate that paired local references and reliability assessment each contribute to the final performance.

Weakness 3: Robustness to synthetic misleading references.

Thank you for this suggestion. We have conducted a robustness analysis with three types of synthetic misleading references, comparing MatchVLM with the no-reference and no-reliability-supervision variants. Starting from 486 Pointing and Judge samples with reliable matcher references, we replace only the target reference patch with a same-size patch that is guaranteed to exclude the ground truth: Random (a random location), Displaced (the original box shifted along one of 8 directions), and Distractor (the most similar wrong region in DINOv2 features for Pointing, or a box centered on a wrong option for Judge). All three variants are SFT models with the same base model, differing only in how references are used. The results and corresponding analysis have been added to Appendix B.3 in the main manuscript.

VariantCleanRandomDisplacedDistractor
No reference75.5175.5175.5175.51
References w/o reliability supervision80.0471.4071.8167.90
References + reliability supervision (Ours, SFT)80.4577.1677.3776.13

Reliability supervision makes MatchVLM robust to misleading references. Without it, misleading references pull accuracy below the no-reference baseline (CroPond-3B), especially for distractors. With it, MatchVLM exploits clean references while remaining above the no-reference baseline under all perturbations.

Thanks for the authors' responses, which largely address my main concerns.

However, since MatchVLM does not outperform existing matching experts, as shown in the responses, its advantages over a tool-use approach remain unclear. Treating these experts as tools within an agent framework might be a simpler and better solution. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.