- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of Cross-View Point Correspondence, aiming to identify the same pysical point across viewpoints. The authors propose MatchVLM as a solution. MatchVLM is trained in 2 training stages. In the first, Reference-Oriented Spatial Perception Learning, the model is supervised through supervised-fine-tuning on a known dataset, CrossPoint-378K, modified with additional evidence from the surroundings produced by an external matcher. In order to improve localization accuracy, in the second training stage, Distance-Aware Point Refinement, the authors employ RL with localization-based reward, on samples that were classified as hard based on performance on the SFT-trained checkpoint. The authors introduce CVPC-Bench to evaluate zero-shot performance, and show higher overall performance.
Critical strengths
- The results seem impressive: the proposed method mostly outperforms pre-trained VLMs in much larger scales, in addition to downstream-tuned baselines with matching (3B) and even larger-scaled (7B) VLMs.
Critical weaknesses
The design-choices and usage of RL is not clear to me.
- The motivation to use RL was stated: "However, token-level cross-entropy measures agreement with the target sequence rather than the spatial distance between predicted and ground-truth points. Smaller coordinate errors therefore do not necessarily incur lower training losses.". This by itself shouldn't be enough to apply RL, as the localization objective is present in the SFT training stage. Did you try to supervise using spatial distance between predicted and GT points in SFT? Supervising using the available GT coordinates seems to be easier than using the suggested reward using GRPO (which is generally more challenging to stabilize).
- A more plausible usage of RL would be to eliminate the need of the external matcher. Currently the usage of the external matcher is not measured in trainable params nor in the inference complexity. Could you provide inference times analysis/comparison between MatchVLM and CroPond-3B/7B?
Questions for authors
Please see the 2 questions from the weaknesses above.
3 comments
Response by Authors (2)
Weakness 2: Inference cost, and using RL to replace the external matcher.
Thank you for this suggestion.
(1) Including the matcher, MatchVLM-3B is more accurate than CroPond-7B at a lower inference cost. We have provided the inference-time comparison with CroPond-3B and CroPond-7B in Appendix C. The reported latency and peak GPU memory of MatchVLM cover the full pipeline, including the matcher and reference construction.
| Model | Total Params | Trainable Params | CrossPoint-Bench Score ↑ | CVPC-Bench EPE (px) ↓ | Latency (ms) ↓ | Peak GPU Mem. (GiB) ↓ |
|---|---|---|---|---|---|---|
| CroPond-3B | 3B | 3B | 71.60 | 36.65 | 197.6 | 7.405 |
| CroPond-7B | 7B | 7B | 76.60 | 27.59 | 253.0 | 15.867 |
| MatchVLM-3B (Ours) | 3B (VLM) + 15M (matcher) | 3B | 77.90 | 22.40 | 222.3 | 8.441 |
Best results are in bold.
The matcher adds 15M parameters in total (about 0.5% of the VLM), but it stays frozen throughout training and adds no trainable parameters. MatchVLM-3B incurs only a modest overhead over CroPond-3B, and outperforms CroPond-7B on both benchmarks with lower latency and about half the peak memory.
(2) We are exploring RL-based policies that invoke the matching expert only when needed. We agree that reducing the reliance on the external matcher is a valuable direction. Under our current training setting, matcher-free GRPO ("Original + GRPO" in Table 3) does not yet reach the level of the full system and remains below MatchVLM on all three cross-view tasks, since the distance reward tells the model how far a prediction is but not which local region corresponds across views. Building on the reliability assessment that MatchVLM already learns, we are exploring policy optimization that invokes the matching expert only when needed, aiming at high-precision cross-view understanding with low computational cost.
I thank the authors for the detailed rebuttal and the additional experiments.
Why RL instead of distance supervision in SFT
I thank the authors for clarifying that the distance of a decoded point cannot be optimized directly in SFT, since decoding is discrete and training uses teacher forcing, and for the comparison with a distance-aware SFT loss (PDCE, Table 10). PDCE yields only marginal gains (Score 76.80 vs. 76.60, CVPC EPE 27.22 vs. 27.90), whereas GRPO improves both (77.90 and 22.40). This addresses my concern.
Inference cost and matcher-free RL
I thank the authors for the inference-cost comparison (Appendix C, Table 12), which includes the matcher and reference construction. MatchVLM-3B adds a modest overhead over CroPond-3B (222.3 vs. 197.6 ms) and is both faster and more accurate than CroPond-7B. I also appreciate the comparison with matcher-free GRPO (Table 3), which shows that RL alone does not yet replace the matcher (Score 75.10 vs. 77.90), and the discussion of invoking the matcher only when needed as future work. This addresses my concern.
Overall
The rebuttal addresses both of my concerns, and the additional experiments strengthen the paper. I am therefore raising my score to 4.
Response by Authors (1)
We sincerely thank the reviewer for recognizing the strong empirical performance of MatchVLM against both much larger VLMs and downstream-tuned baselines. We appreciate the constructive feedback on our RL design and address the concerns below. All the revisions in the main manuscript are marked in blue.
Thank you for this insightful question. Supervising the distance of the predicted point is not differentiable in SFT and naturally leads to RL; we nevertheless tried a state-of-the-art distance-aware SFT loss, which brings only marginal gains, far below those of GRPO.
The results and corresponding analysis have been added to Appendix B.5 in the main manuscript.(1) Directly supervising the distance of the decoded point is not differentiable in SFT. The most direct way is to decode the output tokens (sampling or argmax), parse them into a coordinate, and penalize its distance to the ground truth. However, the VLM outputs coordinates as discrete text tokens, and both decoding and parsing are discrete steps, so the resulting distance is a scalar with zero gradient with respect to the model parameters. Moreover, SFT is trained with teacher forcing: each token is predicted from the ground-truth prefix, and the model never generates its own complete coordinates during training, so there is no predicted point to measure. Using the distance of decoded predictions as a training signal therefore requires treating it as a reward and optimizing it with policy gradients, which is exactly what our GRPO stage does. Within SFT, distance can only be approximated indirectly on the token distribution, which we evaluate next.
(2) Even with a dedicated distance-aware loss, supervising distance in SFT brings only marginal gains. To work around the limitation in (1), recent work injects distance into SFT through specially designed token-level losses. We evaluate the recently proposed PDCE [1], which makes cross-entropy (CE) on digit tokens distance-aware: each digit's one-hot target is replaced by a Gaussian centered on the ground-truth digit, and more significant digits are weighted more. Unlike (1), PDCE never decodes a point: it reshapes the per-digit target distribution under teacher forcing, so it remains differentiable but only approximates the distance digit by digit. We apply it to the coordinate digits and keep CE elsewhere; data, initialization, and hyperparameters are identical to our SFT stage.
PDCE yields only marginal gains on coordinate-output tasks and even degrades the others, leaving the overall performance nearly unchanged. This is because the digit-wise proxy is only loosely aligned with the true point-level distance, and it cannot supervise the option answers of Corr.-Judge at all. In contrast, GRPO consistently improves all tasks and substantially reduces the localization error.
(3) GRPO directly optimizes the localization error of the model's own predictions. GRPO does not need to differentiate through the distance: it samples complete responses, scores each decoded point with Eq. 10, and uses the reward only to weight the gradients of their log-probabilities. This brings three advantages over distance-aware SFT:
[1] Zhang et al. SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models. ICML, 2025.