- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how well VLMs can locate corresponding pixels across two views of the same scene. It introduces AnyPoint-Bench, which covers 12 datasets and evaluates models with 1 to 144 pixel queries in a single response. The authors also propose CorrVLM, which repurposes vocabulary entries as spatial tokens and uses coarse-to-fine training to learn correspondences. The experiments show that general-purpose VLMs struggle with this task, while CorrVLM substantially improves matching accuracy and achieves competitive results against dedicated visual matching models.
Critical strengths
-
Asking models to return coordinates makes localization errors directly measurable. Testing multiple points also gives a broader picture of performance than checking a single selected point. The range of query counts makes it possible to examine how well models handle increasingly long lists of correspondences.
-
The experiments cover 12 datasets with different scenes and imaging conditions, and include both general-purpose VLMs and dedicated matching methods. The “Same as Query” baseline is particularly useful: it shows when a seemingly reasonable score can come from simply copying the input coordinates, especially in image pairs with small displacements.
-
CorrVLM produces substantial gains over the evaluated general-purpose VLMs. It also reaches 66.98% PCK@9px on AnyPoint-Full in the 144-query setting, compared with RoMa’s 64.79%. The same checkpoint handles query counts not used during training, which provides some evidence that it can generalize beyond the specific output lengths it was trained on.
Critical weaknesses
-
The paper shows that fine-tuning improves a VLM’s cross-view pixel matching, but I’m still unsure what we gain by putting this ability inside a VLM. Table 3 shows that dedicated matchers such as RoMa already perform this task well. The current setup takes two images and source coordinates, so it does not show whether language understanding and precise matching can work together on a more useful task. For example, could the model follow a language instruction to identify a target and then locate it in another view? What would it offer over a VLM that calls an existing matching tool? I see the benchmark’s value for diagnosing model limitations, but the practical benefit of training CorrVLM is less clear.
-
Both the training correspondences and the benchmark labels come from UFM, which raises a concern about how reliable the supervision and evaluation are. Section 3.1 filters benchmark labels using cycle consistency and epipolar geometry, while the training pipeline in Section 4 uses only cycle consistency. These checks can remove some errors, but an incorrect match can still pass them, particularly in regions with repeated textures. The paper does not report an independent accuracy check of the retained labels. Without that, it is hard to tell how much the scores reflect correct geometric matching and how much they reflect agreement with UFM’s predictions. This also makes the performance gaps between methods harder to interpret.
-
The ablation in Figure 7 does not separate the effects of the two main design choices. The comparison with VLM3 changes vocabulary repurposing and coarse-to-fine training together, without showing what happens when either is used alone. The full approach clearly helps, but it is unclear whether the gain mainly comes from an easier coordinate representation or from learning correspondences in stages. Since the paper uses this comparison to explain the larger gains with more queries, separate ablations with matched training data and budgets would make that explanation more convincing.
Questions for authors
See the weakness.
Response to Reviewer fAM1
We thank Reviewer fAM1 for the constructive feedback on the motivation, label reliability, and training design.
Thank you for this important question about the purpose of the task. Recently, frontier VLMs are increasingly used for spatial tasks such as vision-and-language navigation, where interpreting relationships across views is fundamental. AnyPoint-Bench evaluates the precision of this capability through coordinate-level correspondence. The benchmark (our AnyPoint-Bench) exposes a clear gap in the fine-grained cross-view capabilities of current frontier VLMs. We therefore investigate whether this gap reflects a fundamental constraint of vision-language learning. CorrVLM provides evidence that task-specific training can enable a VLM to achieve correspondence accuracy competitive with specialized vision-only matchers. We then hope AnyPoint-Bench will serve as a reference for assessing and advancing this capability in future frontier VLMs.
Thank you for emphasizing annotation reliability. Global Response G1 reports validation against UFM-independent reference labels from real 3D geometry and controlled homographies. It also evaluates no-match queries separately and counts rejected positive predictions as PCK failures. For scalable training-data generation, we omit camera-based epipolar verification so that image pairs without camera annotations can be used.
Thank you for suggesting the additional ablation. G4 compares a VLM³-style baseline, VLM³-C2F, and CorrVLM, with the training controls stated explicitly. Adding the coarse stage uses the same image pairs and yields the largest observed correspondence gains. At the larger data scale, vocabulary repurposing gives comparable PCK@9px and improves no-match accuracy by 3.50/2.63 percentage points at Q4/Q8.