acceptodds
Under review as a conference paper at ICLR 2027

AnyPoint: Unpacking Cross-View Understanding Granularity in Vision-Language Models

Abstract

Fine-grained cross-view understanding requires Vision-Language Models (VLMs) to localize arbitrary pixel locations across changes in viewpoint and appearance. This capability is fundamental to spatial reasoning and embodied AI. Existing benchmarks rely on discrete candidate selection or isolated pixel queries. However, discrete candidate selection limits localization precision, while isolated pixel queries yield query-dependent scores and fail to assess global understanding. To address these limitations, we introduce AnyPoint-Bench, the first benchmark to formulate fine-grained cross-view understanding as the cross-view localization of arbitrary pixels. Given an image pair, a VLM jointly localizes an arbitrary set of reference-view pixels by predicting their coordinates in the target view. This unified protocol evaluates both fine-grained localization and global understanding. AnyPoint-Bench covers diverse scenes, viewpoint and appearance changes, and query configurations. Our evaluation reveals a pronounced gap between proprietary and open-source VLMs. GPT-5.6-Sol achieves 36.07% accuracy, compared with 17.19% for Qwen3.8. To probe the potential of the VLM paradigm, we develop CorrVLM, a cross-view specialist trained with vocabulary repurposing and coarse-to-fine learning. Despite using substantially fewer image pairs than specialized visual-geometric models, CorrVLM matches their performance at moderate thresholds and surpasses GPT-5.6-Sol by 31.28 percentage points. These results demonstrate that strong fine-grained cross-view understanding is attainable within the VLM paradigm, yet remains largely unrealized in current general-purpose VLMs. AnyPoint-Bench offers a new lens for evaluating this capability and advancing fine-grained cross-view understanding in VLMs. Our benchmark, code, and models will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-View Understanding, Vision-Language Model, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_fAM1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how well VLMs can locate corresponding pixels across two views of the same scene. It introduces AnyPoint-Bench, which covers 12 datasets and evaluates models with 1 to 144 pixel queries in a single response. The authors also propose CorrVLM, which repurposes vocabulary entries as spatial tokens and uses coarse-to-fine training to learn correspondences. The experiments show that general-purpose VLMs struggle with this task, while CorrVLM substantially improves matching accuracy and achieves competitive results against dedicated visual matching models.

Critical strengths
  1. Asking models to return coordinates makes localization errors directly measurable. Testing multiple points also gives a broader picture of performance than checking a single selected point. The range of query counts makes it possible to examine how well models handle increasingly long lists of correspondences.

  2. The experiments cover 12 datasets with different scenes and imaging conditions, and include both general-purpose VLMs and dedicated matching methods. The “Same as Query” baseline is particularly useful: it shows when a seemingly reasonable score can come from simply copying the input coordinates, especially in image pairs with small displacements.

  3. CorrVLM produces substantial gains over the evaluated general-purpose VLMs. It also reaches 66.98% PCK@9px on AnyPoint-Full in the 144-query setting, compared with RoMa’s 64.79%. The same checkpoint handles query counts not used during training, which provides some evidence that it can generalize beyond the specific output lengths it was trained on.

Critical weaknesses
  1. The paper shows that fine-tuning improves a VLM’s cross-view pixel matching, but I’m still unsure what we gain by putting this ability inside a VLM. Table 3 shows that dedicated matchers such as RoMa already perform this task well. The current setup takes two images and source coordinates, so it does not show whether language understanding and precise matching can work together on a more useful task. For example, could the model follow a language instruction to identify a target and then locate it in another view? What would it offer over a VLM that calls an existing matching tool? I see the benchmark’s value for diagnosing model limitations, but the practical benefit of training CorrVLM is less clear.

  2. Both the training correspondences and the benchmark labels come from UFM, which raises a concern about how reliable the supervision and evaluation are. Section 3.1 filters benchmark labels using cycle consistency and epipolar geometry, while the training pipeline in Section 4 uses only cycle consistency. These checks can remove some errors, but an incorrect match can still pass them, particularly in regions with repeated textures. The paper does not report an independent accuracy check of the retained labels. Without that, it is hard to tell how much the scores reflect correct geometric matching and how much they reflect agreement with UFM’s predictions. This also makes the performance gaps between methods harder to interpret.

  3. The ablation in Figure 7 does not separate the effects of the two main design choices. The comparison with VLM3 changes vocabulary repurposing and coarse-to-fine training together, without showing what happens when either is used alone. The full approach clearly helps, but it is unclear whether the gain mainly comes from an easier coordinate representation or from learning correspondences in stages. Since the paper uses this comparison to explain the larger gains with more queries, separate ablations with matched training data and budgets would make that explanation more convincing.

Questions for authors

See the weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fAM1

We thank Reviewer fAM1 for the constructive feedback on the motivation, label reliability, and training design.

Q1. Motivation for learning pixel localization within a VLM

Thank you for this important question about the purpose of the task. Recently, frontier VLMs are increasingly used for spatial tasks such as vision-and-language navigation, where interpreting relationships across views is fundamental. AnyPoint-Bench evaluates the precision of this capability through coordinate-level correspondence. The benchmark (our AnyPoint-Bench) exposes a clear gap in the fine-grained cross-view capabilities of current frontier VLMs. We therefore investigate whether this gap reflects a fundamental constraint of vision-language learning. CorrVLM provides evidence that task-specific training can enable a VLM to achieve correspondence accuracy competitive with specialized vision-only matchers. We then hope AnyPoint-Bench will serve as a reference for assessing and advancing this capability in future frontier VLMs.

Q2. Reliability of UFM-derived labels

Thank you for emphasizing annotation reliability. Global Response G1 reports validation against UFM-independent reference labels from real 3D geometry and controlled homographies. It also evaluates no-match queries separately and counts rejected positive predictions as PCK failures. For scalable training-data generation, we omit camera-based epipolar verification so that image pairs without camera annotations can be used.

Q3. Contributions of vocabulary repurposing

Thank you for suggesting the additional ablation. G4 compares a VLM³-style baseline, VLM³-C2F, and CorrVLM, with the training controls stated explicitly. Adding the coarse stage uses the same image pairs and yields the largest observed correspondence gains. At the larger data scale, vocabulary repurposing gives comparable PCK@9px and improves no-match accuracy by 3.50/2.63 percentage points at Q4/Q8.

Reviewer_941vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • AnyPoint-Bench evaluates cross-view correspondence by asking VLMs to predict target coordinates for multiple source points in one response. It supports eight query cardinalities and uses UFM-generated labels filtered by cycle consistency and epipolar geometry.

  • CorrVLM combines vocabulary repurposing with coarse-to-fine training. It achieves 67.35% average PCK@9px on AnyPoint-4K and 66.98% in the 144-query setting on AnyPoint-Full, compared with RoMa’s 64.79%.

Critical strengths
  • The benchmark provides a unified interface for evaluating correspondence across query counts and diverse visual conditions. The coordinate-copying baseline and dataset breakdowns help distinguish matching ability from success due to small displacements.

  • CorrVLM substantially improves over the evaluated general-purpose VLMs and is competitive with specialized matchers. Its performance on query counts unseen during training supports the usefulness of the training approach.

Critical weaknesses
  • Global understanding is not sufficiently established. PCK measures individual point accuracy rather than consistency across correspondences, so independent predictions could also score well. The attention visualizations do not resolve this distinction. Comparisons across query counts also lack clearly matched image pairs and query sets, potentially confounding query-context benefits with sample difficulty.

  • Shared UFM supervision may bias evaluation. Both training supervision and benchmark labels originate from UFM. Cycle consistency and epipolar filtering improve reliability but cannot eliminate all incorrect correspondences. Independent ground-truth validation is needed to determine whether the reported gains reflect better physical accuracy or closer agreement with UFM, particularly for the 2.19-point advantage over RoMa.

Questions for authors
  • Clarification: Do comparisons across query counts use the same image pairs and shared evaluation points, particularly in Figure 6? If not, can you provide a controlled comparison that keeps scored points fixed while adding surrounding queries?

  • Validation: Have filtered UFM labels been checked against independent ground truth? Available label-error measurements and model rankings on such a subset would help assess annotation bias.

  • Clarification: How are sparse or semi-dense matcher outputs mapped to the grid-center queries in Table 3? Please specify interpolation or nearest-neighbor rules, confidence thresholds, and handling of missing matches.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 941v

We thank Reviewer 941v for the careful assessment and for suggesting comparisons that more directly test the interpretation and robustness of our results.

Q1. Label reliability under shared UFM supervision

Thank you for raising this potential source of bias. G1 validates the annotation pipeline against independent geometric and homography-derived references, while G2 evaluates VLM rankings directly on independent geometric ground truth. CorrVLM ranks first among the evaluated VLMs in all six dataset–query-count settings. Its advantage therefore cannot be explained solely by agreement with UFM labels.

Q2. Geometric consistency beyond PCK

Thank you for pointing out that per-point accuracy alone does not establish geometric consistency. Global Response G3 reports relative pose estimation from Q144 prediction sets, with AUC computed over all evaluation samples. CorrVLM leads at all five thresholds and reaches 32.22% AUC@10°, compared with 12.22% for the best frontier VLM. This supports the geometric consistency utility of its prediction sets.

Q3. Additional queries with identical evaluation points

Thank you for requesting this control. In Figure 6, AnyPoint-Full uses the same image pairs at Q16 and Q144, but each setting is scored on its own query set. AnyPoint-4K uses different pairs across query counts to broaden coverage. G2 adds a comparison with fixed image pairs and nested queries on two datasets with geometric ground truth independent of UFM. When only the original Q32 or Q64 points are scored, expanding to Q144 improves CorrVLM's PCK@9px by 19.31–25.14 percentage points.

Q4. Comparison with RoMa on independent ground truth

Thank you for suggesting this independent comparison. On ARKitScenes-Geo-4K under Q144, CorrVLM scores 92.31%, compared with 97.16% for RoMa and 84.21% for DKM. We use the official indoor checkpoints and inference settings for RoMa and DKM. Our main finding from AnyPoint-Bench is that current frontier VLMs struggle with fine-grained cross-view understanding. This finding led us to ask whether the limitation is inherent to the vision-language learning paradigm. Our results with CorrVLM show that, with task-specific training, a VLM can be competitive with specialized vision-only matchers on this task. We hope AnyPoint-Bench will help guide and measure future progress in fine-grained cross-view understanding for frontier VLMs.

Q5. Sparse matcher evaluation

Thank you for requesting clarification. G5 describes how sparse matcher are evaluated at the same 144 prescribed grid queries as the other methods. We index the aligned source features directly, without interpolation or nearest-neighbor assignment. A positive query without a returned match counts as a PCK failure.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Global Response (Reply 1/4)

We thank the reviewers for their constructive feedback and for recognizing the value of AnyPoint-Bench's diverse coverage and diagnostic analyses, as well as CorrVLM's strong correspondence performance. Their main concerns center on label reliability, the interpretation of multi-query behavior, and the training and evaluation protocols. We address the shared concerns below and discuss reviewer-specific questions in the individual responses.

G1. Motivation and Core Contributions

Recent work increasingly uses frontier VLMs for spatial tasks such as vision-and-language navigation [1–3]. Cross-view understanding is a basic capability for spatial understanding. Yet a comprehensive benchmark for fine-grained cross-view understanding across diverse scene domains is still missing.

We introduce AnyPoint-Bench to fill this gap. It evaluates VLMs through arbitrary-pixel localization across views with varying numbers of queries. Our evaluation reveals clear limitations in the fine-grained cross-view understanding of current frontier VLMs. We then ask whether this limitation is inherent to vision-language learning. CorrVLM shows that accurate multi-query localization is possible within this paradigm. With task-specific training, it achieves accuracy competitive with specialized vision-only matchers.

We use CorrVLM as a capability probe. Our goal is not to present it as a powerful new multi-query localization model. Its results reveal substantial room for current frontier VLMs to improve their fine-grained cross-view understanding. We hope AnyPoint-Bench and these findings will help guide and measure future improvements in frontier VLMs.


G2. Label reliability and validation against independent ground truth

Relevant reviewers — HGDB, 941v, and fAM1

Thank you for emphasizing this central concern. AnyPoint-Bench aims to cover diverse scene domains, including driving and aerial scenes where sparse or incomplete reconstructions can limit geometry-based correspondence labeling. We therefore use UFM, pretrained on 27.5M image pairs, followed by forward–backward cycle consistency and epipolar verification. To assess label reliability independently of the labeling model, we added three validation sets under the Q144 protocol.

ARKitScenes-Geo-4K and ScanNetPP-Geo-2K derive reference correspondences from metric scene geometry and calibrated cameras. They test real 3D viewpoint changes with depth-dependent parallax and visibility changes. AnyPoint-Homo-4K instead applies known homographies to AnyPoint-4K images, providing exact analytical correspondences under controlled projective warps. The homography set is a localization sanity check, while the geometry-derived sets provide evidence under real 3D viewpoint changes.

Table G1 evaluates label accuracy under Q144, with cumulative filtering stages. Multi-query localization is measured by PCK@3px and PCK@9px in the canonical 2000×20002000 \times 2000 coordinate space, while no-match accuracy measures the correct identification of queries without valid targets. Both cycle consistency and epipolar verification improve correspondence and no-match label accuracy, with all reported scores increasing.

Epipolar verification is particularly effective on the two geometry-derived datasets with real 3D viewpoint changes. Relative to cycle consistency alone, it improves PCK@3px by 10.44 percentage points on ScanNetPP-Geo-2K and 6.49 points on ARKitScenes-Geo-4K, with corresponding gains of 6.81 and 0.97 points in no-match accuracy. The homography set is already near saturation and shows smaller additional gains. Across the three sets, the full pipeline achieves macro-averaged scores of 94.82% PCK@3px, 99.68% PCK@9px, and 93.45% no-match accuracy.

Table G1. Accuracy of derived labels under Q144. Filtering stages are cumulative. No-match accuracy evaluates queries without targets. The full pipeline is shown in bold.

\hline \text{Dataset} & \text{Method} & \text{PCK@3px} & \text{PCK@9px} & \text{No-match} \\\\ \hline \text{ScanNetPP-Geo-2K} & \text{UFM} & 83.33 & 96.48 & 72.28 \\\\ & \text{+ cycle consistency} & 85.64 & 97.72 & 75.60 \\\\ & \text{+ epipolar verification} & \mathbf{96.08} & \mathbf{99.63} & \mathbf{82.41} \\\\ \hline \text{ARKitScenes-Geo-4K} & \text{UFM} & 81.50 & 98.29 & 96.19 \\\\ & \text{+ cycle consistency} & 81.95 & 98.61 & 97.11 \\\\ & \text{+ epipolar verification} & \mathbf{88.44} & \mathbf{99.40} & \mathbf{98.08} \\\\ \hline \text{AnyPoint-Homo-4K} & \text{UFM} & 99.86 & 99.99 & 99.74 \\\\ & \text{+ cycle consistency} & 99.89 & 100.00 & 99.85 \\\\ & \text{+ epipolar verification} & \mathbf{99.95} & \mathbf{100.00} & \mathbf{99.86} \\\\ \hline \end{array}$$ --- Our global response is provided in Reply 1/4.
AuthorsOpenReview ↗

Global Response (Reply 2/4)

G3. Query count and model rankings with fixed image pairs

Relevant reviewers — HGDB and 941v

Thank you for requesting a comparison that controls both the images and the evaluated points. AnyPoint-4K prioritizes broad scene coverage within a practical evaluation budget, and its query-count settings use different image pairs. We therefore added ARKitScenes-Geo-100 and ScanNetPP-Geo-100, each containing 100 scene-balanced image pairs with geometry-derived ground truth independent of UFM. Within each dataset, Q32, Q64, and Q144 share the same image pairs and use nested query sets, Q32⊂Q64⊂Q144Q_{32} \subset Q_{64} \subset Q_{144}.

This protocol supports two distinct comparisons. First, Table G3 isolates the effect of additional query context by scoring the same original 32 or 64 points before and after expansion to Q144. CorrVLM gains 19.31–25.14 percentage points in PCK@9px, with improvements also observed at 3px and 5px in all four dataset–expansion settings. These gains suggest that CorrVLM uses additional queries as context to refine the original correspondences, consistent with interactions among queries rather than fully independent predictions. The evaluated frontier VLMs show no consistent improvement across query counts in the full-query scores (Table G2). Together with CorrVLM's stronger overall accuracy, these findings highlight a substantial gap in joint multi-query correspondence prediction. They provide behavioral evidence of the benefit of query context without establishing a specific internal reasoning mechanism.

Second, Table G2 compares model rankings against UFM-independent geometric ground truth. CorrVLM ranks first among the evaluated VLMs in all six dataset–query-count settings. At Q144, it achieves 64.65% on ScanNetPP-Geo-100 and 89.68% on ARKitScenes-Geo-100, compared with 24.80% and 45.94% for the best frontier VLM, respectively. Its advantage therefore persists under independent reference labels. The relative ordering of frontier VLMs varies between the two datasets, further motivating evaluation across diverse scene domains.

Full construction and evaluation details are provided in the Appendix.

Table G2. Query accuracy on independent geometric ground truth. We report PCK@9px (%) on the same 100 image pairs across query counts within each dataset. †\dagger and ‡\ddagger denote low- and medium-effort inference, respectively. The best result in each setting is shown in bold.

\hline \text{Dataset} & \text{Method} & \text{Q32} & \text{Q64} & \text{Q144} \\\\ \hline \text{ScanNetPP-Geo-100} & \text{Gemini-3.8-Flash}^{\dagger} & 1.67 & 1.08 & 0.95 \\\\ & \text{Gemini-3.8-Flash}^{\ddagger} & 22.22 & 20.54 & 16.35 \\\\ & \text{GPT-5.6-Sol}^{\dagger} & 24.54 & 18.97 & 21.65 \\\\ & \text{GPT-5.6-Sol}^{\ddagger} & 28.86 & 28.96 & 24.80 \\\\ & \text{CorrVLM (Ours)} & \mathbf{46.28} & \mathbf{42.13} & \mathbf{64.65} \\\\ \hline \text{ARKitScenes-Geo-100} & \text{Gemini-3.8-Flash}^{\dagger} & 8.10 & 8.72 & 9.24 \\\\ & \text{Gemini-3.8-Flash}^{\ddagger} & 42.69 & 41.80 & 37.24 \\\\ & \text{GPT-5.6-Sol}^{\dagger} & 44.39 & 37.53 & 45.94 \\\\ & \text{GPT-5.6-Sol}^{\ddagger} & 41.54 & 41.83 & 42.04 \\\\ & \text{CorrVLM (Ours)} & \mathbf{64.81} & \mathbf{64.72} & \mathbf{89.68} \\\\ \hline \end{array}$$ **Table G3. CorrVLM gains on identical query points.** When expanding Q32 or Q64 to Q144, we evaluate only the original 32 or 64 query points in both settings. Values are absolute PCK increases in percentage points at the indicated pixel thresholds. $$\begin{array}{llrrr} \hline \text{Dataset} & \text{Expansion} & \Delta\text{PCK@3px} & \Delta\text{PCK@5px} & \Delta\text{PCK@9px} \\\\ \hline \text{ScanNetPP-Geo-100} & Q_{32}\to Q_{144} & +9.12 & +15.75 & +19.31 \\\\ & Q_{64}\to Q_{144} & +11.52 & +18.52 & +22.38 \\\\ \hline \text{ARKitScenes-Geo-100} & Q_{32}\to Q_{144} & +27.55 & +30.86 & +24.70 \\\\ & Q_{64}\to Q_{144} & +27.25 & +29.82 & +25.14 \\\\ \hline \end{array}$$ --- Our global response is provided in Reply 2/4.
AuthorsOpenReview ↗

Global Response (Reply 3/4)

G4. Geometric evaluation beyond per-point PCK

Relevant reviewers — HGDB and 941v

Thank you for highlighting the distinction between point accuracy and geometric consistency. We added a downstream geometric evaluation on AnyPoint-4K's wide-baseline two-view pairs under Q144. Each model predicts the query correspondences in one response, and we pass the resulting matches to MAGSAC++ for relative camera pose estimation. Pose-error AUC is computed over all evaluation samples.

CorrVLM achieves the highest pose AUC at all five thresholds, scoring 18.55 / 32.22 / 41.01 / 47.03 / 51.26 at 5° / 10° / 15° / 20° / 25°, respectively. Among the evaluated frontier VLMs, increasing thinking effort consistently improves pose estimation for both GPT and Gemini, with GPT-medium performing best. At 10°, GPT-medium achieves 12.22% AUC, compared with 32.22% for CorrVLM. These results provide quantitative evidence that CorrVLM's correspondence sets support a coherent global two-view geometry, beyond individual point accuracy. Because MAGSAC++ estimates pose after robust outlier rejection, this evaluation measures the geometric coherence supported by the predicted sets without requiring every match to be an inlier.

Table G4. Relative camera pose estimation on AnyPoint-4K under Q144. We report pose-error AUC (%) over all evaluation samples, with higher values indicating better performance. †\dagger and ‡\ddagger denote low- and medium-effort inference, respectively. The best result at each threshold is shown in bold.

\hline \text{Method} & \text{AUC@}5^{\circ} & \text{AUC@}10^{\circ} & \text{AUC@}15^{\circ} & \text{AUC@}20^{\circ} & \text{AUC@}25^{\circ} \\\\ \hline \text{Gemini-3.8-Flash}^{\dagger} & 0.19 & 0.59 & 1.04 & 1.45 & 2.14 \\\\ \text{Gemini-3.8-Flash}^{\ddagger} & 4.63 & 8.14 & 10.67 & 12.93 & 15.16 \\\\ \text{GPT-5.6-Sol}^{\dagger} & 3.29 & 7.12 & 9.90 & 12.18 & 14.12 \\\\ \text{GPT-5.6-Sol}^{\ddagger} & 6.93 & 12.22 & 15.79 & 18.68 & 21.22 \\\\ \hline \text{CorrVLM (Ours)} & \mathbf{18.55} & \mathbf{32.22} & \mathbf{41.01} & \mathbf{47.03} & \mathbf{51.26} \\\\ \hline \end{array}$$ --- Our global response is provided in Reply 3/4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.