acceptodds
Under review as a conference paper at ICLR 2027

CVGL-CueBench: A Benchmark for Visual-Cue Dependence and Representation Transfer in Cross-View Geo-Localization

Abstract

Cross-view geo-localization (CVGL) models achieve high retrieval recall on benchmark RGB ground queries. However, standard RGB images intrinsically couple surface appearance with spatial geometric layouts. Evaluating RGB inputs alone obscures which visual cues actually drive alignment, as well as how sensitively models rely on each cue. To systematically diagnose visual cue reliance and model sensitivity, we introduce CVGL-CueBench, a multi-representation diagnostic benchmark comprising two sub-benchmarks, CVUSA-Cue and CVACT-Cue, derived from CVUSA and CVACT. While retaining ground–aerial spatial correspondences and the invariant RGB aerial gallery, CVGL-CueBench provides five parallel query representations: the original RGB image alongside four derived structural representations (class-agnostic region segmentation, estimated relative depth, edge maps, and sketches). We establish a systematic diagnostic protocol evaluating five representative CVGL methods across representation-matched training () to measure localization capacity preserved within each visual cue, and cross-representation transfer () to assess sensitivity to representation shifts, complemented by test-time spatial perturbations on an RGB baseline to examine reliance on high-frequency textures, spatial layout, and regional context. Evaluations demonstrate that derived structural representations retain sufficient geometric cues to support high retrieval recall when paired with capable backbones: on north-aligned CVUSA-Cue, Sample4Geo achieves 90.13%–98.68% Recall@1 across all query representations. However, cross-representation transfer () reveals pronounced sensitivity, varying from 68.12% on sketches to a near-total collapse of 0.06% on relative depth queries. Furthermore, under a field of view on CVACT-Cue, models trained on sketch queries surpass their RGB counterparts across all five evaluated methods by 0.96–15.48 percentage points, though this ranking advantage varies across methods on CVUSA-Cue. These findings demonstrate that high RGB recall masks acute sensitivity to representation shifts, while confirming that structural representations retain sufficient geometric cues to support cross-view alignment, particularly under restricted viewing constraints. CVGL-CueBench thus provides a controlled diagnostic foundation to decouple visual cues and evaluate structural robustness in cross-view geo-localization.

Primary area:
datasets and benchmarks
Keywords:
Cross-view geo-localization, Visual cue analysis, Multi-representation benchmarking, Cross-representation transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9C3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CVGL-CueBench, a benchmark built upon the existing CVUSA and CVACT datasets to investigate visual cue dependence and representation transfer in cross-view geo-localization. The authors generate four additional query representations (sketch, edge, depth, and segmentation) using existing image processing and pretrained vision models, while keeping the aerial reference images unchanged. They evaluate five representative CVGL methods under representation-matched training and different viewing conditions, and further analyze cross-representation transfer and spatial perturbations using Sample4Geo.

Critical strengths
  • The paper provides numerious comparison of multiple query representations across several CVGL methods, datasets, and viewing conditions.
Critical weaknesses
  1. Insufficient motivation for cross-representation transfer. The paper does not convincingly explain why a model trained on one query representation should be expected to handle substantially different representations without adaptation. The practical necessity of this setting remains unclear.

    Although Appendix D provides practical motivations for using structural ground queries with a fixed RGB aerial gallery, these arguments do not adequately justify the necessity of zero-shot transfer between arbitrarily selected ground-query representations. The two settings should not be conflated. Besides, all additional query representations are artificially generated from RGB images rather than obtained from real-world heterogeneous sensors. It is unclear whether these synthetic representation shifts correspond to meaningful deployment challenges in CVGL.

  2. Limited novelty in benchmark construction. The benchmark largely applies existing off-the-shelf methods (Canny, SAM, Depth Anything V2, and Qwen-Image-Edit) to established datasets (CVUSA and CVACT). It fundamentally introduces neither new raw data nor new evaluation methodologies. Every user of the existing datasets can easily handle them by themself.

  3. Limited and largely predictable insights. The main findings, such as the sufficiency of structural information for localization and the degradation caused by large input distribution shifts, are largely expected or consistent with prior knowledge. The paper provides limited new understanding beyond these empirical observations.

  4. Insufficient evidence for visual cue dependence. Comparing performance across different generated representations does not necessarily isolate the contributions of individual visual cues. The results are confounded by differences in representation quality, preprocessing pipelines, pretrained model priors, and the color mapping in the visualizations.

  5. Unclear source of the reported sketch advantage. Sketches are generated using a powerful pretrained generative model, potentially introducing additional structural enhancement. Moreover, representations are generated from full panoramas before FoV cropping. Without appropriate controls, the improvements cannot be confidently attributed to structural information alone.

  6. Limited conclusions. Despite extensive experiments, the paper provides few concrete insights into how the observed representation sensitivities can be addressed. The analysis does not establish generalizable principles for improving CVGL robustness or representation learning.

Overall, I'm tending giving a typical weak reject below borderline reject.

Questions for authors

See the weaknesses

11 comments
AuthorsOpenReview ↗

Response to Reviewer 9C3F

Thank you for taking the time to provide us with a thorough review. We address the specific weaknesses and questions next.

Response to Weakness

Q1: Regarding the motivation for cross-representation transfer (A→BA \rightarrow B), the role of Appendix D, and the use of RGB-derived representations.

A1: We thank the reviewer for this comment and address the three points below:

  1. Why evaluate cross-representation transfer (A→BA \rightarrow B)? We would like to clarify that A→BA \rightarrow B transfer is designed as a diagnostic stress test to uncover what visual features a model actually relies on after training, rather than a claim that arbitrary representation pairs should be interchangeable without adaptation in deployment. Specifically, A→AA \rightarrow A and A→BA \rightarrow B answer two distinct diagnostic questions: representation-matched training (A→AA \rightarrow A) measures whether an isolated visual cue contains sufficient spatial information to support localization when trained in-domain, whereas cross-representation transfer (A→BA \rightarrow B) reveals whether a trained model has learned underlying spatial geometry or overfitted to representation-specific appearance shortcuts (Geirhos et al., ICLR 2019). A→AA \rightarrow A alone cannot expose shortcut learning, whereas the directional asymmetry in A→BA \rightarrow B (Figure 2a and Lines 423–433) directly reveals it: on CVUSA-Cue, an RGB-trained model drops sharply when tested on structural cues (RGB→Sketch\text{RGB} \rightarrow \text{Sketch} drops by 30.56 percentage points to 68.12% R@1, RGB→Edge\text{RGB} \rightarrow \text{Edge} to 10.77%, and RGB→Depth\text{RGB} \rightarrow \text{Depth} to 0.06%), showing that standard RGB training relies heavily on surface color and texture shortcuts. Conversely, models trained on structural contours transfer strongly back to standard RGB queries without any adaptation (Sketch→RGB\text{Sketch} \rightarrow \text{RGB} achieves 93.25% R@1 and Edge→RGB\text{Edge} \rightarrow \text{RGB} achieves 81.51%), demonstrating that training on clean structural outlines guides the backbone to encode cross-view spatial geometry that persists in natural RGB images.

    2. Clarifying the scope of Section 3.1 and Appendix D: We would like to clarify that Section 3.1 (Lines 206–209) and Appendix D (Lines 860–917) were to explain why we vary the ground query while holding the aerial gallery fixed in RGB—both because practical geo-localization systems query pre-existing RGB satellite repositories and because fixing the aerial reference ensures that performance variations come from the ground query side (we also evaluate converting the aerial gallery to Seg in Table R4 in our response to Reviewer f97p, Q2). We will refine the wording in Section 3.1 and Appendix D to ensure their scope is unambiguous.

    3. Why representations are derived from RGB and their relevance to practical CVGL: CVGL-CueBench is designed as a controlled diagnostic benchmark (similar in spirit to ImageNet-C) rather than a multi-sensor hardware benchmark. Representations such as contours (Sketch), gradients (Edge), and regional partitions (Seg) represent structural abstractions of a visual scene rather than separate hardware sensor modalities. Deriving them from the same geo-registered RGB images keeps the underlying geographic location, camera viewpoint, and scene content consistent across all five representations, while preserving the original ground–aerial correspondences. Furthermore, evaluating these RGB-derived representations provides practical value for real-world RGB localization: first, it diagnoses why standard RGB-trained models are vulnerable to appearance variations by exposing their reliance on color and texture shortcuts (Figure 2a); second, under restricted fields of view (FoV=70∘\text{FoV}=70^\circ) in dense urban scenes (within our standardized full-panorama extraction and online-cropping protocol, Table 2), suppressing street-level shadow and texture clutter in favor of continuous structural outlines (Sketch) outperforms raw RGB, while combining complementary structural representations (Table R1 in Reviewer f97p, Q2) substantially mitigates single-cue fragility.

AuthorsOpenReview ↗

Response to Reviewer 9C3F

Q2: Regarding the novelty of benchmark construction, evaluation methodology, and why a standardized benchmark is needed.

A2: We thank the reviewer for this comment and clarify the design rationale, evaluation methodology, and practical necessity of CVGL-CueBench across the three points raised:

1. Why apply off-the-shelf extractors to established datasets, and how we advance beyond prior benchmarks? First, applying mature, off-the-shelf operators or foundation models to established datasets is a widely adopted paradigm for constructing controlled diagnostic benchmarks. For instance, ImageNet-C [1] applies 15 standard filters to ImageNet to evaluate corruption robustness; Background Challenge [2] uses off-the-shelf segmentation masks to study foreground vs. background reliance; and the CVGL Robustness Benchmark [3] applies standard pixel corruptions to CVUSA and CVACT. This paradigm keeps ground-truth correspondences strictly fixed while using validated extractors to isolate target visual factors without introducing artifacts from custom, unverified generators. Second, CVGL-CueBench advances both the representation decomposition and diagnostic scope: whereas prior CVGL benchmarks [3] apply low-level pixel corruptions (e.g., blur, noise, weather) to RGB images—where appearance and geometry remain coupled—we combine foundation models (SAM, Depth Anything V2, Qwen-Image-Edit) and classical operators (Canny) to construct four derived structural representations (Sketch, Edge, Depth, Seg) alongside RGB. This shifts evaluation from pixel-level corruption tolerance to diagnosing which visual cues drive ground-to-aerial alignment.

2. Systematic diagnostic evaluation protocol: Conventional CVGL evaluations test RGB queries alone, coupling surface appearance (color, texture) with spatial geometry (contours, layout, depth) and obscuring which cues models rely on. Instead, we establish a systematic diagnostic protocol spanning 250 primary configurations (5 methods×5 representations×5 viewing conditions×2 datasets5 \text{ methods} \times 5 \text{ representations} \times 5 \text{ viewing conditions} \times 2 \text{ datasets}): (i) Representation-matched training (A→AA \rightarrow A): training and testing on the same representation to measure standalone localization capacity and multi-cue synergy (Table R1, Reviewer f97p W-Q2); (ii) Cross-representation transfer (A→BA \rightarrow B): evaluating models trained on representation AA directly on BB without fine-tuning (Figure 2a), exposing whether models rely on transferable spatial geometry or appearance shortcuts; (iii) Viewing conditions and test-time spatial perturbations: examining how cue reliance shifts from north-aligned panoramas to unknown orientations and restricted FoVs (180∘,90∘,70∘180^\circ, 90^\circ, 70^\circ), alongside test-time spatial perturbations (Gaussian blur, patch shuffling, region occlusion). This joint protocol uncovers empirical findings—directional transfer asymmetry, Sketch outperforming RGB at 70∘70^\circ FoV (under our panoramic-extraction protocol), and architectural sensitivity to Edge vs. Sketch—that standard RGB evaluation cannot observe.

3. Why a precomputed, standardized benchmark cannot be replaced by ad-hoc user processing: Providing a precomputed benchmark is essential for computational accessibility and fair cross-paper comparison. Generating over 355,000 high-resolution derived images across 88,832 panoramas (plus our VIGOR extension, Table R3, Reviewer f97p W-Q5) using heavy foundation models (Qwen-Image-Edit-2511, ViT-H SAM) and running 250 benchmark evaluations requires substantial GPU compute and careful calibration (e.g., structure-preserving prompts against hallucination and deterministic cyclic color mapping, Appendix E). Moreover, ad-hoc generation with varying prompts, Canny thresholds, depth normalizations, or checkpoints would make cross-paper results inconsistent and non-comparable. Releasing CVGL-CueBench with fixed splits, unified representations, and baseline checkpoints establishes a reproducible standard for the community.

References:

[1] Hendrycks D, Dietterich T. Benchmarking neural network robustness to common corruptions and perturbations[J]. ICLR, 2019.

[2] Xiao K, Engstrom L, Ilyas A, et al. Noise or signal: The role of image backgrounds in object recognition[J]. ICLR, 2021.

[3] Zhang Q, Zhu Y. Benchmarking the robustness of cross-view geo-localization models[C]. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 36-53.

AuthorsOpenReview ↗

Response to Reviewer 9C3F

Q3: Regarding the novelty of our empirical insights beyond prior expectations.

A3: We thank the reviewer for this comment. While high-level domain-shift or structural intuition may seem familiar, it cannot explain four counter-intuitive findings revealed by our benchmark:

First, standard domain-shift intuition cannot explain the strong directional asymmetry in cross-representation transfer (A→BA \rightarrow B). A simple distribution gap does not explain why transferring between the exact same representation pair differs by 25–70 percentage points (pp) by direction. On CVUSA-Cue (Figure 2a, Lines 423–433), an RGB-trained model drops to 68.12% R@1 on Sketch and 10.77% on Edge, whereas Sketch- or Edge-trained models achieve 93.25% and 81.51% R@1 directly on unmodified RGB queries without fine-tuning. Thus, the bottleneck is not the domain gap itself: standard RGB training encourages reliance on color and texture shortcuts, whereas contour training guides the network to encode spatial geometry that remains directly effective on natural RGB images.

Second, discarding appearance information in Sketch outperforms raw RGB under a restricted 70∘70^\circ FoV within our standardized panoramic-extraction and online-cropping protocol. While RGB performs best on 360∘360^\circ panoramas (Table 1), the ranking reverses under a 70∘70^\circ FoV on urban CVACT-Cue (Table 2, Lines 362–372): Sketch surpasses raw RGB across all five evaluated methods (e.g., improving R@1 from 18.37% to 22.43% on Sample4Geo, 24.60% to 33.56% on ConGeo, and 22.61% to 38.09% on SinGeo). When wide panoramic context is lost in urban scenes, transient street shadows and facade textures in RGB act as distractions, whereas suppressing them in favor of continuous object outlines improves matching (source and boundary conditions are examined in Q5 below).

Third, structural cues are not uniformly sufficient; our benchmark exposes sharp divergences across viewing conditions and architectures. Comparing four derived structural cues reveals distinct boundary conditions: (1) although Depth, Seg, and Edge achieve high recall on 360∘360^\circ panoramas (Table 1), they drop to 1.21%–13.57% R@1 at 70∘70^\circ FoV on CVACT-Cue (Table 2), showing that depth ordering, region partitioning, and raw pixel gradients rely heavily on wide panoramic context, whereas continuous contours (Sketch) remain resilient under this protocol; and (2) methods with near-identical R@1 on RGB (84.95% vs. 85.43%) and Sketch (79.51% vs. 77.74%)—TransGeo and GeoDTR on CVACT-Cue (Table 1)—diverge by 31.65 pp on fragmented Edge maps (45.21% vs. 76.86%), exposing how differently retrieval architectures respond to high-frequency gradient clutter versus continuous boundaries.

Fourth, prompted by Reviewers f97p and hc3e, our new rebuttal experiments uncover three additional insights: (1) multi-cue structural synergy under narrow FoVs (Table R1, Reviewer f97p W-Q2): although individual non-Sketch cues degrade under unknown orientation and FoV=70∘\text{FoV}=70^\circ on CVUSA-Cue (7.32%–10.61% R@1), combining complementary structural cues consistently boosts retrieval, with Sketch + Depth + Seg + Edge reaching 28.58% R@1 and 90.03% R@1% (+5.72 to +21.26 pp over single cues); (2) asymmetric impact of ground vs. aerial structural conversion (Table R4, Reviewer f97p Q2): converting the aerial gallery to segmentation (RGB–Seg at 80.47% R@1, Seg–Seg at 62.16%) shows that 2D regional layouts support matching in both views, while retaining aerial RGB textures (Seg–RGB at 91.72%) substantially aids fine-grained alignment; and (3) generalization to metropolitan canyons and one-to-many matching (Tables R6–R7, Reviewer hc3e W-Q2 & Q2): evaluations on VIGOR (Chicago) confirm that both representation-matched trends (Sketch at 67.66% vs. RGB at 69.47% R@1) and directional transfer asymmetry hold under severe high-rise occlusions and non-central matching.

While our main quantitative results are reported in the manuscript's tables and figures, we agree that these counter-intuitive takeaways and the newly added findings should be highlighted more prominently. We thank the reviewer for this feedback and will sharpen the takeaway summaries in the revised manuscript.

Reviewer_hc3eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CVGL-CueBench, a benchmark designed to investigate visual cue dependence and representation transfer in cross-view geo-localization. Built upon CVUSA and CVACT, the benchmark provides five query representations: RGB, Sketch, Edge, Depth, and Segmentation, while keeping the aerial RGB reference gallery unchanged.

The authors evaluate five representative CVGL methods under different viewing conditions through representation-matched training (A → A) and cross-representation transfer (A → B).

Critical strengths

The paper addresses an important limitation of existing CVGL evaluations by systematically investigating visual cue dependence and representation transfer beyond standard RGB inputs. The benchmark covers five query representations, five representative CVGL methods, and multiple viewing conditions across CVUSA and CVACT. This comprehensive evaluation provides a valuable foundation for understanding model robustness and generalization.

Critical weaknesses
  1. Potential Bias in the Restricted FoV Evaluation Protocol. The structural representations (Sketch, Depth, Segmentation, and Edge) are generated from complete 360° panoramic images before being cropped to restricted FoVs. Consequently, the representation generation process has access to visual context outside the final query's field of view, particularly for foundation-model-generated representations such as Sketch. This may introduce a potential bias in the comparison with RGB queries under restricted FoV settings. It would be valuable to investigate whether similar conclusions hold when the original RGB images are cropped before generating the structural representations.

  2. Limited Dataset Scale and Diversity. Although CVGL-CueBench contains 444,160 query representations, these are derived from only two existing datasets, CVUSA and CVACT, rather than independently collected images. The number and diversity of unique scenes are therefore relatively limited. This raises concerns about whether the observed visual cue dependencies and representation transfer behaviors generalize to more diverse environments, geographic regions, and real-world imaging conditions. Evaluations on additional datasets would further strengthen the generalizability of the benchmark's findings.

Questions for authors
  1. Restricted FoV Evaluation Protocol: Since the structural representations are generated from complete 360° panoramas before FoV cropping, could the authors clarify whether this introduces additional contextual information into the generated representations? More importantly, can the authors compare the current pipeline with an alternative setting where RGB images are cropped before generating structural representations, particularly for Sketch under the 70° FoV setting? This would help verify the reliability of the main experimental findings.

  2. Dataset Scale and Generalizability: Since CVGL-CueBench is constructed from only two existing datasets, CVUSA and CVACT, could the authors provide additional evidence that the observed visual cue dependencies and representation transfer behaviors generalize to more diverse geographic regions and scene types? For example, would the main findings remain consistent when evaluated on an additional independent cross-view geo-localization dataset?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hc3e

Thank you for taking the time to provide us with a thorough review. We address the specific weaknesses and questions next.

Response to Weakness

Q1: Regarding the cropping order in the restricted FoV evaluation protocol (Generate-then-Crop vs. Crop-then-Generate).

A1: Thank you for highlighting the distinction between the extractor's field of view and the retrieval model's field of view. As documented in Section 3.4 (lines 250–260), our main protocol evaluates restricted views of representations precomputed from full 360∘360^\circ panoramas so that dynamic online azimuth augmentation (θ∼U(0∘,359∘)\theta \sim \mathcal{U}(0^\circ, 359^\circ)) can be applied without prohibitive on-the-fly foundation-model inference, while holding upstream extraction quality constant across FoVs (360∘→70∘360^\circ \rightarrow 70^\circ). Under this protocol, Depth, Seg, and Edge drop sharply at FoV=70∘\text{FoV}=70^\circ (1.21%–13.57% R@1 on CVACT-Cue) and remain consistently below the RGB baseline (18.37% R@1) across all restricted-FoV settings (70∘,90∘70^\circ, 90^\circ, and 180∘180^\circ in Table 2 and Appendix A Tables 4–5); thus, even when extracted from full panoramas (which involves global operations such as depth min–max normalization, full-image mask assignment, or global edge thresholding), they do not outperform RGB under restricted views. In contrast, Sketch surpasses RGB across all five evaluated methods under our panorama-conditioned protocol (achieving 22.43% vs. 18.37% R@1 on Sample4Geo at FoV=70∘\text{FoV}=70^\circ), making it the critical case where extractor field of view could affect the relative comparison with RGB.

To examine the effect of cropping before generation, Table R5 additionally evaluates offline generation from a single fixed 70∘70^\circ crop per location on CVACT-Cue using Sample4Geo (with prompts requesting strict structure preservation without object alteration). Under this fixed-crop setting, Crop-then-Generate Sketch drops to 2.00% R@1, below the corresponding fixed-crop RGB baseline (6.46% R@1), and the non-generative local filter XDoG exhibits a similarly low score (1.25% R@1). Two practical factors make this single-crop setting substantially more challenging: (1) each location is exposed from only one static 70∘70^\circ sector (19.4%19.4\% of the horizon) rather than dynamic online random azimuths, drastically reducing training-view diversity; and (2) cropping a 70∘70^\circ sector from a 2:12:1 equirectangular panorama yields an extreme vertical slice (aspect ratio ≈1:2.57\approx 1:2.57), introducing horizontal stretching or blank-padding boundary artifacts during extraction. Because these factors change simultaneously with the extractor's field of view, this control does not isolate the contribution of full-panorama context. Similarly, while the low XDoG result indicates that fixed narrow-slice extraction is challenging even without learned generative priors, it does not rule out contextual effects in Qwen-Image-Edit-2511.

We therefore interpret the 70∘70^\circ Sketch advantage as specific to panorama-conditioned extraction under our standardized online-cropping protocol, rather than claiming it holds for independently acquired narrow-FoV inputs without further controlled evaluation. The benchmark remains useful for comparing representations under this explicitly defined protocol (answering how localized representations support retrieval under a fixed panoramic extraction pipeline). Complementing this restricted-FoV analysis, a related ranking reversal also occurs without any FoV truncation: with complete 360∘360^\circ queries and unknown orientation on CVACT-Cue (Appendix A, Table 6, where Generate-then-Crop and Crop-then-Generate are identical), Sketch still exceeds RGB across all five evaluated methods by 0.87–6.71 percentage points (and by +7.47 pp on TransGeo on CVUSA-Cue). This confirms that representation rankings depend on the viewing protocol independently of narrow-crop context exclusion.

Table R5: Comparison of generation-cropping order for Sketch queries on CVACT-Cue with FoV=70∘\text{FoV}=70^\circ using Sample4Geo.

Query Representation & ProtocolR@1R@5R@10R@1%
RGB (Online Random Crop)18.3738.8848.9878.55
Sketch (Generate-then-Crop, Online Random Crop)22.4349.2761.5189.80
RGB (Fixed Per-Image Crop)6.4618.7026.8160.81
Sketch (Crop-then-Generate, Fixed Per-Image Crop)2.005.648.7123.05
XDoG (Crop-then-Extract, Fixed Per-Image Crop)1.253.945.7216.32
AuthorsOpenReview ↗

Response to Reviewer hc3e

Q2: Regarding dataset scale, scene diversity, and evaluation on additional benchmarks.

A2: We sincerely thank the reviewer for this constructive suggestion. While our initial submission systematically evaluated 5 representations across 5 viewing conditions (360∘360^\circ–70∘70^\circ FoVs) on two geographically distinct benchmarks—rural/suburban roadways across the US (CVUSA) and urban/residential scenes in Canberra, Australia (CVACT)—we agree that extending our evaluation to an additional dense metropolitan dataset further strengthens the empirical support of our findings.

To expand scene diversity and matching conditions, we extended our representation generation and evaluation pipeline to the VIGOR dataset (Chicago subset, official same-area split) using Sample4Geo (Table R6). Specifically, we evaluate full 360∘360^\circ queries trained independently from scratch under representation-matched training (A→AA \rightarrow A). Unlike the one-to-one center-aligned pairs in CVUSA and CVACT, VIGOR captures dense high-rise metropolitan street canyons with severe building occlusions and adopts a realistic one-to-many non-central matching protocol, where ground queries have arbitrary spatial offsets relative to aerial patches. As shown in Table R6, our findings generalize well to dense metropolitan canyons and off-center matching: (1) all four derived structural representations preserve strong localization capability without color or surface texture, achieving 53.58%–67.66% R@1 and 96.45%–98.71% R@1%; (2) Sketch (67.66% R@1, 77.78% Hit Rate) closely approaches raw RGB (69.47% R@1, within 1.81 percentage points) and remains the strongest structural representation, followed by Edge (62.19%), Depth (59.40%), and Seg (53.58%), as fine-grained structural contours provide more precise spatial anchoring for non-central matching in high-rise street canyons. Together, our evaluations across three distinct benchmarks—spanning open rural roadways (CVUSA), suburban/residential scenes (CVACT), and high-rise metropolitan canyons with off-center matching (VIGOR)—demonstrate that our visual-cue findings replicate consistently across diverse geographic regions, scene layouts, and matching protocols under same-area evaluation.

Table R6: Retrieval performance of query representations on VIGOR (Chicago) under the same-area setting using Sample4Geo.

Query RepresentationR@1R@5R@10R@1%Hit Rate
RGB69.4791.8594.7099.0881.35
Sketch67.6689.9393.0398.7177.78
Edge62.1986.6091.1098.0471.75
Depth59.4082.5487.6097.0966.25
Seg53.5878.3884.6296.4560.17
AuthorsOpenReview ↗

Response to Reviewer hc3e

Response to Questions

Q1: Regarding whether generating representations before FoV cropping introduces external context, and comparing with cropping RGB before generation (particularly for Sketch at 70∘70^\circ FoV).

A1: We thank the reviewer for this question. As detailed in our response to Q1 in Response to Weaknesses above, we distinguish between the extractor's field of view and the retrieval model's field of view:

  1. Restricted-FoV protocol and Crop-then-Generate boundary: In our main protocol, representations are precomputed from full 360∘360^\circ panoramas so that dynamic online azimuth augmentation (θ∼U(0∘,359∘)\theta \sim \mathcal{U}(0^\circ, 359^\circ)) can be applied consistently across FoVs (360∘→70∘360^\circ \rightarrow 70^\circ). Under this setting, Depth, Seg, and Edge still drop well below RGB at FoV=70∘\text{FoV}=70^\circ (1.21%–13.57% vs. 18.37% R@1, Table 2). For Sketch, while it surpasses RGB across all five methods under our panorama-conditioned protocol (22.43% vs. 18.37% R@1 on Sample4Geo), switching to offline single-crop Crop-then-Generate reduces Sketch to 2.00% R@1 (below fixed-crop RGB at 6.46% R@1, Table R5), with the non-generative local filter XDoG exhibiting a similarly low score (1.25% R@1). Two practical factors make this single-crop setting substantially more challenging: (i) each location is exposed from only one static 70∘70^\circ sector (19.4%19.4\% of the horizon) rather than dynamic online random azimuths, drastically reducing training-view diversity; and (ii) cropping a 70∘70^\circ sector from a 2:12:1 equirectangular panorama yields an extreme vertical slice (aspect ratio ≈1:2.57\approx 1:2.57), introducing horizontal stretching or blank-padding boundary artifacts during extraction. Because these factors change simultaneously with the extractor's field of view, this control does not isolate the role of full-panorama context or rule out contextual effects in Qwen-Image-Edit-2511. Accordingly, we explicitly scope Sketch's 70∘70^\circ advantage as specific to panorama-conditioned extraction, noting that independently acquired narrow-FoV inputs require further controlled evaluation.

  2. Supplementary 360∘360^\circ uncropped evidence: Complementing the restricted-FoV analysis, on complete 360∘360^\circ queries with unknown orientation (Appendix A, Table 6, where no FoV cropping is applied and Generate-then-Crop is identical to Crop-then-Generate), Sketch still outperforms RGB across all five evaluated methods on CVACT-Cue (+0.87 to +6.71 pp in R@1; and by +7.47 pp on TransGeo on CVUSA-Cue). This confirms that the ranking reversal between Sketch and RGB is a genuine effect of the viewing protocol itself, rather than an artifact of out-of-view context from narrow cropping.

AuthorsOpenReview ↗

Response to Reviewer hc3e

Q2: Regarding additional evidence on an independent dataset (VIGOR) for visual-cue dependencies and representation transfer.

A2: We thank the reviewer for this question. Building upon the VIGOR (Chicago) evaluation under representation-matched training (A→AA \rightarrow A) presented in Table R6 (in Q2 of Response to Weaknesses above), we further evaluated the 5×55 \times 5 cross-representation transfer matrix (A→BA \rightarrow B) on VIGOR (Chicago) using Sample4Geo without test-time fine-tuning (Table R7). As shown in Table R6 (in Q2 of Response to Weaknesses) and Table R7, both the visual-cue dependencies (A→AA \rightarrow A) and cross-representation transfer behaviors (A→BA \rightarrow B) on VIGOR (Chicago) remain consistent with our findings on CVUSA-Cue and CVACT-Cue: (1) Consistent visual-cue dependencies under representation-matched training (A→AA \rightarrow A, Table R6 and diagonal of Table R7): Matched training and testing along the diagonal achieves the highest recall for each representation (53.58%–69.47% R@1), with Sketch (67.66% R@1) remaining the strongest structural representation and closely approaching RGB (69.47% R@1, within 1.81 percentage points), followed by Edge (62.19%), Depth (59.40%), and Seg (53.58%). (2) Consistent directional transfer asymmetry (A→BA \rightarrow B, off-diagonal of Table R7): Models trained on standard RGB degrade severely when tested zero-shot on structural representations, dropping from 69.47% to 1.34% R@1 on Sketch and 0.00% on Edge, Depth, and Seg. In contrast, models trained solely on structural contours transfer substantially better to unmodified RGB queries without any fine-tuning: Sketch -> RGB achieves 40.03% R@1 (+38.69 percentage points higher than RGB -> Sketch), Edge -> RGB achieves 33.72% R@1 (+33.72 percentage points higher than RGB -> Edge). Furthermore, structurally related contour representations exhibit mutual transferability (Sketch -> Edge at 10.36% and Edge -> Sketch at 7.29%), whereas zero-shot transfer across representations with distinct structural formats and encodings (Depth and Seg, 0.00%–1.03%) remains limited, mirroring the patterns in Figure 2a of the main paper. Together, Tables R6 and R7 confirm that our visual-cue trends and direction-dependent transfer patterns replicate on this metropolitan dataset under same-area training and non-central matching. We will incorporate these results into the revised manuscript.

Table R7: Cross-representation transfer matrix (A→BA \rightarrow B) in R@1 (%) on VIGOR (Chicago) under the same-area setting using Sample4Geo (rows denote training representations AA, and columns denote test representations BB).

Train (AA) \ Test (BB)RGBSketchEdgeDepthSeg
RGB69.471.340.000.000.00
Sketch40.0367.6610.360.000.07
Edge33.727.2962.190.000.00
Depth0.090.000.0059.400.01
Seg7.941.030.180.0253.58
Reviewer_f97pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CVGL-CueBench, a comprehensive benchmark assessing how CVGL models use different visual cues. Based on the original CVUSA and CVACT datasets, the authors provide four representations derived from each original RGB panorama: class-agnostic region segmentation, relative depth, edges, and sketches. The authors evaluate five baseline methods across these five query representations and five viewing conditions, including north-aligned panoramas, unknown orientations, and limited fields of view. The aerial gallery remains RGB throughout. Results show that structural representations can also support retrieval, particularly with full panoramic views.

Critical strengths
  1. The benchmark design is novel. The authors consider several structural representations, including region segmentation maps, depth maps, edge maps, and sketches. The viewing condition configurations are realistic.

  2. The evaluation is solid. The authors evaluate representative baselines with the same query representation used for training and testing, as well as transfer between different query representations.

  3. The paper is well written and easy to follow. The visual examples of the different representations help readers understand the benchmark.

Critical weaknesses
  1. The range of representations is still limited. It would be interesting to also consider 3D representations built from RGB observations using models such as VGGT [1]. These 3D representations are shown effective for CVGL [2]. Including more types of representations could provide further insights.

  2. The paper evaluates each representation separately but does not explore whether combining two or more representations could improve retrieval robustness.

  3. The depth, segmentation, and sketch representations depend on the pretrained models used to generate them. Depth Anything V2 may produce inaccurate depth, while SAM may produce masks that are not useful for localization as shown in Figure 5. Some performance drops may therefore come from errors in these models rather than the choice of representation itself. The paper acknowledges this limitation, but the experiments do not separate these effects. Ablating the models used for the same type of representation can help.

  4. Although the paper aims to study which visual cues models use, the practical motivation for replacing RGB queries is less clear to me. While the authors mention privacy and bandwidth benefits, deriving these representations from RGB also needs additional computation. This is confusing especially considering RGB retrieval already performs well.

  5. The evaluation is limited to CVUSA and CVACT. Results on other datasets, such as VIGOR or a drone-based geo-localization dataset, would help show whether the findings are generalizable.

Questions for authors
  1. Edge maps and sketches both emphasize scene contours, although their generation methods and retrieval performance differ. What additional insight into visual-cue dependence does including both representations provide?

  2. If the aerial gallery is also converted to different representations, how would different models perform?

[1] Wang, Jianyuan, et al. "Vggt: Visual geometry grounded transformer."

[2] Zhang, Yancheng, et al. "Geo2: Geometry-Guided Cross-view Geo-Localization and Image Synthesis."

6 comments
AuthorsOpenReview ↗

Response to Reviewer f97p

Thank you for taking the time to provide us with a thorough review. We address the specific weaknesses and questions next.

Response to Weakness

Q1: Regarding the range of representations.

A1: We sincerely thank the reviewer for this insightful suggestion and for pointing out these two works. VGGT (Wang et al.)  introduces a feed-forward Geometric Foundation Model capable of directly inferring comprehensive 3D scene attributes from RGB images, and Geo² (Zhang et al.) effectively leverages VGGT's 3D geometric priors in a shared 3D-aware latent space to jointly boost CVGL and bidirectional cross-view image synthesis. In this paper, our primary research question is to diagnose how retrieval performance and transfer sensitivity change when RGB ground queries are replaced by derived structural representations (Lines 16-23). To evaluate off-the-shelf 2D CVGL backbones without altering their network architectures, we designed four complementary representations that attenuate photographic appearance while retaining different aspects of scene structure:

  • Edge maps are chosen to represent intensity discontinuities and local boundary contours without retaining explicit color or continuous intensity values. Edges caused by texture, shadows, or reflections may nevertheless remain.
  • Sketches are chosen to preserve clean, continuous structural outlines and global architectural silhouettes while suppressing fragmented texture noise and illumination variations.
  • Segmentation maps (Seg) are chosen to capture discrete 2D regional layouts and spatial partitioning without surface textures or semantic class labels.
  • Depth maps are chosen to encode camera-relative distance ordering and 2.5D surface geometry.

Together with RGB, these representations form a controlled 2D diagnostic suite to probe visual-cue reliance and representation transfer. We fully agree with the reviewer that incorporating explicit 3D representations (such as VGGT and the 3D-guided paradigm in Geo²) will provide deeper geometric insights. We therefore regard explicit 3D representations as a valuable extension of the present 2D benchmark and have cited and discussed both VGGT and Geo² in Sections 2 and 6 of the revised manuscript, and we will explore integrating 3D foundation representations in future work.

Q2: Regarding the exploration of combining two or more representations to improve retrieval robustness.

A2: We sincerely thank the reviewer for this constructive suggestion. In our initial manuscript, we evaluated each representation separately to analyze representation-matched retrieval (A→AA \rightarrow A) and sensitivity to representation shifts  (A→BA \rightarrow B) of each individual visual cue. Following your suggestion, we further conducted new experiments combining multiple structural representations under the challenging 70∘70^\circ FoV setting on CVUSA-Cue using Sample4Geo. For implementation, we adopted a equal-weight score-level fusion strategy at inference time (concatenating the L2-normalized descriptors from independently trained models, which is mathematically equivalent to averaging their cosine similarity scores) without training a fusion module or updating the existing model weights. As shown in Table R1, combining multiple structural representations consistently outperforms each constituent representation across all metrics. For example, the four-representation combination (Edge + Sketch + Depth + Seg) reaches 28.58% R@1, compared with 22.86% for Sketch, the strongest individual structural representation in this setting. It remains below RGB at 36.77% R@1. We report these gains as an empirical multi-representation ensemble benefit in this restricted-FoV setting—reflecting both multi-model aggregation and cross-cue diversity—and have incorporated Table R1 and this discussion into the revised manuscript.

Table R1: Retrieval performance of single and combined structural query representations on CVUSA-Cue with unknown orientation and FoV=70° using Sample4Geo.

Representation CombinationR@1R@5R@10R@1%Delta R@1 vs Edge (pp)Delta R@1 vs Sketch (pp)Delta R@1 vs Depth (pp)Delta R@1 vs Seg (pp)
RGB36.7762.2071.9292.06
Edge10.6126.4835.6471.33
Depth7.6520.6228.4164.81
Seg7.3219.6027.2762.71
Sketch22.8647.6159.3689.01
Edge+Sketch26.3450.9361.8289.10+15.73+3.48
Edge+Depth17.2237.5848.1780.19+6.61+9.57
Edge+Seg15.5134.7745.0677.97+4.90+8.19
Sketch+Depth24.7650.8862.1089.82+1.90+17.11
Sketch+Seg25.3750.0960.7789.05+2.51+18.05
Depth+Seg14.0632.1442.6777.38+6.41+6.74
Edge+Sketch+Depth28.3953.2364.2590.06+17.78+5.53+20.74
Edge+Sketch+Seg27.4351.9462.3889.46+16.82+4.57+20.11
Edge+Depth+Seg19.4441.6352.4582.55+8.83+11.79+12.12
Sketch+Depth+Seg26.4152.0762.8389.77+3.55+18.76+19.09
Sketch+Depth+Seg + Edge28.5853.8863.9790.03+17.97+5.72+20.93+21.26
AuthorsOpenReview ↗

Response to Reviewer f97p

Q3: Regarding the dependency on pretrained generators.

A3: We sincerely thank the reviewer for this insightful comment. In our initial benchmark design, we intentionally selected widely adopted pretrained models and standard operators—Depth Anything V2, SAM (ViT-H), Qwen-Image-Edit, and Canny—to provide strong reference implementations for generating the structural queries. At the same time, we fully agree with your point that ablating different generators within each representation category can help quantify sensitivity to the specific extraction pipeline. To evaluate this, we conducted an intra-representation generator ablation on north-aligned CVACT-Cue using Sample4Geo (Table R2) across all four structural representations. As shown in Table R2, across all four structural representation categories, replacing the default generator with an alternative extractor alters R@1 by only 0.36 to 2.17 percentage points and R@1% by 0.01 to 0.24 percentage points. Specifically, for Sketch, the classical non-generative XDoG filter achieves 87.31% R@1 (vs. 88.57% with Qwen-Image-Edit), demonstrating that strong retrieval performance in this panoramic setting is consistently achieved across both neural and classical mathematical contour extractors. Similarly, substituting boundary-refined HQ-SAM for Seg (86.11% vs. 86.47% R@1), Depth Anything V1 for Depth (83.45% vs. 85.62%), or Sobel for Edge (85.25% vs. 85.78%) yields highly consistent retrieval accuracy. These results provide evidence that strong panoramic retrieval is not unique to a generative pipeline and support robustness to these specific implementation choices in the evaluated north-aligned panoramic setting. We will include Table R2 and this discussion in Appendix of the revised manuscript.

  • Sketch: Comparing the generative Qwen-Image-Edit with XDoG (eXtended Difference-of-Gaussians), a classical non-generative stylization operator that extracts coherent line drawings via band-pass filtering and soft thresholding.
  • Segmentation (Seg): Comparing SAM (ViT-H) with HQ-SAM (ViT-H) [2] (which introduces high-quality token learning and global-local feature fusion for fine-grained mask boundaries) to measure sensitivity to mask boundary refinement.
  • Depth: Replacing Depth Anything V2 (ViT-B) with Depth Anything V1 (ViT-B) to assess retrieval sensitivity to the depth estimator.
  • Edge: Replacing Canny (which includes non-maximum suppression and hysteresis thresholding) with the first-order gradient Sobel operator.

Table R2: Intra-representation generator ablation on north-aligned CVACT-Cue using Sample4Geo.

Representation TypeGeneratorParadigmR@1R@5R@10R@1%
SketchQwen-Image-EditGenerative Foundation Model88.5794.9396.1098.23
SketchXDoG [1]Non-generative Band-pass Filter87.3194.6495.9998.22
SegSAM (ViT-H)Foundation Model (v1)86.4794.4195.6797.90
SegHQ-SAM (ViT-H) [2]Boundary-refined SAM86.1194.3395.8498.14
DepthDepth Anything V2Foundation Model (v2)85.6293.6294.7897.61
DepthDepth Anything V1 [3]Foundation Model (v1)83.4592.5694.3597.43
EdgeCannyMulti-stage Edge Detector85.7894.0095.3597.71
EdgeSobel [4]First-order Gradient Filter85.2594.1895.6097.86

Reference

[1] Winnemöller H, Kyprianidis J E, Olsen S C. XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization[J]. Computers & Graphics, 2012, 36(6): 740-753.

[2] Ke L, Ye M, Danelljan M, et al. Segment anything in high quality[J]. Advances in Neural Information Processing Systems, 2023, 36: 29914-29934.

[3] Yang L, Kang B, Huang Z, et al. Depth anything: Unleashing the power of large-scale unlabeled data[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 10371-10381.

[4] Duda R O, Hart P E. Pattern classification and scene analysis[J]. A Wiley-interscience publication, 1973.

AuthorsOpenReview ↗

Response to Reviewer f97p

Q4: Regarding the practical motivation for evaluating non-RGB query representations.

A4: We sincerely thank the reviewer for raising this fundamental question, which allows us to clarify the primary positioning and practical value of our work. First and foremost, CVGL-CueBench is designed primarily as a scientific diagnostic benchmark rather than a deployment pipeline advocating the universal replacement of RGB queries (Lines 16-23, 89-91). Much like how stylized or contour-only benchmarks are used in visual recognition to expose texture bias and shortcut learning, generating parallel structural representations offline enables a controlled investigation of representation sensitivity and feature reuse—for instance, revealing that methods with similar RGB recall (TransGeo and GeoDTR) diverge by 31.65 pp on Edge queries (Table 1), and that contour-trained models retain transferable features effective on unmodified RGB queries (93.25% R@1 for Sketch -> RGB, Figure 2a).

Second, RGB retrieval is not always optimal within the tested protocols. While RGB performs well on full panoramas, under a restricted 70∘70^\circ FoV on the urban CVACT-Cue dataset (under our standardized panoramic-extraction and online-cropping protocol), Sketch queries outperform RGB baselines across all five evaluated methods by 0.96–15.48 percentage points in R@1. In narrow urban views, filtering out transient shadows and facade textures helps models focus on stable architectural outlines. Furthermore, our transfer results show that models trained on Sketch or Edge retain high recall when tested directly on RGB (93.25% and 81.51% R@1) (see Figure 2(a)), offering useful insights for improving RGB model training.

Finally, regarding the trade-off among privacy, bandwidth, and computation, we agree that deriving structural queries introduces client-side extraction overhead, and we treat privacy desensitization and bandwidth reduction as potential downstream application scenarios rather than already-validated deployment gains. At the same time, privacy protection and uplink bandwidth reduction remain important downstream application motivations (Section 3.1 and Appendix D): in edge-to-cloud localization, ground clients can transmit appearance-desensitized structural queries (removing faces, license plates, and raw photographic textures) to a central server hosting a pre-existing RGB aerial gallery. Furthermore, while we used foundation models offline to construct high-quality benchmark references, lightweight classical operators (such as Canny or XDoG in Table R2) can extract these structural cues on edge devices with minimal computational overhead when client-side privacy desensitization is desired. More importantly, even when the test-time input remains standard RGB, our directional transfer analysis (Sketch -> RGB at 93.25% vs. RGB -> Sketch at 68.12% R@1, Figure 2a) is consistent with contour-trained models retaining features useful for RGB retrieval, motivating future structural augmentation designs. We will revise to clarify this diagnostic positioning.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all Reviewers

Response to all Reviewers

We sincerely thank all the reviewers for their constructive feedback, which has significantly helped us improve our manuscript. We are delighted to see that the reviewers recognize the novelty and core contributions of CVGL-CueBench, specifically: (1) the novel benchmark design that addresses a key limitation of existing CVGL evaluations by decoupling visual cues across five query representations and realistic viewing configurations(acknowledged by Reviewers f97p, hc3e); (2) the solid, extensive, and systematic experimental evaluations spanning multiple representative CVGL methods, datasets (CVUSA and CVACT), representation-matched training, and cross-representation transfer(recognized by all reviewers: f97p, hc3e, 9C3F); and (3) the clear presentation, readability, and informative visual examples that facilitate understanding of the benchmark(highlighted by Reviewer f97p). We greatly appreciate your encouraging feedback and have addressed each reviewer's specific comments and questions in detail in our individual responses below. In the revised manuscript, newly added content is highlighted in orange and revised content is marked in blue. Thank you once again for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.