- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CVGL-CueBench, a benchmark built upon the existing CVUSA and CVACT datasets to investigate visual cue dependence and representation transfer in cross-view geo-localization. The authors generate four additional query representations (sketch, edge, depth, and segmentation) using existing image processing and pretrained vision models, while keeping the aerial reference images unchanged. They evaluate five representative CVGL methods under representation-matched training and different viewing conditions, and further analyze cross-representation transfer and spatial perturbations using Sample4Geo.
Critical strengths
- The paper provides numerious comparison of multiple query representations across several CVGL methods, datasets, and viewing conditions.
Critical weaknesses
-
Insufficient motivation for cross-representation transfer. The paper does not convincingly explain why a model trained on one query representation should be expected to handle substantially different representations without adaptation. The practical necessity of this setting remains unclear.
Although Appendix D provides practical motivations for using structural ground queries with a fixed RGB aerial gallery, these arguments do not adequately justify the necessity of zero-shot transfer between arbitrarily selected ground-query representations. The two settings should not be conflated. Besides, all additional query representations are artificially generated from RGB images rather than obtained from real-world heterogeneous sensors. It is unclear whether these synthetic representation shifts correspond to meaningful deployment challenges in CVGL.
-
Limited novelty in benchmark construction. The benchmark largely applies existing off-the-shelf methods (Canny, SAM, Depth Anything V2, and Qwen-Image-Edit) to established datasets (CVUSA and CVACT). It fundamentally introduces neither new raw data nor new evaluation methodologies. Every user of the existing datasets can easily handle them by themself.
-
Limited and largely predictable insights. The main findings, such as the sufficiency of structural information for localization and the degradation caused by large input distribution shifts, are largely expected or consistent with prior knowledge. The paper provides limited new understanding beyond these empirical observations.
-
Insufficient evidence for visual cue dependence. Comparing performance across different generated representations does not necessarily isolate the contributions of individual visual cues. The results are confounded by differences in representation quality, preprocessing pipelines, pretrained model priors, and the color mapping in the visualizations.
-
Unclear source of the reported sketch advantage. Sketches are generated using a powerful pretrained generative model, potentially introducing additional structural enhancement. Moreover, representations are generated from full panoramas before FoV cropping. Without appropriate controls, the improvements cannot be confidently attributed to structural information alone.
-
Limited conclusions. Despite extensive experiments, the paper provides few concrete insights into how the observed representation sensitivities can be addressed. The analysis does not establish generalizable principles for improving CVGL robustness or representation learning.
Overall, I'm tending giving a typical weak reject below borderline reject.
Questions for authors
See the weaknesses
11 comments
Response to Reviewer 9C3F
Q2: Regarding the novelty of benchmark construction, evaluation methodology, and why a standardized benchmark is needed.
A2: We thank the reviewer for this comment and clarify the design rationale, evaluation methodology, and practical necessity of CVGL-CueBench across the three points raised:
1. Why apply off-the-shelf extractors to established datasets, and how we advance beyond prior benchmarks? First, applying mature, off-the-shelf operators or foundation models to established datasets is a widely adopted paradigm for constructing controlled diagnostic benchmarks. For instance, ImageNet-C [1] applies 15 standard filters to ImageNet to evaluate corruption robustness; Background Challenge [2] uses off-the-shelf segmentation masks to study foreground vs. background reliance; and the CVGL Robustness Benchmark [3] applies standard pixel corruptions to CVUSA and CVACT. This paradigm keeps ground-truth correspondences strictly fixed while using validated extractors to isolate target visual factors without introducing artifacts from custom, unverified generators. Second, CVGL-CueBench advances both the representation decomposition and diagnostic scope: whereas prior CVGL benchmarks [3] apply low-level pixel corruptions (e.g., blur, noise, weather) to RGB images—where appearance and geometry remain coupled—we combine foundation models (SAM, Depth Anything V2, Qwen-Image-Edit) and classical operators (Canny) to construct four derived structural representations (Sketch, Edge, Depth, Seg) alongside RGB. This shifts evaluation from pixel-level corruption tolerance to diagnosing which visual cues drive ground-to-aerial alignment.
2. Systematic diagnostic evaluation protocol: Conventional CVGL evaluations test RGB queries alone, coupling surface appearance (color, texture) with spatial geometry (contours, layout, depth) and obscuring which cues models rely on. Instead, we establish a systematic diagnostic protocol spanning 250 primary configurations (): (i) Representation-matched training (): training and testing on the same representation to measure standalone localization capacity and multi-cue synergy (Table R1, Reviewer f97p W-Q2); (ii) Cross-representation transfer (): evaluating models trained on representation directly on without fine-tuning (Figure 2a), exposing whether models rely on transferable spatial geometry or appearance shortcuts; (iii) Viewing conditions and test-time spatial perturbations: examining how cue reliance shifts from north-aligned panoramas to unknown orientations and restricted FoVs (), alongside test-time spatial perturbations (Gaussian blur, patch shuffling, region occlusion). This joint protocol uncovers empirical findings—directional transfer asymmetry, Sketch outperforming RGB at FoV (under our panoramic-extraction protocol), and architectural sensitivity to Edge vs. Sketch—that standard RGB evaluation cannot observe.
3. Why a precomputed, standardized benchmark cannot be replaced by ad-hoc user processing: Providing a precomputed benchmark is essential for computational accessibility and fair cross-paper comparison. Generating over 355,000 high-resolution derived images across 88,832 panoramas (plus our VIGOR extension, Table R3, Reviewer f97p W-Q5) using heavy foundation models (Qwen-Image-Edit-2511, ViT-H SAM) and running 250 benchmark evaluations requires substantial GPU compute and careful calibration (e.g., structure-preserving prompts against hallucination and deterministic cyclic color mapping, Appendix E). Moreover, ad-hoc generation with varying prompts, Canny thresholds, depth normalizations, or checkpoints would make cross-paper results inconsistent and non-comparable. Releasing CVGL-CueBench with fixed splits, unified representations, and baseline checkpoints establishes a reproducible standard for the community.
References:
[1] Hendrycks D, Dietterich T. Benchmarking neural network robustness to common corruptions and perturbations[J]. ICLR, 2019.
[2] Xiao K, Engstrom L, Ilyas A, et al. Noise or signal: The role of image backgrounds in object recognition[J]. ICLR, 2021.
[3] Zhang Q, Zhu Y. Benchmarking the robustness of cross-view geo-localization models[C]. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 36-53.
Response to Reviewer 9C3F
Q3: Regarding the novelty of our empirical insights beyond prior expectations.
A3: We thank the reviewer for this comment. While high-level domain-shift or structural intuition may seem familiar, it cannot explain four counter-intuitive findings revealed by our benchmark:
First, standard domain-shift intuition cannot explain the strong directional asymmetry in cross-representation transfer (). A simple distribution gap does not explain why transferring between the exact same representation pair differs by 25–70 percentage points (pp) by direction. On CVUSA-Cue (Figure 2a, Lines 423–433), an RGB-trained model drops to 68.12% R@1 on Sketch and 10.77% on Edge, whereas Sketch- or Edge-trained models achieve 93.25% and 81.51% R@1 directly on unmodified RGB queries without fine-tuning. Thus, the bottleneck is not the domain gap itself: standard RGB training encourages reliance on color and texture shortcuts, whereas contour training guides the network to encode spatial geometry that remains directly effective on natural RGB images.
Second, discarding appearance information in Sketch outperforms raw RGB under a restricted FoV within our standardized panoramic-extraction and online-cropping protocol. While RGB performs best on panoramas (Table 1), the ranking reverses under a FoV on urban CVACT-Cue (Table 2, Lines 362–372): Sketch surpasses raw RGB across all five evaluated methods (e.g., improving R@1 from 18.37% to 22.43% on Sample4Geo, 24.60% to 33.56% on ConGeo, and 22.61% to 38.09% on SinGeo). When wide panoramic context is lost in urban scenes, transient street shadows and facade textures in RGB act as distractions, whereas suppressing them in favor of continuous object outlines improves matching (source and boundary conditions are examined in Q5 below).
Third, structural cues are not uniformly sufficient; our benchmark exposes sharp divergences across viewing conditions and architectures. Comparing four derived structural cues reveals distinct boundary conditions: (1) although Depth, Seg, and Edge achieve high recall on panoramas (Table 1), they drop to 1.21%–13.57% R@1 at FoV on CVACT-Cue (Table 2), showing that depth ordering, region partitioning, and raw pixel gradients rely heavily on wide panoramic context, whereas continuous contours (Sketch) remain resilient under this protocol; and (2) methods with near-identical R@1 on RGB (84.95% vs. 85.43%) and Sketch (79.51% vs. 77.74%)—TransGeo and GeoDTR on CVACT-Cue (Table 1)—diverge by 31.65 pp on fragmented Edge maps (45.21% vs. 76.86%), exposing how differently retrieval architectures respond to high-frequency gradient clutter versus continuous boundaries.
Fourth, prompted by Reviewers f97p and hc3e, our new rebuttal experiments uncover three additional insights: (1) multi-cue structural synergy under narrow FoVs (Table R1, Reviewer f97p W-Q2): although individual non-Sketch cues degrade under unknown orientation and on CVUSA-Cue (7.32%–10.61% R@1), combining complementary structural cues consistently boosts retrieval, with Sketch + Depth + Seg + Edge reaching 28.58% R@1 and 90.03% R@1% (+5.72 to +21.26 pp over single cues); (2) asymmetric impact of ground vs. aerial structural conversion (Table R4, Reviewer f97p Q2): converting the aerial gallery to segmentation (RGB–Seg at 80.47% R@1, Seg–Seg at 62.16%) shows that 2D regional layouts support matching in both views, while retaining aerial RGB textures (Seg–RGB at 91.72%) substantially aids fine-grained alignment; and (3) generalization to metropolitan canyons and one-to-many matching (Tables R6–R7, Reviewer hc3e W-Q2 & Q2): evaluations on VIGOR (Chicago) confirm that both representation-matched trends (Sketch at 67.66% vs. RGB at 69.47% R@1) and directional transfer asymmetry hold under severe high-rise occlusions and non-central matching.
While our main quantitative results are reported in the manuscript's tables and figures, we agree that these counter-intuitive takeaways and the newly added findings should be highlighted more prominently. We thank the reviewer for this feedback and will sharpen the takeaway summaries in the revised manuscript.
Response to Reviewer 9C3F
Thank you for taking the time to provide us with a thorough review. We address the specific weaknesses and questions next.
Response to Weakness
Q1: Regarding the motivation for cross-representation transfer (A→B), the role of Appendix D, and the use of RGB-derived representations.
A1: We thank the reviewer for this comment and address the three points below:
Why evaluate cross-representation transfer (A→B)? We would like to clarify that A→B transfer is designed as a diagnostic stress test to uncover what visual features a model actually relies on after training, rather than a claim that arbitrary representation pairs should be interchangeable without adaptation in deployment. Specifically, A→A and A→B answer two distinct diagnostic questions: representation-matched training (A→A) measures whether an isolated visual cue contains sufficient spatial information to support localization when trained in-domain, whereas cross-representation transfer (A→B) reveals whether a trained model has learned underlying spatial geometry or overfitted to representation-specific appearance shortcuts (Geirhos et al., ICLR 2019). A→A alone cannot expose shortcut learning, whereas the directional asymmetry in A→B (Figure 2a and Lines 423–433) directly reveals it: on CVUSA-Cue, an RGB-trained model drops sharply when tested on structural cues (RGB→Sketch drops by 30.56 percentage points to 68.12% R@1, RGB→Edge to 10.77%, and RGB→Depth to 0.06%), showing that standard RGB training relies heavily on surface color and texture shortcuts. Conversely, models trained on structural contours transfer strongly back to standard RGB queries without any adaptation (Sketch→RGB achieves 93.25% R@1 and Edge→RGB achieves 81.51%), demonstrating that training on clean structural outlines guides the backbone to encode cross-view spatial geometry that persists in natural RGB images.
2. Clarifying the scope of Section 3.1 and Appendix D: We would like to clarify that Section 3.1 (Lines 206–209) and Appendix D (Lines 860–917) were to explain why we vary the ground query while holding the aerial gallery fixed in RGB—both because practical geo-localization systems query pre-existing RGB satellite repositories and because fixing the aerial reference ensures that performance variations come from the ground query side (we also evaluate converting the aerial gallery to Seg in Table R4 in our response to Reviewer f97p, Q2). We will refine the wording in Section 3.1 and Appendix D to ensure their scope is unambiguous.
3. Why representations are derived from RGB and their relevance to practical CVGL: CVGL-CueBench is designed as a controlled diagnostic benchmark (similar in spirit to ImageNet-C) rather than a multi-sensor hardware benchmark. Representations such as contours (Sketch), gradients (Edge), and regional partitions (Seg) represent structural abstractions of a visual scene rather than separate hardware sensor modalities. Deriving them from the same geo-registered RGB images keeps the underlying geographic location, camera viewpoint, and scene content consistent across all five representations, while preserving the original ground–aerial correspondences. Furthermore, evaluating these RGB-derived representations provides practical value for real-world RGB localization: first, it diagnoses why standard RGB-trained models are vulnerable to appearance variations by exposing their reliance on color and texture shortcuts (Figure 2a); second, under restricted fields of view (FoV=70∘) in dense urban scenes (within our standardized full-panorama extraction and online-cropping protocol, Table 2), suppressing street-level shadow and texture clutter in favor of continuous structural outlines (Sketch) outperforms raw RGB, while combining complementary structural representations (Table R1 in Reviewer f97p, Q2) substantially mitigates single-cue fragility.