- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces DiG, a generative geolocation approach that performs diffusion in a pretrained location encoder space. This allows the model to have already built priors about Earth semantics without having to rebuild these global semantics.
The authors also propose LAE to be able to derive the coordinates from the location encoder embedding and Levenberg–Marquardt steps for non-trivial inversing decoders.
Critical strengths
The paper is really clear and well-motivated. Performing diffusion in a latent space that already contains semantic information around the globe, knowing where the cities are, if two different areas around the globe look the same.
The inversion of location encoder features is really elegant and might be useful to other tasks.
The results, especially the one on OSV are really strong, especially outperforming retrieval methods.
The ablations are complete and allow a better understanding of the gain of each component.
Critical weaknesses
The main weakness that appears is that these latent spaces are not perfect; it is unclear whether a bijection truly exists between embeddings and locations. Allowing the model to refine this latent space or the decoding into coordinates might be beneficial.
Questions for authors
GeoClip seems to perform well at a fine scale. It is hard to know if this comes from the modality with which it has been aligned or the different architecture with Random Fourier Features. To disentangle this, it would be interesting to enrich Table 5 with RFF.
In Table 5, do you still use REPA on the Spherical Harmonics or not? Have you tried to perform diffusion on SH while doing REPA with pretrained location encoders, to encourage the model to learn semantics from the location encoder while still being more free about its own latent space?
Can you run DIG with only StreetCLIP conditioning against RFM S² using the same conditioning, to isolate the effect of the diffusion space from the conditioning upgrade?
3 comments
The authors have adequately addressed my questions and concerns in their rebuttal. I am satisfied with their clarifications and have no further concerns.
The authors have addressed my questions.
We thank the reviewer for finding our approach clear, well-motivated, and elegant. We especially appreciate the detailed and thoughtful questions which engage closely with our design choices. We address them below:
W1: Imperfect latent spaces and bijection between embeddings and locations
We agree that a discriminative mapping between the location representation and locations is essential to geolocalization performance. We find empirically that the mapping between representations and locations is strongly discriminative and recoverable at the geolocalization task range: our decoders recover locations to within 1 km at 99.9–100% accuracy for all encoders (Tab. 9). With numerical refinement, GeoCLIP and TTE recover at least 99.7% of locations within 10 m, while SatCLIP reaches 90.2% (Tab. 6). For generated embeddings that may not exactly match any location, LM refinement finds the closest match, and this decoding outperforms gallery retrieval (Tab. 10).
Making DIG more fine-grained to exploit sub-kilometer resolution, as well as exploring location representations optimized for diffusion, remain open directions we're excited to explore in future work.
Q1/Q2. RFF baseline and isolating REPA
To clarify, we do not use REPA with non-LAE diffusion spaces in Tab. 5. This is deliberate to isolate any impact of semantic location representations in the diffusion process. We include additional baselines of position-only spaces (including the requested RFF) with REPA in Tab. R2:
Table R2. DIG-B Geolocation performance on OSV-5M (official train/test splits)
RFF follows the random Fourier features as used in GeoCLIP, and RFF w/ REPA and GeoCLIP w/ REPA align with GeoCLIP location encoder features. SH (L=40) w/ REPA and SatCLIP w/ REPA align with SatCLIP (L=40) location encoder features. REPA does improve RFF and SH baselines on all metrics, but it still falls behind pretrained location encoder latent spaces with REPA, demonstrating the importance of diffusing directly in semantic latent spaces.
By isolating RFF as the latent space, we see that it performs worse than the SH (L=40) at fine scales (13.9 vs. 12.4). Yet, GeoCLIP (built on RFF) performs better than SatCLIP (built on SH (L=40)) at city-level (19.8 vs. 18.4), suggesting that the alignment modality impacts GeoCLIPs ability to perform well at fine scales.
Q3. Isolating StreetCLIP conditioning
We compare DIG-B models trained on only with StreetCLIP image conditioning in Tab. 4d. We highlight the comparison here for ease.
Table R3. DIG-B Geolocalization performance on OSV-5M only using StreetCLIP as the vision encoder.
Even at less training (400k steps for DIG-B vs. 1M steps for RFM S²), DIG-B with only StreetCLIP conditioning outperforms RFM S² across the board, especially at finer scales (18.2 vs. 5.4 City acc.).