acceptodds
Withdrawn from ICLR 2027

GeoTAP: Resolving Video Geolocation Ambiguity via Temporal Audio Prototypes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Determining where a video was filmed is a fundamental challenge in spatial intelligence and computer vision. While modern visual foundation models excel at geolocating static images, adapting them to continuous video streams introduces two core failure modes: spatial aggregation noise, where naive per-frame averaging on the sphere degrades accuracy below a single static frame, and visual candidate ambiguity, where visually interchangeable scenes leave top geographic predictions tied with high confidence. To address both challenges, we propose GeoTAP, an encoder-agnostic framework that formulates video geolocation as hypothesis disambiguation within a unified Bayesian pipeline. GeoTAP extracts spatial candidate hypotheses via geodesic clustering and spherical geometric-median filtering to eliminate aggregation noise, then disambiguates candidate hypotheses against temporal audio patterns using learnable acoustic prototypes in our TAP (Temporal Audio Prototypes) module. Visual prior entropy naturally regulates audio influence during Bayesian posterior selection. Evaluated on our new AVG benchmark of 20538}diegetic-audio walking-tour clips across 907 sites, GeoTAP raises city accuracy from 27.9% to 34.2% and country accuracy to 69.4%, outperforming state-of-the-art visual and multimodal baselines without backbone retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual geolocation, audio geolocation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_mYYoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents GeoTAP, a multimodal video geolocation framework that integrates visual candidate generation using frozen visual backbones, temporal audio prototype matching, and entropy-calibrated Bayesian posterior selection. The key idea is to leverage environmental audio as a complementary signal to disambiguate visually similar geographic candidates and improve the accuracy of visual-based geolocation.

Critical strengths
  1. The paper introduces AVG, a new audio-visual geolocation benchmark that provides synchronized video and environmental audio. Unlike existing benchmarks that primarily focus on visual or audio modalities independently, AVG includes paired audio-visual data with diegetic audio filtering and geographically separated training and test splits, providing a valuable resource for multimodal geolocation research.

  2. The idea of leveraging environmental audio as a complementary signal to visual information for geographic disambiguation is well motivated. The authors propose a practical framework that integrates visual candidate generation and audio-based reranking, demonstrating promising improvements over visual-only baselines on the newly introduced AVG benchmark.

  3. Beyond the proposed AVG dataset, the authors conduct additional experiments on existing geolocation benchmarks, including CityGuessr and iNatSounds, to investigate the generalization capability of the proposed framework under out-of-distribution (OOD) settings. These experiments provide additional insights into the framework's transferability beyond the training dataset.

Critical weaknesses
  1. The effectiveness of the proposed method heavily relies on the availability of high-quality, location-specific audio cues. Generic environmental sounds, background noise, or unrelated audio may provide little useful information for geographic disambiguation and could potentially introduce misleading evidence. To construct the AVG dataset, the authors explicitly employ a diegetic audio filtering pipeline to exclude non-diegetic audio overlays (e.g., background music and sound effects), speech-dominant clips, and degraded audio. However, in real-world applications, such favorable audio conditions cannot always be guaranteed. This strong dependence on carefully filtered audio raises concerns about the practicality and general applicability of the proposed framework.

  2. The proposed method is primarily evaluated on the newly collected AVG benchmark, which is specifically designed to contain high-quality diegetic audio. While this careful dataset construction is scientifically valuable, it may also create evaluation conditions that are particularly favorable to the assumptions underlying GeoTAP. As discussed above, the proposed method may be less effective when audio signals are noisy, uninformative, or unrelated to the actual filming location. Although the authors provide out-of-distribution evaluations, these experiments do not adequately demonstrate the robustness of the audio-based disambiguation module under challenging real-world conditions. I suggest that the authors collect and conduct additional experiments on a more challenging benchmark containing silent clips, music- or speech-dominant videos, and videos with misleading or low-quality ambient audio. Such evaluations would provide stronger evidence of the robustness and practical applicability of the proposed approach.

  3. The proposed framework relies on pretrained visual geolocation backbones to generate candidate geographic locations, which are subsequently reranked using acoustic evidence. However, the audio-based disambiguation module can only select among the candidates provided by the visual stage. If the ground-truth location is not included in the generated candidate set, the proposed method cannot recover the correct location, regardless of the quality of the audio information. Consequently, the overall performance remains fundamentally constrained by the accuracy and recall of the visual candidate generation stage. The authors should provide a more detailed analysis of candidate recall, including the proportion of samples for which the ground-truth location is covered by the generated candidate set, and investigate how this limitation affects the overall performance of GeoTAP.

  4. The audio prototypes are learned in a purely data-driven manner without explicit semantic supervision. Although the authors use semantic labels, such as urban rail, wind, and crowd noise, to illustrate the prototypes in the figures, it remains unclear whether the learned prototypes actually capture these meaningful acoustic patterns. The illustrative examples alone do not provide sufficient evidence that individual prototypes consistently correspond to distinct sound categories. A systematic prototype-level analysis, such as visualizing representative audio segments with the highest activation for each prototype, would help validate the semantic interpretability of the learned representations and provide deeper insights into the effectiveness of the proposed TAP module.

  5. The contribution of this work is incremental, as the proposed framework primarily integrates existing techniques such as geodesic clustering and spherical geometric-median filtering. Although the TAP module introduces a learnable audio-based reranking mechanism, its novelty remains relatively limited. Furthermore, the lack of controlled ablation studies and semantic analysis makes it difficult to assess the module's individual contribution and the meaningfulness of its learned representations. The authors should better clarify the technical contributions beyond the integration of established components.

Questions for authors
  1. Have the authors evaluated the proposed method on more challenging real-world samples with low-quality audios? Since AVG employs explicit diegetic audio filtering, to what extent does the proposed method generalize to unfiltered audio-visual data?
  2. The proposed method samples (T=12) video frames to generate geographic candidates through geodesic clustering. How was (T=12) selected? How sensitive is the performance to the number of sampled frames, the clustering radius, and the resulting number of candidates? It would be helpful to report candidate recall at different geographic distance thresholds and analyze how candidate coverage affects the overall geolocation performance.
  3. How much of the reported performance improvement can be attributed specifically to the proposed TAP module rather than robust visual candidate aggregation?
  4. Do the learned audio prototypes consistently correspond to meaningful and distinct acoustic patterns, as suggested by the semantic labels in the figures? Could the authors provide a qualitative analysis of the learned prototypes?
Reviewer_x6Q7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GeoTAP addresses video geolocation by clustering predictions from frozen visual models into geographic candidates and reranking them with temporal audio prototypes. The paper introduces AVG, a dataset of 20,538 audiovisual clips across 907 sites. It reports improvements across several visual backbones, including an increase in GLOBE city accuracy from 27.9% to 34.2%.

Critical strengths

The use of audio to distinguish visually plausible geographic candidates is well motivated. The modular design with frozen encoders offers a practical way to investigate complementary acoustic evidence without retraining the full visual model.

Critical weaknesses
  1. Inconsistencies in the reported evaluation. Equations (11) and (12) define mathematically identical normalized vector means, yet Table 4 reports different country accuracies (56.9% versus 60.5%) and median errors. The GLOBE median error also differs between Table 5 (330 km) and Tables 2 and 4 (462 km). These discrepancies need clarification before the reported gains can be assessed reliably.
  2. The independent contribution of audio is insufficiently established. Although Appendices C.2 and C.6 support audio reranking, visual-only aggregation already achieves 69.0% country accuracy versus 69.4% for GeoTAP. Comparisons using identical visual candidates with audio disabled or replaced by scene-matched audio from other locations, together with uncertainty estimates, would better isolate the contribution of audio.
  3. AI-style drawings.
Questions for authors
  1. Could you reconcile the equivalent mean definitions in Equations (11) and (12) with their different results in Table 4, and explain the GLOBE median-error discrepancy? Table 10 also implies approximately 34.6% overall TAP city accuracy, rather than the 34.2% in the main table. Do these results use different configurations?
  2. With the same frames, candidate sets, and visual priors, how does GeoTAP compare with disabling audio, using silent audio, using audio from another location with a similar scene, and calibrated late fusion? Please report uncertainty across runs or source-video/site groups.
  3. What assumptions justify treating the learned candidate posterior as an acoustic likelihood in Equation (9)? What explicitly ensures that acoustic score differences vanish for a zero waveform, as claimed in Table 3, given Equations (3)-(7)?
  4. How are GeoNames centroid labels checked against the actual filming locations? What is the visual candidate recall at the evaluation thresholds, and how is the training target defined when no candidate is sufficiently close to the ground truth?
Reviewer_czYgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoTAP, a framework for video geolocation that combines multi-frame visual predictions with environmental audio while keeping the backbone encoders frozen. It first constructs geographic candidates through geodesic clustering and spherical geometric median estimation, then uses learnable audio prototypes and temporal attention to score each candidate’s acoustic compatibility. These scores are combined with a visual prior to select the final location. The authors also introduce AVG, a dataset containing 20,538 audiovisual clips across 907 sites, with audio filtering and geographically separated splits. Evaluation includes comparisons across multiple visual backbones, aggregation and fusion baselines, component ablations, and transfer experiments on CityGuessr and iNatSounds. On AVG, the authors report improvements over the single-frame GLOBE baseline from 27.9% to 34.2% city-level accuracy and from 61.2% to 69.4% country-level accuracy.

Critical strengths
  1. A well-motivated role for environmental audio. Using audio to distinguish among plausible visual location hypotheses is a compelling direction. Restricting the task to candidate disambiguation provides a sensible way to exploit acoustic cues that may be insufficient for global localization on their own. Keeping the backbone encoders frozen also makes the proposed approach potentially useful for extending existing geolocation systems.

  2. A relevant benchmark contribution. The proposed AVG dataset provides a concrete setting for studying geolocation from paired video and environmental audio, comprising 20,538 clips across 907 sites. The emphasis on filtering audio and separating training and evaluation locations geographically addresses important considerations for this task. With adequate validation, this benchmark could be a useful resource for investigating when audio helps resolve visual ambiguity.

Critical weaknesses
  1. Some baseline definitions and results need clarification. As written, Eqs. (11) and (12) yield the same prediction for identical inputs with a nonzero vector sum, yet Table 4(a) reports different results. The GLOBE median error also differs between Tables 2/4 (462 km) and Table 5 (330 km), despite identical accuracies. These may reflect implementation or reporting differences, but clarification is needed to verify the comparisons.

  2. The overall audio contribution needs clearer attribution. The headline gains include improvements from visual aggregation. Relative to the spherical median baseline, GeoTAP improves city accuracy from 31.2% to 34.2% and country accuracy from 69.0% to 69.4%. Table 10 and Appendix C.6 support acoustic reranking, but their relationship to the main results is unclear. Calibrated late fusion is also competitive in Table 12. Together, these results leave the magnitude and robustness of TAP’s contribution insufficiently established.

Questions for authors
  1. Could the authors clarify the implementation differences underlying Eqs. (11) and (12), and reconcile the GLOBE median errors of 462 km and 330 km?

  2. What configurations and averaging conventions are used in Table 10 and Appendix C.6? How do their results relate to the main tables? Overall results before and after acoustic reranking on identical candidates, together with calibrated late fusion results for GLOBE, would help clarify the contribution.

  3. How is i∗i^* in Eq. (8) assigned when all candidates are far from the ground truth? Please specify the evaluation scales and candidate oracle recall.

  4. Should the acoustic score be interpreted as a likelihood approximation or a learned compatibility score? For Table 3, is an explicit mask or bypass used to eliminate the acoustic contribution when audio is missing?

  5. Could the authors clarify the advances over earlier audiovisual geolocation work, including the 2011 ICSI system and Sevillano et al. (2015), and specify the properties underlying AVG’s “first benchmark” claim?

Reviewer_GPq9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of audio-visual geolocalization of videos. Existing image-based grolocalization models, when applied to each frame of a video, produce scattered guesses, and averaging these guesses can land on an unrelated place. Vision alone often cannot decide between a few plausible locations.

The two main contributions of the paper are: a new dataset called AVG, which consists of 20.5k YouTube walking-tour clips across 907 locations which are automatically labelled, with a minimum separation between training and test samples to prevent leakage. The second contribution is a new audio-visual geolocalization method called GeoTAP. GeoTAP builds on top of an existing image geolocation model that stays frozen. It first runs that model on frames sampled from the video, clusters the resulting predictions geographically, and treats each cluster as a candidate location, with larger clusters given a stronger prior. To choose between these candidates, a lightweight audio module encodes short windows of the soundtrack and matches them against a set of learned acoustic prototypes. It then scores how consistent the audio is with each candidate location and combines this score with the visual prior to make the final prediction.

The method is evaluated mainly on the new dataset, using three different visual location models as the base, and is compared against single-frame visual models, audio-only location models, and simple ways of combining vision and audio. The best configuration improves city-level accuracy from about 28 to 34 percent and country-level accuracy from about 61 to 69 percent over the single-frame visual baseline.

Critical strengths
  • New dataset for the audio-visual geolocalization task, consisting of 20.5k YouTube walking-tour clips
  • The idea of using visual and audio signals sequentially is novel. Audio is used to pick among candidate locations, not to predict location directly. This makes sense as Audio is a very noisy signal for gelocalization task.
Critical weaknesses
  • Dataset labels are generated automatically and seemingly not human verified ?

  • Most of the baselines evaluated are image geolocalization baselines, video geolocalization methods like CityGuessr and VidTAG are not evaluated.

  • Most of the headline gain comes from combining 12 frames, not from audio. (There is only +3.0 city accuracy gain against the fair multi-frame baseline). Table 2 compares against single-frame GLOBE. Table 10 splits the 2,054 test clips into 1,303 low-entropy and 751 high-entropy clips. Weighting those back up, the visual prior alone (cluster prior, no audio) gets about 33.1% city and TAP gets about 34.6%. So impact of the audio component might be only 1% to 1.5%.

  • Figure 2 is LLM generated and seems to have a few errors, e.g. Weiszfeld Median text is struck through and the arrow is only an arrowhead.

Questions for authors
  1. Can we get a human review of at least a subset of data ?
  2. Could you reproduce a video based GeoLocalization baseline from CityGuessr [Kulkarni et al. (2024)] or VidTAG[Kulkarni et al. (2026)] ?
  3. Can you provide some simple Audio as a tie-breaker baselines, e.g. using GLOBE per frame and then using GeoCLAP among those predictions as tiebreaker ?
  4. can you report multi-frame GLOBE baseline using Cluster prior (largest cluster) results instead of 12-frame spherical median ?

I will be willing to raise my rating if these specific concerns are addressed satisfactorily.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.