acceptodds
Under review as a conference paper at ICLR 2027

AVLoc: Robust Audio-Visual Floorplan Localization under Sensor Noise and Environmental Change

Abstract

Floorplan localization estimates an agent's pose within a 2D floorplan from egocentric observations. Existing methods draw every prediction from a single sensor, a camera, whose limited field of view and vulnerability to dim corridors, motion blur, sensor noise, and occluded walls mean that one corrupted image corrupts every cue the model has. We present AVLoc, which pairs vision with binaural acoustic echoes. Echoes are governed by geometry rather than appearance, they cover the full , and are largely unaffected by the conditions that disable a camera. Complementarity alone does not give robustness, since a joint encoder can propagate corruption from one modality into the other. AVLoc therefore combines unimodal and multimodal experts, each predicting the surrounding layout with its uncertainty. The router weights the experts by these uncertainty estimates, so the model dynamically shifts its reliance toward whichever modality the current observation supports, and the fused uncertainty carries through to floorplan matching. % rather than being discarded at fusion. Under maximum visual corruption, AVLoc retains recall@1m against for the strongest vision-only baseline, and when vision and audio are degraded at once. Because echoes also constrain geometry beyond the field of view, AVLoc improves clean-condition accuracy as well, raising recall@1m from to on Gibson(f) and from to on HM3D. Code and benchmarks will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual, floorplan localization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_riEzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AVLoc, an audio-visual framework for floorplan localization that augments egocentric RGB observations with binaural acoustic echoes. The central motivation is that visual floorplan localization is fundamentally limited by the camera’s restricted field of view and vulnerability to environment, whereas acoustic echoes provide complementary geometric information and fail under different conditions. AVLoc employs three experts (i.e., vision-only, audio-only, and audio-visual), each predicting directional wall distances together with heteroscedastic uncertainty. Their predictions are combined through a ray-wise uncertainty-conditioned router, and the fused prediction and uncertainty are subsequently used for floorplan matching.

Critical strengths

The paper identifies an important limitation of vision-only floorplan localization: uncertainty estimation can indicate unreliable visual evidence but cannot recover geometry outside the camera field of view. Introducing binaural echoes therefore provides genuinely complementary geometric information and improves observability of the localization problem. Besides, it provides strong empirical performance with a reasonably controlled robustness evaluation.

Critical weaknesses

1 The use of audio-visual sensing, binaural echoes, and acoustic cues for recovering geometry beyond the visual field of view has substantial precedent in audio-visual spatial perception and floorplan-related tasks. Within this paper, the architecture itself follows a relatively standard design with vision-only, audio-only, and joint audio-visual experts followed by adaptive fusion.
Therefore, the main novelty seems to lie in adapting these ideas to known-floorplan localization.

2 The joint audio-visual expert alone already achieves 72.3% Recall@1m, compared with 74.7% for the full model, while the global router achieves 73.5%. In contrast, introducing audio produces much larger gains over the vision-only expert. This suggests that the additional acoustic observation may be the dominant source of improvement, whereas the incremental benefit of ray-wise uncertainty routing is comparatively small.

3 The paper argues that reliability should be resolved independently for each ray because modality quality is direction dependent. However, the strongest evidence for this claim is qualitative weight visualization, while the quantitative router ablation is mainly reported under clean conditions. A full comparison of ray-wise, global, confidence-based, and uniform routing under the same corruption grid is necessary to demonstrate that per-ray routing is specifically responsible for robustness.

4 Uncertainty is treated as expert reliability without direct calibration evidence. The predicted Laplace scale is used both as an uncertainty estimate and as the input to the routing mechanism. Although it is trained using heteroscedastic NLL, the paper does not establish whether this uncertainty is calibrated or strongly correlated with actual per-ray prediction errors under clean and corrupted conditions.

5 The manuscript states that visual degradation does not affect out-of-FOV directions. However, while this is true for the audio-only expert, the joint audio-visual expert allows all ray tokens to attend to visual features and therefore may still propagate visual corruption into out-of-FOV predictions. The paper should distinguish architectural independence from empirical trends in router weights.

6 The real-world evaluation is too small to support strong generalization claims.

Questions for authors

1.Can the authors provide robustness-grid results for the ray-wise router, global router, direct-confidence router, and uniform router under identical visual/audio corruption settings? This would directly test whether per-ray routing is essential under degradation.

2.How well calibrated are the predicted uncertainty values? Can the authors report the correlation between predicted scale and actual ray error, or calibration/coverage metrics under clean and corrupted conditions?

3.Since the audio-visual expert attends to visual tokens for all 36 rays, how should the statement that visual corruption does not affect out-of-FOV directions be interpreted?

4.Can the authors more clearly disentangle the contribution of additional acoustic sensing from the contribution of the proposed uncertainty-guided routing? For example, how much of the robustness gain remains when comparing against a strong joint audio-visual model with a simpler fusion strategy?

5.Can the authors report real-world results per floor/map and provide paired AVLoc-vs-UnLoc success statistics, together with the complete acoustic acquisition protocol?

6.Please reconcile the inconsistent Gibson scene/floor statistics across the main text, Table 3, and Appendix D.2

7.More broadly, what methodological insight from the proposed formulation is expected to transfer beyond this specific floorplan localization setting, beyond the observation that adding a complementary 360° acoustic sensor mitigates limited visual observability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer riEz 1

Q1 & W3: Robustness Comparison Across Routing Strategies

Following the reviewer's request, we evaluated adaptive per-ray (ours), global, direct-confidence, and uniform routing under identical conditions across a 6 × 5 corruption grid of audio SNRs (clean, 30, 10, 0, −10, −15 dB) and visual severities (clean, 0.3, 0.6, 0.9, 1.0) on Gibson(f). The full results are provided in Appendix H, Fig. 8 (new).

Our results show that per-ray routing consistently performs best, with its advantage becoming more pronounced under severe degradation. Direct-confidence and uniform routing underperform our method by at least 5.6 and 4.4 percentage points, respectively, across all settings. While global routing performs similarly under clean conditions, its performance drops substantially relative to per-ray routing as corruption increases, particularly under severe visual degradation.

These results confirm that our per-ray routing is especially important for robustness, as it allows the model to adapt expert weights to the reliability of each direction rather than relying on a single set of global weights.

Q2 & W4: Can the authors report the correlation between predicted scale and actual ray error?

We measured the Spearman rank correlation between predicted per-ray Laplace scale and absolute ray error on the same 500 Gibson(f) test observations across a 3 × 3 visual/audio corruption grid (18,000 rays per condition). The results are:

Vision: CleanVision: 0.6Vision: 1.0
Audio: Clean0.6470.6450.642
Audio SNR: 00.6630.6500.627
Audio SNR: −150.6690.6370.583

The correlations are positive in every setting, ranging from 0.583 to 0.669, including when both modalities are heavily degraded. This supports using predicted scale as a signal of relative ray difficulty.

While rank correlation does not establish probabilistic calibration, it demonstrates that the predicted uncertainty provides a meaningful signal of relative prediction reliability, which is what our adaptive expert router uses to determine fusion weights.

Q3 & W5: How Does Visual Corruption Affect Out-of-FOV Predictions?

We thank the reviewer for pointing out this distinction. Visual corruption can indeed affect out-of-FOV predictions from the audio-visual expert, since all 36 ray tokens attend to both audio and visual features. Our intended point is that the audio-only expert remains independent of visual corruption, providing a reliable prediction path for all 360° directions.

Importantly, the audio-visual expert can also benefit from visual context outside the FOV by extrapolating nearby geometry, such as walls extending beyond the camera's view (Sec. 4.6). However, this also makes its predictions potentially sensitive to visual degradation. Our uncertainty-guided router is designed to handle this by adjusting the weights between the audio-only and audio-visual experts based on their predicted reliability.

Empirically, Fig. 6 and Appendix M (Figs. 11-13) show that out-of-FOV routing weights remain relatively stable under increasing visual corruption, suggesting that the model relies primarily on acoustic evidence in these directions. We will clarify the wording in the paper to distinguish the architectural independence of the audio-only expert from the observed behavior of the joint expert.

Q4: Disentangling the Contributions of Acoustic Sensing and Uncertainty-Guided Routing

We agree that adding audio explains the larger part of the clean-condition gain. On Gibson(f), the vision-only expert reaches 48.2% Recall@1m, while the joint audio-visual expert reaches 72.3% (Table 3). The full model reaches 74.7%, compared with 73.5% for global routing, 67.8% for direct-confidence routing, and 62.8% for uniform routing.

The two contributions address different problems. Acoustic observations add geometric evidence, especially outside the camera FOV. The unimodal experts preserve prediction paths that cannot be contaminated by the other sensor, and the learned per-ray router changes their weights when reliability varies by direction. These design choices matter most under corruption, not necessarily when both inputs are clean.

We compare AVLoc directly with the joint audio-visual expert across the corruption grid (Appendix L, Fig. 10) and compare routing strategies under identical corruptions (Appendix H, Fig. 8). The full model outperforms the joint expert across the tested conditions, and the per-ray router has a larger advantage over global routing under severe visual degradation. These results support the fusion contribution separately from the benefit of adding audio, without claiming that routing explains the full gain in clean conditions.

AuthorsOpenReview ↗

Response to Reviewer riEz 2

Q5 & W6: Detailed Real-World Results and Acquisition Protocol

We expanded the real-world evaluation from 44 observations on 7 maps to 95 observations on 10 floor maps. Overall Recall@1m is 25.3% for AVLoc and 17.9% for UnLoc (updated Table 2). We report the breakdown by environment type since some individual maps have very few observations:

EnvironmentFloorplansSamplesUnLocAVLoc
Home12642.350.0
Lounge31414.314.3
Classroom3296.917.2
TA office1714.30.0
Lab1714.342.9
Large map1120.08.3
Overall109517.925.3

We also provide paired success statistics on the same 95 observations to show how often AVLoc succeeds when UnLoc fails:

CriterionBoth correctOnly AVLoc correctOnly UnLoc correct
[email protected]5123
Recall@1m8169
Recall@1m, 30°6134

This provides further evidence that acoustic observations help recover cases where vision alone is insufficient. We discuss the sim-to-real gap in more detail in our response to Q1 of Reviewer xnLp.

Acquisition protocol: We use a RealSense camera, a 3Dio binaural microphone, and a speaker emitting a 50 ms logarithmic sine sweep (20 Hz–20 kHz). We record at 48 kHz and use the first 100 ms of the response, as in simulation (Appendix C). The sensors are mounted on a common rig at a nominal height of 1.7 m. We capture multiple observations on each floor map and label their ground-truth positions on the corresponding 2D floorplans.

Q6: Reconcile the inconsistent Gibson scene/floor statistics across the main text, Table 3, and Appendix D.2

We have clarified and reconciled the dataset statistics in Sec. 4 and Appendix D (Table 4 and Sec. D.2).

The original Gibson benchmark contains 118 floorplans (100 train / 9 val / 9 test). SoundSpaces 2 cannot render 24 of them (23 train / 1 val), leaving 94 floorplans (77 / 8 / 9) from 78 scenes/buildings (61 / 8 / 9) for AVLoc.

The nine test floorplans are unchanged. AVLoc uses the reduced audio-compatible training split, while the vision-only baselines use the original training data. AVLoc therefore does not have an advantage from training on more environments.

Q7: What Methodological Insights Transfer Beyond Floorplan Localization?

The key methodological insight of AVLoc goes beyond adding a complementary sensor: robust multimodal fusion requires preserving independent prediction paths and adapting to the failure mode, reliability, and spatial coverage of each modality.

AVLoc addresses this by retaining independent unimodal prediction paths, keeping a joint expert for cross-modal information, and learning weights at the spatial resolution where reliability varies. Its predicted uncertainty is also carried into the downstream matching step instead of being discarded after fusion. The corruption-grid experiments in Appendices H and L support these choices in our setting.

This design could apply to other tasks where sensors have different coverage and failure modes, such as camera-LiDAR or RGB-thermal fusion.

Thank you for your detailed responses and additional experiments. My concerns have been adequately addressed. One minor suggestion is that the manuscript layout appears somewhat crowded after the revision (e.g. fig.7) and could benefit from some further refinement. I appreciate the authors' efforts and am happy to increase my rating.

We thank the reviewer for engaging with our rebuttal and reassessing our work. We will address the presentation issue in the final manuscript.

Reviewer_ZToiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AVLoc, a pipeline for using audio and visual signals for 2D floorplan localization. These signals are pushed through a network that predits a ray-map from each. AVLoc uses three Transformer-based branches: a vision-only expert (predicting 11 rays within the 106 FOV), an audio-only expert (predicting 36 rays across 360), and an audio-visual expert (predicting 36 rays from concatenated tokens). These rays are the fused by uncertainty-weighted fusion. The authors augment Gibson and HM3D (including a multi-floor split, HM3D-Multi) with simulated echoes from SoundSpaces 2.0 and demonstrate gains over vision-only baselines in both clean and synthetically degraded settings, alongside a small-scale zero-shot test on real-world samples.

Critical strengths
  • The core idea of combining vision and active audio for floorplan localization is interesting and well-motivated. Binaural echoes naturally complement perspective cameras by providing 360-degree omnidirectional geometric cues and remaining unaffected by visual degradations such as low lighting or motion blur. Let me note that, I believe, they should be affected by the materials in the environment, however, this is not addressed at all in the paper (I don't know how much of this is actually a problem in practice.)
Critical weaknesses
  • Floorplan localization frequently relies on RF signals such as Wi-Fi, Bluetooth , or Ultra-Wideband, which also provide omnidirectional constraints. It is unclear how sound compares/complements these. While audio may generalize to environments without access to RF infrastructure, the authors mainly evaluate in synthetic environments where comparing against or analyzing such signals is straightforward. I don't say audio is not more practical in some cases, but I see both pros (e.g., works when we have no access to RF) and cons (what happens in noisy environments, e.g., hospital, concert halls, subway stations, .....?) for using audio. AVLoc requires continuously emitting audible 20 Hz-20 kHz chirps, which is intrusive and impractical in human-occupied indoor space. Of course, this can probably be changed, but it is what it is at the moment. Also, it is unclear to me what happens when there are non-geometric occluders (e.g., dynamic objects that will break the sounds pattern).
  • Since, to my understanding, there is nothing particularly novel theoretically or methodologically in the architecture, the paper requires rocksolid experiments, showing that audio-visual ray prediction transfers to the real world. Almost the entire evaluation is done in simulation. While Section 4.4 includes a small zero-shot test on 44 static real-world samples across 7 floor maps, AVLoc's Recall@1m collapses from 74.7% to 15.9% in the real world (I see that other methods also collapse). Rather than demonstrating practical benefits, a 15.9% recall highlights how severe the sim-to-real gap is. There are now some methods that work zero-shot with visual signals (ZFLoc; I know it is unpublished so I don't ask for comparison). At the moment, AVLoc has to be trained environment-by-environment which was already hard with visual-only method (requiring full 3D reconstruction of the scene and a floorplan). Now, this method also needs to capture sound. This is making it even less practical.
  • There is no comparison with audio-visual depth/echolocation architectures - such as VisualEchoes (ECCV 2020), BatVision (ICRA 2020 / IROS 2023), or EchoScan (TASLP 2024) - adapted to predict rays within the localization pipeline.
  • Both the audio-to-ray and vision-to-ray processes and described on a very very high level, so I actually have no idea what is going on before the "each provides a ray map" point. Then, it is clear, the method fuses the measurements with uncertainty-weighted fusion, but the most interesting part: how to get rays from audio is mostly missing. Or is it simply pushing all signals through the transformer and hoping for the network to figure out the connection?
Questions for authors
  • What happens with method in very noisy environments?
  • What happens in dynamic environments where the echoes may not reflect actual geometry? These are reasonably simple to filter in the visual signal.
  • What happens with the method if one modality fails completely (i.e., weighted fusion will not help unless the weight is exactly 0)
  • How would active audio compare to or complement ubiquitous indoor RF signals (Wi-Fi/BLE) in your simulation setup, and can AVLoc operate with opportunistic ambient sound sources in the environment rather than intrusive 50 ms chirps emitted by the robot?
  • What is the exact process to make it work in new environments? Full data capture, reconstruction, and fine-tuning/training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZToi 1

Q1 & W1: Robustness to Very Noisy Environments

We explicitly consider this setting and evaluate AVLoc under different types of ambient noise from ESC-50 [6], as well as Gaussian noise, at SNRs of 10, 0, −10, and −15 dB (severe noise) (Sec. 4.2, Fig. 4). While strong noise can mask acoustic reflections and reduce localization accuracy, AVLoc's performance degrades gradually rather than collapsing and better than the vision-only baseline under the tested corruption levels.

One reason is the spectral structure of the noise. The chirp covers a wide frequency range, while many ambient sounds occupy only parts of the spectrum, leaving some acoustic information available even under strong noise (Fig. 16). Broadband Gaussian noise can be particularly challenging because it interferes with the entire frequency spectrum.

Importantly, AVLoc dynamically adjusts its reliance on each modality as noise increases. As shown in Fig. 6 and Fig. 11, when audio quality decreases, the router shifts more weight toward the vision-only expert within the camera's FOV. This adaptive behavior allows AVLoc to rely more on visual information when acoustic observations become unreliable, improving robustness in noisy environments.

Q2 & W1: What happens when there are non-geometric occluders?

SoundSpaces 2.0 [7] simulates acoustic reflections from all scene meshes, including furniture and other occluders. However, our supervision uses only the distances to walls obtained from the floorplan (Lines 771-772). Thus, the audio expert therefore learns to predict wall geometry despite some intervening objects.

We further investigate this behavior qualitatively in Appendix I (Fig. 9). Our audio expert successfully ignores common occluders, such as furniture (Fig. 9a) and columns (Fig. 9b), predicting the walls behind them. However, a challenging case arises when a large, wall-like object is very close to the agent (Fig. 9c). Such objects can produce strong reflections that resemble those of actual walls, causing incorrect ray predictions in the affected directions.

Further, each observation uses the first 100 ms of the recorded response, including the 50 ms sweep (Appendix C). This short window limits motion during acquisition. However, we didn't evaluate this setup with moving objects directly.

Overall, the previous results show that AVLoc can handle common occluders but remains sensitive to large, nearby wall-like objects. We have added this analysis and discussion of the limitation in the new Appendix I.

Q3: What Happens if One Modality Fails Completely?

The unimodal experts provide paths that are independent of the other sensor, so the router can reduce the weight of an unreliable expert. Because the weights come from a softmax, they approach zero but are not exactly zero.

Our robustness evaluation includes extreme degradation levels that simulate near-complete modality failure. Fig. 13 and 12 illustrates how AVLoc adapts its routing weights under these conditions.

For visual failure, the middle example in Fig. 13 shows increasing visual degradation, reaching complete darkness (black image). In this case, the vision-only expert's weight drops to nearly zero, and the router shifts its reliance toward the audio and audio-visual experts.

For audio failure, the top example shows severe Gaussian noise at −15 dB SNR, which heavily masks the acoustic response. As audio quality decreases, the router reduces the audio-only expert's contribution and relies more on vision within the FOV. Interestingly, the audio-visual expert retains substantial weight outside the FOV. Unlike visual failure, where audio can directly replace missing visual information, audio is the only modality providing direct geometric cues outside the FOV. We believe the audio-visual expert can partially compensate by using visual context to infer geometry beyond the FOV, together with any remaining acoustic information.

We also examine complete input removal in Appendix M (Fig. 12), using all-zero audio and a black image, separately and together. With zero audio, the router almost suppresses the audio-only expert and increases the vision-only weight within the FOV. With a black image, it suppresses the vision-only expert and shifts weight toward experts using audio. When both inputs are removed, it puts most weight on the joint expert. Note, these are routing-weight results, not measurements of localization success under total sensor failure. In particular, choosing the joint expert when both inputs are absent does not imply that its predictions remain accurate.

AuthorsOpenReview ↗

Response to Reviewer ZToi 2

Q4.a & W1: Comparison with RF-Based Localization

As the reviewer noted, many RF-based localization methods rely on existing infrastructure, such as Wi-Fi access points or deployed beacons. In contrast, AVLoc uses only onboard sensors (camera, speaker, and binaural microphones), making it applicable to environments where RF infrastructure is unavailable or costly to deploy.

We see the approaches as complementary. RF could provide a coarse position prior, while AVLoc could use local geometric constraints to refine it.

Q4.b: Can AVLoc Use Ambient Sounds Instead of Emitting a Chirp?

We test other known probe signals in Appendix F (Table 6). When trained and evaluated with emitted speech and music clips instead of chirps, AVLoc reaches 60.9% and 61.6% Recall@1m, compared with 74.7% for chirps. Both remain above DisCo-FLoc's 56.7% on Gibson(f).

Using naturally occurring ambient sounds without actively emitting a signal is also an interesting direction. However, passive sensing introduces additional challenges: unknown source locations, timing, and content. While our current experiments use controlled sound emissions, exploring localization from passive ambient sounds is a promising direction for future work.

Q5: What Is the Process for Deploying AVLoc in New Environments?

AVLoc does not need training or fine-tuning for each new building. Like prior ray-based methods (F3Loc [1], UnLoc [2]), AVLoc is trained on one set of buildings and evaluated on disjoint buildings in Gibson and HM3D (Appendix D). All reported results are evaluated on buildings unseen during training. At deployment, AVLoc needs a 2D floorplan and an audio-visual observation. It predicts the surrounding wall distances and matches them to candidate poses on the floorplan. No posed-image database or building-specific 3D reconstruction is required.

Z-FLoc [3] also aims to avoid scene-specific training, but uses multiple images (15 to 100 frames) and relative camera poses to form local geometric evidence. AVLoc can make a single-view prediction and can also integrate observations over time (Fig. 5). We will add this distinction to related work.

W2: Novelty of AVLoc

While individual components such as Transformers and uncertainty estimation are established, AVLoc's methodological novelty lies in how it combines complementary sensing modalities that have different spatial coverage and failure modes to achieve robust floorplan localization.

First, we introduce audio-visual floorplan localization using omnidirectional acoustic echoes, which provide geometric information beyond the camera's FOV. Unlike prior audio-visual depth estimation methods, AVLoc directly predicts 360° wall distances for matching against floorplans.

Second, our multi-expert design preserves independent prediction paths for each modality, preventing corruption in one modality from contaminating all predictions, while the joint expert captures cross-modal interactions. Our ray-wise uncertainty-guided router then dynamically adjusts expert contributions based on their reliability and spatial coverage, propagating the fused uncertainty into floorplan matching.

Importantly, we distinguish the benefits of adding audio from those of our fusion design. While audio provides the largest accuracy gains under clean conditions, our architecture is particularly important for robustness. The full model outperforms the joint audio-visual expert alone (74.7% vs. 72.3% Recall@1m), while uniform routing drops to 62.8% (Table 3). More importantly, Appendices H and L show that the gains from our design become larger under modality degradation, where simpler fusion strategies and the joint expert are less reliable.

W2: Sim-to-Real Performance Drop

To better understand the sim-to-real gap, we expanded our real-world evaluation from 44 to 95 samples across 10 floor maps. On this larger dataset, AVLoc achieves 25.3% Recall@1m, compared to 17.9% for UnLoc (updated Table 2), an improvement of 7.4 percentage points.

The environment breakdown in our response to xnLp Q1 shows that both models work better in the home environment than in most other settings. Since Gibson mainly contains residential scenes, this suggests that differences in scene types and visual appearance contribute to the performance drop. Acoustic domain differences may also play a role, but the fact that vision-only methods experience a similar drop indicates that this is a broader challenge for floorplan localization, rather than a limitation specific to acoustic sensing.

Despite this challenging domain shift, AVLoc outperforms UnLoc overall on the expanded real-world dataset, showing that adding acoustic cues remains beneficial beyond simulation.

AuthorsOpenReview ↗

Response to Reviewer ZToi 3

W3: Comparison with Audio-Visual Depth and Echolocation Methods

We thank the reviewer for pointing out these works. VisualEchoes learns visual representations using echoes but relies only on RGB at inference, while BatVision predicts depth from acoustic echoes alone. Unlike AVLoc, neither combines audio and vision at inference time, and both focus on FOV-restricted depth rather than omnidirectional wall geometry.

To investigate how well FOV-restricted depth supports floorplan localization, we conducted an additional experiment using ground-truth simulator depth, extracting 11 forward-facing rays from the center row of the depth image (with 5×5 patch averaging) and matching them against the floorplan. On Gibson(f), this achieves only 51.5% Recall@1m, compared to 74.7% for AVLoc. This is also comparable to vision-only methods (UnLoc: 50.2%, DisCo-FLoc: 56.7%). We attribute this gap to two limitations: (1) forward-facing depth cannot capture geometry outside the FOV, and (2) depth measures visible surfaces, including furniture and other occluders, rather than the underlying walls. These results highlight why accurate FOV-restricted depth alone is insufficient for our localization task. (new: Appendix J)

EchoScan predicts top-down room geometry from acoustic echoes using fixed-size segmentation output that is designed for single-room layouts. Adapting such a representation to multi-room floorplan matching is not straightforward, whereas directional wall distances can be compared directly with floorplan rays. Our audio-only expert gives an acoustic-only reference point (46.7% Recall@1m), and combining both modalities reaches 74.7% (Table 3). We will clarify these distinctions in related work.

W4: How Are Audio Observations Mapped to Rays?

While the main text briefly described the audio branch, further details were provided in the appendix. We have now expanded Sec. 3.2 to clarify this process.

The mapping from echoes to rays is learned end-to-end, but we do not rely on the Transformer to discover directional information from raw audio alone. Following the Spatial-AST front-end [4], we represent the binaural response using left/right log-magnitude spectrograms and the sine and cosine of the interaural phase difference (Lines 294–296). The phase difference between the two microphones provides cues about the direction from which sound reflections arrive, giving the model explicit spatial information.

These features are encoded into audio tokens, which are processed alongside 36 learnable ray tokens, each associated with a specific direction through positional embeddings. Through self-attention, each ray token learns to extract relevant acoustic information for its direction, supervised by the corresponding ground-truth wall distance. Thus, the Transformer learns the mapping between acoustic reflections and wall geometry using both explicit binaural spatial cues and directional ray supervision, rather than learning this relationship from unstructured audio alone.

[1] Chen, Changan, et al. "F 3 Loc: fusion and filtering for floorplan localization." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Wüest, Matthias, et al. "Unloc: Leveraging depth uncertainties for floorplan localization." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Umemura, Ayumi, et al. "Z-FLoc: Zero-Shot Floorplan Localization via Geometric Primitives." arXiv preprint arXiv:2606.04788 (2026).

[4] Zheng, Zhisheng, et al. "Bat: Learning to reason about spatial sounds with large language models." arXiv preprint arXiv:2402.01591 (2024).

[5] Sarlin, Paul-Edouard, et al. "Lamar: Benchmarking localization and mapping for augmented reality." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[6] Piczak, Karol J. "ESC: Dataset for environmental sound classification." Proceedings of the 23rd ACM international conference on Multimedia. 2015.

[7] Chen, Changan, et al. "Soundspaces 2.0: A simulation platform for visual-acoustic learning." Advances in Neural Information Processing Systems 35 (2022): 8896-8911.

Reviewer_xnLpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AVLoc, an audio-visual floorplan localization framework that combines RGB observations with binaural acoustic echoes to estimate an agent's position+orientation on a 2D floorplan. Authors introduce three experts (unimodal vision and audio, and audio-visual) that predict wall distances and uncertainties along different directions. The experimental improvements on Gibson and HM3D are substantial, with improvements over vision-only floorplan localization methods under clean and degraded conditions with corrupted visual observations.

Critical strengths

Using acoustic echoes to provide complementary geometric information outside the camera's FoV is intuitive and potentially useful.

The robustness evaluation is reasonably comprehensive. Authors evaluate different levels of visual and acoustic corruption. Ablations in Table 2 provide useful evidence for the contribution of the individual experts and uncertainty-guided routing.

Critical weaknesses

The practical motivation is real-world floorplan localization, and the large sim-to-real gap raises questions about whether the learned acoustic representations generalize to actual environmental conditions and microphone configurations. Authors should comment on what drives the large sim-to-real drop from 74.7% to 15.9% Recall@1m

The active acoustic sensing design relies on strict, idealized conditions: Appendix C details the use of a 50 ms logarithmic sine sweep. Emitting a precise probe signal to measure room reflections presents practical challenges in real-world indoor settings with dynamic audio environments, moving occupants, varying room acoustics, or differing microphone placements. The reliance on this specific probing signal is evident in Appendix F's ablation study, substituting the chirp with speech or music causes recall@1m to decrease from 74.7% to 60.9% and 61.6%, respectively.

Ln 274–276 states that Gibson covers 118 scenes (100 training, 9 validation, 9 test) with 24,779 samples for Gibson(f) and 49,558 for Gibson(g). Appendix D.1 (Table 3) and D.2 (Ln 759–761) state that 24 scenes were excluded due to SoundSpaces 2, leaving 20,305 and 40,610 samples across 78 scenes (61 train, 8 val, 9 test) and 94 floors. Authors should clarify if Table 1 baselines are trained on the 100-scene split or 61-scene subset.

Questions for authors

Given the relatively small real-world evaluation set of 44 samples, it would help to report uncertainty estimates or additional results across different environments. Also how much is attributable to differences in microphone placement, or visual domain shift?

In table 2, why does Direct Confidence Routing degrade performance (-6.9) if the per-ray router MLP only receives scalar uncertainties as input, ?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xnLp 1

Q1 & W1: Sim-to-Real Gap and Analysis

We expanded the real-world test set from 44 to 95 observations across 10 floor maps. On this larger set, AVLoc achieves 25.3% Recall@1m (95% CI [17.6, 34.8]) compared with 17.9% (95% CI [11.5, 26.8]) for UnLoc. For Recall@1m&30°, AVLoc reaches 20.0% (95% CI [13.2, 29.1]) versus 10.5% (95% CI [5.8, 18.3]) for UnLoc. AVLoc obtains higher recall on both metrics, with a larger margin on the stricter joint metric (+9.5 vs. +7.4 points). The gains are encouraging, but the confidence intervals reflect the size of the real-world set, and the new larger evaluation narrows them compared to the original 44 samples.

We break down the results by environment type below (all values are percentages):

EnvironmentFloorplansSamplesR@1mR@1m&30°
UnLocAVLocUnLocAVLoc
Home12642.350.026.938.5
Lounge31414.314.37.17.1
Classroom3296.917.26.917.2
TA office1714.30.00.00.0
Lab1714.342.90.028.6
Large map1120.08.30.08.3
Overall109517.925.310.520.0

Both methods perform much better in the home environment than in most other settings. As Gibson consists mainly of residential scenes, this suggests that changes in scene type and visual appearance contribute to the sim-to-real gap. The drop in UnLoc also shows that the gap is not caused by acoustic sensing alone. While these results do not tell us how much exactly of AVLoc's drop comes from visual shift, acoustic shift, or their interaction, AVLoc maintains its advantage over UnLoc, showing that acoustic cues and our audio-visual approach remain beneficial even under domain shift.

In the real-world setup, the camera, binaural microphones, and speaker are mounted on a shared rig at a height of 1.7 m. In simulation, the sound source and binaural listener are placed at the same position and height (Appendix D.2, Lines: 759-760). The physical setup cannot reproduce this geometry exactly; source--microphone spacing and orientation, may contribute to the remaining gap. We have not isolated these factors experimentally. However, we view the expanded results as promising evidence of real-world feasibility.

Q2: "Why does Direct Confidence Routing degrade performance?"

Direct Confidence Routing converts each predicted Laplace scale into a weight using a fixed inverse-uncertainty rule. This assumes that the same rule gives suitable fusion weights for every expert and every direction. But a predicted scale describes an expert's ray error, not necessarily how useful that expert's prediction will be when combined with the others. For example, the audio-only and joint experts have different dependencies on the input modalities, and the vision-only expert is available only within the FOV.

Our per-ray MLP takes the available experts' uncertainty scales together and learns the mapping from uncertainty to fusion weights for each direction. It can therefore use different relationships for different experts and rays, instead of applying a fixed inverse-scale rule. This improves Recall@1m from 67.8% to 74.7%, a gain of 6.9 points (Table 3).

AuthorsOpenReview ↗

Response to Reviewer xnLp 2

W2: "The active acoustic sensing design relies on strict, idealized conditions"

AVLoc is not evaluated only in quiet rooms. During training, we add Gaussian noise and environmental sounds from ESC-50[1], with training and test sound categories kept disjoint (Appendix C). We test SNRs down to −15 dB, including held-out sounds such as laughter, dog barking, rain, sirens, and clock alarms (Sec. 4.2; Appendix C, Fig. 16). Strong noise lowers accuracy, but the model can shift weight toward vision when the acoustic input becomes unreliable.

Regarding environmental changes, our training data includes furnished scenes with different room sizes, layouts, and objects that may occlude walls (e.g., columns, closets, shelves, and refrigerators). The model is trained to predict wall distances despite these objects, rather than simply predicting distances to the nearest visible surfaces (Lines: 771-772). Appendix I (Fig. 9) shows cases where the audio expert predicts walls behind occluders, as well as a failure caused by a large wall-like object nearby. These experiments support robustness to some static occluders. Moreover, training and test scenes are disjoint, allowing us to evaluate generalization to unseen environments.

For moving occupants, AVLoc uses only the first 100 ms of the recorded echo response (Lines: 698-699). Within this short time window, most moving objects can be considered approximately stationary, limiting the effect of motion during acoustic acquisition. While changes in object positions can still alter the reflected signals, we expect training on furnished environments could provide some robustness to such occlusions.

W2: Reliance on Chirp Probing Signal

The chirp gives a known signal with broad frequency coverage, which helps the model interpret reflections, but it is not the only signal we tested. In Appendix F (Table 6), we replace the chirp with controlled speech or music clips during both training and evaluation. AVLoc still outperforms the strongest vision-only baseline, DisCo-FLoc (56.7% Recall@1m), by 4.2 and 4.9 percentage points, respectively. Our results show that chirps are beneficial but not strictly required for AVLoc to improve localization accuracy.

Passive sensing from ambient sounds has its own challenges since it would require handling unknown source positions and signal content. Developing models that can leverage passive audio effectively remains an open question.

W3: Clarification of Gibson Training Subset

We acknowledge the discrepancy and have clarified it in Sec. 4 and Appendix D (Table 4). The original Gibson floorplan benchmark contains 118 floorplans (100 train / 9 val / 9 test). SoundSpaces 2 cannot render audio for 24 of them (23 train / 1 val), leaving 94 floorplans (77 train / 8 val / 9 test) across 78 scenes/buildings (61 / 8 / 9).

AVLoc uses this reduced audio-compatible split. The vision-only baselines use the original training split, and all methods are evaluated on the same nine test floorplans, none of which was removed. Thus, AVLoc does not benefit from additional training environments. We will make the distinction between scenes and floorplans explicit throughout the paper.

[1] Piczak, Karol J. "ESC: Dataset for environmental sound classification." Proceedings of the 23rd ACM international conference on Multimedia. 2015.

We thank the reviewer again for the thoughtful review. Since the discussion phase ends today, we would like to check whether the reviewer has any additional questions or concerns that we can address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.