- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AVLoc, an audio-visual framework for floorplan localization that augments egocentric RGB observations with binaural acoustic echoes. The central motivation is that visual floorplan localization is fundamentally limited by the camera’s restricted field of view and vulnerability to environment, whereas acoustic echoes provide complementary geometric information and fail under different conditions. AVLoc employs three experts (i.e., vision-only, audio-only, and audio-visual), each predicting directional wall distances together with heteroscedastic uncertainty. Their predictions are combined through a ray-wise uncertainty-conditioned router, and the fused prediction and uncertainty are subsequently used for floorplan matching.
Critical strengths
The paper identifies an important limitation of vision-only floorplan localization: uncertainty estimation can indicate unreliable visual evidence but cannot recover geometry outside the camera field of view. Introducing binaural echoes therefore provides genuinely complementary geometric information and improves observability of the localization problem. Besides, it provides strong empirical performance with a reasonably controlled robustness evaluation.
Critical weaknesses
1 The use of audio-visual sensing, binaural echoes, and acoustic cues for recovering geometry beyond the visual field of view has substantial precedent in audio-visual spatial perception and floorplan-related tasks. Within this paper, the architecture itself follows a relatively standard design with vision-only, audio-only, and joint audio-visual experts followed by adaptive fusion.
Therefore, the main novelty seems to lie in adapting these ideas to known-floorplan localization.
2 The joint audio-visual expert alone already achieves 72.3% Recall@1m, compared with 74.7% for the full model, while the global router achieves 73.5%. In contrast, introducing audio produces much larger gains over the vision-only expert. This suggests that the additional acoustic observation may be the dominant source of improvement, whereas the incremental benefit of ray-wise uncertainty routing is comparatively small.
3 The paper argues that reliability should be resolved independently for each ray because modality quality is direction dependent. However, the strongest evidence for this claim is qualitative weight visualization, while the quantitative router ablation is mainly reported under clean conditions. A full comparison of ray-wise, global, confidence-based, and uniform routing under the same corruption grid is necessary to demonstrate that per-ray routing is specifically responsible for robustness.
4 Uncertainty is treated as expert reliability without direct calibration evidence. The predicted Laplace scale is used both as an uncertainty estimate and as the input to the routing mechanism. Although it is trained using heteroscedastic NLL, the paper does not establish whether this uncertainty is calibrated or strongly correlated with actual per-ray prediction errors under clean and corrupted conditions.
5 The manuscript states that visual degradation does not affect out-of-FOV directions. However, while this is true for the audio-only expert, the joint audio-visual expert allows all ray tokens to attend to visual features and therefore may still propagate visual corruption into out-of-FOV predictions. The paper should distinguish architectural independence from empirical trends in router weights.
6 The real-world evaluation is too small to support strong generalization claims.
Questions for authors
1.Can the authors provide robustness-grid results for the ray-wise router, global router, direct-confidence router, and uniform router under identical visual/audio corruption settings? This would directly test whether per-ray routing is essential under degradation.
2.How well calibrated are the predicted uncertainty values? Can the authors report the correlation between predicted scale and actual ray error, or calibration/coverage metrics under clean and corrupted conditions?
3.Since the audio-visual expert attends to visual tokens for all 36 rays, how should the statement that visual corruption does not affect out-of-FOV directions be interpreted?
4.Can the authors more clearly disentangle the contribution of additional acoustic sensing from the contribution of the proposed uncertainty-guided routing? For example, how much of the robustness gain remains when comparing against a strong joint audio-visual model with a simpler fusion strategy?
5.Can the authors report real-world results per floor/map and provide paired AVLoc-vs-UnLoc success statistics, together with the complete acoustic acquisition protocol?
6.Please reconcile the inconsistent Gibson scene/floor statistics across the main text, Table 3, and Appendix D.2
7.More broadly, what methodological insight from the proposed formulation is expected to transfer beyond this specific floorplan localization setting, beyond the observation that adding a complementary 360° acoustic sensor mitigates limited visual observability?
4 comments
Response to Reviewer riEz 2
Q5 & W6: Detailed Real-World Results and Acquisition Protocol
We expanded the real-world evaluation from 44 observations on 7 maps to 95 observations on 10 floor maps. Overall Recall@1m is 25.3% for AVLoc and 17.9% for UnLoc (updated Table 2). We report the breakdown by environment type since some individual maps have very few observations:
| Environment | Floorplans | Samples | UnLoc | AVLoc |
|---|---|---|---|---|
| Home | 1 | 26 | 42.3 | 50.0 |
| Lounge | 3 | 14 | 14.3 | 14.3 |
| Classroom | 3 | 29 | 6.9 | 17.2 |
| TA office | 1 | 7 | 14.3 | 0.0 |
| Lab | 1 | 7 | 14.3 | 42.9 |
| Large map | 1 | 12 | 0.0 | 8.3 |
| Overall | 10 | 95 | 17.9 | 25.3 |
We also provide paired success statistics on the same 95 observations to show how often AVLoc succeeds when UnLoc fails:
| Criterion | Both correct | Only AVLoc correct | Only UnLoc correct |
|---|---|---|---|
| [email protected] | 5 | 12 | 3 |
| Recall@1m | 8 | 16 | 9 |
| Recall@1m, 30° | 6 | 13 | 4 |
This provides further evidence that acoustic observations help recover cases where vision alone is insufficient. We discuss the sim-to-real gap in more detail in our response to Q1 of Reviewer xnLp.
Acquisition protocol: We use a RealSense camera, a 3Dio binaural microphone, and a speaker emitting a 50 ms logarithmic sine sweep (20 Hz–20 kHz). We record at 48 kHz and use the first 100 ms of the response, as in simulation (Appendix C). The sensors are mounted on a common rig at a nominal height of 1.7 m. We capture multiple observations on each floor map and label their ground-truth positions on the corresponding 2D floorplans.
Q6: Reconcile the inconsistent Gibson scene/floor statistics across the main text, Table 3, and Appendix D.2
We have clarified and reconciled the dataset statistics in Sec. 4 and Appendix D (Table 4 and Sec. D.2).
The original Gibson benchmark contains 118 floorplans (100 train / 9 val / 9 test). SoundSpaces 2 cannot render 24 of them (23 train / 1 val), leaving 94 floorplans (77 / 8 / 9) from 78 scenes/buildings (61 / 8 / 9) for AVLoc.
The nine test floorplans are unchanged. AVLoc uses the reduced audio-compatible training split, while the vision-only baselines use the original training data. AVLoc therefore does not have an advantage from training on more environments.
Q7: What Methodological Insights Transfer Beyond Floorplan Localization?
The key methodological insight of AVLoc goes beyond adding a complementary sensor: robust multimodal fusion requires preserving independent prediction paths and adapting to the failure mode, reliability, and spatial coverage of each modality.
AVLoc addresses this by retaining independent unimodal prediction paths, keeping a joint expert for cross-modal information, and learning weights at the spatial resolution where reliability varies. Its predicted uncertainty is also carried into the downstream matching step instead of being discarded after fusion. The corruption-grid experiments in Appendices H and L support these choices in our setting.
This design could apply to other tasks where sensors have different coverage and failure modes, such as camera-LiDAR or RGB-thermal fusion.
Thank you for your detailed responses and additional experiments. My concerns have been adequately addressed. One minor suggestion is that the manuscript layout appears somewhat crowded after the revision (e.g. fig.7) and could benefit from some further refinement. I appreciate the authors' efforts and am happy to increase my rating.
We thank the reviewer for engaging with our rebuttal and reassessing our work. We will address the presentation issue in the final manuscript.
Response to Reviewer riEz 1
Q1 & W3: Robustness Comparison Across Routing Strategies
Following the reviewer's request, we evaluated adaptive per-ray (ours), global, direct-confidence, and uniform routing under identical conditions across a 6 × 5 corruption grid of audio SNRs (clean, 30, 10, 0, −10, −15 dB) and visual severities (clean, 0.3, 0.6, 0.9, 1.0) on Gibson(f). The full results are provided in Appendix H, Fig. 8 (new).
Our results show that per-ray routing consistently performs best, with its advantage becoming more pronounced under severe degradation. Direct-confidence and uniform routing underperform our method by at least 5.6 and 4.4 percentage points, respectively, across all settings. While global routing performs similarly under clean conditions, its performance drops substantially relative to per-ray routing as corruption increases, particularly under severe visual degradation.
These results confirm that our per-ray routing is especially important for robustness, as it allows the model to adapt expert weights to the reliability of each direction rather than relying on a single set of global weights.
Q2 & W4: Can the authors report the correlation between predicted scale and actual ray error?
We measured the Spearman rank correlation between predicted per-ray Laplace scale and absolute ray error on the same 500 Gibson(f) test observations across a 3 × 3 visual/audio corruption grid (18,000 rays per condition). The results are:
The correlations are positive in every setting, ranging from 0.583 to 0.669, including when both modalities are heavily degraded. This supports using predicted scale as a signal of relative ray difficulty.
While rank correlation does not establish probabilistic calibration, it demonstrates that the predicted uncertainty provides a meaningful signal of relative prediction reliability, which is what our adaptive expert router uses to determine fusion weights.
Q3 & W5: How Does Visual Corruption Affect Out-of-FOV Predictions?
We thank the reviewer for pointing out this distinction. Visual corruption can indeed affect out-of-FOV predictions from the audio-visual expert, since all 36 ray tokens attend to both audio and visual features. Our intended point is that the audio-only expert remains independent of visual corruption, providing a reliable prediction path for all 360° directions.
Importantly, the audio-visual expert can also benefit from visual context outside the FOV by extrapolating nearby geometry, such as walls extending beyond the camera's view (Sec. 4.6). However, this also makes its predictions potentially sensitive to visual degradation. Our uncertainty-guided router is designed to handle this by adjusting the weights between the audio-only and audio-visual experts based on their predicted reliability.
Empirically, Fig. 6 and Appendix M (Figs. 11-13) show that out-of-FOV routing weights remain relatively stable under increasing visual corruption, suggesting that the model relies primarily on acoustic evidence in these directions. We will clarify the wording in the paper to distinguish the architectural independence of the audio-only expert from the observed behavior of the joint expert.
Q4: Disentangling the Contributions of Acoustic Sensing and Uncertainty-Guided Routing
We agree that adding audio explains the larger part of the clean-condition gain. On Gibson(f), the vision-only expert reaches 48.2% Recall@1m, while the joint audio-visual expert reaches 72.3% (Table 3). The full model reaches 74.7%, compared with 73.5% for global routing, 67.8% for direct-confidence routing, and 62.8% for uniform routing.
The two contributions address different problems. Acoustic observations add geometric evidence, especially outside the camera FOV. The unimodal experts preserve prediction paths that cannot be contaminated by the other sensor, and the learned per-ray router changes their weights when reliability varies by direction. These design choices matter most under corruption, not necessarily when both inputs are clean.
We compare AVLoc directly with the joint audio-visual expert across the corruption grid (Appendix L, Fig. 10) and compare routing strategies under identical corruptions (Appendix H, Fig. 8). The full model outperforms the joint expert across the tested conditions, and the per-ray router has a larger advantage over global routing under severe visual degradation. These results support the fusion contribution separately from the benefit of adding audio, without claiming that routing explains the full gain in clean conditions.