- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of echo-based panoramic depth estimation, in which depth at 360 degrees around the microphone(s) is estimated from reflections of sound emitted by the speaker. The proposed method of this paper is named HEAR360, composed of (1) the use of multiple binaural microphones, (2) pose-conditioned acoustic embedding, and (3) direction-aware panoramic reconstruction. The second component embeds the positions of time-frequency blocks and microphones, integrating them into acoustic features. Meanwhile, the third component employs a cross-attention architecture, in which a learnable ray-specific embedding is used as the query to obtain panoramic features. Experimental evaluations with synthetic datasets demonstrate that the proposed method outperforms simple baselines (just introducing pre-trained visual encoders for obtaining tokens from spectrograms) and several previous methods for non-panoramic eho-based depth estimation.
Critical strengths
S1. The proposed method is simple but reasonable. In particular, pose-conditioned embedding would be required for the current problem setting, since the position and direction of a microphone are critical to determining the corresponding ray in the panoramic output. Simple methods should be appreciated as a strength, especially from a practical viewpoint.
S2. The current paper is basically well-written and easy to follow. The conciseness of the proposed method and the readability of the paper are of great importance, particularly for engineers interested in the problems addressed here.
Critical weaknesses
W1. The most critical concern is the validity and novelty of the problem setting.
- I am afraid that depth estimation from a panoramic perspective does not seem to be a problem that significantly differs from the conventional 2D depth estimation. I understand that the above issue might be wrong in image-based depth estimation due to the geometric deformation. However, I could not understand the significant difference specific to acoustic-based prediction.
- As far as I know, the use of more than three microphones or ambisonic microphones for echo-based depth estimation is now becoming a standard approach. For example, [Chen+ ECCV2020, SoundSpaces: Audio-Visual Navigation in 3D Environments] is a popular infrastructure for generating ambisonics from a given indoor room structure, and [Honma+ ICIP2024, Estimating Indoor Scene Depth Maps From Ultrasonic Echoes] introduces a microphone array. Also, in the audio signal processing community, impulse response estimation using microphone arrays and ambisonic microphones is the first choice for predicting indoor 3D structures.
W2. Another critical concern is related to the experimental evaluations.
- Table 2 indicates that no proposed component performs poorly on any metric. Meanwhile, Table 1 indicates that using pretrained networks performs better than the baseline in Table 2. This implies that the baseline shown in Table 2 is either too weak or unfair.
- Table 2 also indicates that time-frequency embedding and ray embedding do not contribute to the performance improvement.
W3. The current paper does not clearly describe the position of speakers, which is critical for understanding the adverse effect of the direct sound from the speakers on the microphones. As the authors know, direct sound does not contain any information about room structure, including depth from the microphones. We have to carefully remove or reduce this direct sound from audio signals captured by the microphone. We can easily do this if we know the relative positions of the speakers from the microphones. However, the current description does not provide sufficient information for this issue.
Questions for authors
None
4 comments
[W2] Fairness of Baseline and Ablation Comparisons
(1) Fairness of Baseline and Ablation Comparisons
Purpose of the ablation study: We appreciate the reviewer's concern regarding the baseline in Table 2. We would like to emphasize that Table 1 and Table 2 serve different experimental purposes. Table 1 evaluates our full framework against existing methods, including pretrained ResNet and ViT models, whereas Table 2 examines the contributions of our proposed components within a controlled architecture.
Specifically, Table 2(a) represents a minimal configuration without the proposed embeddings and geometric bias, rather than a competitive baseline intended to outperform existing methods. All variants in Table 2(i) are trained from scratch under identical settings, allowing us to examine how individual components contribute to the full framework. Therefore, the relatively weak performance of Table 2(a) does not imply that the comparisons in Table 1 are unfair. The purpose of Table 2 is to examine the effectiveness of our design choices, rather than establish the competitiveness of the minimal configuration itself.
Additional comparison with geometric metadata: Following another reviewer's suggestion regarding baseline fairness, we additionally evaluate BatVision + pose/ear and EchoDiffusion + pose/ear. These strengthened baselines incorporate receiver heading (pose) and left/right microphone identity (ear) as additional input channels, concatenated with the original acoustic inputs along the channel dimension. This provides the same geometric metadata available to HEAR 360°, without introducing our explicit geometry-aware association.
Comparison on Replica (8 observations, Abs Rel ↓):
| Method | Abs Rel ↓ |
|---|---|
| BatVision + pose/ear | 0.166 |
| EchoDiffusion + pose/ear | 0.153 |
| HEAR 360° (Ours) | 0.127 |
Our method maintains the lowest Abs Rel in this comparison, suggesting that the performance improvement is not solely attributable to the availability of geometric metadata.
(2) Contributions of Time-Frequency and Ray Embeddings
Interpretation of individual components: We agree that the individual contributions of time-frequency and ray embeddings appear relatively modest in Table 2(i). However, we would like to clarify that the time-frequency embedding is an auxiliary design component, while our main contribution lies in explicitly associating panoramic directions with posed acoustic observations.
The time-frequency embedding preserves positional information within the acoustic features, whereas the ray embedding represents the viewing direction of each panoramic query. The geometric bias further models the directional relationship between the panoramic query and each acoustic observation (Secs. 3.2–3.3).
As shown in Table 2(i), removing the time-frequency embedding increases Abs Rel from 0.127 to 0.129, removing the ray embedding to 0.132, and removing the geometric bias to 0.136. Importantly, removing both the ray embedding and geometric bias increases Abs Rel to 0.161, indicating a larger degradation than removing either component individually.
These results suggest that the ray embedding and geometric bias provide complementary information for geometry-aware association. We therefore emphasize that our intended contribution is not the individual positional embeddings themselves, but the framework that explicitly relates acoustic observations to panoramic viewing directions.
We kindly ask the reviewer to consider the baseline definitions in Sec. 4.2 and the ablation analysis in Sec. 4.5, particularly Tables 1–2, of the revised manuscript. We would appreciate any further feedback or discussion.
[W3] Speaker Position and Direct-Sound Suppression
Speaker–microphone configuration: As described in Sec. 4.1 and Appendix F.1 of our original submission, we follow the active echolocation setup of VisualEchoes, placing the sound source and binaural receiver at the same listener center (1.5 m height) in simulation. We use a known 3-ms sweep to generate binaural echo observations. (In our real-world capture setup, the speaker is mounted directly above the binaural microphone, as described in Appendix H of the revised manuscript.)
Treatment of direct sound: We agree that direct sound does not directly encode the surrounding geometry and can, in principle, be suppressed using known source–receiver geometry. However, as in prior echo-based depth estimation setups, we do not explicitly suppress the direct-sound component. We intentionally avoid configuration-specific direct-sound suppression to minimize reliance on heuristic preprocessing and facilitate future extensions to diverse acoustic sensing hardware. Instead, we allow the network to learn depth-relevant features directly from the acoustic responses.
Additional direct-sound suppression experiment: To further examine this design choice, we conducted a controlled ablation comparing models trained with and without direct-sound suppression under identical settings. Specifically, we suppress a 0.31-ms window around the direct-sound peak before extracting acoustic features.
Direct-sound suppression ablation (Replica, 8 observations):
| Setting | RMSE ↓ | Abs Rel ↓ |
|---|---|---|
| Without suppression (Ours) | 0.483 | 0.1307 |
| With suppression | 0.481 | 0.1309 |
The results show negligible performance differences between the two settings, with a slight improvement in RMSE but nearly identical Abs Rel. These findings suggest that explicit direct-sound suppression is not essential for achieving competitive depth estimation performance under our evaluated setup, supporting our choice to avoid configuration-specific preprocessing.
Effect of early-response suppression in real-world recordings (Sec. 4.7, Table 3, and Appendix H of the revised manuscript): In contrast, early-response suppression was beneficial in real-world recordings, where the physical speaker, microphones, and mounting apparatus introduce additional early acoustic responses. We attenuate these recording-system responses to reduce their interference with room-reflection cues. Applying this correction reduced the scale-and-shift-aligned Abs Rel from 0.487 to 0.449 (−7.9%) with 8 observations, and from 0.563 to 0.498 (−11.5%) with 4 observations. The same correction was applied consistently to all methods, with improvements also observed for BatVision and EchoDiffusion. Their corrected performance is reported in Table 3 of the revised manuscript.
These findings suggest that while explicit direct-sound suppression provides little additional benefit in our simulated setting, suppressing early recording-system responses can improve real-world acoustic depth estimation.
Summary of Responses and Discussion Reminder
Dear Reviewer nYqP,
Thank you again for your detailed review. We would like to briefly summarize our responses to your main concerns.
- Panoramic motivation: We clarified why panoramic depth better reflects the spatial extent of acoustic observations. Our additional transfer experiment shows that panoramic pretraining improves real-world perspective depth estimation (Abs Rel 0.408 vs. 0.457).
- Ablation fairness: We clarified the different purposes of Tables 1 and 2 and provided comparisons with pose-aware baselines. Removing both ray embeddings and geometric bias increases Abs Rel from 0.127 to 0.161.
- Direct sound: We clarified the speaker–microphone configuration and evaluated direct-sound suppression. Its effect is negligible in simulation (0.1307 vs. 0.1309), while early-response suppression improves real-world performance (0.487 to 0.449).
We hope these explanations and additional experiments address your concerns. As the discussion period approaches its end, we would appreciate your consideration of our responses and welcome any further feedback.
Best regards, The Authors
We would first like to thank you for your valuable feedback. We hope our responses below address your concerns and look forward to further discussion.
[W1] Motivation and Novelty of Panoramic Acoustic Depth Estimation
(1) Motivation for Panoramic Acoustic Depth Estimation
Alignment with the spatial extent of acoustic observations: We would like to emphasize that our motivation stems from the distinct spatial characteristics of acoustic observations compared to visual observations. In image-based depth estimation, each pixel corresponds to a specific viewing direction, so the input image and predicted depth naturally share the same field of view (FoV). In contrast, acoustic signals are formed through sound propagation and reflections from the surrounding environment, including regions outside a conventional perspective FoV. While predicting depth within a limited FoV is still valid, it only reconstructs a subset of the surrounding space contributing to the acoustic observations. We therefore consider panoramic depth a more natural listener-centric representation that better reflects the spatial extent of acoustic observations. This provides a modality-specific motivation distinct from the geometric deformation discussed by the reviewer. Furthermore, a single binaural observation provides limited directional constraints due to directional ambiguity and reverberation. Acquiring complementary observations at different receiver headings provides additional directional evidence for panoramic reconstruction (Sec. 1 and Fig. 1).
Experimental benefits of panoramic depth supervision: To examine whether broader spatial supervision leads to more informative acoustic representations, we conducted an additional experiment. Specifically, we pretrained the same HEAR 360° architecture on synthetic Matterport3D using identical binaural audio inputs with either panoramic (360°) or perspective (87° × 58°, horizontal × vertical) depth supervision. Both models were then fine-tuned and evaluated under the same perspective depth setting on BatVision-V2, a real-world dataset containing binaural audio recordings and depth maps.
Transfer to real-world BatVision-V2 (2 channels, Abs Rel ↓):
Across three random seeds, panoramic pretraining achieved the lowest Abs Rel. These results suggest that panoramic supervision provides additional spatial learning signals beyond the restricted perspective FoV, enabling the model to learn acoustic representations that benefit depth estimation even within a limited FoV.
(2) Novelty Beyond Multiple Microphones
Distinction from prior studies: We thank the reviewer for highlighting these relevant studies. As discussed in the Introduction and Sec. 2 of our original submission, we acknowledge the importance of microphone arrays and multichannel acoustic sensing for spatial perception. We would like to emphasize that our contribution is not simply increasing the number of microphones, but extending acoustic depth estimation to multiple posed observations and effectively integrating their complementary evidence for panoramic depth estimation.
We have reviewed the studies mentioned by the reviewer and would like to clarify how they differ from our work. Chen et al. (ECCV 2020) introduce SoundSpaces for spatial audio simulation and audio-visual navigation toward sound-emitting targets, rather than estimating scene geometry from actively emitted echoes. Honma et al. (ICIP 2024) investigate binaural ultrasonic echo-based depth estimation, focusing on improving accuracy by using audible echoes during training rather than integrating acoustic observations acquired at multiple receiver headings.
Geometry-aware integration of acoustic observations: In contrast, our approach explicitly models the geometric relationships between acoustic observations and panoramic rays, rather than simply stacking acoustic channels, as illustrated in Fig. 1 and detailed in Secs. 3.2–3.3. Furthermore, Table 1 demonstrates consistent performance improvements with increasing numbers of observations, while Fig. 4 shows robustness to irregular observation configurations and generalization to unseen receiver headings.
We kindly ask the reviewer to consider the motivation and methodological distinctions in the revised manuscript, particularly Secs. 1–3.3 and Fig. 1, together with the additional transfer experiment above. We would appreciate any further feedback or discussion.