acceptodds
Under review as a conference paper at ICLR 2027

HEAR 360: Geometry-Aware Panoramic Depth Estimation from Multiple Posed Acoustic Observations

Abstract

We address the problem of audio-only panoramic depth estimation from multiple posed binaural observations. Existing audio-to-depth methods typically rely on a single forward-facing binaural pair and predict depth over a limited field of view, providing insufficient directional constraints for panoramic reconstruction. The challenge is further amplified by directional ambiguity and reverberation, which entangle geometric evidence from multiple propagation paths. We propose HEAR 360, which formulates multichannel audio depth estimation as a geometry-aware association rather than a conventional channel-stacking approach. Our framework introduces: (i) pose-conditioned acoustic encoding that preserves observation-specific evidence while incorporating receiver heading and ear identity, and (ii) geometry-aware cross-attention that represents each panoramic location as a 3D ray and associates it with the most relevant posed acoustic observations. We evaluate HEAR 360 on panoramic depth benchmarks derived from Matterport3D and Replica. Experiments show that HEAR 360 outperforms existing echo-based methods and controlled channel-stacking baselines with substantially fewer parameters. Further analyses show consistent gains from additional observations, as well as robustness to irregular observation subsets and unseen receiver headings, supporting the effectiveness of pose-aware association.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene understanding, acoustic scene understanding, depth Estimation, echolocation, spatial Audio

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nYqPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of echo-based panoramic depth estimation, in which depth at 360 degrees around the microphone(s) is estimated from reflections of sound emitted by the speaker. The proposed method of this paper is named HEAR360, composed of (1) the use of multiple binaural microphones, (2) pose-conditioned acoustic embedding, and (3) direction-aware panoramic reconstruction. The second component embeds the positions of time-frequency blocks and microphones, integrating them into acoustic features. Meanwhile, the third component employs a cross-attention architecture, in which a learnable ray-specific embedding is used as the query to obtain panoramic features. Experimental evaluations with synthetic datasets demonstrate that the proposed method outperforms simple baselines (just introducing pre-trained visual encoders for obtaining tokens from spectrograms) and several previous methods for non-panoramic eho-based depth estimation.

Critical strengths

S1. The proposed method is simple but reasonable. In particular, pose-conditioned embedding would be required for the current problem setting, since the position and direction of a microphone are critical to determining the corresponding ray in the panoramic output. Simple methods should be appreciated as a strength, especially from a practical viewpoint.

S2. The current paper is basically well-written and easy to follow. The conciseness of the proposed method and the readability of the paper are of great importance, particularly for engineers interested in the problems addressed here.

Critical weaknesses

W1. The most critical concern is the validity and novelty of the problem setting.

  • I am afraid that depth estimation from a panoramic perspective does not seem to be a problem that significantly differs from the conventional 2D depth estimation. I understand that the above issue might be wrong in image-based depth estimation due to the geometric deformation. However, I could not understand the significant difference specific to acoustic-based prediction.
  • As far as I know, the use of more than three microphones or ambisonic microphones for echo-based depth estimation is now becoming a standard approach. For example, [Chen+ ECCV2020, SoundSpaces: Audio-Visual Navigation in 3D Environments] is a popular infrastructure for generating ambisonics from a given indoor room structure, and [Honma+ ICIP2024, Estimating Indoor Scene Depth Maps From Ultrasonic Echoes] introduces a microphone array. Also, in the audio signal processing community, impulse response estimation using microphone arrays and ambisonic microphones is the first choice for predicting indoor 3D structures.

W2. Another critical concern is related to the experimental evaluations.

  • Table 2 indicates that no proposed component performs poorly on any metric. Meanwhile, Table 1 indicates that using pretrained networks performs better than the baseline in Table 2. This implies that the baseline shown in Table 2 is either too weak or unfair.
  • Table 2 also indicates that time-frequency embedding and ray embedding do not contribute to the performance improvement.

W3. The current paper does not clearly describe the position of speakers, which is critical for understanding the adverse effect of the direct sound from the speakers on the microphones. As the authors know, direct sound does not contain any information about room structure, including depth from the microphones. We have to carefully remove or reduce this direct sound from audio signals captured by the microphone. We can easily do this if we know the relative positions of the speakers from the microphones. However, the current description does not provide sufficient information for this issue.

Questions for authors

None

4 comments

We would first like to thank you for your valuable feedback. We hope our responses below address your concerns and look forward to further discussion.


[W1] Motivation and Novelty of Panoramic Acoustic Depth Estimation

(1) Motivation for Panoramic Acoustic Depth Estimation

Alignment with the spatial extent of acoustic observations: We would like to emphasize that our motivation stems from the distinct spatial characteristics of acoustic observations compared to visual observations. In image-based depth estimation, each pixel corresponds to a specific viewing direction, so the input image and predicted depth naturally share the same field of view (FoV). In contrast, acoustic signals are formed through sound propagation and reflections from the surrounding environment, including regions outside a conventional perspective FoV. While predicting depth within a limited FoV is still valid, it only reconstructs a subset of the surrounding space contributing to the acoustic observations. We therefore consider panoramic depth a more natural listener-centric representation that better reflects the spatial extent of acoustic observations. This provides a modality-specific motivation distinct from the geometric deformation discussed by the reviewer. Furthermore, a single binaural observation provides limited directional constraints due to directional ambiguity and reverberation. Acquiring complementary observations at different receiver headings provides additional directional evidence for panoramic reconstruction (Sec. 1 and Fig. 1).

Experimental benefits of panoramic depth supervision: To examine whether broader spatial supervision leads to more informative acoustic representations, we conducted an additional experiment. Specifically, we pretrained the same HEAR 360° architecture on synthetic Matterport3D using identical binaural audio inputs with either panoramic (360°) or perspective (87° × 58°, horizontal × vertical) depth supervision. Both models were then fine-tuned and evaluated under the same perspective depth setting on BatVision-V2, a real-world dataset containing binaural audio recordings and depth maps.

Transfer to real-world BatVision-V2 (2 channels, Abs Rel ↓):

Pretraining supervisionAbs Rel ↓
From scratch0.480
Perspective (87° × 58°)0.457
Panoramic (360°, Ours)0.408

Across three random seeds, panoramic pretraining achieved the lowest Abs Rel. These results suggest that panoramic supervision provides additional spatial learning signals beyond the restricted perspective FoV, enabling the model to learn acoustic representations that benefit depth estimation even within a limited FoV.

(2) Novelty Beyond Multiple Microphones

Distinction from prior studies: We thank the reviewer for highlighting these relevant studies. As discussed in the Introduction and Sec. 2 of our original submission, we acknowledge the importance of microphone arrays and multichannel acoustic sensing for spatial perception. We would like to emphasize that our contribution is not simply increasing the number of microphones, but extending acoustic depth estimation to multiple posed observations and effectively integrating their complementary evidence for panoramic depth estimation.

We have reviewed the studies mentioned by the reviewer and would like to clarify how they differ from our work. Chen et al. (ECCV 2020) introduce SoundSpaces for spatial audio simulation and audio-visual navigation toward sound-emitting targets, rather than estimating scene geometry from actively emitted echoes. Honma et al. (ICIP 2024) investigate binaural ultrasonic echo-based depth estimation, focusing on improving accuracy by using audible echoes during training rather than integrating acoustic observations acquired at multiple receiver headings.

Geometry-aware integration of acoustic observations: In contrast, our approach explicitly models the geometric relationships between acoustic observations and panoramic rays, rather than simply stacking acoustic channels, as illustrated in Fig. 1 and detailed in Secs. 3.2–3.3. Furthermore, Table 1 demonstrates consistent performance improvements with increasing numbers of observations, while Fig. 4 shows robustness to irregular observation configurations and generalization to unseen receiver headings.

We kindly ask the reviewer to consider the motivation and methodological distinctions in the revised manuscript, particularly Secs. 1–3.3 and Fig. 1, together with the additional transfer experiment above. We would appreciate any further feedback or discussion.


[W2] Fairness of Baseline and Ablation Comparisons

(1) Fairness of Baseline and Ablation Comparisons

Purpose of the ablation study: We appreciate the reviewer's concern regarding the baseline in Table 2. We would like to emphasize that Table 1 and Table 2 serve different experimental purposes. Table 1 evaluates our full framework against existing methods, including pretrained ResNet and ViT models, whereas Table 2 examines the contributions of our proposed components within a controlled architecture.

Specifically, Table 2(a) represents a minimal configuration without the proposed embeddings and geometric bias, rather than a competitive baseline intended to outperform existing methods. All variants in Table 2(i) are trained from scratch under identical settings, allowing us to examine how individual components contribute to the full framework. Therefore, the relatively weak performance of Table 2(a) does not imply that the comparisons in Table 1 are unfair. The purpose of Table 2 is to examine the effectiveness of our design choices, rather than establish the competitiveness of the minimal configuration itself.

Additional comparison with geometric metadata: Following another reviewer's suggestion regarding baseline fairness, we additionally evaluate BatVision + pose/ear and EchoDiffusion + pose/ear. These strengthened baselines incorporate receiver heading (pose) and left/right microphone identity (ear) as additional input channels, concatenated with the original acoustic inputs along the channel dimension. This provides the same geometric metadata available to HEAR 360°, without introducing our explicit geometry-aware association.

Comparison on Replica (8 observations, Abs Rel ↓):

MethodAbs Rel ↓
BatVision + pose/ear0.166
EchoDiffusion + pose/ear0.153
HEAR 360° (Ours)0.127

Our method maintains the lowest Abs Rel in this comparison, suggesting that the performance improvement is not solely attributable to the availability of geometric metadata.

(2) Contributions of Time-Frequency and Ray Embeddings

Interpretation of individual components: We agree that the individual contributions of time-frequency and ray embeddings appear relatively modest in Table 2(i). However, we would like to clarify that the time-frequency embedding is an auxiliary design component, while our main contribution lies in explicitly associating panoramic directions with posed acoustic observations.

The time-frequency embedding preserves positional information within the acoustic features, whereas the ray embedding represents the viewing direction of each panoramic query. The geometric bias further models the directional relationship between the panoramic query and each acoustic observation (Secs. 3.2–3.3).

As shown in Table 2(i), removing the time-frequency embedding increases Abs Rel from 0.127 to 0.129, removing the ray embedding to 0.132, and removing the geometric bias to 0.136. Importantly, removing both the ray embedding and geometric bias increases Abs Rel to 0.161, indicating a larger degradation than removing either component individually.

These results suggest that the ray embedding and geometric bias provide complementary information for geometry-aware association. We therefore emphasize that our intended contribution is not the individual positional embeddings themselves, but the framework that explicitly relates acoustic observations to panoramic viewing directions.

We kindly ask the reviewer to consider the baseline definitions in Sec. 4.2 and the ablation analysis in Sec. 4.5, particularly Tables 1–2, of the revised manuscript. We would appreciate any further feedback or discussion.

[W3] Speaker Position and Direct-Sound Suppression

Speaker–microphone configuration: As described in Sec. 4.1 and Appendix F.1 of our original submission, we follow the active echolocation setup of VisualEchoes, placing the sound source and binaural receiver at the same listener center (1.5 m height) in simulation. We use a known 3-ms sweep to generate binaural echo observations. (In our real-world capture setup, the speaker is mounted directly above the binaural microphone, as described in Appendix H of the revised manuscript.)

Treatment of direct sound: We agree that direct sound does not directly encode the surrounding geometry and can, in principle, be suppressed using known source–receiver geometry. However, as in prior echo-based depth estimation setups, we do not explicitly suppress the direct-sound component. We intentionally avoid configuration-specific direct-sound suppression to minimize reliance on heuristic preprocessing and facilitate future extensions to diverse acoustic sensing hardware. Instead, we allow the network to learn depth-relevant features directly from the acoustic responses.

Additional direct-sound suppression experiment: To further examine this design choice, we conducted a controlled ablation comparing models trained with and without direct-sound suppression under identical settings. Specifically, we suppress a 0.31-ms window around the direct-sound peak before extracting acoustic features.

Direct-sound suppression ablation (Replica, 8 observations):

SettingRMSE ↓Abs Rel ↓
Without suppression (Ours)0.4830.1307
With suppression0.4810.1309

The results show negligible performance differences between the two settings, with a slight improvement in RMSE but nearly identical Abs Rel. These findings suggest that explicit direct-sound suppression is not essential for achieving competitive depth estimation performance under our evaluated setup, supporting our choice to avoid configuration-specific preprocessing.

Effect of early-response suppression in real-world recordings (Sec. 4.7, Table 3, and Appendix H of the revised manuscript): In contrast, early-response suppression was beneficial in real-world recordings, where the physical speaker, microphones, and mounting apparatus introduce additional early acoustic responses. We attenuate these recording-system responses to reduce their interference with room-reflection cues. Applying this correction reduced the scale-and-shift-aligned Abs Rel from 0.487 to 0.449 (−7.9%) with 8 observations, and from 0.563 to 0.498 (−11.5%) with 4 observations. The same correction was applied consistently to all methods, with improvements also observed for BatVision and EchoDiffusion. Their corrected performance is reported in Table 3 of the revised manuscript.

These findings suggest that while explicit direct-sound suppression provides little additional benefit in our simulated setting, suppressing early recording-system responses can improve real-world acoustic depth estimation.

AuthorsOpenReview ↗

Summary of Responses and Discussion Reminder

Dear Reviewer nYqP,

Thank you again for your detailed review. We would like to briefly summarize our responses to your main concerns.

  • Panoramic motivation: We clarified why panoramic depth better reflects the spatial extent of acoustic observations. Our additional transfer experiment shows that panoramic pretraining improves real-world perspective depth estimation (Abs Rel 0.408 vs. 0.457).
  • Ablation fairness: We clarified the different purposes of Tables 1 and 2 and provided comparisons with pose-aware baselines. Removing both ray embeddings and geometric bias increases Abs Rel from 0.127 to 0.161.
  • Direct sound: We clarified the speaker–microphone configuration and evaluated direct-sound suppression. Its effect is negligible in simulation (0.1307 vs. 0.1309), while early-response suppression improves real-world performance (0.487 to 0.449).

We hope these explanations and additional experiments address your concerns. As the discussion period approaches its end, we would appreciate your consideration of our responses and welcome any further feedback.

Best regards, The Authors

Reviewer_C5WKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HEAR 360° predicts panoramic depth from multiple binaural echo recordings captured at known receiver headings. It combines pose-conditioned acoustic features with panoramic ray queries and geometry-aware cross-attention. On simulated Matterport3D and Replica benchmarks, it improves depth prediction over adapted baselines while using fewer parameters.

Critical strengths

A well-motivated approach to combining acoustic observations for panoramic depth estimation. Consistent improvements with additional observations across both datasets with a compact model. Component ablations support the usefulness of pose conditioning and directional information.

Critical weaknesses
  1. Efficiency is assessed mainly through parameter count. Inference latency, memory, and the acquisition cost of multiple recordings are not reported.
  2. HEAR 360° receives actual test-time headings, while the baselines do not. This does not isolate the advantage of the proposed geometric attention.
  3. Baseline adaptations limit the conclusions. In particular, EchoDiffusion is evaluated using a deterministic implementation rather than its described iterative diffusion procedure.
  4. Evaluation is entirely simulated. Robustness to real recordings, sensor noise, pose errors, and acoustic material changes remains untested.
  5. The core attention formulation is inconsistent. The text describes a geometric bias applied to attention logits, but Equation 8 adds it to the attention output. These operations have different effects.
Questions for authors
  1. Is the geometric bias added before attention softmax or after attention aggregation? Please provide the exact implemented equation.
  2. How does HEAR 360° compare with baselines given the same heading and ear metadata?
  3. Do gains persist against a parameter-matched pose-aware transformer without the proposed geometric bias?
  4. How sensitive is performance to heading errors, background noise, and unseen acoustic materials?
  5. Can you provide per-scene results and confidence intervals across training seeds?
  6. What are the inference and total acquisition costs for two, four, and eight observations?
11 comments

Before responding to your comments, we thank you for taking the time to review our work. We would be grateful if you could review our responses below and share any further thoughts.


[W1, Q6] Computational Efficiency and Acquisition Cost

(1) Inference Cost

Inference latency and memory: We thank the reviewer for highlighting the need to evaluate computational efficiency beyond parameter count. We additionally evaluate inference latency (ms) and peak GPU memory (MB) against existing methods under matched 2/4/8-channel settings. All experiments are conducted on an NVIDIA RTX A6000.

Method#P (M)2ch (ms / MB)4ch (ms / MB)8ch (ms / MB)
EchoScan41.37.0 / 3105.9 / 3107.7 / 310
BatVision54.52.2 / 3962.2 / 6152.3 / 835
EchoDiffusion*156.131.9 / 95932.5 / 100830.1 / 1057
HEAR 360° (Ours)15.419.8 / 34429.1 / 61852.8 / 643

EchoDiffusion uses the deterministic single-pass implementation evaluated in our experiments.

Comparison with prior methods: HEAR 360° has the fewest parameters (15.4M) among the compared methods and achieves lower inference latency than EchoDiffusion at 2 and 4 channels. At 8 channels, its latency increases to 52.8 ms, while peak GPU memory usage remains at 643 MB.

Inference latency and memory efficiency: Unlike conventional approaches that concatenate acoustic channels before encoding, HEAR 360° independently extracts features from each acoustic observation using a shared encoder to preserve observation-specific directional information. Consequently, computational cost increases with the number of observations despite the small parameter count. However, adjusting the number of observations processed in parallel allows a flexible trade-off between inference latency and GPU memory consumption, without modifying the architecture or retraining.

(2) Rendering and Acquisition Cost

Acoustic rendering cost: As described in Supplementary Appendix F of our original submission, our binaural acoustic and depth rendering pipeline requires approximately 8.5–12.8 seconds per capture. Using four parallel workers, we generated 4,800 captures in approximately 5.4 hours, corresponding to an estimated 27 minutes per scene with 400 captures.

Our data-generation pipeline builds on the same acoustic rendering framework as SoundSpaces 2.0. SoundSpaces 2.0 reports approximately 0.25 seconds per frame in its high-quality setting and 0.03 seconds in its high-speed setting. Our configuration places greater emphasis on modeling acoustic reflections, with rendering costs depending on simulation parameters such as ray count. Although these timings reflect different rendering configurations, such parameters can be adjusted to reduce computational overhead, as demonstrated by the faster rendering settings in SoundSpaces 2.0.

Acquisition cost of multiple observations: Furthermore, we use a fixed set of receiver headings across all scenes, regardless of room size, and evaluate 2-, 4-, and 8-channel configurations by selecting subsets of the available observations. These correspond to one, two, and four binaural recordings, respectively. This requires no additional rendering when evaluating different channel configurations from the same captured set, while acquiring fewer observations from the outset could further reduce acquisition costs. Thus, observation count and simulation fidelity offer practical trade-offs between acquisition cost and depth estimation performance.

[W2, Q2] Fairness of Heading-Aware Baseline Comparisons

(1) Comparison with Matched Geometric Metadata

We appreciate the reviewer's concern regarding the fairness of providing receiver headings only to our method. To address this, we additionally provide the same receiver heading (pose) and left/right microphone identity (ear) metadata to BatVision and EchoDiffusion by concatenating these cues with their acoustic inputs. This allows the baselines to access the same geometric metadata without introducing our explicit geometry-aware association.

Comparison with pose/ear cues (Replica, 8 observations):

ModelRMSE ↓Abs Rel ↓δ1 ↑
BatVision0.5160.1640.792
BatVision +pose/ear0.5080.1660.787
EchoDiffusion0.5130.1500.803
EchoDiffusion +pose/ear0.5110.1530.797
HEAR 360° (Ours)0.4810.1270.847

Providing pose/ear metadata yields mixed performance changes across the baselines. Nevertheless, HEAR 360° achieves the best performance across all three metrics, suggesting that our improvements cannot be explained solely by providing additional geometric metadata.

(2) Generalization to Unseen Receiver Headings

We further evaluate generalization to unseen receiver orientations using two rotated heading configurations and one irregular configuration, following Fig. 4(iii) of the revised manuscript, without retraining.

Generalization to unseen receiver headings (Replica, 8 observations):

Microphone orientationsModelRMSE ↓Abs Rel ↓log10 ↓
{45°, 135°, 225°, 315°}BatVision0.7110.2550.093
BatVision + pose/ear0.6950.2560.095
EchoDiffusion0.7090.2230.092
EchoDiffusion + pose/ear0.6980.2220.091
Ours0.6160.1630.076
{30°, 120°, 210°, 300°}BatVision0.7310.2640.095
BatVision + pose/ear0.7110.2610.096
EchoDiffusion0.7260.2220.092
EchoDiffusion + pose/ear0.7120.2230.091
Ours0.6260.1630.073
{−45°, 135°, 120°, 300°}BatVision0.9200.4100.138
BatVision + pose/ear1.6810.6810.325
EchoDiffusion0.8710.3380.129
EchoDiffusion + pose/ear1.2180.4910.183
Ours0.6850.1990.093

Benefit of geometry-aware association: HEAR 360° achieves the best results across all three unseen configurations. While directly concatenating pose/ear metadata provides limited benefits under rotated headings and can substantially degrade performance under irregular configurations, our framework explicitly associates each acoustic observation with panoramic viewing directions using its acquisition geometry.

We would therefore like to emphasize that our contribution is not merely providing geometric metadata, but effectively utilizing this information to integrate complementary acoustic observations. These results further support the benefit of our geometry-aware design beyond conventional channel stacking.

We kindly ask the reviewer to consider the comparisons in Table 1 and Fig. 4(iii) of the revised manuscript, together with the additional matched-metadata experiments above. We would appreciate any further feedback or discussion.

[W3] Deterministic Implementation of EchoDiffusion

Clarification of the baseline implementation: We appreciate the reviewer's concern regarding our adaptation of EchoDiffusion. As described in Sec. 4.2, the publicly available implementation does not include the full iterative diffusion procedure described in the original paper. We therefore adapt its released deterministic single-pass architecture, retaining the complementary waveform- and spectrogram-based pathways, and acknowledge this limitation.

Comparison with the original results: Nevertheless, the original EchoDiffusion paper reports a relatively small RMSE improvement over Bat-Net on Replica (0.956 to 0.913; 4.5%). Similarly, our two-channel results in Table 1 show an RMSE improvement from BatVision to our adapted EchoDiffusion (0.574 to 0.547; 4.7%). Although the evaluation settings differ, the comparable relative improvements provide supporting context for our adapted baseline, while not establishing equivalence to the full iterative implementation.

[W4, Q4] Robustness to Real-World Conditions

(1) Real World Demonstration

To address this concern, we have added real-world experiments in Sec. 4.7 (Table 3 and Fig. 5) of the revised manuscript. We collected 92 binaural recordings across 23 indoor locations with diverse geometries and acoustic conditions. All models were trained on Replica and evaluated directly on real recordings without fine-tuning.

Real-world quantitative results (8 observations):

MethodOrd. (%) ↑Abs Rel ↓Δρ ↑
BatVision62.90.4850.07
EchoDiffusion55.40.5550.05
HEAR 360° (Ours)69.90.4490.30

Our method achieves the best results across all three metrics, providing additional evidence of sim-to-real transfer under naturally varying recording conditions. Since metric ground-truth depth is unavailable, we evaluate agreement with image-derived depth references. We kindly ask the reviewer to carefully examine the revised manuscript, particularly Sec. 4.7, Table 3, Fig. 5, and Appendix H, which provide our real-world quantitative results, qualitative comparisons, and detailed acquisition protocol.

(2) Acoustic noise (SNR)

We add Gaussian noise to waveforms before STFT, varying SNR from clean to 5 dB. This range covers the SNR distribution measured from 288 real indoor recordings (4.0–40.7 dB; median 26.8 dB). Following another reviewer's suggestion regarding baseline fairness, we additionally evaluate BatVision + pose/ear and EchoDiffusion + pose/ear. These strengthened baselines incorporate receiver heading (pose) and left/right microphone identity (ear) as additional input channels, concatenated with the original acoustic inputs along the channel dimension. This provides the same geometric metadata available to HEAR 360°, while retaining the baselines' original architectures without our explicit geometry-aware association.

Noise robustness on Replica (8 observations, Abs Rel ↓):

SNR (dB)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
Clean0.1270.1660.153
300.1270.1660.154
200.1270.171 (+3.1%)0.155 (+0.9%)
150.129 (+1.4%)0.194 (+17.0%)0.157 (+2.6%)
100.142 (+11.2%)0.275 (+65.6%)0.172 (+12.4%)
70.158 (+23.9%)0.341 (+105.3%)0.200 (+30.7%)
50.174 (+36.5%)0.379 (+128.4%)0.231 (+50.9%)

HEAR 360° achieves the lowest Abs Rel across all tested SNR conditions. At 5 dB, its error increases by 36.5%, compared with 128.4% for BatVision and 50.9% for EchoDiffusion. This indicates a smaller relative degradation under the evaluated noise conditions.

(3) Independent heading perturbations (σ)

We add independent Gaussian heading errors to each binaural observation, ε ~ N(0, σ²), while keeping the audio unchanged. The same error is applied to both ears, and results are averaged over three perturbation seeds.

Heading perturbation robustness on Replica (8 observations, Abs Rel ↓):

σ (°)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
00.1270.1660.153
20.1270.1660.154 (+0.3%)
50.129 (+1.1%)0.168 (+1.1%)0.157 (+2.2%)
100.132 (+3.8%)0.216 (+29.9%)0.169 (+10.3%)
200.141 (+11.1%)0.427 (+157.1%)0.214 (+39.4%)

A similar trend is observed under independent heading perturbations. At σ = 10°, our error increases by 3.8%, compared with 29.9% and 10.3% for the baselines. At σ = 20°, HEAR 360° maintains the lowest Abs Rel (0.141), showing a relatively smaller sensitivity to heading perturbations in these experiments.

(4) Systematic heading offsets (δ)

We apply a shared angular offset to all observation headings to simulate reference-orientation misalignment, evaluating δ = 0°, ±2°, ±5°, ±10°, and ±20°. Unlike independent perturbations, this setting represents a consistent calibration error across observations.

Heading offset robustness on Replica (8 observations, Abs Rel ↓):

δ (°)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
−200.187 (+47.0%)0.582 (+250.3%)0.277 (+80.9%)
−100.148 (+16.3%)0.263 (+58.3%)0.213 (+38.6%)
−50.135 (+6.0%)0.168 (+1.0%)0.175 (+14.1%)
−20.128 (+0.8%)0.1660.156 (+1.9%)
00.1270.1660.153
+20.1270.1660.158 (+3.2%)
+50.133 (+4.1%)0.179 (+7.8%)0.188 (+22.8%)
+100.141 (+10.7%)0.355 (+113.7%)0.243 (+58.5%)
+200.178 (+40.3%)0.602 (+262.2%)0.303 (+97.7%)

(Continued in the next comment.)

Reviewer_VibxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HEAR 360°, a framework for panoramic depth estimation from multiple binaural recordings with known receiver headings. It combines pose-conditioned acoustic features with ray-based panoramic queries through geometry-aware attention. Experiments on simulated Matterport3D and Replica benchmarks show improvements over the evaluated baselines, with ablations and unseen-heading tests supporting the proposed design.

Critical strengths
  • The paper proposes a novel, well-motivated architecture that uses receiver poses and panoramic ray directions to guide acoustic feature fusion. The ablations support the benefit of this design.

  • The method consistently outperforms the evaluated baselines on Matterport3D and Replica, while using substantially fewer parameters.

Critical weaknesses

The experiments are all conducted with simulated audio, and the authors do not evaluate how sensitive the method is to changes in the recording conditions. Relevant tests include adding noise to the test audio at different SNRs before computing the spectrograms and introducing small errors in the input heading angles. Another useful test is to change the acoustic properties of the floor or walls and regenerate the echoes, keeping the geometry and ground-truth depth fixed. This would show how much material changes alone affect the predictions. These tests would help establish whether the improvements over the baselines hold under less ideal conditions.

Questions for authors
  • Section 3.3 text mentioned that the geometric bias is added to the attention logits, but Eq. (8) placed it after the cross-attention output. Where is the bias actually applied in the implementation?

  • Could the authors clarify the dataset and evaluation protocol used in Table S2? It is labeled as Replica, but the reported Abs Rel values are much higher than the four-observation Replica result in Table 1 and closer to the Matterport3D results. Also, setup (a) reports 0.389 for HEAR 360° in Table S2, compared with 0.376 in Fig. 4(i). Please explain whether these differences reflect different experimental settings or reporting errors.

5 comments

Before addressing your comments, we sincerely thank you for your thoughtful review. We would appreciate your consideration of our responses below and welcome further discussion.


[W1] Robustness to Real-World Conditions

(1) Real World Demonstration

To address this concern, we have added real-world experiments in Sec. 4.7 (Table 3 and Fig. 5) of the revised manuscript. We collected 92 binaural recordings across 23 indoor locations with diverse geometries and acoustic conditions. All models were trained on Replica and evaluated directly on real recordings without fine-tuning.

Real-world quantitative results (8 observations):

MethodOrd. (%) ↑Abs Rel ↓Δρ ↑
BatVision62.90.4850.07
EchoDiffusion55.40.5550.05
HEAR 360° (Ours)69.90.4490.30

Our method achieves the best results across all three metrics, providing additional evidence of sim-to-real transfer under naturally varying recording conditions. Since metric ground-truth depth is unavailable, we evaluate agreement with image-derived depth references. We kindly ask the reviewer to carefully examine the revised manuscript, particularly Sec. 4.7, Table 3, Fig. 5, and Appendix H, which provide our real-world quantitative results, qualitative comparisons, and detailed acquisition protocol.

(2) Acoustic noise (SNR)

We add Gaussian noise to waveforms before STFT, varying SNR from clean to 5 dB. This range covers the SNR distribution measured from 288 real indoor recordings (4.0–40.7 dB; median 26.8 dB). Following another reviewer's suggestion regarding baseline fairness, we additionally evaluate BatVision + pose/ear and EchoDiffusion + pose/ear. These strengthened baselines incorporate receiver heading (pose) and left/right microphone identity (ear) as additional input channels, concatenated with the original acoustic inputs along the channel dimension. This provides the same geometric metadata available to HEAR 360°, while retaining the baselines' original architectures without our explicit geometry-aware association.

Noise robustness on Replica (8 observations, Abs Rel ↓):

SNR (dB)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
Clean0.1270.1660.153
300.1270.1660.154
200.1270.171 (+3.1%)0.155 (+0.9%)
150.129 (+1.4%)0.194 (+17.0%)0.157 (+2.6%)
100.142 (+11.2%)0.275 (+65.6%)0.172 (+12.4%)
70.158 (+23.9%)0.341 (+105.3%)0.200 (+30.7%)
50.174 (+36.5%)0.379 (+128.4%)0.231 (+50.9%)

HEAR 360° achieves the lowest Abs Rel across all tested SNR conditions. At 5 dB, its error increases by 36.5%, compared with 128.4% for BatVision and 50.9% for EchoDiffusion. This indicates a smaller relative degradation under the evaluated noise conditions.

(3) Independent heading perturbations (σ)

We add independent Gaussian heading errors to each binaural observation, ε ~ N(0, σ²), while keeping the audio unchanged. The same error is applied to both ears, and results are averaged over three perturbation seeds.

Heading perturbation robustness on Replica (8 observations, Abs Rel ↓):

σ (°)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
00.1270.1660.153
20.1270.1660.154 (+0.3%)
50.129 (+1.1%)0.168 (+1.1%)0.157 (+2.2%)
100.132 (+3.8%)0.216 (+29.9%)0.169 (+10.3%)
200.141 (+11.1%)0.427 (+157.1%)0.214 (+39.4%)

A similar trend is observed under independent heading perturbations. At σ = 10°, our error increases by 3.8%, compared with 29.9% and 10.3% for the baselines. At σ = 20°, HEAR 360° maintains the lowest Abs Rel (0.141), showing a relatively smaller sensitivity to heading perturbations in these experiments.

(4) Systematic heading offsets (δ)

We apply a shared angular offset to all observation headings to simulate reference-orientation misalignment, evaluating δ = 0°, ±2°, ±5°, ±10°, and ±20°. Unlike independent perturbations, this setting represents a consistent calibration error across observations.

Heading offset robustness on Replica (8 observations, Abs Rel ↓):

δ (°)Ours ↓BatVision + pose/ear ↓EchoDiffusion + pose/ear ↓
−200.187 (+47.0%)0.582 (+250.3%)0.277 (+80.9%)
−100.148 (+16.3%)0.263 (+58.3%)0.213 (+38.6%)
−50.135 (+6.0%)0.168 (+1.0%)0.175 (+14.1%)
−20.128 (+0.8%)0.1660.156 (+1.9%)
00.1270.1660.153
+20.1270.1660.158 (+3.2%)
+50.133 (+4.1%)0.179 (+7.8%)0.188 (+22.8%)
+100.141 (+10.7%)0.355 (+113.7%)0.243 (+58.5%)
+200.178 (+40.3%)0.602 (+262.2%)0.303 (+97.7%)

(Continued in the next comment.)

(Continued from the previous comment.)

A comparable trend is observed under systematic heading offsets. Although larger offsets generally increase prediction errors, HEAR 360° achieves the lowest Abs Rel across all tested conditions. At ±20°, its error increases by 40.3–47.0%, compared with 250.3–262.2% for BatVision and 80.9–97.7% for EchoDiffusion. These results complement the independent heading perturbation analysis by considering a different form of orientation error.

(5) Sensitivity to Unseen Acoustic Materials

Following the reviewer's suggestion, we evaluate the effect of acoustic material changes while keeping room geometry, ground-truth depth, listener poses, source/receiver placement, and renderer settings fixed. We re-render the full Matterport3D test set (9 scenes × 100 positions × 4 binaural headings), varying only the acoustic materials, and evaluate the original checkpoints without retraining. We examine overall absorption, surface material replacement, absorber placement, scattering, and frequency-dependent absorption.

Material sensitivity on Matterport3D (8 observations, Abs Rel ↓):

Material configurationOurs ↓BatVision ↓EchoDiffusion ↓
Reference (re-rendered)0.3370.4690.504
Overall absorption scaling
(a) ×0.25 (more reverberant)0.327 (−0.010)0.472 (+0.003)0.513 (+0.009)
(b) ×0.50.323 (−0.014)0.466 (−0.003)0.505 (+0.001)
(c) ×20.428 (+0.091)0.708 (+0.239)0.644 (+0.140)
(d) ×4 (heavily damped)0.505 (+0.168)0.845 (+0.377)0.678 (+0.174)
Surface material replacement
(e) Hard finishes0.336 (−0.001)0.467 (−0.002)0.511 (+0.007)
(f) Plaster ceiling0.328 (−0.009)0.465 (−0.004)0.490 (−0.014)
(g) Soft finishes0.424 (+0.087)0.554 (+0.085)0.586 (+0.082)
(h) Soft + soundproof ceiling0.523 (+0.186)0.801 (+0.332)0.838 (+0.334)
Absorber placement
(i) Swap ceiling/floor materials0.340 (+0.003)0.479 (+0.010)0.519 (+0.015)
(j) Move ceiling absorption to walls0.323 (−0.014)0.437 (−0.032)0.463 (−0.042)
Surface scattering
(k) Low (0.1)0.352 (+0.015)0.489 (+0.020)0.538 (+0.034)
(l) High (0.9)0.371 (+0.034)0.514 (+0.045)0.530 (+0.026)
Frequency-dependent absorption
(m) Stronger high-frequency absorption0.377 (+0.040)0.603 (+0.135)0.682 (+0.177)
(n) Stronger low-frequency absorption0.310 (−0.027)0.421 (−0.048)0.443 (−0.061)

A similar trend is observed under acoustic material variations. HEAR 360° achieves the lowest Abs Rel across all 15 evaluated configurations, including the reference. This trend also holds with 2, 4, and 8 observations (45/45 settings). Although stronger absorption generally increases prediction errors, our method maintains the lowest Abs Rel even under ×4 absorption (0.505 vs. 0.845/0.678). These results indicate that the relative performance advantage is maintained across the tested material changes.

Evaluation note. Since SoundSpaces 2.0 uses stochastic Monte Carlo ray tracing, independently re-rendered reference scores differ from those in Table 1 (Ours: 0.337 vs. 0.346; BatVision: 0.469 vs. 0.446; EchoDiffusion: 0.504 vs. 0.386). We therefore assess material-induced changes relative to each model's re-rendered reference. The contribution of rendering stochasticity to these differences has not been isolated.

Summary

Overall, the additional evaluations show a consistent trend across acoustic noise, independent and systematic heading errors, and material variations, with HEAR 360° maintaining lower prediction errors than the compared baselines under the tested conditions. Together with the real-world experiments in Sec. 4.7 of the revised manuscript, these results provide further evidence of the method's performance under less ideal acoustic conditions.


[Q1] Clarification of Geometric Bias

We thank the reviewer for identifying the inconsistency. The geometric bias is added to the attention logits before softmax, rather than to the attention output. We have corrected Eq. (8) in Sec. 3.3 of the revised manuscript to accurately reflect the implementation.

Specifically, the bias is computed from the panoramic ray transformed into each observation's local receiver frame and its ear identity (Eqs. (6)–(7)). It is then added to the query-key attention logits, shared across the acoustic tokens of each observation. Softmax is applied jointly over all acoustic tokens from all observations, allowing the bias to influence attention weights across observations.

We kindly refer the reviewer to Sec. 3.3, Eqs. (6)–(8), of the revised manuscript for the corrected formulation.

[Q2] Inconsistencies in Table S2

We thank the reviewer for identifying these reporting errors, which have been corrected in the revised manuscript.

(1) Dataset and evaluation protocol. Table S2 evaluates observation selection on Matterport3D, not Replica. The dataset label was incorrect. The evaluation uses the same Matterport3D test set and four-observation protocol as Table 1, with different subsets selected from the eight available ear-specific observations.

(2) Discrepancy with Fig. 4(i). The discrepancy arose because we inadvertently reported a result from a different random seed in Table S2(a). We have corrected the value from 0.389 to 0.376, consistent with Fig. 4(i). The configurations in Fig. 4(i), (a)–(c), correspond to Table S2 (a), (b), and (e), respectively.

These corrections do not change the conclusions regarding observation selection. We kindly ask the reviewer to refer to Fig. 4(i) and the corrected Table S2 in Appendix C of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.