- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a system for acoustic-based mapping, named EchoBelief, which uses echos to gradually create the physical map (walls) of the environment, and decide where to move to sense next. It stores map updates from actual observations separately from the estimated map state based on predictions to ensure that information actually observed is not forgotten or erased. A benchmark is created based on simulated data and the results of the proposed system show significant benefits compared to baselines.
Critical strengths
A new benchmark for acoustic mapping with a variety of layouts, from more regular to more challenging ones
A complete system that addresses the sensing, active exploration and map state estimation steps
Significant results in numbers of rooms mapped successfully under a limited sensing budget above a required accuracy threshold
Critical weaknesses
-
Not clear how the proposed and competing algorithms trade the % of successfully mapped environment vs the available time/sensing budget. It would be great to run the experiments with
-
A more detailed ablation study could be added to show where the main benefits come from. There are four steps (polar echo encoder, local geometry mapper, evidence anchored spatial memory, and belief guided exploration. An ablation study replacing each of these four steps with state of the art could show more clearly where most of the benefit comes from.
-
In the simulated data it would have been good to simulate noise from people, furniture etc to show performance a more realistic setting
Questions for authors
Is the main novelty in the overall system architecture or each individual step? It would be worth highlighting if some of the steps in the architecture could be replaced successfully with other state of the art methods, and what would be the overall impact on performance.
What would be the impact of people / furniture and so on on the performance? Is it possible to simulate rooms with more dynamic traffic and show the impact on the accuracy of the mapping?
A more detailed ablation study would be useful to show the impact of the novelty in each step wrt to existing competing algorithms for each step.
5 comments
Response (2/3)
W2 / Q1 / Q3. Contribution and component replacements.
Our contribution is the complete acoustic mapping system, with the geometry-supervised polar echo field and evidence-anchored spatial memory as its central design contributions. The mapper and policy use established neural architectures. The polar field locates first-order wall reflections in range and bearing, providing a geometric signal for mapping. Sparse echoes still leave substantial structure unobserved, so the system also needs predictive context. Support-gated fusion protects the evidence map from unsupported prediction writes, while a separate recurrent memory retains that context for reconstruction and exploration.
To answer where the gains come from and whether components can be replaced successfully, we evaluated each stage through controlled comparisons: alternative encoder supervision, a geometric mapper replacement, a retrained memory ablation, and a hybrid exploration planner. The whole memory pathway produces the largest tested policy-ablation effect: removing it and retraining the belief and policy lowers success from 70.1% to 47.3%, a difference of 22.8 points [18.8, 27.0]. The table identifies the comparison and evaluation level for each stage.
| Stage | Comparison and evidence | Contribution supported |
|---|---|---|
| Polar echo encoder | In a token-only local-mapping probe, near-wall F1 is 71.3% with polar supervision, 42.2% with masked reconstruction, and 35.2% with waveform reconstruction; polar minus masked: +29.1 points [28.4, 29.8]. | Geometric supervision makes the acoustic features more useful for local mapping. This compares training objectives. |
| Local geometry mapper | Replacing the U-Net with a validation-tuned, non-learned echo-to-occupancy projection gives 72.4% versus 80.5% test near-wall F1 at 0.125 m; U-Net gain: +8.1 points [7.8, 8.4]. | The learned mapper improves local reconstruction over geometric projection. This is a local substitution. |
| Evidence-anchored memory | Removing the whole memory pathway, with belief and policy retrained, gives 47.3% versus 70.1% success. | Retaining predictive context alongside supported evidence benefits the complete system. The ablation measures the whole pathway. |
| Belief-guided exploration | Replacing the learned explorer with the hybrid, using the same belief, gives 69.1% versus 70.1% full-budget success, with an unresolved difference; subsequent SENSE actions are 122.9 versus 59.3. | The belief supports another successful exploration method, while EchoBelief's joint strategy reduces measurement use. |
We also include an external acoustic reconstruction comparison using an adaptation of Nguyen et al. (IROS 2019) under EchoRooms' sensor interface. To isolate reconstruction from the chosen trajectory, we score identical acquisition trajectories with both estimators. Replacing its geometric reconstruction with our frozen belief and decoder raises success from 28.0% to 43.7% (+15.7 points [12.9, 18.7]) and wall F1 by 20.5 points [18.7, 22.4]. Thus, the reconstruction pipeline extracts more useful geometry from the same measurements.
Together, these comparisons identify the benefits of geometric supervision, learned local mapping, and retained predictive context. They also demonstrate a successful explorer replacement: the same belief supports the hybrid planner, while EchoBelief's joint strategy uses fewer measurements. We will add the four-stage table, distinguish local reconstruction tests from complete-system evaluations, and include the strongest hybrid in the main comparison.
Response (3/3)
W3 / Q2. People, furniture, and noise.
EchoBelief retains higher success than raycast IG across all 12 tested noise conditions. On 120 test rooms with three starts each, we add white, pink, and motor-like noise to the raw 12-channel RIRs at 40, 30, 20, and 10 dB. All models remain frozen, with no retraining. The success advantage ranges from 10.8 points [4.7, 16.9] to 22.5 points [15.0, 30.0].
Matched clean success is 71.9% for EchoBelief and 52.2% for raycast. At 40 dB, neither method has a statistically resolved success change. At 20 dB, EchoBelief loses 5–16 points and raycast 0–10; at 10 dB, success falls to 20–27% and 8–11%, respectively. Thus, EchoBelief retains an absolute success advantage, while strong noise still substantially degrades mapping. SNR is defined over the 100 ms input window, including the direct path; early-reflection SNR is about 13 dB lower.
For interior obstacles, EchoBelief retains higher structural-wall recall: 84.5% versus 75.3% (+9.3 points [0.6, 19.2]). All three obstacle-study conditions use the same 24 test rooms with one start each and live rendering at every acquisition. The added obstacles are static full-height boxes that reflect sound and block motion. Box sides range from 0.4 to 1.5 m, with one box per 25 m² of free floor and 1–30 boxes per room. The table scores the original structural walls.
| Condition | EchoBelief success (%) | Raycast IG success (%) |
|---|---|---|
| Clean live rendering | 75.0 | 79.2 |
| Static reflecting obstacles | 50.0 | 50.0 |
| Obstacles and intermittent interference | 33.3 | 33.3 |
Obstacles therefore reduce success for both methods. EchoBelief's reduction is 25.0 points [8.3, 41.7], and its collisions increase from 3.4 to 22.6 per episode. These frozen models were trained without the added obstacles. The higher structural-wall recall shows that EchoBelief recovers more of the room boundary in this test, while the success and collision results identify the remaining difficulty of exploring clutter.
Intermittent interference affects 25% of acquisitions, using babble-like noise or 5–30 ms broadband bursts at 10 dB. With obstacles and interference combined, structural-wall recall remains higher for EchoBelief by 11.0 points [2.1, 20.4], although both methods reach 33.3% success.
These tests address added acoustic noise and static reflecting obstacles. The boxes behave like pillars, and the interference models intermittent sound; ordinary three-dimensional furniture, moving people, and changing occlusion remain untested. We will state this scope explicitly and include both the retained performance and the degradation in the revised paper.
We hope the budget comparisons, component evidence, and noise and obstacle tests address your concerns and clarify the contribution. We will incorporate these results and explanations into the main text so that each comparison and its implications are easy to follow. Please let us know if any concern remains unresolved or would benefit from further clarification. In light of these clarifications and additional results, we would be grateful if you would consider raising your score.
Response (1/3)
Thank you for recognizing the value of EchoRooms, the complete acoustic mapping system, and the results under a limited sensing budget. Your questions concern how performance changes with the available budget, where the component benefits come from, and how noise and obstacles affect mapping. We address each with the comparisons below.
W1. Success versus sensing budget.
To make the tradeoff explicit, we compare EchoBelief with the three original planners and the strongest tested frontier-plus-raycast hybrid at six acquisition caps. The evaluation uses 355 test rooms with three starts each. Success means at least 90% wall recall in the final map. Throughout this response, differences are computed before rounding; brackets give paired room-bootstrap 95% confidence intervals conditional on the trained checkpoints.
EchoBelief improves success over the hybrid at 16, 32, and 96 acquisitions, while the differences at 48, 64, and 128 are not statistically resolved. Its success advantage over each of the three original planners is resolved at every tested cap.
The joint exploration and stopping strategy also reduces measurement use. At the 128-acquisition cap, EchoBelief uses 59.3 subsequent SENSE actions versus 122.9 for the hybrid, approximately 52% fewer, with no statistically resolved success difference. At the 64-acquisition cap, it uses 40.9 versus 61.9. These counts exclude the shared initial acquisition; the cap K includes it. Together, the results show both where EchoBelief maps more rooms successfully and where it uses fewer measurements.
For each cap, all methods have a decision limit of 8K. EchoBelief is evaluated separately at each cap. Planner trajectories are truncated at acquisition K or decision 8K because those planners do not read the remaining budget; at K = 64, this reproduces the independently evaluated success outcome in every episode. We will add this table and the recall-versus-acquisitions figure, with final maps carried forward after an episode ends. These evaluate sensing budgets and measurement use; physical execution time was not measured.