- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an evaluation benchmark not only tested whether the model would take a detour when necessary, but also examined whether the model could maintain the original route when the sound did not require a detour. This criterion was then used to evaluate existing multimodal models. The authors evaluated existing multimodal models that had not undergone specific training. The experiments were very thorough.
Critical strengths
- A new task was defined, and a new benchmark was constructed for this task. The benchmark includes scenarios that require avoidance and those that should follow the original route, to test whether the model can determine when to adjust the route using sound.
- The experiments were extensive, and the ablation design was convincing. The paper uniformly evaluated 14 multimodal models and conducted ablations for silence, interchanging left and right channels, and visual input. The experiments were very thorough.
Critical weaknesses
- The trajectory design is very simple and does not include timestamps, making it impossible to assess whether timely avoidance can be achieved. The experiment can only indicate whether sound affects the shape and direction of the local path, but it cannot demonstrate whether the model can adjust the route in a dynamic environment in a timely manner.
- The scoring indicators that rely on the reference trajectory may underestimate other reasonable detour routes. In the same scenario, there may be multiple safe and feasible paths. Does a low score indicate that the model will not take a detour, or did it choose another reasonable route?
Questions for authors
- Why did you choose to represent a trajectory with 10 waypoints? Have you checked whether the relative performance of the model would change after altering the number or spacing of waypoints?
- Besides these 536 CARLA trajectories, have you conducted validations in other scenarios, different sound conditions, or real street data? If so, do the main conclusions of the paper still hold under these conditions?
- The model predicts the entire route at once and the waypoints do not have timestamps. Under such a setting, can your evaluation distinguish between timely avoidance and delayed avoidance? Or can it only determine whether the predicted route avoids obstacles?
- If the model selects a route that is different from the human reference trajectory but still safe and feasible, how will your metrics be scored? If this route scores lower, how can you determine whether the avoidance is unreasonable or simply different from the reference trajectory?
2 comments
Q3: Timely & delayed reactions.
Thank you for pointing this out. Our current evaluation asks what path a model predicts at a given decision point, rather than when it chooses to react. We manually mark this point and provide all models with the same preceding observation history. The waypoints describe path geometry without execution timestamps, so the scores do not distinguish timely from delayed execution or measure inference latency. This controlled setting lets us examine whether the available acoustic and visual evidence leads to an appropriate trajectory response. We see streaming and interactive evaluation as the next step for measuring how quickly models respond to new acoustic evidence while navigating.
Q4: Multiple valid avoidance paths.
Thank you for raising this point. Multiple avoidance paths can be reasonable, but their directions are often constrained by the scene layout in our dataset. The pedestrian can normally continue along the original walking direction and pass beside the potential hazard. Sound provides a warning that may motivate additional clearance, rather than requiring an entirely different route. For example, a small construction area on the right may leave enough sidewalk space for the pedestrian to walk straight past it when the machinery is inactive. Once it starts operating, shifting left increases clearance while preserving the overall route. Reaching the other side may instead require moving closer to the active machinery. This could increase collision risk, particularly when the pedestrian is already close to the sound source. In most collected scenes, walls, buildings or road boundaries further constrain the available avoidance side. Examples include a shouting pedestrian beside a wall or a sounding vehicle along the road edge.
Path length also matters. Across all trajectory instances, the mean recorded path length from the decision point to the local goal is 11.45 m. These paths involve local adjustments along a sidewalk or another constrained passage, rather than choices between substantially different routes. People may begin adjusting at different positions, choose different clearances or follow different curves. Over these short planning distances, however, the source location, available walking space and destination often constrain the alternatives to a similar passage and the same general avoidance side. This motivates evaluating the overall response without requiring an exact match to the recorded trajectory.
Accordingly, our evaluation allows different path shapes without requiring a pointwise match to the recorded trajectory. For Adapt, A-Q can reward paths that begin adjusting at different positions or follow different curves on the demonstrated avoidance side, while accounting for goal progress, efficiency and effective waypoint use. For Preserve, K-Q considers overall deviation from the reference route rather than exact waypoint correspondence. We report nDTW$_{GT}$ separately to measure similarity to the recorded human trajectory.
We hope our replies answer your questions and are happy to discuss any remaining points.
We thank you for your thoughtful questions and provide our responses below.
Thank you for these comments. Our contribution is to establish a setting for studying whether environmental sound should change a route toward an independent goal. To our knowledge, HearVGo is the first outdoor audio-visual benchmark dedicated to this form of route adaptation. We built a library of 1,288 clips across 32 classes, focusing on common outdoor sounds relevant to the task and associating them with semantically matched scene assets. The local trajectory setting provides a consistent prediction task across models and lets us examine both path coherence and responses to acoustic evidence. The current evaluation already reveals inconsistent sound conditioned responses and trajectory degeneration in several tested models. These findings offer the community concrete problems to investigate and a shared setting for developing and comparing possible solutions. We recognize that broader coverage would strengthen the benchmark and are considering additional scenes and weather conditions as future extensions.
Thank you for this suggestion. Across all 536 trajectory instances, the mean recorded path length from the decision point to the local goal is 11.45 m. We chose 10 waypoints to provide reasonable spatial spacing. To directly address your question about sensitivity to this choice, we additionally tested 8 and 12 waypoints for Qwen2.5 Omni and MiniCPM o 4.5, the two models with the highest Adapt A-Q in our main experiment.
Changing the point count does not consistently resolve trajectory generation problems. Although point count affects the diagnostic values, Table R2 shows that Qwen2.5 retains substantial template reuse at all three settings. With 12 points, its WDS improves, but template reuse increases relative to 10 points. MiniCPM maintains high WDS and low template reuse across these settings. These results reflect the differences in trajectory generation ability discussed in the paper, rather than suggesting that changing the point count consistently resolves the observed problems.
Table R2. Trajectory generation diagnostics across waypoint counts (%).
Brief acoustic evidence still does not reliably translate into improved Adapt quality. Table R3 reports the paired score changes from Silence to Full. Audio brings modest directional gains, but the paired 95% A-Q confidence intervals include zero for both models at all three point counts. Their relatively strong Full performance therefore does not by itself establish a clear Adapt benefit from audio. Preserve effects also vary with the model and point count. Increasing or decreasing the output length does not reveal a clear Adapt quality gain from audio in either model.
Table R3. Paired score changes from Silence to Full (%). Subscripts give 95% confidence intervals.
Together, these results reinforce our findings on the tested models’ limitations in generating coherent trajectories and using brief acoustic evidence to guide route adjustments.
Thank you for asking about generalization. All 536 trajectory instances are used exclusively for testing. We evaluate every model in a zero shot setting without training or fine tuning on HearVGo. The benchmark includes different outdoor scene contexts and environmental sounds, providing an initial assessment of existing models on this task. We have not yet evaluated on real world recordings. We are considering additional scenes, sound conditions and real outdoor data to examine whether the observed difficulties in trajectory generation and sound conditioned route adjustment persist beyond the current setting.