acceptodds
Under review as a conference paper at ICLR 2027

HearVGo: Benchmarking Sound-Conditioned Trajectory Adaptation For Urban Walking

Abstract

Environmental sound can reveal nearby activity and potential hazards along a route. Yet outdoor navigation largely centers on vision, while audio-visual navigation often treats sound as a cue to the destination. How sound should instead guide route adjustments toward an independent goal remains underexplored. We introduce HearVGo, a benchmark that tests this capability in omni models using brief multimodal observations. We augment CARLA with 1,288 environmental sound clips rendered as spatial audio and collect human walking demonstrations. The resulting 536 trajectory instances ask models to Adapt the path when sound warrants a change or Preserve the path otherwise. We evaluate 14 omni models under a common zero-shot protocol and complement the main comparison with controlled audio and vision ablations. These results show model-dependent gains from audio and few reversals of predicted avoidance direction when the left and right audio channels are exchanged. Some models retain substantial performance using only motion and the goal. Trajectory diagnostics show that goal progress can mask poorly formed waypoint sequences. Together, these results expose limits in the evaluated omni models' ability to determine when sound should alter a route and to express that decision as a coherent local trajectory. Code and data are publicly available: https://anonymous.4open.science/r/HearVGo.

Primary area:
datasets and benchmarks
Keywords:
Sound Understanding, Audio-visual Navigation, Dataset and benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ERHUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an evaluation benchmark not only tested whether the model would take a detour when necessary, but also examined whether the model could maintain the original route when the sound did not require a detour. This criterion was then used to evaluate 1414 existing multimodal models. The authors evaluated 1414 existing multimodal models that had not undergone specific training. The experiments were very thorough.

Critical strengths
  1. A new task was defined, and a new benchmark was constructed for this task. The benchmark includes scenarios that require avoidance and those that should follow the original route, to test whether the model can determine when to adjust the route using sound.
  2. The experiments were extensive, and the ablation design was convincing. The paper uniformly evaluated 14 multimodal models and conducted ablations for silence, interchanging left and right channels, and visual input. The experiments were very thorough.
Critical weaknesses
  1. The trajectory design is very simple and does not include timestamps, making it impossible to assess whether timely avoidance can be achieved. The experiment can only indicate whether sound affects the shape and direction of the local path, but it cannot demonstrate whether the model can adjust the route in a dynamic environment in a timely manner.
  2. The scoring indicators that rely on the reference trajectory may underestimate other reasonable detour routes. In the same scenario, there may be multiple safe and feasible paths. Does a low score indicate that the model will not take a detour, or did it choose another reasonable route?
Questions for authors
  1. Why did you choose to represent a trajectory with 10 waypoints? Have you checked whether the relative performance of the model would change after altering the number or spacing of waypoints?
  2. Besides these 536 CARLA trajectories, have you conducted validations in other scenarios, different sound conditions, or real street data? If so, do the main conclusions of the paper still hold under these conditions?
  3. The model predicts the entire route at once and the waypoints do not have timestamps. Under such a setting, can your evaluation distinguish between timely avoidance and delayed avoidance? Or can it only determine whether the predicted route avoids obstacles?
  4. If the model selects a route that is different from the human reference trajectory but still safe and feasible, how will your metrics be scored? If this route scores lower, how can you determine whether the avoidance is unreasonable or simply different from the reference trajectory?
2 comments

We thank you for your thoughtful questions and provide our responses below.

W1+W2: Benchmark scale and trajectory setting.

Thank you for these comments. Our contribution is to establish a setting for studying whether environmental sound should change a route toward an independent goal. To our knowledge, HearVGo is the first outdoor audio-visual benchmark dedicated to this form of route adaptation. We built a library of 1,288 clips across 32 classes, focusing on common outdoor sounds relevant to the task and associating them with semantically matched scene assets. The local trajectory setting provides a consistent prediction task across models and lets us examine both path coherence and responses to acoustic evidence. The current evaluation already reveals inconsistent sound conditioned responses and trajectory degeneration in several tested models. These findings offer the community concrete problems to investigate and a shared setting for developing and comparing possible solutions. We recognize that broader coverage would strengthen the benchmark and are considering additional scenes and weather conditions as future extensions.

Q1: Waypoint representation and sensitivity.

Thank you for this suggestion. Across all 536 trajectory instances, the mean recorded path length from the decision point to the local goal is 11.45 m. We chose 10 waypoints to provide reasonable spatial spacing. To directly address your question about sensitivity to this choice, we additionally tested 8 and 12 waypoints for Qwen2.5 Omni and MiniCPM o 4.5, the two models with the highest Adapt A-Q in our main experiment.

Changing the point count does not consistently resolve trajectory generation problems. Although point count affects the diagnostic values, Table R2 shows that Qwen2.5 retains substantial template reuse at all three settings. With 12 points, its WDS improves, but template reuse increases relative to 10 points. MiniCPM maintains high WDS and low template reuse across these settings. These results reflect the differences in trajectory generation ability discussed in the paper, rather than suggesting that changing the point count consistently resolves the observed problems.

Table R2. Trajectory generation diagnostics across waypoint counts (%).

MetricQwen2.5 OmniMiniCPM o 4.5
K8101281012
WDS82.5687.8694.8694.8894.1793.36
Share37.7929.7612.507.118.237.25
Reuse55.2224.4478.174.485.606.53

Brief acoustic evidence still does not reliably translate into improved Adapt quality. Table R3 reports the paired score changes from Silence to Full. Audio brings modest directional gains, but the paired 95% A-Q confidence intervals include zero for both models at all three point counts. Their relatively strong Full performance therefore does not by itself establish a clear Adapt benefit from audio. Preserve effects also vary with the model and point count. Increasing or decreasing the output length does not reveal a clear Adapt quality gain from audio in either model.

Table R3. Paired score changes from Silence to Full (%). Subscripts give 95% confidence intervals.

MetricQwen2.5 OmniMiniCPM o 4.5
K8101281012
Δ A-Q+0.45$_{[-1.21,2.09]}$+0.79$_{[-0.37,1.97]}$+0.73$_{[-0.02,1.50]}$+1.33$_{[-0.05,2.89]}$+0.84$_{[-1.15,2.73]}$+0.60$_{[-1.21,2.55]}$
Δ BDS+0.45+1.16+2.24+0.95+1.76+2.67
Δ K-Q−0.76+1.72+1.67+1.68−0.99+1.15

Together, these results reinforce our findings on the tested models’ limitations in generating coherent trajectories and using brief acoustic evidence to guide route adjustments.

Q2: Generalization beyond the benchmark.

Thank you for asking about generalization. All 536 trajectory instances are used exclusively for testing. We evaluate every model in a zero shot setting without training or fine tuning on HearVGo. The benchmark includes different outdoor scene contexts and environmental sounds, providing an initial assessment of existing models on this task. We have not yet evaluated on real world recordings. We are considering additional scenes, sound conditions and real outdoor data to examine whether the observed difficulties in trajectory generation and sound conditioned route adjustment persist beyond the current setting.

Q3: Timely & delayed reactions.

Thank you for pointing this out. Our current evaluation asks what path a model predicts at a given decision point, rather than when it chooses to react. We manually mark this point and provide all models with the same preceding observation history. The waypoints describe path geometry without execution timestamps, so the scores do not distinguish timely from delayed execution or measure inference latency. This controlled setting lets us examine whether the available acoustic and visual evidence leads to an appropriate trajectory response. We see streaming and interactive evaluation as the next step for measuring how quickly models respond to new acoustic evidence while navigating.

Q4: Multiple valid avoidance paths.

Thank you for raising this point. Multiple avoidance paths can be reasonable, but their directions are often constrained by the scene layout in our dataset. The pedestrian can normally continue along the original walking direction and pass beside the potential hazard. Sound provides a warning that may motivate additional clearance, rather than requiring an entirely different route. For example, a small construction area on the right may leave enough sidewalk space for the pedestrian to walk straight past it when the machinery is inactive. Once it starts operating, shifting left increases clearance while preserving the overall route. Reaching the other side may instead require moving closer to the active machinery. This could increase collision risk, particularly when the pedestrian is already close to the sound source. In most collected scenes, walls, buildings or road boundaries further constrain the available avoidance side. Examples include a shouting pedestrian beside a wall or a sounding vehicle along the road edge.

Path length also matters. Across all trajectory instances, the mean recorded path length from the decision point to the local goal is 11.45 m. These paths involve local adjustments along a sidewalk or another constrained passage, rather than choices between substantially different routes. People may begin adjusting at different positions, choose different clearances or follow different curves. Over these short planning distances, however, the source location, available walking space and destination often constrain the alternatives to a similar passage and the same general avoidance side. This motivates evaluating the overall response without requiring an exact match to the recorded trajectory.

Accordingly, our evaluation allows different path shapes without requiring a pointwise match to the recorded trajectory. For Adapt, A-Q can reward paths that begin adjusting at different positions or follow different curves on the demonstrated avoidance side, while accounting for goal progress, efficiency and effective waypoint use. For Preserve, K-Q considers overall deviation from the reference route rather than exact waypoint correspondence. We report nDTW$_{GT}$ separately to measure similarity to the recorded human trajectory.

We hope our replies answer your questions and are happy to discuss any remaining points.

Reviewer_1PiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HearVGo, a benchmark for evaluating whether multimodal models can use environmental sounds to adapt walking trajectories toward a predefined destination. The authors extend the CARLA simulator with 1,288 environmental sound clips and collect 536 human-demonstrated trajectory instances, including 379 Adapt and 157 Preserve cases. Given RGB-D observations, binaural audio, motion history, and a local goal, models predict 10 future waypoints. The benchmark evaluates 14 omni models using metrics for goal progress, avoidance behavior, trajectory preservation, and waypoint quality, alongside audio and visual ablations. Results show that models often achieve reasonable goal progress but struggle with sound-conditioned trajectory adaptation, exhibit inconsistent benefits from audio, and show limited sensitivity to spatial sound direction.

Critical strengths
  1. I think this is an interesting benchmark that evaluates an important aspect of real-world navigation: whether AI models can recognize and react to potential hazards through environmental sounds. This capability could also be useful for assistive navigation, particularly for people with hearing difficulties.

  2. The paper provides several interesting findings about the limitations of current multimodal models. For example, models can achieve high goal progress but still fail to make appropriate adjustments in response to hazard-related sounds. The authors also show that adding audio does not consistently improve performance, and models rarely change their avoidance direction when the left and right audio channels are swapped. In addition, some models produce poorly structured trajectories despite making good progress toward the goal. I think these findings are useful because they reveal different weaknesses in how current models understand and use audio information for navigation, and highlight directions for future research.

Critical weaknesses
  1. The human demonstrations appear to have limited diversity. The 536 trajectory instances are collected by only one person. This raises concerns about the diversity and reliability of the ground truth. Different people may react differently to the same hazard, and multiple trajectories could be equally safe and reasonable. Evaluating models against a single human trajectory per scenario may therefore penalize valid alternative behaviors. I think the authors should ideally include demonstrations from multiple individuals and evaluate the variability in human navigation decisions.

  2. The evaluation does not cover some important models and baselines. Although the paper evaluates 14 omni models, it misses some important multimodal models, such as GPT-4o, and does not include dedicated audio-visual navigation models or traditional trajectory-planning baselines. While the focus on omni models is understandable, including these additional baselines would provide a more complete picture of current capabilities and help determine whether the observed limitations are specific to general-purpose multimodal models or reflect broader challenges in sound-aware navigation.

Questions for authors

Please see weakness.

1 comment

We appreciate your careful review and respond to your comments below.

W1: Limited demonstration diversity.

Thank you for raising this concern. We would like to clarify that the demonstrations were collected by 4 participants, not a single person. The phrase a human operator refers to the operator of each recording, and we will revise this wording. Participants chose how to respond to the sounds they encountered while moving toward the goal, without prescribed avoidance directions or trajectory shapes. Each instance retains one reference, with different human choices represented across participants and scenes rather than repeated demonstrations of the same situation.

We also recognize that different paths can be reasonable. In many collected scenes, walls, road boundaries and nearby obstacles constrain the general avoidance side while leaving room for different adjustment positions, clearances and curves. A-Q can reward these alternative curves on the demonstrated side without requiring an exact reproduction of the reference. For Preserve, K-Q considers overall route deviation rather than matching individual waypoints. We report nDTW$_{GT}$ separately to describe reference similarity, rather than using similarity alone to judge the response.

W2: Incomplete model and baseline coverage.

Thank you for this suggestion. We also considered broader model coverage when designing the experiments. We initially focused on omni models because their joint audio and visual inputs and textual outputs closely match our task requirements. Many multimodal models do not accept audio, which is central to the question we study. For example, the standard GPT-4o API supports text and image inputs but does not accept audio.

Dedicated audio-visual navigation models present a different challenge. The indoor policies we considered typically navigate toward a sounding source and receive fresh observations after each action. Our task instead involves an independent goal and offline recordings. These recordings cannot supply observations along alternative paths chosen by a policy, making a fair rollout comparison difficult. Some learned navigation policies also predict short local segments rather than the complete path required here. For traditional map based planners, we would need to construct a map and specify how acoustic evidence changes planning costs. Using privileged scene information or manually defining the avoidance response would introduce additional assumptions beyond our current protocol. These differences motivated our initial focus on omni models.

Nevertheless, we agree that including additional models would make the comparison more comprehensive. Following your suggestion, we evaluated GPT-4o, Cosmos3 Nano and Qwen3 VL 8B on all 536 trajectory instances (please kindly refer to Table R1 below). They receive the same walker RGB-D observations, motion history and local goal, with no audio or drone observations. We retain the shared task prompt, coordinate system, 10 waypoint output contract and evaluation metrics.

Table R1. Results for additional models (%).

ModelAdaptPreserveObserved Space
A-Q↑SoftRPE↑BDS↑ESU↑nDTW$_{GT}$↑K-Q↑SoftRPE↑KeepPath↑ESU↑nDTW_{GT}\↑OVR↓Coverage
GPT-4o15.2398.8416.2797.6842.7792.7599.6293.9899.0493.876.9097.23
Cosmos3 Nano5.9914.5556.8797.413.477.2618.8320.3197.687.5659.8991.49
Qwen3 VL 8B11.9731.2159.7591.4112.2147.4971.0073.4583.9663.4736.1996.55

We hope these clarifications are helpful and would be glad to discuss any remaining questions.

Reviewer_hUGKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes HearVGo, a benchmark for the task of sound-conditioned trajectory adaptation for urban walking. The CARLA is extended with spatial audio and 536 trajectories spanning adapt and preserve test cases. Performance of 14 omni models is analysed on this task. While the paper proposes an interesting direction; the dataset and evaluation, which is the main contribution of this paper, are limited.

Critical strengths
  1. An interesting, less-explored use of audio for trajectory adaptation.
  2. Contains interesting analysis about the lack of (spatial) audio understanding in omni models, specifically for such a task.
Critical weaknesses
  1. Limited contribution: While it is important to establish a dataset and that omni models are currently not capable of converting acoustic evidence into actionable trajectory adaptation, the finding is not surprising. The paper is taking a good direction and I would encourage the authors to take it a step further. This can be done in several ways:

    • scale the data, incorporate trajectory variability.
    • propose at least one way to adapt these models for the task.
    • It would also be good to evaluate other approaches for multimodal navigation to put things in perspective.
  2. Related to my point above, the use case and motivation warrants real-time trajectory adaptation. In that respect, latency/reaction time of the model is a crucial factor. Is this taken into account?

  3. Were the human demonstrations done by a single person? If yes, it does not account for the variability in reaction to an event. Is the output variability accounted for in evaluation metrics i.e several trajectories could be “safe adaptations”.

Questions for authors

Please see weaknesses

2 comments

We thank you for your constructive feedback and address each point below.

W1: Limited contribution, dataset scale, model adaptation, and additional baselines.

Thank you for your valuable suggestions and for encouraging us to take this work further.

On the data side, we agree that broader coverage would be valuable. Our starting point is to ask whether models can use environmental sound to adjust a route toward an independent goal. To our knowledge, HearVGo is the first outdoor audio-visual benchmark dedicated to this task. We collected and classified common outdoor sounds relevant to this setting, resulting in 1,288 clips across 32 classes. These sounds are associated with semantically matched scene assets so that they fit the surrounding environment. The current evaluation already reveals inconsistent responses to acoustic cues and poorly formed trajectories in several tested models. We are considering more scenes and weather conditions as future extensions.

Regarding a new method, we agree that adapting existing models would be a natural next step. Our focus here is to introduce the problem and examine existing approaches under a shared protocol. The experiments identify concrete gaps in using acoustic evidence and generating coherent paths. By making these limitations explicit, we aim to give the community a clear starting point for discussing the problem and developing suitable methods. The benchmark then provides a common basis for evaluating their progress.

We also agree that comparisons beyond omni models would be useful. We initially chose omni models because they support the joint audio and visual inputs and textual trajectory output required by our task. The indoor audio-visual navigation policies we considered require new observations after each action. Some other policies also rely on interactive rollouts and predict short local movements without audio input. These methods assume access to observations along their chosen paths, whereas our offline dataset provides fixed recordings. Following your suggestion, we have broadened the evaluation with vision language models that can predict trajectories from the recorded observations under the same offline protocol. Please see our response to Reviewer 1PiY's W2 for the results.

W2: Real time trajectory adaptation and model latency.

We agree that latency and reaction time are important for practical route adaptation. They are not evaluated in the current study. We assess multimodal large language models offline and focus on the paths they predict, without including inference time in the scores.

We manually mark a decision point on each recorded trajectory. The model receives the observation history available up to that point and predicts the subsequent local path. All models use the same decision points and history windows. This provides a controlled comparison of their trajectory responses, but does not measure whether those responses would arrive in time during navigation.

This setting evaluates trajectory quality without imposing a real time navigation deadline. Even under this condition, our experiments reveal trajectory degeneration and inconsistent use of acoustic evidence. These findings highlight limitations in route adaptation that need to be addressed alongside inference latency and reaction time.

W3. Demonstration diversity and alternative trajectories.

We would like to clarify that the demonstrations were collected by 4 participants. The phrase a human operator describes the operator of an individual recording, not one person collecting the entire dataset. We will clarify this wording. Participants chose how to respond to the sounds they encountered while moving toward the goal. We did not prescribe an avoidance direction or trajectory shape. Each instance retains one recorded trajectory as its reference. Diversity is therefore represented through different human choices across participants and scenes, rather than multiple demonstrations of the same situation.

In many collected scenes, walls, road boundaries and nearby obstacles constrain the general avoidance side while allowing different adjustment positions, clearances and curves. Our evaluation accommodates these variations without requiring predictions to reproduce the reference point by point. A-Q can reward different avoidance curves on the demonstrated side, while K-Q considers overall deviation from the reference route rather than exact waypoint correspondence. We report nDTW$_{GT}$ separately to describe reference similarity. This allows alternative path shapes to receive credit without treating one recorded trajectory as the only acceptable geometry.

We hope our responses address your concerns and welcome further discussion.

Thank you for your responses, including the additional VLM baseline results and the clarification regarding the 4 human operators.

However, my core concerns regarding the experimental design (W2) and overall scope (W1) remain. Specifically, while offline evaluation is a start, real-time action-reaction dynamics (such as closed-loop replanning and inference latency) are a central part of the problem. Combined with the relatively small scale of the benchmark, I believe the contribution remains limited in its current form. I will therefore maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.