acceptodds
Under review as a conference paper at ICLR 2027

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

Abstract

While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial audio-visual reasoning, audio-visual large language model, egocentric multi-modal perception, 2D floormap

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xaQ4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FloorSAV, a training-free framework for spatial audio-visual reasoning. It builds a dynamic 2D floormap from the AEA dataset’s 3D point cloud, camera pose, estimated sound-source position, camera trajectory, and VLM-grounded object labels. The floormap is synchronized with the egocentric video and passed to an AV-LLM as a second video stream. The paper also introduces SAVED-Bench, with dynamic relativity, regional, and path reasoning tasks. FloorSAV improves Gemini-3.6-Flash from 58.3 to 65.3 overall on SAVED-Bench.

Critical strengths
  1. The method is intuitive. A top-down map is a natural way to expose global geometry, camera motion, sound locations, and landmarks for spatial reasoning.
  2. SAVED-Bench is a meaningful benchmark contribution. Compared with SAVVY-Bench, which mainly focuses on direction and distance at a queried moment, SAVED-Bench includes cross-agent reasoning, region-level reasoning, and trajectory/path reasoning across longer temporal intervals.
  3. The paper makes a useful attempt to integrate audio localization into spatial intelligence. It combines acoustic tracking, camera geometry, and semantic object grounding into a unified spatial map to support reasoning.
Critical weaknesses
  1. The main evaluation uses serious privileged test-time input. FloorSAV constructs its floormap from the AEA-provided global 3D point cloud and dataset camera pose. In contrast, the AV-LLM baseline only receives the egocentric audiovisual stream. This changes the information available to the compared methods, rather than only changing its representation. It also appears inconsistent with the standard audiovisual benchmark setting, where the task is to answer from just egocentric video, audio, and the question. Therefore, the reported gains cannot be attributed to the method itself, but more like a performance gain from leaked or privileged input. This issue affects both SAVVY-Bench and SAVED-Bench. On SAVED-Bench, the paper explicitly gives the floormap only to FloorSAV, while the compared baseline is not given these privileged information.
    I consider this the most fundamental issue. A fair evaluation should reconstruct geometry from the same test-time observations available to other methods, e.g., using an inference-time 3D reconstruction model (like VGGT-Omega), instead of directly accessing the dataset-provided global point cloud. If the authors could provide such a fair experiment, I would consider to rise my rating.
  2. The method itself is quite incremental. The pipeline mainly takes dataset geometry and poses, applies existing acoustic tracking and VLM grounding, renders them into a top-down video, and feeds this video to an existing AV-LLM. Top-down maps are already common in spatial reasoning. Given the privileged input above, the performance gain can also be explained by stronger input information rather than a new reasoning method.
  3. The empirical advantage is weak even under this favorable setting. In Table 4, SAVVY does not receive the same point-cloud-derived floormap, yet Gemini-2.5-Pro with SAVVY obtains 58.0, while FloorSAV obtains only 54.7 overall. This makes the claimed benefit of FloorSAV over existing rule-based spatial reasoning unclear.
  4. The ablations do not establish the effectiveness of FloorSAV itself. None of the ablations isolates whether the proposed FloorSAV design is actually responsible for the performance gain. In contrast, the partial-GT experiment shows that replacing the off-the-shelf acoustic tracking with ground-truth agent positions leads to substantial improvements, suggesting that a major bottleneck comes from the chosen audio localization pipeline rather than the proposed reasoning framework. Together with the privileged dataset-provided point cloud discussed above, the current experiments mainly show just one thing: providing the model with increasingly accurate spatial information (both source pointmap and GT object location) improves performance. This goes without saying. The ablations do not provide convincing evidence that FloorSAV itself is effective or necessary.
Questions for authors
  1. Could the authors clarify the input protocol of SAVVY-Bench and SAVED-Bench? Are the standard inputs only audio, egocentric video, and the question (like a standard VQA with audio), or do the benchmarks allow access to the global point cloud and camera pose at inference time? If the latter, why are these inputs not also provided to the baseline methods, but only to FloorSAV?
  2. Could the floormap be constructed solely from the given egocentric video using test-time 3D reconstruction, without accessing the dataset-provided point cloud? Does FloorSAV still provide gains under this fair-input setting?
  3. Could the authors provide the same map information to the compared models in alternative formats? For example, (A) FloorSAV provides the information as a floormap video, while (B) the same underlying information used to construct the floormap, such as object coordinates and camera positions at relevant timestamps, is formatted as text and provided to the baseline model or SAVVY. If (A) consistently outperforms (B), this would provide stronger evidence that the benefit comes from FloorSAV itself rather than simply from access to additional spatial information. This comparison would substantially improve both the fairness of the evaluation and the claimed contribution of FloorSAV.
  4. Is Gemini-3.6-Flash sufficiently strong and representative as the main baseline in Table 1? Could the authors report FloorSAV results with more AV-LLMs to demonstrate its generality?
  5. Could the authors report performance using only the floormap video, without the egocentric RGB video? Seems these map videos already contain sufficient information to get the answer, making the ego video less useful.
  6. Could the authors provide more comparison and discussion with closely related spatial-reasoning methods based on cognitive maps or top-down maps? This line of work is currently under-discussed.
  7. Could the authors clarify the methodological contribution of FloorSAV itself? It is unclear. Currently, it seems to simply provide the model with additional spatial information in the form of a floormap, and then compare against methods that do not have access to the same information and get advantage over them.
8 comments
Reviewer_xaQ4OpenReview ↗

More information on the rebuttal

I raised several questions during the review because I find this work interesting. However, I recognize that addressing all of them may place a substantial burden on the rebuttal. I would therefore like to highlight my main concerns: Weaknesses 1 and 4, and Questions 1, 2, 3, and 7. Addressing other concerns would also be helpful, as they are mainly some explanations and do not require much extra experiments.

My primary concerns are the fairness of the point-cloud input setting and the benchmark comparisons. If input and benchmark setup is not fair the contribution of the work cannot be established. If the authors can adequately address these issues, I would be willing to reconsider my assessment and potentially raise my score.

AuthorsOpenReview ↗

Author Response to Reviewer xaQ4 (1/3)

First of all, we sincerely appreciate Reviewer xaQ4’s time and effort in reviewing our paper and providing detailed comments and questions. The review has genuinely helped us conduct additional ablations and experiments that strengthen our paper and clarify the contributions of FloorSAV and SAVED-Bench with a clear vision.

We also thank you for recognizing the intuitive design of FloorSAV, which integrates diverse spatial information into a top-down floormap, its useful attempts to render audio localization, and SAVED-Bench’s meaningful contribution to broad spatial tasks.

The reviewer’s concerns include (1) FloorSAV’s privileged test-time inputs compared with the baseline using only egocentric video/audio, (2) whether its gains come from additional information rather than its methodological contribution, (3) its lower score than SAVVY despite these additional inputs and favorable setting, and (4) the lack of design ablations beyond ground-truth studies, whose gains may simply benefit from more accurate inputs.

Below, we address these concerns by:

  • (1) Clarifying the permitted use of AEA data in these benchmarks and evaluating a baseline with the same spatial information rendered in the floormap, which FloorSAV consistently outperforms across models and benchmarks.
  • (2) Demonstrating FloorSAV’s benefits under the same spatial information conditions and showing that VGGT-Ω FloorSAV with a floormap reconstructed from egocentric video outperforms the baseline using AEA-provided geometry and camera pose.
  • (3) Explaining SAVVY’s task-specific and favorable design on SAVVY-Bench, compared with FloorSAV’s advantages, including its task-agnostic design.
  • (4) Providing system-level ablations that support FloorSAV’s design choices and clarifying that the ground-truth studies assess its potential with accurate inputs.

W1-1 & Q1 & Q3: Request for clarification of the input protocols for SAVVY-Bench and SAVED-Bench, and a baseline with identical spatial information to clearly report the contribution of FloorSAV’s method.

Thank you for these concrete comments. Your concerns are reasonable and important, as differences in the spatial information can obscure FloorSAV’s methodological contribution.

Before analyzing the experimental results, we clarify that SAVVY-Bench and SAVED-Bench allow the use of data from the AEA dataset, including spatial information such as 3D point clouds and camera poses. SAVVY also uses AEA-provided spatial information. However, our original baseline does not use this additional spatial information.

Following your recommendation in Q3, we constructed a text-structured baseline using the same spatial information rendered in the floormap. Specifically, we provide egocentric video/audio, a question, structured spatial information in text form, and a guide prompt for interpreting structured information (see our General Response for implementation details).

Below are the results with Gemini-Flash and Qwen3-Omni-30B across the benchmarks. FloorSAV consistently outperforms the text-structured baseline while using fewer tokens to represent the same spatial information. Especially for Qwen3-Omni-30B, which has a 65.5K-token context window, the text-structured baseline underperforms the baseline only with egocentric video/audio and questions, whereas FloorSAV improves performance. These results further highlight the benefits of FloorSAV’s compact representation for the model with limited context length.

In Section 5 and Appendix A of our paper, we added results of the text-structured baseline and its implementation details.

Table 1. Comparison with text baseline on Qwen3-Omni-30B (Context length limit: 65,536)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline37.748.1-
Text-structured baseline35.046.623,140
FloorSAV43.249.615,405

Table 2. Comparison with text baseline on Gemini-2.5/3.6-Flash (Context length limit: 1,048,576)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline58.345.7-
Text-structured baseline63.247.697,320
FloorSAV65.350.97,829

(Additional tokens refer to required token counts to represent spatial information provided on the floormap and text-structured baseline, excluding egocentric video and audio)

AuthorsOpenReview ↗

Author Response to Reviewer xaQ4 (2/3)

W1-2 & Q2: Request to evaluate FloorSAV under the standard audiovisual benchmark setting, using egocentric video/audio and the question, to demonstrate its benefits with inference-time 3D reconstruction.

Although SAVVY-Bench and SAVED-Bench allow the use of all data provided by the AEA dataset, your comments offer an opportunity to evaluate FloorSAV under more restricted input conditions. Therefore, we evaluate FloorSAV with geometry and camera poses reconstructed from egocentric video along with object landmarks. As some recent works use binaural [1] or first-order ambisonic (FOA) audio [2], we incorporate acoustic estimates. We construct and evaluate VGGT-Ω FloorSAV on both SAVVY-Bench and SAVED-Bench using the Gemini-Flash series (details of the construction procedure are provided below).

VGGT-Ω FloorSAV shows competitive performance with the original FloorSAV, outperforms a baseline provided with the same acoustic estimates, and even surpasses the text-structured baseline that uses point clouds and camera poses from the AEA dataset. These results demonstrate that FloorSAV remains effective when dataset-provided geometry and camera pose are replaced with estimates reconstructed from egocentric video and suggest opportunities to extend its applicability.

In Section 5 of our paper, we added analysis and results of VGGT-Ω FloorSAV.

Table 3. Comprehensive comparison of VGGT-Ω FloorSAV with Gemini-Flash

SAVED-BenchSAVVY-Bench
Baseline58.345.7
Baseline with acoustic estimates61.343.0
Text-structured baseline63.247.6
VGGT-Ω FloorSAV64.250.3
FloorSAV65.350.9

Floormap rendering procedure for VGGT-Ω FloorSAV

  1. We use VGGT-Ω [3] to estimate relative depth and camera poses from 128 frames sampled from each recording.
  2. To handle dynamic agents (humans) appearing in some frames, we use Grounding DINO [4] to detect human bodies, arms, and hands, and SAM2 [5] to generate human masks for each frame.
  3. We exclude human-masked regions from the depth maps and reconstruct 3D point clouds using the estimated camera poses and remaining depth values.
  4. Since acoustic estimates and spatial QAs require metric distances, we use UniDepth V2 [6] to calculate a metric scale for each scene. Specifically, we estimate metric depth for all frames and fit a single scene-level scalar that minimizes the gap between the metric depth and the scaled VGGT-Ω depth.
  5. Using the metrically scaled reconstruction and estimated camera poses, we generate floor plane candidates using RANSAC and refine them through least-squares fitting. Candidates are assessed considering geometry such as walls and tables, and the camera wearer's height.
  6. We project the 3D point clouds onto the estimated floor plane to obtain a floormap, then render the estimated camera poses, semantically grounded objects, and acoustic estimates.

In Appendix A of our paper, we provide a detailed procedure for 2D floormap rendering from RGB egocentric video, acoustic estimates, and object landmarks.


W2 & Q6 & Q7: Request for a detailed comparison with existing map strategies in spatial reasoning, and concern about the method’s novelty and whether its gains come from richer input information rather than a new reasoning method.

We agree that top-down maps are reported on spatial reasoning; However, compared to static cognitive maps [7] and bird’s-eye-view (BEV) maps with object grounding [8], FloorSAV renders an integrated floomap video including a dynamic agent that is synchronized with the egocentric video/aduio stream, allowing explicit representation of dynamic agents with various spatial information.

Furthermore, the comparison between FloorSAV and the text-structured baseline (see our responses to W1-1 & Q1 & Q3), and results of VGGT-Ω FloorSAV (see our responses to W1-2 & Q2), support the benefits of FloorSAV’s methodology.

In Sections 1 and 2 of our paper, we added more comments on comparisons with existing map strategies.

Reviewer_sySCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FloorSAV renders geometry, sound-source positions, and semantic object information into a dynamic 2D map, which is provided to an AV-LLM alongside synchronized egocentric audio/video. The framework improves spatial question answering without fine-tuning. The paper also introduces SAVED-Bench, containing 1,988 questions covering cross-agent perspective transformations, regional understanding, and path-related reasoning.

Critical strengths

1)FloorSAV provides a practical way to improve spatial reasoning without fine-tuning. It raises the SAVED-Bench score from 58.3 to 65.3 and improves average performance for three base models on SAVVY-Bench. Component ablations support the usefulness of the proposed design.

2)SAVED-Bench contributes 1,988 questions covering cross-agent transformations, regional understanding, and path reasoning, extending evaluation beyond the direction and distance questions in SAVVY-Bench.

Critical weaknesses

1)The distinctive benefit of map rendering remains insufficiently isolated from the additional spatial information it supplies. A comparison with coordinate-text or programmatic baselines using the same perception outputs would clarify the methodological contribution. FloorSAV-Pro also scores below SAVVY overall (54.7 versus 58.0), although differing prompts limit this comparison.

2)Both benchmarks derive from AEA, and SAVED-Bench is evaluated with only one model. This limits the evidence that the reported gains extend to other models and environments.

Questions for authors

1)What evidence distinguishes the benefit of map rendering from that of the additional spatial information? Do you have a comparison with coordinate-text or programmatic reasoning using the same perception outputs? 2)How many independent physical environments are represented in SAVED-Bench, and were the evaluation recordings also used to select prompts or pipeline parameters?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer sySC

First of all, we thank Reviewer sySC for reviewing our paper, providing valuable comments, and recognizing FloorSAV’s design for spatial reasoning, as well as SAVED-Bench’s contribution to extending task coverage.

Regarding the reviewer’s concerns about the benefits of additional spatial information and generalization across models and environments, we show that FloorSAV outperforms the text-structured baseline with identical spatial information on both Gemini-Flash and Qwen3-Omni-30B across 55 real-world recordings from the AEA dataset.


W1-1 & Q1: Comparison with text baselines using the same additional spatial information is needed

We construct a text-structured baseline using the same spatial information rendered in the floormap to assess the benefits of the FloorSAV methodology beyond access to additional spatial information. Specifically, we provide egocentric video/audio, a question, the spatial information rendered in the floormap as a JSON-style structured text, and a guide prompt for interpreting the structured information (see our General Response for implementation details).

Below are the results with Gemini-Flash and Qwen3-Omni-30B across the benchmarks.

Table R1. Comparison with text baseline on Gemini-2.5/3.6-Flash (Context length limit: 1,048,576)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline58.345.7-
Text-structured baseline63.247.697,320
FloorSAV65.350.97,829

Table R2. Comparison with text baseline on Qwen3-Omni-30B (Context length limit: 65,536)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline37.748.1-
Text-structured baseline35.046.623,140
FloorSAV43.249.615,405

(Additional tokens refer to required token counts to represent spatial information provided on the floormap and text-structured baseline, excluding egocentric video and audio.)

FloorSAV consistently outperforms the text-structured baseline while using fewer tokens to represent the same spatial information. Especially for Qwen3-Omni-30B, which has a 65.5K-token context window, the text-structured baseline underperforms the baseline with only egocentric video/audio and questions, whereas FloorSAV improves performance. These further highlight the benefits of FloorSAV’s compact representation for the model with limited context length.

In Section 5 and Appendix A of our revision, we have added results of the text-structured baseline and its implementation details.


W1-2: Concern of FloorSAV’s lower score vs. SAVVY (54.7 vs. 58.0) on SAVVY-Bench

We clarify that SAVVY is task-specifically designed for SAVVY-Bench. Specifically, it assumes a closed question set and branches according to predefined task types in SAVVY-Bench. It then uses an AV-LLM for sound-based temporal grounding and target localization in video frames. Subsequent calculations use fixed, deterministic arithmetic, and the results are mapped to a predefined answer set. Therefore, SAVVY has strengths on SAVVY-Bench (Detailed explanation of SAVVY in our General Response).

Although SAVVY’s task-specific design demonstrates strengths on SAVVY-Bench, it cannot be directly applied to other spatial tasks without modifying the whole pipeline, including those in SAVED-Bench. In contrast, FloorSAV provides egocentric and floormap videos to an AV-LLM, which acts as a task-agnostic spatial reasoner. Furthermore, while SAVVY requires running the entire pipeline for each question, FloorSAV renders the floormap video once per recording, regardless of the task.

Therefore, the experiment's primary goal is to demonstrate the generality of FloorSAV as an effective and efficient spatial representation for a broad range of spatial tasks, even under SAVVY's evaluation protocol, which is unfavorable to FloorSAV. The results in Tables R1 and R2 further support benefit of FloorSAV’s representation across tasks and models.

In Appendix B of our revision, we provide a detailed comparison between SAVVY and FloorSAV.


W2 & Q2: Generalization of models and environments from the AEA dataset, and the use of evaluation recordings to select prompts or pipeline parameters.

We demonstrate that FloorSAV consistently improves performance across models (Tables R1 and R2). We constructed SAVED-Bench using the AEA dataset because it captures diverse real-world activities across regions, agent movements, sounds, and spatial interactions, along with rich spatial information from wearable devices. SAVVY-Bench and SAVED-Bench use 59 and 55 video sequences, respectively, covering diverse activities and interpersonal interactions across three distinct indoor environments. Extending the benchmarks to outdoor would allow further assessment of FloorSAV’s generalizability in future work. We used one evaluation recording, loc1_script2_seq1_rec1, to visually inspect the floormap and refine the parameters of FloorSAV’s rendering pipeline.

AuthorsOpenReview ↗

Thank you for your commitment

Dear Reviewer sySC,

Thank you again for the review and valuable comments. With only a few hours left before the discussion window closes, we would very much appreciate hearing whether our response has addressed your concern. We know how busy this period is, and we sincerely appreciate your time and consideration.

Best regards,
Authors

Thank you for the detailed response and for the new text-structured baseline experiments.

The comparison with a text baseline that uses the same perception outputs was the main missing piece in my review. FloorSAV outperforms this baseline on both Gemini-Flash and Qwen3-Omni-30B, and the Qwen results are particularly informative because the text baseline can hurt performance under a limited context window while the floormap representation still helps. This evidence better isolates the benefit of map rendering from that of simply providing additional spatial information. The clarification of SAVVY’s task-specific pipeline also puts the FloorSAV-Pro comparison in a fairer context.

Two residual limitations remain. SAVED-Bench and SAVVY-Bench still come from the AEA dataset and cover three indoor environments, so cross-environment generalization is not yet strongly established. It is also useful that the authors disclose the use of one evaluation recording to refine rendering parameters; this should be stated clearly in the paper.

These points do not outweigh the new evidence on the rendering-versus-text comparison, which addresses my primary concern. I will keep my rating of 3 (Weak acceptance).

Reviewer_Ubs6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FloorSAV, a training-free framework that provides audio-visual language models with a synchronized 2D floormap alongside egocentric video. The map integrates scene geometry, camera motion, estimated sound-source positions, and semantic object landmarks. The paper also introduces SAVED-Bench, covering cross-agent viewpoint transformations, regional understanding, and path-related reasoning, and evaluates the framework on both SAVED-Bench and SAVVY-Bench.

Critical strengths
  1. The paper addresses a meaningful problem with plausible applications to embodied assistants: understanding spatial relations involving moving people, changing viewpoints, and audio events in real environments.
  2. The framework improves aggregate performance over the corresponding egocentric-only baselines across multiple models on SAVVY-Bench, and from 58.3 to 65.3 on SAVED-Bench. The benchmark construction and component studies provide useful evidence about the feasibility and limitations of map-assisted reasoning.
Critical weaknesses
  1. The relative benefit of the integrative method remains insufficiently established. The methodological contribution appears primarily incremental, and the paper lacks a clear horizontal comparison with competitive spatial reasoning pipelines. Table 4 reports an aggregate score of 54.7 for FloorSAV with Gemini-2.5-Pro, compared with the previously reported SAVVY score of 58.0, despite FloorSAV’s stronger exocentric direction result. However, this comparison is incomplete because SAVVY’s results are imported from prior work and the two methods are not evaluated under fully matched settings. The paper should therefore provide a direct, side-by-side comparison with representative existing pipelines on the same tasks, using consistent models, metrics, and evaluation protocols. Such a comparison would clarify whether FloorSAV offers advantages in task coverage, efficiency, robustness, or overall effectiveness, rather than merely presenting results against unaugmented foundation-model baselines.
  2. The distinct value of SAVED-Bench needs stronger comparative evidence. Its evaluation primarily compares the proposed framework with unaugmented foundation-model baselines, without a clear horizontal comparison against representative existing spatial reasoning pipelines. As a result, it remains difficult to determine which additional challenges SAVED-Bench exposes relative to existing benchmarks, whether current methods already address those challenges, and how much FloorSAV improves over prior approaches when evaluated on the same tasks. The authors should include applicable existing methods in a unified comparison, or explain in detail why such methods cannot be evaluated under the benchmark’s setting. These comparisons would clarify the benchmark’s added value and distinguish the contributions of SAVED-Bench from those of the proposed method.
Questions for authors
  1. Can the SAVVY comparison be clarified or reproduced under matched model, input, and prompting conditions? How should readers interpret the 54.7 versus 58.0 aggregate scores, and what practical benefit compensates for the lower aggregate performance?
  2. Could representative existing spatial reasoning methods, such as SAVVY on task-compatible subsets or a baseline using the same geometric estimates with deterministic calculations, be evaluated on SAVED-Bench? A task-wise comparison would help identify the difficulties that remain unsolved.
  3. Which specific capabilities or real-world conditions in SAVED-Bench are insufficiently assessed by existing benchmarks? A concise task-level comparison would help substantiate its distinct contribution. Where feasible, evaluation on another compatible benchmark could also strengthen the evidence for the framework’s broader applicability.

I will raise my score if my concern is resolved.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer Ubs6 (1/3)

First of all, we thank Reviewer Ubs6 for reviewing our paper and recognizing FloorSAV as a plausible approach to a meaningful problem: spatial interactions involving dynamic agents in real environments. We also appreciate their comments on the SAVED-Bench design and FloorSAV’s ground-truth studies, which support the feasibility and potential of our work.

The reviewer’s concerns include (1) a horizontal comparison of FloorSAV with competitive spatial reasoning, (2) a side-by-side comparison with SAVVY, (3) the distinct value of SAVED-Bench compared to SAVVY-Bench, and (4) why SAVVY cannot be directly evaluated on SAVED-Bench.

Below, we address these concerns by:

  • (1) Comparing FloorSAV with a text-structured baseline outlining the same spatial information; results show that FloorSAV consistently performs better.
  • (2) and (4) Explaining SAVVY’s favorable setting and task-specific constraints on SAVVY-Bench, which limit direct comparison with FloorSAV and its applicability to SAVED-Bench, while clarifying the advantages of FloorSAV’s task-agnostic design.
  • (3) Detailed explanation of how SAVED-Bench extends spatial tasks beyond the basic direction and distance calculations covered by SAVVY-Bench.

W1-1 & W2-1 & Q1-1: Request for a direct comparison with SAVVY on the same tasks (or why such comparisons are not applicable) to establish distinct contributions of FloorSAV and SAVED-Bench.

The SAVVY framework heavily depends on the tasks in SAVVY-Bench. Specifically, SAVVY assumes fixed question templates and branches according to predefined task types (e.g., egocentric direction). It then uses an AV-LLM for sound-based temporal grounding and target localization in video frames. Subsequent calculations use fixed, deterministic arithmetic, and the results are mapped to a predefined answer set. Therefore, evaluating SAVVY on SAVED-Bench requires rebuilding the whole framework (see our General Response for details of SAVVY).

Since SAVVY cannot be applied to SAVED-Bench, we alternatively constructed a text-structured baseline using the same spatial information rendered in the floormap to assess the benefits of the FloorSAV methodology beyond access to additional spatial information. Specifically, we provide egocentric video/audio, a question, the spatial information rendered in the floormap as a JSON-style structured text representation, and a guide prompt for interpreting the structured information (see our General Response for implementation details). Below are the results with Gemini-Flash and Qwen3-Omni-30B across the benchmarks.

Table R1. Comparison with text baseline on Gemini-2.5/3.6-Flash (Context length limit: 1,048,576)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline58.345.7-
Text-structured baseline63.247.697,320
FloorSAV65.350.97,829

Table R2. Comparison with text baseline on Qwen3-Omni-30B (Context length limit: 65,536)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline37.748.1-
Text-structured baseline35.046.623,140
FloorSAV43.249.615,405

(Additional tokens refer to required token counts to represent spatial information provided on the floormap and text-structured baseline, excluding egocentric video and audio.)

FloorSAV consistently outperforms the text-structured baseline while using fewer tokens to represent the same spatial information. Especially for Qwen3-Omni-30B, which has a 65.5K-token context window, the text-structured baseline underperforms the baseline only with egocentric video/audio and questions, whereas FloorSAV improves performance. These results further highlight the benefits of FloorSAV’s compact representation for the model with limited context length. In Section 5 and Appendix A of our revision, we have added results of the text-structured baseline and its implementation details.


Based on the effectiveness of FloorSAV demonstrated in Tables R1 and R2, and the task-specific SAVVY framework, we address the following concern:

W1-2 & Q1-2: Concern of FloorSAV’s lower score vs. SAVVY (54.7 vs. 58.0) on SAVVY-Bench

Although SAVVY’s task-specific design demonstrates strengths on SAVVY-Bench, it cannot be applied to other spatial tasks, including those in SAVED-Bench. In contrast, FloorSAV provides egocentric and floormap videos to an AV-LLM, which acts as a task-agnostic spatial reasoner. Furthermore, while SAVVY requires running the entire pipeline for each question, FloorSAV renders the floormap video once per recording, regardless of the task.

Therefore, the experiment's primary goal is to demonstrate the generality of FloorSAV as an effective and efficient spatial representation for a broad range of spatial tasks, even under SAVVY's evaluation protocol, which is unfavorable to FloorSAV.

AuthorsOpenReview ↗

Author Response to Reviewer Ubs6 (2/3)

W2-2 & Q3-1: Task-level clarification of SAVED-Bench’s contribution beyond SAVVY-Bench and whether SAVVY can address the tasks in SAVED-Bench.

We clarify that SAVED-Bench assesses broader spatial reasoning about dynamic agents in real-world scenarios through three categories: dynamic relativity, regional, and path reasoning QAs. These extend beyond the basic direction and distance tasks of SAVVY-Bench, as we discussed in Section 4, Figure 4, and Appendix A of our paper. In addition, SAVVY cannot address tasks of SAVED-Bench (see our response on W1-2 & Q1-2).

First, we explain SAVVY-Bench, which evaluates two types of viewpoints: egocentric and allocentric.

Egocentric tasks evaluate the relative direction or distance to another agent from the camera wearer’s viewpoint at the query time.

  • Question: “Imagine you are the camera wearer. When the speech topic <sound event> comes up, relative to where you are facing, where is the speaker?”
  • Answer: “(A) Front-left.”

Allocentric tasks evaluate the relative direction or distance to another agent from the viewpoint described in the question at the query time.

  • Question: “Imagine you are a robot standing by <object1> and facing <object2>. When the speech topic <sound event> comes up, relative to where you are facing, where is the speaker?”
  • Answer: “(A) Front-left.”

Both types of tasks are limited to basic direction or distance estimation from a single viewpoint at a queried moment. However, SAVED-Bench evaluates broader spatial tasks, including cross-agent viewpoint transformation, region recognition, and environmental awareness across various temporal windows.

We explain SAVED-Bench with concise task-level explanations and corresponding QA pairs below.

Dynamic relativity QAs: An embodied agent interacting with other agents in real-world scenarios must understand their spatial reference frames to interpret the shared spatial context and communicate correctly. For example, when two agents face each other, one agent’s right corresponds to the other’s left. Dynamic relativity QAs evaluate cross-agent viewpoint transformation capabilities through questions about spatial relations between the camera wearer’s and another agent’s viewpoints.

  • Question: “At the query time, imagine an object 3 m away, 10 degrees clockwise from your forward direction. In which direction is the object from the other person’s viewpoint?”
  • Answer: “(A) Front-left.”

Dynamic relativity QAs also include metric distance estimation and transformations in the reverse direction: from the other agent’s viewpoint to the camera wearer’s viewpoint.

Regional QAs: People usually describe locations using semantic regions, such as kitchens and living rooms. Therefore, an embodied agent should recognize these regions and integrate them into spatial reasoning to successfully interact with human agents.

  • Question: ”Imagine you are the camera wearer; consider the interval from when the <sound event 1> comes up until when the <sound event 2> comes up. During this interval, in which order did you make stable visits to areas of the home?”
  • Answer: “(B) bedroom - living room - dining area - kitchen”

This Visit Order question-answer pair evaluates whether a model can identify semantic regions and reason about the order of visits along a dynamic agent’s trajectory. Beyond Visit Order, the regional category also includes Unvisited, which asks the model to identify regions not visited by the agent during a temporal interval, and Location Awareness, which asks which region the agent is in at the query time.

Path reasoning QAs: Embodied agents deployed in the physical world have their movements constrained by the surrounding environment, including object landmarks. The path reasoning category evaluates spatial awareness of relationships among dynamic agents, surrounding objects, and paths.

  • Question: “Consider the interval from when the <sound event 1> comes up until when the <sound event 2> comes up. During this interval, how many meters did you and the other person travel? Answer in the form {your distance, other person distance}”
  • Answer: “{12.71, 17.43} (meters)”

This Trajectory Distance task asks for the metric distance traveled by each agent during a temporal window. The model must infer the actual path rather than only the displacement between the trajectory’s endpoints. The path reasoning category also includes the Line-Path Search task, which requires spatial reasoning to localize the camera wearer and the target, consider the hypothetical line segment connecting them, and compare candidate objects’ distances to that segment. The target can be either a static object or a dynamic agent.

In Appendix G of the revision, we have added a concise comparison of SAVED-Bench with SAVVY-Bench, including a task-level explanation to clarify the contribution of SAVED-Bench.

AuthorsOpenReview ↗

Author Response to Reviewer Ubs6 (3/3)

Q2: Request for a comparison with existing spatial reasoning methods or a deterministic approach on SAVED-Bench to identify the unsolved difficulties of the proposed benchmark.

To the best of our knowledge, SAVVY is a prominent method for spatial reasoning about dynamic agents, but it cannot be directly applied to SAVED-Bench (see our responses to W1-1 & W2-1 & Q1-1). Therefore, we provide a comparison with a text-structured baseline (see our responses to W1-1 & W2-1 & Q1-1).

Nonetheless, in response to the reviewer’s comment, we have constructed a deterministic baseline with acoustic estimates for applicable tasks on SAVED-Bench. Since object-aware and region-aware tasks require semantic alignment, we conduct experiments on tasks from dynamic relativity QAs and the Trajectory Distance task from path reasoning QAs.

Table R3. Comparison between FloorSAV and deterministic baseline

TaskDeterministic baselineFloorSAV
Ego-to-Exo Direction70.065.0
Ego-to-Exo Distance28.634.6
Exo-to-Ego Direction47.550.5
Exo-to-Ego Distance38.644.7
Trajectory Distance37.325.0

For the deterministic baseline, we use the same acoustic estimates that were rendered on the floor map and agent poses nearest to the ground-truth query time to derive the answers.

Although the deterministic baseline is grounded at the ground-truth query time, its improvements are limited to some tasks. While FloorSAV reasons over integrated spatial information, a substantial gap remains in fully solving these tasks, highlighting their difficulty. Nevertheless, since FloorSAV is training-free, additional fine-tuning or similar procedures may improve spatial reasoning performance, including on SAVED-Bench.

In Appendix H of the revised paper, we have added a comparison between FloorSAV and a deterministic baseline on SAVED-Bench, along with details of the baseline setup, to help identify the remaining difficulties in SAVED-Bench.


Q3-2: Request validates on other benchmarks to strengthen a FloorSAV’s applicability if feasible

AEA provides 3D point clouds, multi-microphone audio, IMU measurements, and synchronized camera recordings from two dynamic agents engaged in spatial and interpersonal interactions. SAVVY-Bench and SAVED-Bench are built on AEA, allowing FloorSAV to be directly evaluated on both benchmarks. To the best of our knowledge, related benchmarks [1, 2] provide RGB images and multi-channel audio, but they lack other required information, including time-varying poses and trajectories of moving agents aligned with scene geometry, and therefore require additional processing to directly evaluate FloorSAV. We leave these adaptations and further evaluation to future work.

References

[1] Ryu et al. Hear you are: Teaching LLMs Spatial Reasoning with Vision and Spatial Sound. CVPR, 2026.
[2] Liu et al. JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments. ICML, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (2/2)

2. Comparison of SAVVY and FloorSAV Frameworks

SAVVY has a favorable and task-specific design on SAVVY-Bench. Its entire pipeline assumes fixed question templates. Below, we explain how SAVVY computes its answers.

  1. SAVVY first branches according to a manually specified task type and extracts relevant phrases from SAVVY-Bench questions using predefined token positions. For example, given an egocentric direction question, “Imagine you are the camera wearer, when the speech topic <sound event> comes up, relative to where you are facing, where is the other person?”, SAVVY extracts <sound event>.
  2. Using the extracted phrase, AV-LLM temporally grounds the sound event and estimates the other person’s location from video frames near the grounded time. SAVVY then uses camera poses to transform the visual location estimates and acoustic estimates into global coordinates and averages them.
  3. Finally, it deterministically calculates the direction or distance relative to the queried camera pose and converts the result into a fixed output format. For example, if the estimated location is front-left of the camera wearer, it outputs “A”.
  4. For an allocentric direction question, “Imagine you are a robot standing by <object1> and facing <object2>, when the speech topic <sound event> comes up, relative to where you are facing, where is the speaker?”, the AV-LLM additionally estimates the locations of <object1> and <object2> across the recording. These estimates are also transformed into global coordinates using given camera poses.

Consequently, SAVVY strongly depends on the task types and question forms in SAVVY-Bench and is not designed to integrate the provided 3D point clouds. It underutilizes AV-LLM only for temporal grounding and target localization in 2D frames, and the entire SAVVY framework runs individually for each question.

In contrast, FloorSAV renders a dynamic floormap video once per recording and provides it to the AV-LLM alongside the egocentric video. As AV-LLM serves as the central spatial reasoner, FloorSAV supports task-agnostic spatial reasoning which fully leverages the model’s reasoning capabilities (see Figure 2 in our paper).

Further comparisons between SAVVY and FloorSAV are provided in the Introduction and Appendix B of our paper.


References

[1] Zeng et al. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language. ICLR, 2023
[2] Chen et al. CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning. WACV, 2026, pp. 2295–2305

AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank Reviewers sySC, Ubs6, and xaQ4 for their efforts in reviewing our paper and providing concrete comments on our work, including FloorSAV and SAVED-Bench.

In this General Response, we’d like to address common comments and concerns regarding a fair comparison between the baseline and FloorSAV, and clarify the differences between the SAVVY framework and FloorSAV.

In addition to the discussion below, we provide detailed individual responses to the weaknesses and questions raised by each reviewer. We kindly ask the reviewers to refer to those responses as well. Also, changes in the revised paper are marked in blue.


 

1. Text-structured Baseline

We construct a text-structured baseline using the same spatial information as FloorSAV to assess the benefit of FloorSAV’s representation strategy beyond accessing richer spatial information.

Compared to the original baseline, which takes egocentric video/audio and a question, the text-structured baseline additionally receives spatial information, including acoustic estimates, 3D point clouds, camera poses, trajectories, and field-of-view (FoV), as structured text. This text-based approach follows the reviewers’ suggestions and is inspired by Socratic Models [1], in which pretrained models exchange information through language to solve multimodal tasks.

Specifically, we provide coordinate-based information aligned with a floormap:

  • We define a coordinate system with directions (+x, +y) and metric resolution (e.g., 0.1 m). We also denote the camera’s FoV in degrees.
  • At each timestamp corresponding to the 128 frames per recording, we provide the camera direction (dx, dy), position (x, y), trajectory over the preceding 15 seconds {(x1, y1, time1), (x2, y2, time2), …}, and acoustic estimates (x, y), if available; otherwise, we inject null.
  • For objects, we provide each name and its corresponding coordinates as (<object_name>, x, y).
  • For 3D point clouds, consistent with FloorSAV’s rendering of filtered points associated with dynamic agents in gray, we use three categories: 0 for empty locations (no detected points), 1 for points from static physical objects, and 2 for filtered points from dynamic agents.

Since point clouds require a large amount of text to express, inspired by CVP [2], we aggregate points into a 2D grid, assigning each cell the majority category among its points. We then represent each cell by its center coordinates, majority category, and the number of points belonging to that category: {(x_grid1, y_grid1, <category>, <number_of_points>), …}.

Considering the models’ context length limits, we use a grid resolution of 10 cm × 10 cm for the Gemini-Flash series and 25 cm × 25 cm for Qwen3-Omni-30B. Since the input to Qwen3-Omni-30B exceeds its context limit even with a coarser grid, we additionally omit repeated trajectories while maintaining the camera poses, acoustic estimates, and timestamps for all frames.

 

Result

We utilize the Gemini-Flash series and Qwen3-Omni-30B for both SAVED-Bench and SAVVY-Bench for comparison between the text-structured baseline and FloorSAV.

Table R1. Comparison with text baseline on Gemini-2.5/3.6-Flash (Context length limit: 1,048,576)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline58.345.7-
Text-structured baseline63.247.697,320
FloorSAV65.350.97,829

Table R2. Comparison with text baseline on Qwen3-Omni-30B (Context length limit: 65,536)

SAVED-BenchSAVVY-BenchAdditional tokens
Baseline37.748.1-
Text-structured baseline35.046.623,140
FloorSAV43.249.615,405

(Additional tokens means the number of tokens to express the identical spatial information that is provided on the floormap and text-structured form)

FloorSAV consistently improves performance across models and benchmarks while using fewer tokens. We found that for Qwen3-Omni-30B, whose context length is limited to 65.5K tokens, the text-structured baseline underperforms the original baseline. These results further highlight FloorSAV under limited context lengths.

 

Revision

In Section 5 and Appendix A of our paper, we added results and implementation details of the text-structured baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.