- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FloorSAV, a training-free framework for spatial audio-visual reasoning. It builds a dynamic 2D floormap from the AEA dataset’s 3D point cloud, camera pose, estimated sound-source position, camera trajectory, and VLM-grounded object labels. The floormap is synchronized with the egocentric video and passed to an AV-LLM as a second video stream. The paper also introduces SAVED-Bench, with dynamic relativity, regional, and path reasoning tasks. FloorSAV improves Gemini-3.6-Flash from 58.3 to 65.3 overall on SAVED-Bench.
Critical strengths
- The method is intuitive. A top-down map is a natural way to expose global geometry, camera motion, sound locations, and landmarks for spatial reasoning.
- SAVED-Bench is a meaningful benchmark contribution. Compared with SAVVY-Bench, which mainly focuses on direction and distance at a queried moment, SAVED-Bench includes cross-agent reasoning, region-level reasoning, and trajectory/path reasoning across longer temporal intervals.
- The paper makes a useful attempt to integrate audio localization into spatial intelligence. It combines acoustic tracking, camera geometry, and semantic object grounding into a unified spatial map to support reasoning.
Critical weaknesses
- The main evaluation uses serious privileged test-time input. FloorSAV constructs its floormap from the AEA-provided global 3D point cloud and dataset camera pose. In contrast, the AV-LLM baseline only receives the egocentric audiovisual stream. This changes the information available to the compared methods, rather than only changing its representation. It also appears inconsistent with the standard audiovisual benchmark setting, where the task is to answer from just egocentric video, audio, and the question. Therefore, the reported gains cannot be attributed to the method itself, but more like a performance gain from leaked or privileged input. This issue affects both SAVVY-Bench and SAVED-Bench. On SAVED-Bench, the paper explicitly gives the floormap only to FloorSAV, while the compared baseline is not given these privileged information.
I consider this the most fundamental issue. A fair evaluation should reconstruct geometry from the same test-time observations available to other methods, e.g., using an inference-time 3D reconstruction model (like VGGT-Omega), instead of directly accessing the dataset-provided global point cloud. If the authors could provide such a fair experiment, I would consider to rise my rating. - The method itself is quite incremental. The pipeline mainly takes dataset geometry and poses, applies existing acoustic tracking and VLM grounding, renders them into a top-down video, and feeds this video to an existing AV-LLM. Top-down maps are already common in spatial reasoning. Given the privileged input above, the performance gain can also be explained by stronger input information rather than a new reasoning method.
- The empirical advantage is weak even under this favorable setting. In Table 4, SAVVY does not receive the same point-cloud-derived floormap, yet Gemini-2.5-Pro with SAVVY obtains 58.0, while FloorSAV obtains only 54.7 overall. This makes the claimed benefit of FloorSAV over existing rule-based spatial reasoning unclear.
- The ablations do not establish the effectiveness of FloorSAV itself. None of the ablations isolates whether the proposed FloorSAV design is actually responsible for the performance gain. In contrast, the partial-GT experiment shows that replacing the off-the-shelf acoustic tracking with ground-truth agent positions leads to substantial improvements, suggesting that a major bottleneck comes from the chosen audio localization pipeline rather than the proposed reasoning framework. Together with the privileged dataset-provided point cloud discussed above, the current experiments mainly show just one thing: providing the model with increasingly accurate spatial information (both source pointmap and GT object location) improves performance. This goes without saying. The ablations do not provide convincing evidence that FloorSAV itself is effective or necessary.
Questions for authors
- Could the authors clarify the input protocol of SAVVY-Bench and SAVED-Bench? Are the standard inputs only audio, egocentric video, and the question (like a standard VQA with audio), or do the benchmarks allow access to the global point cloud and camera pose at inference time? If the latter, why are these inputs not also provided to the baseline methods, but only to FloorSAV?
- Could the floormap be constructed solely from the given egocentric video using test-time 3D reconstruction, without accessing the dataset-provided point cloud? Does FloorSAV still provide gains under this fair-input setting?
- Could the authors provide the same map information to the compared models in alternative formats? For example, (A) FloorSAV provides the information as a floormap video, while (B) the same underlying information used to construct the floormap, such as object coordinates and camera positions at relevant timestamps, is formatted as text and provided to the baseline model or SAVVY. If (A) consistently outperforms (B), this would provide stronger evidence that the benefit comes from FloorSAV itself rather than simply from access to additional spatial information. This comparison would substantially improve both the fairness of the evaluation and the claimed contribution of FloorSAV.
- Is Gemini-3.6-Flash sufficiently strong and representative as the main baseline in Table 1? Could the authors report FloorSAV results with more AV-LLMs to demonstrate its generality?
- Could the authors report performance using only the floormap video, without the egocentric RGB video? Seems these map videos already contain sufficient information to get the answer, making the ego video less useful.
- Could the authors provide more comparison and discussion with closely related spatial-reasoning methods based on cognitive maps or top-down maps? This line of work is currently under-discussed.
- Could the authors clarify the methodological contribution of FloorSAV itself? It is unclear. Currently, it seems to simply provide the model with additional spatial information in the form of a floormap, and then compare against methods that do not have access to the same information and get advantage over them.
8 comments
Author Response to Reviewer xaQ4 (1/3)
First of all, we sincerely appreciate Reviewer xaQ4’s time and effort in reviewing our paper and providing detailed comments and questions. The review has genuinely helped us conduct additional ablations and experiments that strengthen our paper and clarify the contributions of FloorSAV and SAVED-Bench with a clear vision.
We also thank you for recognizing the intuitive design of FloorSAV, which integrates diverse spatial information into a top-down floormap, its useful attempts to render audio localization, and SAVED-Bench’s meaningful contribution to broad spatial tasks.
The reviewer’s concerns include (1) FloorSAV’s privileged test-time inputs compared with the baseline using only egocentric video/audio, (2) whether its gains come from additional information rather than its methodological contribution, (3) its lower score than SAVVY despite these additional inputs and favorable setting, and (4) the lack of design ablations beyond ground-truth studies, whose gains may simply benefit from more accurate inputs.
Below, we address these concerns by:
- (1) Clarifying the permitted use of AEA data in these benchmarks and evaluating a baseline with the same spatial information rendered in the floormap, which FloorSAV consistently outperforms across models and benchmarks.
- (2) Demonstrating FloorSAV’s benefits under the same spatial information conditions and showing that VGGT-Ω FloorSAV with a floormap reconstructed from egocentric video outperforms the baseline using AEA-provided geometry and camera pose.
- (3) Explaining SAVVY’s task-specific and favorable design on SAVVY-Bench, compared with FloorSAV’s advantages, including its task-agnostic design.
- (4) Providing system-level ablations that support FloorSAV’s design choices and clarifying that the ground-truth studies assess its potential with accurate inputs.
W1-1 & Q1 & Q3: Request for clarification of the input protocols for SAVVY-Bench and SAVED-Bench, and a baseline with identical spatial information to clearly report the contribution of FloorSAV’s method.
Thank you for these concrete comments. Your concerns are reasonable and important, as differences in the spatial information can obscure FloorSAV’s methodological contribution.
Before analyzing the experimental results, we clarify that SAVVY-Bench and SAVED-Bench allow the use of data from the AEA dataset, including spatial information such as 3D point clouds and camera poses. SAVVY also uses AEA-provided spatial information. However, our original baseline does not use this additional spatial information.
Following your recommendation in Q3, we constructed a text-structured baseline using the same spatial information rendered in the floormap. Specifically, we provide egocentric video/audio, a question, structured spatial information in text form, and a guide prompt for interpreting structured information (see our General Response for implementation details).
Below are the results with Gemini-Flash and Qwen3-Omni-30B across the benchmarks. FloorSAV consistently outperforms the text-structured baseline while using fewer tokens to represent the same spatial information. Especially for Qwen3-Omni-30B, which has a 65.5K-token context window, the text-structured baseline underperforms the baseline only with egocentric video/audio and questions, whereas FloorSAV improves performance. These results further highlight the benefits of FloorSAV’s compact representation for the model with limited context length.
In Section 5 and Appendix A of our paper, we added results of the text-structured baseline and its implementation details.
Table 1. Comparison with text baseline on Qwen3-Omni-30B (Context length limit: 65,536)
| SAVED-Bench | SAVVY-Bench | Additional tokens | |
|---|---|---|---|
| Baseline | 37.7 | 48.1 | - |
| Text-structured baseline | 35.0 | 46.6 | 23,140 |
| FloorSAV | 43.2 | 49.6 | 15,405 |
Table 2. Comparison with text baseline on Gemini-2.5/3.6-Flash (Context length limit: 1,048,576)
| SAVED-Bench | SAVVY-Bench | Additional tokens | |
|---|---|---|---|
| Baseline | 58.3 | 45.7 | - |
| Text-structured baseline | 63.2 | 47.6 | 97,320 |
| FloorSAV | 65.3 | 50.9 | 7,829 |
(Additional tokens refer to required token counts to represent spatial information provided on the floormap and text-structured baseline, excluding egocentric video and audio)
Author Response to Reviewer xaQ4 (2/3)
W1-2 & Q2: Request to evaluate FloorSAV under the standard audiovisual benchmark setting, using egocentric video/audio and the question, to demonstrate its benefits with inference-time 3D reconstruction.
Although SAVVY-Bench and SAVED-Bench allow the use of all data provided by the AEA dataset, your comments offer an opportunity to evaluate FloorSAV under more restricted input conditions. Therefore, we evaluate FloorSAV with geometry and camera poses reconstructed from egocentric video along with object landmarks. As some recent works use binaural [1] or first-order ambisonic (FOA) audio [2], we incorporate acoustic estimates. We construct and evaluate VGGT-Ω FloorSAV on both SAVVY-Bench and SAVED-Bench using the Gemini-Flash series (details of the construction procedure are provided below).
VGGT-Ω FloorSAV shows competitive performance with the original FloorSAV, outperforms a baseline provided with the same acoustic estimates, and even surpasses the text-structured baseline that uses point clouds and camera poses from the AEA dataset. These results demonstrate that FloorSAV remains effective when dataset-provided geometry and camera pose are replaced with estimates reconstructed from egocentric video and suggest opportunities to extend its applicability.
In Section 5 of our paper, we added analysis and results of VGGT-Ω FloorSAV.
Table 3. Comprehensive comparison of VGGT-Ω FloorSAV with Gemini-Flash
| SAVED-Bench | SAVVY-Bench | |
|---|---|---|
| Baseline | 58.3 | 45.7 |
| Baseline with acoustic estimates | 61.3 | 43.0 |
| Text-structured baseline | 63.2 | 47.6 |
| VGGT-Ω FloorSAV | 64.2 | 50.3 |
| FloorSAV | 65.3 | 50.9 |
Floormap rendering procedure for VGGT-Ω FloorSAV
- We use VGGT-Ω [3] to estimate relative depth and camera poses from 128 frames sampled from each recording.
- To handle dynamic agents (humans) appearing in some frames, we use Grounding DINO [4] to detect human bodies, arms, and hands, and SAM2 [5] to generate human masks for each frame.
- We exclude human-masked regions from the depth maps and reconstruct 3D point clouds using the estimated camera poses and remaining depth values.
- Since acoustic estimates and spatial QAs require metric distances, we use UniDepth V2 [6] to calculate a metric scale for each scene. Specifically, we estimate metric depth for all frames and fit a single scene-level scalar that minimizes the gap between the metric depth and the scaled VGGT-Ω depth.
- Using the metrically scaled reconstruction and estimated camera poses, we generate floor plane candidates using RANSAC and refine them through least-squares fitting. Candidates are assessed considering geometry such as walls and tables, and the camera wearer's height.
- We project the 3D point clouds onto the estimated floor plane to obtain a floormap, then render the estimated camera poses, semantically grounded objects, and acoustic estimates.
In Appendix A of our paper, we provide a detailed procedure for 2D floormap rendering from RGB egocentric video, acoustic estimates, and object landmarks.
W2 & Q6 & Q7: Request for a detailed comparison with existing map strategies in spatial reasoning, and concern about the method’s novelty and whether its gains come from richer input information rather than a new reasoning method.
We agree that top-down maps are reported on spatial reasoning; However, compared to static cognitive maps [7] and bird’s-eye-view (BEV) maps with object grounding [8], FloorSAV renders an integrated floomap video including a dynamic agent that is synchronized with the egocentric video/aduio stream, allowing explicit representation of dynamic agents with various spatial information.
Furthermore, the comparison between FloorSAV and the text-structured baseline (see our responses to W1-1 & Q1 & Q3), and results of VGGT-Ω FloorSAV (see our responses to W1-2 & Q2), support the benefits of FloorSAV’s methodology.
In Sections 1 and 2 of our paper, we added more comments on comparisons with existing map strategies.
More information on the rebuttal
I raised several questions during the review because I find this work interesting. However, I recognize that addressing all of them may place a substantial burden on the rebuttal. I would therefore like to highlight my main concerns: Weaknesses 1 and 4, and Questions 1, 2, 3, and 7. Addressing other concerns would also be helpful, as they are mainly some explanations and do not require much extra experiments.
My primary concerns are the fairness of the point-cloud input setting and the benchmark comparisons. If input and benchmark setup is not fair the contribution of the work cannot be established. If the authors can adequately address these issues, I would be willing to reconsider my assessment and potentially raise my score.