- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EgoAudit, a framework for auditing the dependence of 3D vision-language models on viewpoint information. It compares absent, predicted, and ground-truth viewpoints and uses controlled pose interventions to examine egocentric spatial reasoning. The evaluation focuses on SQA3D, with additional transfer experiments.
Critical strengths
The distinction between inferring a viewpoint from text and using an externally supplied pose is useful for interpreting benchmark performance. The experiments are reasonably thorough, particularly the matched training and evaluation settings and controlled viewpoint perturbations, which strengthen the evidence beyond simply removing pose information at inference.
Critical weaknesses
-
My main concern is the limited new insight. Viewpoint localization and coordinate alignment have already been studied in works such as SIG3D and MSR3D. Although this paper provides a more systematic audit, the findings that accurate viewpoints help and incorrect headings hurt are largely expected. Moreover, the ground-truth viewpoint gains involve both improved localization and precomputed relative geometry, making their implications for spatial reasoning ability less clear. I would like to see a stronger explanation of what the audit reveals beyond these established observations.
-
The comparison with recent methods is incomplete. Table 1 omits Video-3D LLM (CVPR 2025), 3DGraphLLM (ICCV 2025), 3D-SLIM (CVPR 2026), and GPT4Scene (ICLR 2026), despite their results on the same benchmarks. Several are close to the reported performance, so their inclusion would give a more balanced picture of the model’s competitiveness. The SQA3D comparison also mixes data splits and viewpoint conditions. Although this is acknowledged, it limits the conclusions that can be drawn from the table.
Questions for authors
- Which finding most clearly goes beyond prior work on viewpoint localization and alignment, and how does it distinguish localization difficulties from limitations in spatial reasoning?
- Can the authors include the omitted recent baselines and clarify the comparison under matched data splits and viewpoint availability, particularly for SQA3D?
7 comments
Response to Reviewer eSES (Part 2/3)
W1 & Q1 (continued): Moreover, the ground-truth viewpoint gains involve both improved localization and precomputed relative geometry, making their implications for spatial reasoning ability less clear. [...] how does it distinguish localization difficulties from limitations in spatial reasoning?
Response 1 (continued): Thank you for this observation. You are right that the GT viewpoint gives the model two things at once. Following your comment, we separate them, and we separate both from spatial reasoning, with four models from the submission (Table C1).
Three steps (Fig. 1): to answer "What is on my left?", a system has to (1) locate the user (position and heading), (2) transform the coordinates, so that each object is expressed relative to the user, and (3) reason over the result. A supplied GT viewpoint performs steps 1 and 2 outside the model.
Why the comparison separates them: the four models share the architecture, the data and the training schedule, and differ only in who performs steps 1 and 2. Each comparison with M_GT keeps one step exact and changes only who performs the other.
Table C1: Who performs each step, with SQA3D EM (%). Bracketed values indicate changes relative to M_GT.
| Model | Step 1: who locates the user | Step 2: who transforms the coordinates | EM |
|---|---|---|---|
| M_GT | GT annotation | outside the model | 64.21 |
| Raw-pose model (GT pose given as raw numbers) | GT annotation | the model | 60.38 (-3.83) |
| M_pred | learned localizer | outside the model | 60.44 (-3.77) |
| M_none | the model | the model | 58.69 (-5.52) |
Results:
- Localization is a bottleneck (M_pred row). The transformation is exact, yet 3.77 points are lost because the viewpoint is estimated. Inside this fixed model, replacing the predicted viewpoint by the GT one raises EM from 60.44 to 63.35 without retraining, and correcting the heading alone gives 62.93. The deficit of M_pred is therefore localization error, mostly in the heading, and not weaker reasoning.
- Precomputed geometry also matters (raw-pose row). The viewpoint is exact, yet 3.83 points are lost because the model has to transform the coordinates itself.
- Spatial reasoning (step 3) is not the limiting step. When steps 1 and 2 are supplied, the model answers 84.2% of two-option direction questions correctly, against 50.2% without a viewpoint (Table 22), and 88.0% of its answers follow a swapped viewpoint to the label that is correct for it.
The implication for spatial reasoning ability: the model reasons well from a viewpoint, and it obtains neither the viewpoint nor the user-relative scene by itself. An evaluation with annotated viewpoints credits it with all three steps. This is why we propose that egocentric results state the viewpoint source and any transformation applied before the model.
Revision: Table C1 is now part of the paper (a raw-pose row and a Frame column in Table 2, discussed in Sec. 5.2, with both training runs in the new Appendix J). Related Work compares with SIG3D, MSR3D and Real-3DQA, and the contribution list is rewritten around the findings of Part 1.
Presentation: Generally understandable, but requires substantial effort in places.
Response on presentation: Thank you for this feedback on readability. Following it, the revised paper (i) adds a notation table for the matched models and the four training and testing settings (Table 55), (ii) opens each protocol with the question it answers (Sec. 3.3) and each result with its takeaway (Sec. 5), and (iii) adds one table that maps each claim to its evidence (Table 56), so that the appendix results can be found from the main text.
Response to Reviewer eSES (Part 3/3)
W2 & Q2: The comparison with recent methods is incomplete. Can the authors include the omitted recent baselines and clarify the comparison under matched data splits and viewpoint availability, particularly for SQA3D?
Response 2: Thank you for listing these four methods. Following your suggestion, we include them in Table 1 of the revised paper with the values reported in their papers (Table C2), and we reorganize the SQA3D comparison by data split and by the viewpoint that each model receives (Table C3).
Table C2: Grounding and captioning on the validation splits. Best results are highlighted in bold.
| Method | LLM | ScanRefer [email protected] | Multi3DRefer [email protected] | Scan2Cap [email protected] |
|---|---|---|---|---|
| Chat-Scene | Vicuna-7B | 50.2 | 52.4 | 77.1 |
| Chat-Scene++ | Vicuna-7B | 53.4 | 58.7 | 83.0 |
| Video-3D LLM [C5] | LLaVA-Video-7B | 51.7 | 52.7 | 83.8 |
| 3DGraphLLM [C6] | Llama3-8B | 56.6 | 59.9 | 81.0 |
| GPT4Scene-HDM [C7] | Qwen2-VL-7B | 57.0 | 59.8 | 86.3 |
| 3D-SLIM on Chat-Scene [C8] | Vicuna-7B | 54.1 | 58.7 | 84.2 |
| 3D-SLIM on 3DGraphLLM [C8] | Llama3-8B | 57.7 | 62.0 | 82.2 |
| Ours | Qwen3-VL-8B | 58.19 | 62.95 | 87.40 |
Results: With the four methods included, our model still obtains the highest score on each of the three metrics, by 0.5 to 1.1 points over the strongest entry. This is what Table 1 is meant to show: the audit concerns a competitive model and not a weak baseline.
Table C3: SQA3D EM (%) by data split and by the viewpoint given to the model. Every model also reads the situation description as text. "-" means not reported.
| Method | Viewpoint given to the model | val | test |
|---|---|---|---|
| Chat-Scene | none | 53.2 | 54.6 |
| Chat-Scene++ | none | - | 55.8 |
| 3DGraphLLM | none | 55.9 | - |
| 3D-SLIM on Chat-Scene | none | 55.9 | 55.5 |
| 3D-SLIM on 3DGraphLLM | none | 56.8 | 56.1 |
| Video-3D LLM | none | - | 58.6 |
| GPT4Scene-HDM | none | - | 59.4 |
| Ours, M_none | none | 58.69 | - |
| Ours, M_pred | predicted by a localizer | 60.44 | - |
| Ours, M_GT | annotated (GT) | 64.21 | - |
Validation values of Chat-Scene, 3DGraphLLM and 3D-SLIM are those listed in the 3D-SLIM paper. For GPT4Scene-HDM, the full SQA3D table of its paper (test set) lists 59.4 and its main QA table lists 60.6.
Results:
- Matched comparison (validation split, no viewpoint given): M_none obtains 58.69, against 53.2 to 56.8 for the object-centric methods that report this split.
- Not comparable: the 64.21 of M_GT uses the annotated viewpoint and serves only as an upper reference. No published row in the table receives it.
- Why the reorganized table matters: one architecture moves from 58.69 to 64.21 depending only on where its viewpoint comes from, a range as wide as the spread among the published methods. A table that mixes viewpoint sources ranks annotations as much as models, which is why we propose that every egocentric score state its viewpoint source.
Thank you again for your time and constructive suggestions.
References
- [C1] Situational Awareness Matters in 3D Vision Language Reasoning (SIG3D). CVPR 2024.
- [C2] Multi-modal Situated Reasoning in 3D Scenes (MSR3D). NeurIPS 2024.
- [C3] Do 3D Large Language Models Really Understand 3D Spatial Relationships? (Real-3DQA). ICLR 2026.
- [C4] Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models. arXiv 2026.
- [C5] Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding. CVPR 2025.
- [C6] 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding. ICCV 2025.
- [C7] GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models. ICLR 2026.
- [C8] Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding (3D-SLIM). CVPR 2026.
Response to Reviewer eSES (Part 1/3)
Dear Reviewer
eSES,Thanks for your valuable reviews! We address your concerns as follows. Please let us know if anything remains unclear.
Response 1: Thank you for this feedback. We have revised the Related Work section and the contribution list to separate what prior work establishes from what is new, and we refine our statement to:
Although the topic is shared, the works differ in goals and controls:
Notation: M_none, M_pred and M_GT are one architecture trained and tested with no viewpoint, with the viewpoint predicted by a learned localizer (Loc3R-VLM [C4]), and with the annotated (GT) viewpoint. EM is the exact-match accuracy (%) on SQA3D.
The finding that most clearly goes beyond prior work: with the architecture, the data and the training fixed, EM is 64.21 when the benchmark supplies the GT viewpoint, 60.44 when a learned localizer supplies it, and 58.69 when nothing does. Half to two thirds of the gain disappears as soon as the viewpoint is not annotated. Why this is new: prior work reports that a viewpoint helps; it does not measure how much of the resulting score belongs to the annotation. A score obtained with annotated viewpoints measures how well a model uses a viewpoint, not whether it can take the user's perspective.
Beyond "accurate viewpoints help and incorrect headings hurt": we agree that these two facts are expected, and they serve as sanity checks. The audit adds results that are not:
In short, prior works show that viewpoint information helps and how to encode it; EgoAudit shows what the resulting scores measure. Part 2 answers the second half of your question: how localization is told apart from spatial reasoning.