acceptodds
Under review as a conference paper at ICLR 2027

EgoAudit: Auditing Egocentric Spatial Reasoning in 3D Vision-Language Models

Abstract

3D vision-language models (3D VLMs) have made rapid progress in grounding objects and answering questions in 3D indoor scenes. Many queries that people pose are egocentric: "Which lamp is on my left?" has a definite answer only once the model knows where the speaker stands and which way they face. Existing benchmarks and models often provide this viewpoint as an annotated pose, so a high score may reflect how well a model uses a given viewpoint rather than whether it can infer the viewpoint from the description. We present EgoAudit, a framework for auditing egocentric spatial reasoning in 3D VLMs. EgoAudit equips a 3D VLM with a viewpoint-conditioned object encoding, through which the viewpoint can be removed, perturbed, or replaced while the scene, the query, and the model stay fixed, and it evaluates the model with three protocols: viewpoint intervention, matched training with and without the viewpoint, and substitution of the viewpoint source. Extensive experiments on public 3D grounding and 3D visual question answering benchmarks show that the audited model, a strong 3D VLM, follows a supplied viewpoint closely but does not infer it: much of its advantage on egocentric queries is lost when the annotated viewpoint is withheld or replaced by the estimate of a learned localizer. These results indicate that scores obtained with annotated viewpoints measure viewpoint use rather than perspective taking, and we propose a reporting protocol for egocentric 3D evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasoning, 3D Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eSESOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EgoAudit, a framework for auditing the dependence of 3D vision-language models on viewpoint information. It compares absent, predicted, and ground-truth viewpoints and uses controlled pose interventions to examine egocentric spatial reasoning. The evaluation focuses on SQA3D, with additional transfer experiments.

Critical strengths

The distinction between inferring a viewpoint from text and using an externally supplied pose is useful for interpreting benchmark performance. The experiments are reasonably thorough, particularly the matched training and evaluation settings and controlled viewpoint perturbations, which strengthen the evidence beyond simply removing pose information at inference.

Critical weaknesses
  1. My main concern is the limited new insight. Viewpoint localization and coordinate alignment have already been studied in works such as SIG3D and MSR3D. Although this paper provides a more systematic audit, the findings that accurate viewpoints help and incorrect headings hurt are largely expected. Moreover, the ground-truth viewpoint gains involve both improved localization and precomputed relative geometry, making their implications for spatial reasoning ability less clear. I would like to see a stronger explanation of what the audit reveals beyond these established observations.

  2. The comparison with recent methods is incomplete. Table 1 omits Video-3D LLM (CVPR 2025), 3DGraphLLM (ICCV 2025), 3D-SLIM (CVPR 2026), and GPT4Scene (ICLR 2026), despite their results on the same benchmarks. Several are close to the reported performance, so their inclusion would give a more balanced picture of the model’s competitiveness. The SQA3D comparison also mixes data splits and viewpoint conditions. Although this is acknowledged, it limits the conclusions that can be drawn from the table.

Questions for authors
  1. Which finding most clearly goes beyond prior work on viewpoint localization and alignment, and how does it distinguish localization difficulties from limitations in spatial reasoning?
  2. Can the authors include the omitted recent baselines and clarify the comparison under matched data splits and viewpoint availability, particularly for SQA3D?
7 comments
AuthorsOpenReview ↗

Response to Reviewer eSES (Part 1/3)

Dear Reviewer eSES,

Thanks for your valuable reviews! We address your concerns as follows. Please let us know if anything remains unclear.

W1 & Q1: My main concern is the limited new insight. Which finding most clearly goes beyond prior work on viewpoint localization and alignment?

Response 1: Thank you for this feedback. We have revised the Related Work section and the contribution list to separate what prior work establishes from what is new, and we refine our statement to:

  • Research problem: SIG3D [C1] and MSR3D [C2] study how to supply a viewpoint to a model: how to estimate it and how to encode it. EgoAudit studies what a score obtained with a supplied viewpoint measures.
  • Methodology: We take one trained 3D VLM and change only its viewpoint input (its value, its presence in training, its source), so that the part of the performance that belongs to the model can be told apart from the part that belongs to the annotation.

Although the topic is shared, the works differ in goals and controls:

  • SIG3D [C1] builds a situation estimator and re-encodes the scene tokens with its estimate. It does not replace the viewpoint of a trained model by a different one, and it has a single ablation row with a predicted situation.
  • MSR3D [C2] compares four ways to encode a GT pose and finds that transforming the scene works best. The GT pose is present in all of them, so the comparison cannot say how much of a score depends on the pose being annotated.
  • Real-3DQA [C3] rotates the viewpoint by rewriting the description and supplies no viewpoint, so it cannot tell a failure to read the viewpoint from a failure to reason from it.

Notation: M_none, M_pred and M_GT are one architecture trained and tested with no viewpoint, with the viewpoint predicted by a learned localizer (Loc3R-VLM [C4]), and with the annotated (GT) viewpoint. EM is the exact-match accuracy (%) on SQA3D.

The finding that most clearly goes beyond prior work: with the architecture, the data and the training fixed, EM is 64.21 when the benchmark supplies the GT viewpoint, 60.44 when a learned localizer supplies it, and 58.69 when nothing does. Half to two thirds of the gain disappears as soon as the viewpoint is not annotated. Why this is new: prior work reports that a viewpoint helps; it does not measure how much of the resulting score belongs to the annotation. A score obtained with annotated viewpoints measures how well a model uses a viewpoint, not whether it can take the user's perspective.

Beyond "accurate viewpoints help and incorrect headings hurt": we agree that these two facts are expected, and they serve as sanity checks. The audit adds results that are not:

  • A wrong viewpoint is worse than none. A reversed heading costs 17.39 points and leaves the model 11.8 points below M_none. The model does not fall back on the description when the supplied viewpoint is wrong.
  • The answers are decided by the supplied viewpoint. With the scene, the text and the weights fixed, 88.0% of the answers move to the label that is correct for a swapped viewpoint. Prior work shows that a viewpoint raises the average score, not that the model answers from it.
  • Heading matters, position hardly does. A 1 m position error costs 0.86 points, and correcting the heading of a predicted viewpoint recovers most of its loss.
  • Training with GT viewpoints does not teach a model to find them. Deprived of the viewpoint, the GT-trained model falls to the level of M_none or below (57.68 vs. 58.66).

In short, prior works show that viewpoint information helps and how to encode it; EgoAudit shows what the resulting scores measure. Part 2 answers the second half of your question: how localization is told apart from spatial reasoning.

AuthorsOpenReview ↗

Response to Reviewer eSES (Part 2/3)

W1 & Q1 (continued): Moreover, the ground-truth viewpoint gains involve both improved localization and precomputed relative geometry, making their implications for spatial reasoning ability less clear. [...] how does it distinguish localization difficulties from limitations in spatial reasoning?

Response 1 (continued): Thank you for this observation. You are right that the GT viewpoint gives the model two things at once. Following your comment, we separate them, and we separate both from spatial reasoning, with four models from the submission (Table C1).

Three steps (Fig. 1): to answer "What is on my left?", a system has to (1) locate the user (position and heading), (2) transform the coordinates, so that each object is expressed relative to the user, and (3) reason over the result. A supplied GT viewpoint performs steps 1 and 2 outside the model.

Why the comparison separates them: the four models share the architecture, the data and the training schedule, and differ only in who performs steps 1 and 2. Each comparison with M_GT keeps one step exact and changes only who performs the other.

Table C1: Who performs each step, with SQA3D EM (%). Bracketed values indicate changes relative to M_GT.

ModelStep 1: who locates the userStep 2: who transforms the coordinatesEM
M_GTGT annotationoutside the model64.21
Raw-pose model (GT pose given as raw numbers)GT annotationthe model60.38 (-3.83)
M_predlearned localizeroutside the model60.44 (-3.77)
M_nonethe modelthe model58.69 (-5.52)

Results:

  • Localization is a bottleneck (M_pred row). The transformation is exact, yet 3.77 points are lost because the viewpoint is estimated. Inside this fixed model, replacing the predicted viewpoint by the GT one raises EM from 60.44 to 63.35 without retraining, and correcting the heading alone gives 62.93. The deficit of M_pred is therefore localization error, mostly in the heading, and not weaker reasoning.
  • Precomputed geometry also matters (raw-pose row). The viewpoint is exact, yet 3.83 points are lost because the model has to transform the coordinates itself.
  • Spatial reasoning (step 3) is not the limiting step. When steps 1 and 2 are supplied, the model answers 84.2% of two-option direction questions correctly, against 50.2% without a viewpoint (Table 22), and 88.0% of its answers follow a swapped viewpoint to the label that is correct for it.

The implication for spatial reasoning ability: the model reasons well from a viewpoint, and it obtains neither the viewpoint nor the user-relative scene by itself. An evaluation with annotated viewpoints credits it with all three steps. This is why we propose that egocentric results state the viewpoint source and any transformation applied before the model.

Revision: Table C1 is now part of the paper (a raw-pose row and a Frame column in Table 2, discussed in Sec. 5.2, with both training runs in the new Appendix J). Related Work compares with SIG3D, MSR3D and Real-3DQA, and the contribution list is rewritten around the findings of Part 1.

Presentation: Generally understandable, but requires substantial effort in places.

Response on presentation: Thank you for this feedback on readability. Following it, the revised paper (i) adds a notation table for the matched models and the four training and testing settings (Table 55), (ii) opens each protocol with the question it answers (Sec. 3.3) and each result with its takeaway (Sec. 5), and (iii) adds one table that maps each claim to its evidence (Table 56), so that the appendix results can be found from the main text.

AuthorsOpenReview ↗

Response to Reviewer eSES (Part 3/3)

W2 & Q2: The comparison with recent methods is incomplete. Can the authors include the omitted recent baselines and clarify the comparison under matched data splits and viewpoint availability, particularly for SQA3D?

Response 2: Thank you for listing these four methods. Following your suggestion, we include them in Table 1 of the revised paper with the values reported in their papers (Table C2), and we reorganize the SQA3D comparison by data split and by the viewpoint that each model receives (Table C3).

Table C2: Grounding and captioning on the validation splits. Best results are highlighted in bold.

MethodLLMScanRefer [email protected]Multi3DRefer [email protected]Scan2Cap [email protected]
Chat-SceneVicuna-7B50.252.477.1
Chat-Scene++Vicuna-7B53.458.783.0
Video-3D LLM [C5]LLaVA-Video-7B51.752.783.8
3DGraphLLM [C6]Llama3-8B56.659.981.0
GPT4Scene-HDM [C7]Qwen2-VL-7B57.059.886.3
3D-SLIM on Chat-Scene [C8]Vicuna-7B54.158.784.2
3D-SLIM on 3DGraphLLM [C8]Llama3-8B57.762.082.2
OursQwen3-VL-8B58.1962.9587.40

Results: With the four methods included, our model still obtains the highest score on each of the three metrics, by 0.5 to 1.1 points over the strongest entry. This is what Table 1 is meant to show: the audit concerns a competitive model and not a weak baseline.

Table C3: SQA3D EM (%) by data split and by the viewpoint given to the model. Every model also reads the situation description as text. "-" means not reported.

MethodViewpoint given to the modelvaltest
Chat-Scenenone53.254.6
Chat-Scene++none-55.8
3DGraphLLMnone55.9-
3D-SLIM on Chat-Scenenone55.955.5
3D-SLIM on 3DGraphLLMnone56.856.1
Video-3D LLMnone-58.6
GPT4Scene-HDMnone-59.4
Ours, M_nonenone58.69-
Ours, M_predpredicted by a localizer60.44-
Ours, M_GTannotated (GT)64.21-

Validation values of Chat-Scene, 3DGraphLLM and 3D-SLIM are those listed in the 3D-SLIM paper. For GPT4Scene-HDM, the full SQA3D table of its paper (test set) lists 59.4 and its main QA table lists 60.6.

Results:

  • Matched comparison (validation split, no viewpoint given): M_none obtains 58.69, against 53.2 to 56.8 for the object-centric methods that report this split.
  • Not comparable: the 64.21 of M_GT uses the annotated viewpoint and serves only as an upper reference. No published row in the table receives it.
  • Why the reorganized table matters: one architecture moves from 58.69 to 64.21 depending only on where its viewpoint comes from, a range as wide as the spread among the published methods. A table that mixes viewpoint sources ranks annotations as much as models, which is why we propose that every egocentric score state its viewpoint source.

Thank you again for your time and constructive suggestions.

References

  • [C1] Situational Awareness Matters in 3D Vision Language Reasoning (SIG3D). CVPR 2024.
  • [C2] Multi-modal Situated Reasoning in 3D Scenes (MSR3D). NeurIPS 2024.
  • [C3] Do 3D Large Language Models Really Understand 3D Spatial Relationships? (Real-3DQA). ICLR 2026.
  • [C4] Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models. arXiv 2026.
  • [C5] Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding. CVPR 2025.
  • [C6] 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding. ICCV 2025.
  • [C7] GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models. ICLR 2026.
  • [C8] Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding (3D-SLIM). CVPR 2026.
Reviewer_ftyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoAudit, an evaluation framework designed to distinguish viewpoint inference from viewpoint use in egocentric spatial reasoning. The authors introduce a viewpoint-conditioned object encoding and three auditing protocols: viewpoint intervention, matched training with and without viewpoint information, and evaluation using different viewpoint sources (none, predicted, and ground truth). Experiments on public 3D grounding and situated 3D VQA benchmarks show that the evaluated 3D VLM effectively uses supplied viewpoint information but struggles to infer it independently. In particular, performance drops when ground-truth viewpoints are removed or replaced with predicted ones. These findings highlight potential limitations in existing egocentric spatial reasoning evaluations and motivate more explicit reporting of viewpoint sources and inference capabilities.

Critical strengths
  1. Important and well-motivated evaluation perspective. The paper highlights an important distinction between viewpoint inference and viewpoint use in egocentric 3D spatial reasoning. It raises a meaningful concern that strong performance with ground-truth viewpoints does not necessarily reflect a model's ability to infer the observer's perspective. This distinction provides valuable insights into how existing 3D VLM benchmarks should be interpreted.

  2. Systematic and well-controlled experimental analysis. The authors design three complementary evaluation protocols, including viewpoint intervention, matched training, and viewpoint-source comparison, to investigate the role of viewpoint information. The experiments provide convincing evidence that the evaluated model strongly relies on supplied viewpoints, while predicted viewpoints recover only part of the performance gain. The additional cross-dataset evaluations further strengthen the empirical findings.

Critical weaknesses
  1. Insufficient evidence to isolate viewpoint inference limitations. While the experiments demonstrate that the evaluated model benefits substantially from ground-truth viewpoints, the performance gap between models with and without viewpoint information does not necessarily establish that the model cannot infer viewpoints from scene descriptions. In particular, the ground-truth viewpoint is converted into object-relative geometric features outside the model, meaning that the GT setting also benefits from explicit coordinate transformations. This makes it difficult to disentangle limitations in viewpoint inference from limitations in geometric representation and reasoning.

  2. Limited generalizability of the findings. The main experiments are conducted using a specific object-centric 3D VLM architecture with viewpoint-conditioned geometric encoding. Although the authors provide additional evaluations with a Vicuna-based model and across datasets, it remains unclear whether the observed limitations are representative of broader 3D VLM architectures, particularly models that jointly learn viewpoint estimation and spatial reasoning. This limits the strength of the paper's broader conclusions about egocentric spatial reasoning capabilities.

Questions for authors
  1. Can the authors distinguish failures of viewpoint inference from failures of coordinate transformation? The paper concludes that the audited model uses a supplied viewpoint but does not infer it. However, the GT-viewpoint setting provides object-relative geometry computed outside the model, effectively bypassing both viewpoint localization and coordinate transformation. The raw-pose ablation (60.38% vs. 64.21%) further suggests that coordinate transformation itself is a significant difficulty. Could the authors clarify which existing evidence isolates viewpoint inference as the primary bottleneck? A direct evaluation of viewpoint localization from scene and situation descriptions, or a controlled decomposition of localization and coordinate-transformation errors, would substantially affect my assessment.

  2. Does viewpoint following establish a lack of perspective-taking, or primarily a reliance on the supplied geometric representation? The 88.0% viewpoint-following result is compelling, but the intervention deliberately creates a conflict between the unchanged situation description and the injected viewpoint-relative geometry. Since the model is explicitly trained to consume this geometry, following the swapped viewpoint may be an expected behavior rather than direct evidence that it cannot infer the viewpoint from language. How do the authors distinguish these interpretations? Stronger evidence that the model fails to recover the correct perspective even when the viewpoint must be inferred from consistent scene-language inputs would help substantiate the central claim.

  3. How broadly do the conclusions apply beyond the proposed viewpoint-conditioned architecture? The main results rely on an object-centric 3D VLM with an explicitly designed geometry pathway. Although the paper includes Vicuna-based replication and cross-dataset evaluations, could the authors clarify which findings remain consistent across substantially different viewpoint representations or architectures? In particular, does the evidence support a general limitation of existing 3D VLMs, or primarily characterize models trained to rely on externally constructed viewpoint-relative features? Clarifying this scope would materially affect my assessment of the paper's broader contribution.

I consider Questions 1 and 2 particularly important, as they directly concern the validity of the paper's central claim. In my view, distinguishing viewpoint inference from coordinate transformation and reliance on externally provided geometric features is essential to establishing the significance of this work. I would appreciate a careful and evidence-based response from the authors to these questions. The authors' responses will substantially influence my final recommendation regarding acceptance.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ftyW (Part 1/4)

Dear Reviewer ftyW,

Thanks for your careful and constructive comments! We address your concerns and questions as follows. Please kindly let us know if you have further concerns.

W1 & Q1: Can the authors distinguish failures of viewpoint inference from failures of coordinate transformation?

Response 1: Thank you for this insightful question, which goes to the core of our claim. You are right: the GT viewpoint does two jobs at once, so the gap between the models with and without it cannot tell the two apart. Following your suggestion, we provide a controlled decomposition of localization and coordinate-transformation errors (Table B1). It uses four models that are already in the submission but were reported in separate places (Table 2, App. B.1 and App. D.3).

The two steps: before a model can answer "What is on my left?", it needs (1) viewpoint localization: where the user stands and which way they face, and (2) coordinate transformation (the "change frame" step of Fig. 1): where each object lies relative to the user. Only then can it reason about "left".

Experimental setup: The four models share the architecture, the training data and the schedule. They differ only in who performs the two steps:

  • M_GT: both steps are done outside the model. It receives the annotated (GT) viewpoint, already converted into each object's distance and direction relative to the user.
  • Raw-pose model: step 1 outside, step 2 by the model. It receives the same GT viewpoint as raw numbers (position and heading) and has to relate each object to it by itself.
  • M_pred: step 1 by a learned localizer, step 2 outside. Loc3R-VLM [B1] estimates the viewpoint from video frames and the situation description, and the estimate is converted exactly as for M_GT.
  • M_none: both steps are left to the model. It receives no viewpoint, only the scene and the description.

Why this separates the two steps: each comparison with M_GT keeps one step exact and changes only who performs the other. The raw-pose model has a perfect viewpoint, so its loss can only come from the transformation. M_pred has an exact transformation, so its loss can only come from the viewpoint estimate.

Table B1: Controlled decomposition on SQA3D. EM (%) is the exact-match accuracy on the 3,261 validation questions, for two independent training runs. Bracketed values indicate changes relative to M_GT.

ModelWho locates the userWho transforms the coordinatesRun 1Run 2
M_GTGT annotationoutside the model64.2164.58
Raw-poseGT annotationthe model60.38 (-3.83)59.67 (-4.91)
M_predlearned localizeroutside the model60.44 (-3.77)61.24 (-3.34)
M_nonethe modelthe model58.69 (-5.52)57.62 (-6.96)

Results:

  • Coordinate transformation is a real difficulty, as you suspected. With a perfect viewpoint, leaving the transformation to the model loses 3.83 and 4.91 points.
  • Viewpoint localization is a difficulty of the same size. With an exact transformation, replacing the GT viewpoint by the estimate of a localizer trained for this very task loses 3.77 and 3.34 points.
  • Supplying one step alone is not enough. The raw GT pose recovers about 30% of the gap between M_none and M_GT, and the predicted viewpoint 31.7% and 52.0%. The two losses do not add up to the total because the steps are chained: a correct answer needs both.

The decomposition makes our message more specific: the annotated viewpoint gives the model two things, and the model supplies neither of them well by itself. The revised paper states the two steps separately (Table 2 and Sec. 5.2; see Part 2 for the revised claim).

AuthorsOpenReview ↗

Response to Reviewer ftyW (Part 2/4)

Q1 (continued): Could the authors clarify which existing evidence isolates viewpoint inference as the primary bottleneck?

Response 1 (continued): Thank you for spelling out what evidence would help. Part 1 gives the controlled decomposition. Here we point to the evidence that isolates viewpoint inference, answer whether it is the primary bottleneck, and report direct evaluations of localization.

Which evidence isolates viewpoint inference: the comparisons in which the coordinate transformation is identical on both sides, so that it cannot explain the difference.

  • Matched models: M_pred and M_GT use the same external transformation and differ only in where the viewpoint comes from: 60.44 vs. 64.21 (run 2: 61.24 vs. 64.58).
  • Same weights, viewpoint corrected: giving the trained M_pred the GT viewpoint at test time raises its EM from 60.44 to 63.35 (run 2: 61.24 to 64.12) without any retraining (App. D.1, Table 12).
  • Same weights, viewpoint degraded: giving the GT-trained model the localizer's estimate lowers its EM from 64.21 to 60.04 (App. B).
  • Which part of the viewpoint: inside the fixed M_pred, the GT heading alone gives 62.93 and the GT position alone 61.02. The heading is the part of the estimate that costs the most.

Is viewpoint inference the primary bottleneck? In size, the two steps are comparable (Table B1), and the revised paper says so. In kind, they differ, and this is why the paper centers on inference:

  • The transformation is a fixed formula (Eq. 2). Once a viewpoint is known, a few lines of geometry compute it exactly outside the model, as our encoding does. This difficulty can be removed by design, at no annotation cost.
  • The viewpoint has no formula. It has to be estimated from the scene and the description, and 3.3 to 3.8 points remain lost even with a dedicated localizer trained with pose supervision on SQA3D.

Viewpoint inference is therefore the step that an annotated viewpoint hides and that engineering does not remove. In a deployed system, where no annotation exists, it is what separates the realistic score (M_pred) from the annotated one (M_GT).

Direct evaluations of viewpoint localization: this step has been measured directly on SQA3D, from the scene and the situation description. Table B2 lists the share of situations in which the estimate is close to the annotation, as reported in the Loc3R-VLM paper [B1] (its Table 1, test split).

Table B2: Published localization accuracy on SQA3D (%), selected rows.

MethodPosition within 0.5 mHeading within 30°
Random guess7.216.9
SQA3D baseline9.516.5
3D-VisTA11.724.2
SIG3D [B2]27.442.5
Loc3R-VLM [B1] (the viewpoint source of M_pred)42.663.0

The SQA3D baseline, which is trained with a localization objective, stays close to a random guess, and the best dedicated localizer still misses the heading by more than 30° in over a third of the situations. A wrong heading is costly: rotating it by 90° costs 10.06 points and reversing it 17.39 (Table 3). This agrees with Table B1 and with the heading result above: the viewpoint is hard to infer, and the heading is the hard part. The audited 3D VLM answers questions and does not output a pose, so its perspective taking is tested through its answers (Part 3, Table B3).

A model without a viewpoint: Table B1 measures each step when the other one is supplied. M_none performs both, so its answers alone do not say how its gap divides between them. Inside our GT-trained model run without a viewpoint, a linear probe reads where the objects are in scene coordinates (0.44 m error) but cannot reliably read where they are relative to the user at any of the 36 decoder blocks (App. G): the scene is encoded, the user's frame is not readable.

Revision: The decomposition is now in the main text (a raw-pose row and a Frame column in Table 2, discussed in Sec. 5.2); both runs of Table B1 and the accuracies of Table B2 are in the new Appendix J. Secs. 1, 3.3, 5.1 and 6 now state the two steps separately. The central claim now reads: "The audited model follows a supplied viewpoint closely but does not take the described perspective on its own. [...] Without the annotation it loses its advantage, which lies on viewpoint-dependent queries; given the pose in raw form it recovers under a third of that advantage; and with the estimate of a learned localizer, a third to a half."

AuthorsOpenReview ↗

Response to Reviewer ftyW (Part 3/4)

Q2: Does viewpoint following establish a lack of perspective-taking, or primarily a reliance on the supplied geometric representation?

Response 2: Thank you for drawing this distinction and we agree with your reading. The 88.0% result comes from a test that sets the description against the supplied viewpoint on purpose, and the model was trained to read that viewpoint. It shows reliance on the supplied viewpoint, the "uses" half of our claim. By itself it does not show that the model cannot infer the viewpoint from language.

The other half rests on tests of the kind you ask for: the scene, the description and the question are consistent, no viewpoint is supplied, and the perspective has to come from the description. Following your suggestion, we gather these results, reported in Sec. 5.1 and Appendices D.5, E and H, in Table B3.

Experimental setup:

  • Model: M_none is trained and tested with the scene, the description and the question only. It has no supplied geometry to rely on, and training gives it every reason to use the description.
  • SQA3D by question type: human-reviewed labels split the questions into 2,892 viewpoint-dependent ones, whose answer needs the user's position or heading (e.g. "What is on my left?"), and 308 viewpoint-independent ones. Why: a model that recovers the perspective from the description keeps up with M_GT on the first group, whereas a model that is simply weaker at QA falls behind on both.
  • Described-viewpoint pairs: in each of 14 reviewed scenes, two situations with their original descriptions receive the same two-option direction question about the same object, chosen so that the correct answer differs between them. Why: both answers are right only if the model takes each described perspective; ignoring the description gives the same answer twice, and guessing gives about 3.5/14. The options are also listed in reverse order, to check whether the model answers by option position.

Table B3: Consistent inputs without a supplied viewpoint (M_none), against the same questions answered by the GT-trained model with the GT viewpoint. Two independent training runs each.

TestM_noneWith GT viewpointSource
SQA3D, 2,892 viewpoint-dependent questions (EM, %)58.78, 57.8164.87, 65.39Table 20
SQA3D, 308 viewpoint-independent questions (EM, %)61.36, 59.4261.04, 60.39Table 20
Described-viewpoint pairs, both answers correct3/14, 1/1412/14, 11/14Table 47
Same pairs, options in reverse order0/14, 0/1412/14, 10/14Table 48

Results:

  • The deficit appears exactly where the perspective is needed. M_none is 6.09 and 7.57 points below on viewpoint-dependent questions and level on the others.
  • From the description alone, the perspective is not recovered. On the pairs, M_none is at or below guessing level, and at 0/14 once the options are reversed: each of the two models then picks the first-listed option in all 56 of its answers, so the answer follows the option order, not the description. The same holds with object identifiers as references: 7/14 and 0/14 fall to 1/14 and 0/14 after reversal. With the GT viewpoint the same pairs are answered correctly in 12/14 and 11/14 cases, so the questions themselves are answerable.

Could the model hold its own estimate, which the supplied geometry merely overrides? If so, performance should recover where nothing overrides it. It does not recover in M_none, which was trained and tested without the geometry. Nor does it recover in the GT-trained model once the viewpoint is taken away: on two-option direction questions its accuracy falls from 84.2% to 50.2%, which is chance (Table 22), and on SQA3D it falls to the level of M_none or below in all three runs (57.68 vs. 58.66, 57.04 vs. 57.62, 58.39 vs. 58.39; Table 14). Reliance on a supplied viewpoint and the inability to obtain one are thus both supported, each by its own experiments.

Boundary: these tests assess perspective taking as a whole; Response 1 covers how the failure divides between the two steps.

Revision: Sec. 5.2 now presents the 88.0% following rate strictly as evidence of use, and a new paragraph, "Without a supplied viewpoint, the described perspective is not recovered", collects the evidence of Table B3.

Reviewer_Uv5xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoAudit studies whether egocentric 3D VLM scores measure the use of a supplied viewpoint or the ability to infer that viewpoint. It combines fixed-weight pose interventions, matched training with and without pose inputs, and substitution of ground-truth versus predicted viewpoints within an object-centric 3D VLM. The contribution is a controlled auditing protocol and evidence about what existing evaluation scores measure, rather than a new state-of-the-art architecture.

Critical strengths
  1. Fixed-weight pose interventions provide a useful, more targeted measure than simply removing an input. With scene, text, weights and token layout fixed, the direction diagnostic reports 88.0% agreement with the substituted frame on 517 changed-label questions. This supports a clear statement about supplied-frame dependence, while keeping it distinct from accuracy under semantically coherent inputs.

  2. The combination of matched training, predicted/raw-pose comparisons and three training seeds in each of the Qwen and Vicuna model lines makes the measured effects more credible. The checkpoint lineage and conditional uncertainty analyses are also helpful. This constitutes a useful incremental auditing and reporting contribution.

Critical weaknesses
  1. The incremental knowledge over prior diagnostics is narrower than the broad framing suggests. SIG3D already removes/corrupts situation information and compares direct versus encoded oracle poses (Section 3; Table 6). MSR3D explicitly compares pose encodings with external coordinate transformation (Appendix C.1.3; Table 17), and Real-3DQA audits shortcut dependence and viewpoint rotation. EgoAudit adds a controlled combination of numeric-pose interventions and matched training, but the manuscript should identify the new conclusions this enables. In particular, little average benefit from injecting pose into a model whose pose-dependent inputs were always zero during training is partly expected from the design; replication strengthens reliability without making that outcome a new mechanism discovery.

  2. The central 'does not infer' interpretation is not established as strongly as supplied-frame use. Following contradictory external geometry can coexist with an implicit estimate that is overridden, and a GT-versus-no-pose performance gap does not establish complete absence of inference. The existing MLP checks and full-layer ridge scan are valuable but bounded diagnostics. Together with the focus on a modified object-centric model family, this limits the broader significance of the negative conclusion. I therefore lean toward weak acceptance for a useful but incremental audit, with the absence-of-inference claim narrowed.

Questions for authors
  1. What specific finding goes beyond SIG3D's situation/oracle controls, MSR3D's encoding/transformation comparisons, and Real-3DQA's viewpoint audit? Please distinguish the new controlled measurements from already established conclusions about pose provision and input dependence. The reference-use diagnostic in Seeing the Other Side (LREC 2026, https://aclanthology.org/2026.lrec-1.737/) is also relevant, although its multi-image task is different.

  2. Which existing results distinguish failed or partial viewpoint inference from prioritizing a supplied frame? Please separate the text-versus-geometry conflict probe, coherent natural-language pairs, and representation diagnostics, and qualify conclusions that remain unidentifiable. A focused synthesis of existing evidence is sufficient; no new solution architecture or exhaustive retraining is requested.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Uv5x (Part 1/3)

Dear Reviewer Uv5x,

Thanks for your valuable and detailed comments! We address your concerns as follows. Please let us know if anything remains unclear.

W1 & Q1: The incremental knowledge over prior diagnostics is narrower than the broad framing suggests.

Response 1: Thank you for the precise pointers to the sections and tables of these works, and for bringing Seeing the Other Side to our attention. We have added a dedicated comparison to the Related Work section (Sec. 2 of the revised paper), and we refine our statement to:

  • Research problem: We ask what a score obtained with an annotated viewpoint measures: the model's own perspective taking, or its use of a viewpoint that was handed to it.
  • Methodology: We audit one trained 3D VLM by controlling three things about its viewpoint input: its value at fixed weights, its presence in training, and its source (none, predicted by a learned localizer, or annotated).

Although these prior works are closely related, they differ in goals and controls:

  • SIG3D [A1]
    • Problem scope: How to estimate the situation and feed it to a model. Its pilot study shows that an early SQA3D baseline hardly reacts when the situation text is removed, when its supervision is corrupted, or when the GT situation is fed as an input token (its Sec. 3). Its Table 6 shows that a GT situation helps when it re-encodes the scene tokens and not when it is fed directly.
    • Controls: It does not replace the viewpoint of a trained model by a different one, and it does not separate position from heading for QA. One ablation row uses a predicted situation.
  • MSR3D [A2]
    • Problem scope: How to encode a given GT pose. It compares four separately trained models and finds that transforming the scene into the user's frame works best (its App. C.1.3 and Table 17).
    • Controls: The GT pose is present in every model. There is no predicted viewpoint, and training with and without the situation is not crossed with testing with and without it.
  • Real-3DQA [A3]
    • Problem scope: Whether 3D LLMs stay accurate once language shortcuts are filtered out and the described viewpoint is rotated.
    • Controls: The rotation is applied by rewriting the situation text, and no viewpoint is supplied to the model. An inconsistent answer therefore cannot be traced to reading the viewpoint from the text or to reasoning from it.
  • Seeing the Other Side [A4]
    • It removes the reference image from a multi-image viewpoint task and finds that strong VLMs still answer many questions correctly. It relates to our viewpoint-removal condition only: it works on 2D images and does not supply or alter a viewpoint.

In short, prior works show that viewpoint information helps and how to obtain and encode it; they do not measure what a score obtained with it stands for. Answering that question takes three controls that none of them applies: replacing the viewpoint of a trained model by a different one, crossing training and testing with and without the viewpoint, and comparing viewpoint sources within one architecture. Part 2 lists what these controls establish.

AuthorsOpenReview ↗

Response to Reviewer Uv5x (Part 2/3)

W1 & Q1 (continued)

Response 1 (continued): Thank you for asking us to name the new conclusions. Following your suggestion, we separate the conclusions that prior work already establishes from the new controlled measurements (Table A1).

Notation: M_none, M_pred and M_GT are one architecture trained and tested with no viewpoint, with the viewpoint predicted by a learned localizer (Loc3R-VLM [A5]), and with the annotated (GT) viewpoint. EM is the exact-match accuracy (%) on SQA3D.

Table A1: Conclusions established by prior work and measurements added by EgoAudit.

TopicEstablished by prior workNew in EgoAudit
Providing a viewpointIt helps when it is encoded well (SIG3D, MSR3D)The gain lies on viewpoint-dependent questions only (+7.30 vs. -1.62 points) and appears only in a model trained with the viewpoint
Changing the viewpoint of a trained modelNot tested88.0% of the answers move to the label that is correct for the new viewpoint. A reversed heading costs 17.39 points and leaves the model 11.8 points below M_none; a 1 m shift costs 0.86
Source of the viewpointOne ablation row with a predicted situation (SIG3D)Matched models: 58.69 (M_none), 60.44 (M_pred), 64.21 (M_GT). The localizer keeps 31.7% of the gain (52.0% in a second run), and the heading matters more than the position (62.93 vs. 61.02)
Encoding of a GT poseA transformed pose beats a directly fed one (SIG3D, MSR3D)With the same GT pose given as raw numbers, EM is 60.38: the coordinate transformation alone accounts for 3.83 of the 5.52 points
Rotated descriptions3D LLMs answer inconsistently (Real-3DQA)The failure is located: reasoning from a viewpoint works, obtaining the viewpoint does not

New conclusions that these measurements enable:

  1. A model that uses a viewpoint answers from whichever one it is given. Prior work shows that a supplied viewpoint raises the average score. It does not show that the answers are decided by it. With the scene, the text and the weights fixed, the answers follow a swapped viewpoint, and a wrong viewpoint is worse than none.
  2. The gain from an annotated viewpoint belongs largely to the annotation. Once the viewpoint has to be predicted, half to two thirds of the gain is lost, mostly through the heading. MSR3D keeps the GT pose in all its models and SIG3D has a single predicted row, so neither can quantify this.
  3. Scores with annotated viewpoints measure viewpoint use. Localization and coordinate transformation are done outside the model, and the model performs neither well by itself (58.69 without a viewpoint, 60.38 with the raw pose).

On the expected null effect: we agree with you. Little benefit from injecting a viewpoint into a model whose viewpoint entries were zero in training is expected by design. The revised paper labels it as a control (Secs. 3.3 and 5.2): it rules out that the gain of M_GT comes from the extra input alone. The informative cell of matched training is the opposite one. The model trained with GT viewpoints, once deprived of them, falls to the level of M_none or below (57.68 vs. 58.66 on SQA3D; 35.49 vs. 39.25 on MSQA-ScanNet). Why this matters: training with annotated viewpoints teaches a model to use them and adds no ability to recover them.

Revision: Related Work now includes the comparison of Part 1 and cites Seeing the Other Side. The contribution list names the three conclusions above, and Secs. 3.3 and 5.2 present the injection result as a control.

AuthorsOpenReview ↗

Response to Reviewer Uv5x (Part 3/3)

W2 & Q2: The central 'does not infer' interpretation is not established as strongly as supplied-frame use.

Response 2: Thank you for this constructive suggestion. We agree that supplied-frame use and "does not infer" rest on different kinds of evidence. Following your suggestion, we separate them (Table A2), state what remains unidentifiable, and refine the claim.

Why the kinds of evidence must be kept apart: in the conflict probe a viewpoint is supplied, so the model may simply give it priority. Such a test can show use but cannot show a lack of inference. Only tests without a supplied viewpoint can speak to inference, because there nothing takes priority.

Table A2: Existing evidence, by what the model receives. Pair counts are for two independent training runs.

EvidenceViewpoint suppliedResultWhat it shows
Conflict probe (Eq. 5): description unchanged, viewpoint swappedyes88.0% of the answers take the swapped viewpoint's label (14.1% with the original viewpoint)When text and viewpoint disagree, the supplied viewpoint decides
Coherent natural-language pairs (App. H): description and viewpoint change togetheryesBoth answers correct in 12/14 and 11/14 pairsUse holds without any conflict
The same pairs for M_noneno3/14 and 1/14; 0/14 in both runs with reversed answer optionsThe described perspective is not recovered
QA without a viewpoint (App. D.5)noM_none is 6.09 points below M_GT on the 2,892 viewpoint-dependent questions and level on the 308 othersThe deficit is specific to questions that need the viewpoint
Representation diagnostics (App. G)noScene-frame object positions are readable from hidden states (0.44 m error); user-relative positions are not (0 of 108 readouts beat a constant predictor)The scene is encoded; the user's frame is not readable

Can an implicit estimate coexist with the supplied frame and be overridden? In the conflict probe alone, yes; this test cannot exclude it. The rows without a supplied viewpoint speak to it: if the model formed a usable estimate from the description, it should surface when nothing overrides it. It does not. The GT-trained model run without a viewpoint answers two-option direction questions at 50.2% (84.2% with it; Table 22) and drops to the level of M_none on SQA3D (57.68 vs. 58.66). M_none, which has no geometry to defer to, shows the same deficit.

What is identified and what is not:

  1. Identified: supplied-frame use, its dependence on training (Table 4), and the gap between viewpoint sources (Table 2).
  2. Supported: without a supplied viewpoint, the model does not arrive at the described perspective (rows 3 to 5). The readouts are supporting evidence, since a failed readout does not prove absence.
  3. Not identifiable: a complete absence of inference, and how the deficit of a model without a viewpoint divides between localization and coordinate transformation.

Refined claim: following your recommendation, "does not infer it" now reads "does not take the described perspective on its own" throughout the revised paper, with the tests named and the two limits of item 3 stated (Sec. 5.2).

On the model family: what an annotated-viewpoint score measures does not depend on the architecture, since localization and transformation then happen outside any model. The findings about the model are stated for the audited object-centric family.

Thank you again for your thoughtful review.

References

  • [A1] Situational Awareness Matters in 3D Vision Language Reasoning (SIG3D). CVPR 2024.
  • [A2] Multi-modal Situated Reasoning in 3D Scenes (MSR3D). NeurIPS 2024.
  • [A3] Do 3D Large Language Models Really Understand 3D Spatial Relationships? (Real-3DQA). ICLR 2026.
  • [A4] Seeing the Other Side: Diagnostic Tasks for Viewpoint Reasoning in Vision-Language Models. LREC 2026.
  • [A5] Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models. arXiv 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We appreciate all the reviewers for their insightful and constructive reviews of our manuscript. We are glad that the reviewers found that:

  • The distinction between inferring a viewpoint and using a supplied one is important, well motivated and useful for interpreting benchmark scores (Reviewer ftyW and Reviewer eSES).
  • The fixed-weight viewpoint interventions are a more targeted measure than simply removing an input (Reviewer Uv5x and Reviewer eSES).
  • The experimental analysis is systematic, well controlled and credible, with matched training, predicted and raw-pose comparisons, repeated training runs on two language models, and cross-dataset evaluations (Reviewer Uv5x, Reviewer ftyW, and Reviewer eSES).
  • The paper is clear and easy to follow (Reviewer Uv5x and Reviewer ftyW).

Based on all the reviews, here we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

  • Regarding questions about the central claim, we have addressed the concern as follows:
    • We provide a controlled decomposition of viewpoint localization and coordinate transformation with four matched models (For Reviewer ftyW and Reviewer eSES).
    • We gather the evidence from consistent inputs without a supplied viewpoint, which separates the inability to take the described perspective from reliance on supplied geometry (For Reviewer ftyW and Reviewer Uv5x).
    • We refine the central claim from "does not infer it" to "does not take the described perspective on its own", with the evidence for each step (For Reviewer Uv5x and Reviewer ftyW).
    • We clarify the scope: what annotated-viewpoint scores measure does not depend on the architecture, and the findings about the model are stated for the audited family (For Reviewer ftyW and Reviewer Uv5x).
  • Regarding questions about novelty and contribution, we have addressed the concern as follows:
    • We further clarify how our work differs from SIG3D, MSR3D, Real-3DQA and Seeing the Other Side, and we separate the conclusions that these works establish from the new controlled measurements (For Reviewer Uv5x and Reviewer eSES).
  • Regarding questions about the comparison with recent methods, we have addressed the concern as follows:
    • We include four recent baselines (Video-3D LLM, 3DGraphLLM, 3D-SLIM and GPT4Scene) and report SQA3D by data split and viewpoint availability (For Reviewer eSES).

The message of the paper is unchanged and now more specific: an annotated viewpoint performs localization and coordinate transformation outside the model, so the score it yields measures viewpoint use, and the predicted-viewpoint score is the one that measures progress in egocentric 3D reasoning.

We have also uploaded a revised version of our submission according to the suggestions provided by reviewers, with major changes highlighted in blue: the decomposition (Table 2, Sec. 5.2 and the new Appendix J), the evidence without a supplied viewpoint (Sec. 5.2), the refined claim and its scope (abstract, Secs. 1 and 6), the comparison with prior diagnostics (Sec. 2), the extended Table 1, and the notation and claim-to-evidence tables (Appendix J). The numbering of the existing tables, sections and equations is unchanged. We look forward to further feedback and discussion. Please feel free to let us know if further details or explanations are helpful.

Once again, we thank all reviewers for their thoughtful feedback and valuable guidance in improving our work.

Best regards,

Submission 58440 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.