acceptodds
Under review as a conference paper at ICLR 2027

Anchoring the Observer: Observer-Centeric Positional Encoding for Situated 3D Reasoning

Abstract

We present an observer-centric framework for situated question answering (SQA), coined OC-SQA, that reasons about spatial relations from an observer’s position and orientation. Existing SQA approaches are largely insensitive to situation descriptions, and observer-specific situation information is insufficiently exploited. To address this limitation, we explicitly incorporate observer poses for the positional encoding of visual tokens via a novel observer-centric rotary position embedding (OC-RoPE). OC-RoPE projects 3D locations of visual tokens onto a shared bird’s-eye-view (BEV) plane and then transforms projected positions into the observer’s reference frame for observer-centric positional encoding. We further introduce a learnable observer anchor at the head of the visual tokens of each frame to explicitly indicate the observer’s origin. Together, these two designs provide direct geometric cues for situated reasoning. Extensive experiments demonstrate that our method achieves state-of-the-art performance on SQA3D and Real3DQA, while also demonstrating strong performance on MSQA. Further in-depth analyses show that our design increases the reliance on visual tokens relevent to the situated questions when predicting answers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Situated Question Answering, 3D Spatial Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jw83OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OC-SQA for situated 3D question answering. It encodes visual tokens using observer-relative BEV coordinates in RoPE and adds a learnable token at the observer’s origin. Results are reported on SQA3D, Real3DQA, and MSQA.

Critical strengths

The motivation makes sense: answering observer-dependent questions requires knowing where the observer is and which direction they face. The method is easy to follow, and the gains on directional questions are encouraging. The comparison between sequence-based and spatially positioned anchors also helps justify the anchor design.

Critical weaknesses
  1. My main concern is whether the experiments isolate the contribution of the proposed encoding. The LoRA baseline does not receive the same explicit pose information through another encoding mechanism. The random-rotation ablation shows that correct alignment matters, but does not establish why OC-RoPE is preferable to a simpler embedding of observer-relative coordinates. This comparison is particularly relevant given the existing work on observer-centric representations.

  2. The component ablations are all conducted with GT poses, while the overall accuracy drops from 71.4 to 64.9 with predicted poses. The pose-noise analysis is useful, but I would still like to see whether both components contribute consistently with the predicted poses actually used at inference. Table 3 also lacks the stronger recent baselines included in Table 1, making the viewpoint-robustness improvement less convincing.

Questions for authors

Would the method still outperform a conventional embedding of the same observer-relative coordinates under matched conditions?

What are the individual gains from OC-RoPE and the anchor when using predicted poses?

Were the MSQA baselines evaluated on the same test data with the same judge?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jw83

W1(Q1)

The random-rotation ablation shows that correct alignment matters, but does not establish why OC-RoPE is preferable to a simpler embedding of observer-relative coordinates. This comparison is particularly relevant given the existing work on observer-centric representations.

Response:

Thank you for raising this point.We conducted an additional ablation using a SIG3D-style observer-relative positional encoding baseline, denoted as (OC-RoPE -> OC-XYZ-MLP). This baseline replaces the OC-RoPE with an embedding that encodes observer-relative XYZ coordinates with a two-layer MLP while retaining the observer-centric coordinate system and anchor tokens.

On the full SQA3D test set, our OC-RoPE design outperforms the OC-XYZ-MLP design by 0.7 percentage points in EM@1 and 0.8 percentage points in EM@R1.

EM@1EM@R1
OC-RoPE ->OC-XYZ-MLP70.773.8
Ours(OC-RoPE)71.474.6

On the more challenging subset of SQA3D described in Table 6, the gains increase to 1.2 and 1.5 percentage points in EM@1 and EM@R1, respectively.

EM@1EM@R1
OC-RoPE ->OC-XYZ-MLP45.247.0
Ours(OC-RoPE)46.448.5

W2(Q2)

I would still like to see whether both components contribute consistently with the predicted poses actually used at inference.

Table 3 also lacks the stronger recent baselines included in Table 1, making the viewpoint-robustness improvement less convincing.

Response:

Thank you for these suggestions. We conducted component ablations under both Top-1 and Top-3 predicted-pose settings following Stage 2 training and extended the Real3DQA comparison to include stronger recent baselines.

(1) Ablations with Predicted Poses.

Under the Top-1 predicted-pose setting, OC-RoPE improves EM@1 and EM@R1 over the LoRA baseline by 0.9 and 0.3 percentage points, respectively. Adding the anchor token provides further gains of 1.3 and 1.5 percentage points.

EM@1EM@R1
lora62.766.3
OC-RoPE63.666.6
OC-RoPE+anchor64.968.1

Under the Top-3 predicted-pose setting, OC-RoPE improves EM@1 and EM@R1 over LoRA by 2.9 and 2.5 percentage points, respectively. Adding the anchor token further improves both metrics by 1.1 percentage points.

EM@1EM@R1
lora62.766.3
OC-RoPE65.668.8
OC-RoPE+anchor66.769.9

These results show that both OC-RoPE and the anchor token consistently improve performance under predicted poses in both settings. The overall gains are smaller in the Top-1 setting. This may be partly because SQA3D situation descriptions are not always clear: the same description can match several possible observer positions and orientations in a scene. This ambiguity can cause the Top-1 predicted pose to deviate substantially from the annotated pose, thereby limiting downstream QA performance. Considering the Top-3 pose hypotheses ranked by our pose estimator helps mitigate this ambiguity, providing a plausible explanation for the larger overall gains observed in this setting.

(2) Stronger Baselines on Real3DQA.

We also evaluated 3DRS, SR-3D, Ross3D, and OneCanvas on Real3DQA. Our method achieves the highest scores across all reported metrics. Compared with OneCanvas, it improves VRS, which measures average per-view accuracy, by 6.3 percentage points. It also increases all-four-correct accuracy from 1.3% to 3.3%, showing a higher proportion of question groups answered correctly across all four viewpoints.

modelVRS (%)k=1 (%)k=2 (%)k=3 (%)k=4 (%)
3DRS14.845.111.52.30.4
SR-3D19.3756.2716.6740.53
Ross3D16.250.811.61.90.5
OneCanvas28.371.632.77.61.3
Ours34.676.542.7163.3

Q3

Were the MSQA baselines evaluated on the same test data with the same judge?

Response:

Yes. Our method and all baselines were evaluated under this same configuration on the same MSQA test split using the same official judge provided by MSQA. Since our method accepts only text-based questions, we converted MSQA’s original multimodal questions, which contain both text and images, into a text-only format while keeping the RGB-D scene inputs unchanged.

Thanks for the clarification. I read the rebuttal, and I think my main concerns are addressed. My attitude is towards acceptance.

We sincerely thank you for your positive response and valuable comments. We greatly appreciate your constructive suggestions, which have helped improve the quality of our manuscript.

Reviewer_qxHJOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OC-SQA, a framework for situated 3D question answering that explicitly incorporates the observer’s position and orientation into visual-token positional encoding. Motivated by the limited sensitivity of existing VLMs to situation descriptions, the method introduces observer-centric rotary positional embedding (OC-RoPE), which maps visual tokens from posed RGB-D frames onto an observer-aligned bird’s-eye-view plane and uses the resulting coordinates to replace the spatial indices in M-RoPE. It additionally inserts a learnable observer anchor at the origin of this coordinate system for each frame, providing an explicit reference for observer–scene spatial relationships. Experiments on SQA3D, Real3DQA, and MSQA report improved question-answering performance, with ablations and diagnostic analyses examining the contributions of the proposed components and sensitivity to situation information.

Critical strengths
  1. Clear motivation. The paper highlights the distinction between representing scene geometry and reasoning from a specified observer’s perspective, supported by a diagnostic study of existing models’ sensitivity to situation descriptions.

  2. Simple and lightweight design. OC-RoPE and the observer anchor incorporate observer-relative geometry into a pretrained VLM without requiring an additional spatial encoder or changes to the visual backbone.

  3. Evaluation across multiple benchmarks. Experiments on SQA3D, Real3DQA, and MSQA demonstrate competitive performance, complemented by component ablations and analyses of reliance on relevant visual evidence.

Critical weaknesses
  1. Limited ablation study. The ablations establish the effectiveness of the proposed components within the authors’ framework, but do not isolate their incremental benefit over closely related prior methods. For example, Table 4 does not isolate the gains over closely related methods. The authors should compare OC-RoPE against OneCanvas’s panoramic positional encoding and SIG3D-style observer-relative positional encoding, using the same backbone, training setup, and pose inputs. Adding the proposed observer anchor to the OneCanvas baseline would further clarify its contribution beyond prior work.
  2. The viewpoint-robustness metric does not measure cross-view consistency. By definition, VRS averages the fractions of groups with at least kk correct answers for k=1,...,4k=1,...,4, which mathematically equals average accuracy across individual viewpoint variants. It therefore cannot distinguish consistent performance within a question group from scattered successes across groups. Although Table 3 separately reports all-four-correct accuracy, this remains only 0.8% with predicted poses and 3.3% with GT poses. The robustness claim would benefit from paired analysis showing whether answers change correctly when the observer changes, while remaining stable for viewpoint-invariant questions.
  3. The BEV formulation discards geometric information relevant to general situated 3D reasoning. OC-RoPE drops height and models observer orientation only through yaw. Consequently, vertically separated tokens at the same horizontal location receive identical spatial indices, and pitch/roll cannot affect the observer-relative encoding. Visual features may retain some of this information, but the proposed positional mechanism cannot explicitly represent it. The paper should evaluate height-dependent relations and non-horizontal viewpoints, and compare against a full observer-centric XYZ positional encoding to establish whether BEV is a beneficial inductive bias or a benchmark-specific simplification.
Questions for authors
  1. Training fairness: Which training stages are used for each row of Table 4? Do the LoRA baseline and all proposed variants receive the same training data and number of optimization steps?

  2. Source of gains: How much does Stage 2 improve Top-1 predicted-pose performance? Are the reported gains primarily attributable to the proposed encoding or to additional adaptation to pose errors?

  3. Anchor mechanism: Under the backbone’s causal attention mask, which tokens can attend to each observer anchor? How does placing the anchor before each frame’s visual tokens implement the observer–scene interaction claimed in Section 3.2?

6 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer qxHJ

W1

The authors should compare OC-RoPE against OneCanvas's panoramic positional encoding and SIG3D-style observer-relative positional encoding, using the same backbone, training setup, and pose inputs. Adding the proposed observer anchor to the OneCanvas baseline would further clarify its contribution beyond prior work.

Response:

Thank you for your valuable feedback. We compared our method with the panoramic positional encoding proposed in OneCanvas (a concurrent NeurIPS 2026 paper) and SIG3D-style observer-relative positional encoding, using the same backbone, training configuration, and pose inputs.

For the OneCanvas-style panoramic positional encoding, a 3D point is projected onto the panorama according to its viewing direction relative to the observer. For 3D points with x=0 or (x=0,y=0), the azimuth and elevation are mathematically undefined. For these cases, we evaluate this encoding without the anchor token, denoted as Panoramic-PE in the table. To assess the effect of retaining the anchor token, we also evaluate Panoramic-PE+Anchor, in which the anchor token is manually assigned to (0,0).
For SIG3D-style observer-relative positional encoding, we implemented an OC-XYZ-MLP variant that uses a two-layer MLP to encode 3D coordinates and adds the resulting 3D embeddings to the visual tokens while retaining both the observer-centric coordinate system and the anchor token.

As shown in the table, our full OC-RoPE configuration achieves the best performance on both metrics, outperforming Panoramic-PE by 2.2 and 1.2 percentage points in EM@1 and EM@R1, respectively. Adding an anchor token with manually assigned panoramic coordinates yields no consistent improvement. OC-RoPE also outperforms OC-XYZ-MLP with the same anchor setting, indicating that the gains arise from our positional encoding rather than merely including an anchor token.

EM@1EM@R1
Panoramic-PE69.273.4
Panoramic-PE+anchor token69.372.9
OC-RoPE -> OC-XYZ-MLP70.773.8
Ours (OC-RoPE)71.474.6

W2

VRS measures average per-view accuracy rather than cross-view consistency. All-four-correct accuracy is also low (0.8% with predicted poses and 3.3% with GT poses). Paired analysis should assess whether answers adapt correctly to observer changes while remaining stable for viewpoint-invariant questions.

Response:

We sincerely thank the reviewer for this valuable suggestion. We verified that all questions in Real3DQA are viewpoint-dependent. The absence of viewpoint-invariant questions prevents paired analysis as proposed. Below, we analyze the reasons for the low all-four-correct accuracy (k=4) and present a more direct evaluation of cross-view consistency.

The reason for the low all-four-correct accuracy (k=4k=4).

For each group of Real3DQA questions, the wording of the questions remains identical across the four views, while the ground-truth answers vary across views. For example, at a given location, the same question is asked from four different viewpoints: “What is the closest object in front of me?” Each viewpoint yields a distinct answer. Assume that predictions are independent and equally accurate across views. With a VRS of approximately 35% obtained using GT poses, the expected all-four-correct accuracy is $0.35^4 \approx 1.5\%$. This is on the same order as the observed $3.3\%$ with GT poses.

A more direct evaluation of cross-view consistency.

To evaluate cross-view consistency on viewpoint-dependent questions, we sampled 100 observer locations from Real3DQA and evaluated with GT poses. We retained the first-view question and adjusted the directional wording for the other three views, preserving the same ground truth answer across all four viewpoints. For example, after a 90° right turn, “the closest object in front of me” becomes “the closest object to my left.” This tests whether the model consistently identifies the same target as the observer’s orientation and directional wording change together.

The results are shown below, where each column reports the percentage of groups with at least $k$ correct answers. Our method achieves the highest all-four-correct accuracy of 17%, exceeding OneCanvas by 5 percentage points. It also achieves the highest proportions of groups with at least two and three correct answers, at 40% and 25%, respectively. These results support improved cross-view consistency of correct predictions in this controlled evaluation.

modelk=1 (%)k=2 (%)k=3 (%)k=4 (%)VRS (%)
3DRS533413225.5
SR-3D452813823.5
Ross3d503216525.8
OneCanvas5635161229.8
Ours4640251732
AuthorsOpenReview ↗

(2/2) Response to Reviewer qxHJ

W3

OC-RoPE ignores height and pitch/roll, limiting its ability to encode general 3D relations. Please evaluate height-dependent relations and non-horizontal viewpoints, and compare against full observer-centric XYZ encoding to determine whether BEV is a useful inductive bias or a benchmark-specific simplification.

Response:

We appreciate the reviewer’s thoughtful comments. Our decision to omit explicit height encoding was motivated by the spatial characteristics of indoor scenes. Objects involved in above/below relationships often have nearby projections on the BEV plane and may be visible within the same frame. Horizontal spatial relationships often span larger areas and require information from multiple frames. We therefore designed our spatial representation to support reasoning about these relationships across frames. The same rationale motivates our use of yaw to represent observer orientation. Yaw provides a horizontal reference for integrating spatial information across frames, while visual features can capture view-specific cues associated with pitch and roll, including those arising from non-horizontal viewpoints.

To evaluate performance on height-dependent questions, we used GPT-6 to screen the full SQA3D test set, identifying 213 questions involving above/below relationships. We evaluated the models on this subset, with the results reported below. Ours (OC-XYZ-MLP) corresponds to the previously described SIG3D-style ablation, which uses a two-layer MLP to encode observer-centric XYZ coordinates and adds the resulting embeddings to the visual tokens while retaining the anchor token.

Our method achieves the highest scores among the evaluated methods on this subset, outperforming the OC-XYZ-MLP variant by 1.0 percentage point in EM@1 and 1.4 percentage points in EM@R1. These results show that our BEV-based formulation remains effective on questions involving vertical spatial relationships.

EM@1EM@R1
SR-3D61.565.7
3DRS62.464.8
Ross3D62.966.2
OneCanvas57.762.4
Ours(OC-XYZ-MLP)62.466.7
Ours63.468.1

Q1

Training fairness: Which training stages are used for each row of Table 4? Do the LoRA baseline and all proposed variants receive the same training data and number of optimization steps?

Response:

All methods in Table 4, including the LoRA baseline and all proposed variants, are trained in Stage 1 using ground-truth observer poses. To ensure a fair comparison, all methods use the same training configuration, including the training data and the number of optimization steps.


Q2

Source of gains: How much does Stage 2 improve Top-1 predicted-pose performance? Are the reported gains primarily attributable to the proposed encoding or to additional adaptation to pose errors?

Response:

Thank you for this question. Stage 2 improves Top-1 predicted-pose accuracy from 60.8% to 64.9% (+4.1 percentage points) and Top-3 predicted-pose accuracy from 65.9% to 66.7% (+0.8 percentage points).

SQA3D situation descriptions are not always clear: the same description can match several possible observer positions and orientations in a scene. As a result, the Top-1 predicted pose can deviate substantially from the annotated pose, while considering the Top-3 pose hypotheses ranked by our pose estimator helps mitigate this ambiguity. This explains why Stage 2 training yields a larger performance improvement in the Top-1 setting.


Q3

Anchor mechanism: Under the backbone's causal attention mask, which tokens can attend to each observer anchor? How does placing the anchor before each frame's visual tokens implement the observer–scene interaction claimed in Section 3.2?

Response:

We use the backbone’s standard causal attention mask without modification. Each observer anchor can be attended to by all subsequent tokens, including the visual tokens of its own frame and tokens in later frames, as the observer token is placed immediately before each frame’s visual tokens.

As shown in Figure 4 of the revised paper, the anchor provides a dedicated token representation of the observer as a spatial reference. Without an anchor, attention captures relative spatial relationships among visual tokens, but there is no dedicated observer token for them to attend to directly. Their spatial relationship with the observer is encoded only implicitly through the coordinate offsets used in RoPE. By assigning the observer’s reference position to the anchor in RoPE, we introduce visual–observer attention pairs whose scores incorporate their relative spatial relationship. This enables visual tokens to attend directly to the observer representation and form observer-conditioned scene features, implementing the observer–scene interaction described in Section 3.2.

Thank you for the additional experiments and clarifications. They address several of my concerns, but I am still unsure how much OC-RoPE adds over existing ways of encoding observer-relative positions.

With the same observer-centric coordinates and anchor token, the XYZ-MLP baseline achieves 70.7 EM@1, compared with 71.4 for OC-RoPE. The gap is small.

I also have two questions about the new comparisons:

  1. Does the Panoramic-PE baseline include OneCanvas’s metric 3D positional embedding, in addition to its panoramic encoding?
  2. Do the gains over XYZ-MLP and Panoramic-PE also hold with Top-1 predicted poses after the same Stage 2 adaptation? The added predicted-pose ablations are helpful, but they compare the proposed components against LoRA rather than these closer alternatives.

These points matter to my assessment because the current results suggest that observer-centric alignment and the anchor account for much of the improvement, while the additional benefit of OC-RoPE remains less clear.

Reviewer_qFhiOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces observer-centric positional encoding (OC-RoPE) for improving models' performance on situated question answering tasks. OC-RoPE projects 3D locations of visual tokens onto a shared BEV and transform projected positions into observer's reference frame for observer-centric positional encoding. Experiments on SQA3D and Real3DQA demonstrates the effectiveness of OC-RoPE.

Critical strengths
  1. Simple and works well for Qwen as Qwen use M-RoPE including temporal, width, and height, where OC-RoPE project width and height to observer-related BEW coordinates.

  2. Ablation (4.4) is comprehensive and informative, demonstrating observer alignment is necessary.

Critical weaknesses
  1. Seems like the observer pose is necessary as the input, which might limit the usefulness extending this work to a more general situated egocentric video understanding task such as [1]. Is that possible to use geometric foundation model such as [2] to estimate observer's pose and orientation? I am interested to see if this kind of positional encoding can be extended to video understanding.

[1] SAW-BENCH: Learning Situated Awareness in the Real World, ICML 2026.

[2] VGGT: Visual Geometry Grounded Transformer, CVPR 2025.

  1. Need more experiments to show OC-RoPE is the most effective. Random rotation is a pretty weak baseline, it destroys the observer-centric information. Also, inputing the pose token to Qwen and finetune it end-to-end is another strong baseline.
Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer qFhi

W1

Requiring observer pose may limit applicability to egocentric video understanding tasks such as Saw-Bench. Could VGGT estimate pose and orientation to extend this positional encoding to video understanding?

Response:

For observer pose estimation, there are two cases: (1) the observer’s pose is specified by text and does not depend on the camera pose, e.g., SQA3D; and (2) the observer’s pose is determined by the camera pose, e.g., Saw-Bench and UCS-Bench. For the first case, there is no need to use geometric foundation models for pose estimation. For the second case, geometric foundation models such as VGGT can be used to estimate camera poses, and the pose of the frame temporally closest to the question timestamp is then selected as the observer pose.

Evaluation on SAW-Bench

On SAW-Bench, Route Shape and Spatial Memory questions are less dependent on the observer’s pose at a specific moment. We therefore evaluated results on the remaining four question types: Self-Localization, Relative Direction, Reverse Route Plan, and Spatial Affordance. We selected the first or last frame according to the question and used the corresponding estimated camera pose as the observer’s pose, as summarized in the table.

Our method improves the overall accuracy from 28.21% to 29.89%, with larger gains on Self-Localization and Spatial Affordance (+4.50 and +4.94 percentage points, respectively). These questions are based on the observer’s location at a fixed moment, aligning well with our static observer-centric spatial representation. The gains on Relative Direction and Reverse Route Plan are smaller (+0.72 and +0.44 percentage points, respectively). Both tasks require reasoning across different observer poses and challenge our single static, observer-centric representation. For example, Reverse Route Plan requires tracking changes in observer position and orientation, while Relative Direction requires locating one observer pose relative to another.

Question CategoryReference FrameQwen3-VL-8B Accuracy (%)Ours Accuracy (%)Accuracy Gain
Self-LocalizationFirst frame38.0042.50+4.50
Spatial AffordanceFirst frame50.6255.56+4.94
Relative DirectionFirst or last frame, depending on the question22.1822.90+0.72
Reverse Route PlanLast frame25.7626.20+0.44
Overall (four categories)—28.2129.89+1.68

Evaluation on UCS-Bench

We also evaluated our method on another video understanding dataset, UCS-Bench (Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams), that grounds questions on a specific frame. UCS-Bench draws scenes from multiple datasets, including ScanNet, and provides newly annotated question–answer pairs. Since our method is fine-tuned on SQA3D, which also uses ScanNet scenes, we conducted a zero-shot evaluation on the ScanNet-derived subset of UCS-Bench. Given that UCS-Bench associates each question with a specific timestamp in an egocentric video, we use the camera pose of the frame closest in time to the question’s timestamp as the observer pose.

The results below show that our method improves performance across all three spatially oriented categories in UCS-Bench’s official categorization, achieving an overall accuracy gain of 6.05 percentage points. In particular, our method achieves a substantial gain of 17.94 percentage points in Ego-centric Position & Orientation Memory, the category most directly related to egocentric spatial relationships. Performance on Object Recall & Quantity Tracking decreases slightly, primarily due to counting questions.

Question CategoryQwen3-VL-8B Accuracy (%)Ours Accuracy (%)Change
Ego-centric Position & Orientation Memory40.7658.70+17.94
Embodied trajectory & Movement Memory33.9341.07+7.14
Object Recall & Quantity Tracking54.3351.92-2.41
Perceived Proximity & Reachability Memory51.8354.88+3.05
Overall47.7153.76+6.05
AuthorsOpenReview ↗

(2/2) Response to Reviewer qFhi

W2

Need more experiments to show OC-RoPE is the most effective. Also, inputting the pose token to Qwen and fine-tuning it end-to-end is another strong baseline.

Response:
Thank you for this suggestion. We evaluate two stronger learned baselines:

  • (OC-RoPE->XYZ-MLP): We encode 3D positions for each visual token in a non-observer-centric coordinate frame using a two-layer MLP. Each positional embedding is added to its corresponding visual token, while the observer’s 3D embedding is added to the anchor token to inject the observer pose into Qwen.
  • (OC-RoPE->OC-XYZ-MLP): We first transform the 3D coordinates into the observer-centric frame and then encode them using the same two-layer MLP. This baseline explicitly preserves observer-centric information, providing a direct comparison between MLP-based encoding and OC-RoPE.

For a fair comparison, all variants use the same anchor token and identical training configurations. The results on the full SQA3D test set and the harder subset described in Table 6 are shown below.

Full test EM@1Full test EM@R1Harder subset EM@1Harder subset EM@R1
OC-RoPE -> XYZ-MLP61.765.133.335.6
OC-RoPE -> OC-XYZ-MLP70.773.845.247.0
Ours(OC-RoPE)71.474.646.448.5

On the full test set, our OC-RoPE design outperforms the XYZ-MLP baseline 9.7 and 9.5 percentage points in EM@1 and EM@R1, respectively. Crucially, our results also outperform OC-XYZ-MLP, which already preserves observer-centric information, by 0.7 and 0.8 percentage points. These gains over the observer-centric MLP baseline increase to 1.2 and 1.5 percentage points on the harder subset.

We are also evaluating a variant that directly feeds an observer-pose token into Qwen without relying on an anchor token. The results will be reported later.

Following your suggestion, we evaluate a direct pose-token baseline, which encodes the observer’s 3D position and orientation into a pose token prepended to the visual tokens of each frame. We use the same training configuration as our method.

Full test EM@1Full test EM@R1Harder subset EM@1Harder subset EM@R1
Direct pose-token baseline63.266.434.936.1
Ours(OC-RoPE)71.474.646.448.5

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews

We sincerely thank all reviewers for their constructive feedback and thoughtful discussions. We are encouraged by reviewers’ recognition of our sound motivation (qxHJ, Jw83), simple design (qFhi, qxHJ), and competitive performance across multiple benchmarks (qxHJ). Reviewer qFhi also appreciated the comprehensive ablations, while reviewer Jw83 highlighted the gains on directional questions and the evidence supporting the observer-anchor design.

In rebuttal, we demonstrated that the good performance of our method can be extended to general situated egocentric video benchmarks (SAW-Bench and UCS-Bench) (see response to qFhi-W1) and our method is superior to other alternatives, e.g., OneCanavas (a concurrently accepted NeurIPS 2026 paper), in different suggested settings (see response#3 to qxHJ).

After rebuttal, reviewers qFhi and Jw83 explicitly expressed support for acceptance, with qFhi raising their score from 3 to 4 and Jw83 confirming that their main concerns were addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.