- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes OC-SQA for situated 3D question answering. It encodes visual tokens using observer-relative BEV coordinates in RoPE and adds a learnable token at the observer’s origin. Results are reported on SQA3D, Real3DQA, and MSQA.
Critical strengths
The motivation makes sense: answering observer-dependent questions requires knowing where the observer is and which direction they face. The method is easy to follow, and the gains on directional questions are encouraging. The comparison between sequence-based and spatially positioned anchors also helps justify the anchor design.
Critical weaknesses
-
My main concern is whether the experiments isolate the contribution of the proposed encoding. The LoRA baseline does not receive the same explicit pose information through another encoding mechanism. The random-rotation ablation shows that correct alignment matters, but does not establish why OC-RoPE is preferable to a simpler embedding of observer-relative coordinates. This comparison is particularly relevant given the existing work on observer-centric representations.
-
The component ablations are all conducted with GT poses, while the overall accuracy drops from 71.4 to 64.9 with predicted poses. The pose-noise analysis is useful, but I would still like to see whether both components contribute consistently with the predicted poses actually used at inference. Table 3 also lacks the stronger recent baselines included in Table 1, making the viewpoint-robustness improvement less convincing.
Questions for authors
Would the method still outperform a conventional embedding of the same observer-relative coordinates under matched conditions?
What are the individual gains from OC-RoPE and the anchor when using predicted poses?
Were the MSQA baselines evaluated on the same test data with the same judge?
3 comments
Thanks for the clarification. I read the rebuttal, and I think my main concerns are addressed. My attitude is towards acceptance.
We sincerely thank you for your positive response and valuable comments. We greatly appreciate your constructive suggestions, which have helped improve the quality of our manuscript.
Response to Reviewer Jw83
W1(Q1)
The random-rotation ablation shows that correct alignment matters, but does not establish why OC-RoPE is preferable to a simpler embedding of observer-relative coordinates. This comparison is particularly relevant given the existing work on observer-centric representations.
Response:
Thank you for raising this point.We conducted an additional ablation using a SIG3D-style observer-relative positional encoding baseline, denoted as (OC-RoPE -> OC-XYZ-MLP). This baseline replaces the OC-RoPE with an embedding that encodes observer-relative XYZ coordinates with a two-layer MLP while retaining the observer-centric coordinate system and anchor tokens.
On the full SQA3D test set, our OC-RoPE design outperforms the OC-XYZ-MLP design by 0.7 percentage points in EM@1 and 0.8 percentage points in EM@R1.
On the more challenging subset of SQA3D described in Table 6, the gains increase to 1.2 and 1.5 percentage points in EM@1 and EM@R1, respectively.
W2(Q2)
I would still like to see whether both components contribute consistently with the predicted poses actually used at inference.
Table 3 also lacks the stronger recent baselines included in Table 1, making the viewpoint-robustness improvement less convincing.
Response:
Thank you for these suggestions. We conducted component ablations under both Top-1 and Top-3 predicted-pose settings following Stage 2 training and extended the Real3DQA comparison to include stronger recent baselines.
(1) Ablations with Predicted Poses.
Under the Top-1 predicted-pose setting, OC-RoPE improves EM@1 and EM@R1 over the LoRA baseline by 0.9 and 0.3 percentage points, respectively. Adding the anchor token provides further gains of 1.3 and 1.5 percentage points.
Under the Top-3 predicted-pose setting, OC-RoPE improves EM@1 and EM@R1 over LoRA by 2.9 and 2.5 percentage points, respectively. Adding the anchor token further improves both metrics by 1.1 percentage points.
These results show that both OC-RoPE and the anchor token consistently improve performance under predicted poses in both settings. The overall gains are smaller in the Top-1 setting. This may be partly because SQA3D situation descriptions are not always clear: the same description can match several possible observer positions and orientations in a scene. This ambiguity can cause the Top-1 predicted pose to deviate substantially from the annotated pose, thereby limiting downstream QA performance. Considering the Top-3 pose hypotheses ranked by our pose estimator helps mitigate this ambiguity, providing a plausible explanation for the larger overall gains observed in this setting.
(2) Stronger Baselines on Real3DQA.
We also evaluated 3DRS, SR-3D, Ross3D, and OneCanvas on Real3DQA. Our method achieves the highest scores across all reported metrics. Compared with OneCanvas, it improves VRS, which measures average per-view accuracy, by 6.3 percentage points. It also increases all-four-correct accuracy from 1.3% to 3.3%, showing a higher proportion of question groups answered correctly across all four viewpoints.
Q3
Were the MSQA baselines evaluated on the same test data with the same judge?
Response:
Yes. Our method and all baselines were evaluated under this same configuration on the same MSQA test split using the same official judge provided by MSQA. Since our method accepts only text-based questions, we converted MSQA’s original multimodal questions, which contain both text and images, into a text-only format while keeping the RGB-D scene inputs unchanged.