- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EgoAlign, a 3D language model for situated question-answering. The contributions are three-fold: (1) Proposed a model by combining Concerto with Qwen2.5, following Liu et al. 2026 to enable ego centric information by both transforming the point cloud to an agentic pose, and adding ego-centric position embedding on the input tokens. (2) Extended the idea of Ma et al. 2026 to use egocentric alignment for improving situated directional reasoning, which is done by actually rotating the ego pose by multiples of 90 degrees and determining the expressions of [front, left, back, right] in-situ. This method proved to be the single most important design to achieve sota (Table. 3). (3) A scale-up experiment covering multiple dimensions, signaling a scalable benchmark performance (SQA3D, Real-3DQA, MSQA) but some metrics on the egocentric alignment data remain stagnating across 1.5B to 7B (relation acc, relation consistency, etc.)
Critical strengths
- The problem is well-stated, where a numerical pose should be input to the model rather than sending the model a situated text descriptions, especially in realistic application scenarios. (Note: the flip side of this formulation is also discussed in W2)
- Presentation is clear, paper is generally well-written and easy to follow. Visualization of Figure 3 is well-drawn.
- The results are consistent and strong on all datasets, with a clean scaling curve of model capability with the model weight.
- The biggest, most valuable findings are the counterfactual group training on rotated ego poses and the synergetic effect with aligned point cloud and fusing the metric coordinates in the superpoint tokens. The results are rigously validated in Table 3 and provide valuable direction for development of future methods.
Critical weaknesses
- Figure 1(b) phenomenon and L74 claim are not precisely fair. Pose insensitivity of PoseAlign on SQA3D is because it used random poses for training on SQA3D, which is fixed recently in its official code. The authors are suggested to follow the fixed code and weights. LEO is trained with optional ego pose encoding only enabled on SQA3D and MP3D but with no pose on other mixed datasets, therefore its ego pose insensitivity is expected. Also see Q6.
- Ego-aligned evaluation on situated VQA benchmark kind of defeats its own purpose: The ego pose no longer needs inference and is provided as model input, yet implicit inference of ego pose from language description is the central idea of these datasets. From my perspective, these datasets mainly focus on a benchmark/offline-training aspect to improve model performance, rather than defining their problem formulation as the system model to use during actual deployment, and it is incorrect to discredit their formulation in L068. Furthermore, ego-aligned evaluation on SQA3D and Read-3DQA degrades to simple question-answering tasks such as “What is to your left” while the situation description is less relevant to the task (because it's known at input). This leaves only one task as sufficiently complex, which is MSQA. An extended experiment suite is highly recommended, e.g., on other benchmarks used by 3D-LLaVA or LEO.
- Contribution 4 “without requiring textual situation descriptions” is misleading, since you did use the ego situation (and in a stronger, explicit form) as model input.
- Table 4 experimental setting is not clearly stated and possibly unfair. Specifically, the numbers in Table 4 are even lower than original performance for LEO and PoseAlign in Table 1, which goes beyond simply 'learn the alignment tasks substantially less effectively' (L424) and raises further questions about implementation fairness. For LEO, it is unclear if the authors either (1) rotate the input to the embodied token, or (2) rotate the input point cloud as did in EgoAlign, where the latter could bring in significant distribution shift which can also cause deterioration. For PoseAlign, it uses a slightly different ego coordinate convention (camera with human height and elevation) compared to egocentric alignment data (floor height pose with zero elevation) which can also cause distributional shift in training data. If these questions are unanswered, the fairness of Table 4 can be compromised. Finally, if these questions persist, this will in turn lead to a new question of the general applicability of your training method.
Questions for authors
- The relation task deviates from dataset norms. In your definition, given a ‘bed to the left of a desk’ when the agent is facing them, the agent should answer the opposite direction, i.e., ‘bed is to the right of desk’, when the agent turns to its back. While I agree that this definition is consistent and clear under an agent-centric reference frame, this is not what a normal person would answer; an average person would prefer turning back first and answer from an-object-facing direction, and therefore human-annotated benchmarks would give a different answer from that defined by your relation task. Given this discrepancy maybe it’s better to identify this task as a new task, named ‘egocentric relation’ rather than regular ‘relation’, which most people would assume the latter with a target-facing ego pose.
- How are the poses selected? I would assume it’s (4,262+493)*4=19,020 positions according to L689. The authors report 18,750 training and 250 validation poses which sum up to 19,000. Under what rule are the remaining 20 poses pruned?
- How are multi-instances handled for your alignment data? For example, there are many cases in ScanQA where a wall has many pictures, or a conference room has many chairs, or a storage room has many boxes. Will two objects of the same class be clearly pinpointed to the exact one you meant in the relation task? Do you have a way of specifying the very target you are referring to in the direction and distance tasks?
- Are there human-in-the-loop quality checks for the alignment data?
- To further support your core claim that you ‘address pose insensitivity’, can you provide the pose-shuffling performance as an additional column on your ablation experiment (Table 3)?
- Following W1, the authors are suggested to conduct the experiment in a fairer setup where the ego poses in SQA3D used by the authors are equally added to training data of all methods, including Figure 1(b), Table 1, Table 2, and Table 4. Figure 7 is also recommended to add the two baseline models, LEO and PoseAlign.
- The authors are suggested to display some samples of diagonal relationships, e.g. front-left, front-right, and vertical relationships, e.g., above and below, in the Relation row of Figure 3, because the claimed 8-direction and two vertical relationships (L209) are not fully visualized.
5 comments
3. Controlled pose-interface ablation (W4, Q5)
To complement the cross-architecture comparison, we added a matched-backbone ablation using the same Concerto/Qwen2.5-1.5B backbone and training schedule.
| Setting | Cat. ↑ | Dir. ↑ | Dist. ↓ | Rel. ↑ | SQA EM | SQA Δpose | Real EM | Real Δpose |
|---|---|---|---|---|---|---|---|---|
| Default coords | 28.5 | 12.5 | 1.15 | 11.6 | 48.6 | — | 16.4 | — |
| Default + agent token | 33.9 | 12.5 | 1.17 | 12.0 | 50.4 | 0.11 | 19.1 | −0.20 |
| Full EgoAlign | 61.8 | 59.0 | 0.47 | 42.2 | 56.8 | 9.72 | 30.2 | 10.30 |
Adding a LEO-inspired pose token improves downstream accuracy but leaves direction accuracy at chance level for eight directions. The small pose-shuffle drops further indicate that simply exposing this backbone to pose through an agent token does not establish effective egocentric alignment. In contrast, EgoAlign's agent-centric representation substantially improves alignment accuracy and downstream pose dependence.
Regarding Q5, the original Table 3 already reports Δpose for component ablations. Removing alignment reduces SQA3D/Real-3DQA Δpose from 9.72/10.30 to 0.14/0.07.
4. Numerical pose and benchmark setting (W2, W3)
Our contribution is situated 3D understanding without situation descriptions, not without situation information. Numerical pose is explicitly provided throughout our problem formulation. We agree that supplying pose removes the need to infer the agent's reference frame from linguistic descriptions, an important component of the original SQA3D formulation. We study a complementary setting relevant to embodied agents, where localization supplies position and orientation directly. Even with a known reference frame, recognizing objects, grounding references, and reasoning about agent-relative spatial relationships remain nontrivial.
SQA3D, Real-3DQA, and MSQA evaluate these capabilities across different question distributions and scene settings. We agree that additional general-purpose 3D benchmarks would broaden the evaluation scope, while our current experiments focus specifically on pose-conditioned situated reasoning.
5. Data construction and relation semantics (Q1–Q4, Q7; Revised: §3.3)
Q1: We agree with the terminology egocentric spatial relation and now adopt it explicitly. Relations use the supplied agent frame without an implicit turn toward the objects. Consequently, a 180° heading change reverses horizontal egocentric relations.
Q2: The 4,262/493 counts refer to eligible training/validation source positions, whereas 18,750/250 refer to sampled four-heading question groups per task. Multiple groups can originate from one position. Four tasks × 18,750 groups × four headings produce 300K training examples, with 4K validation examples. No 20 source positions are missing.
Q3: Each referring expression must uniquely identify its target instance. Repeated categories are disambiguated using permitted attributes such as color or distance; ambiguous candidates are excluded. For relation questions, both references are checked independently, and same-category pairs are excluded.
Q4: We manually inspected generated examples during development before scaling up data construction. For the full dataset, we apply deterministic quality controls: (1) object references must uniquely identify annotated instances, with ambiguous candidates excluded; (2) color attributes are used only when the dominant color is supported by at least 35% of an instance's RGB points; (3) object centers use coordinate-wise medians and bounding boxes use the 5th–95th point percentiles to reduce outlier sensitivity; and (4) spatial labels follow explicit geometric rules, including eight 45° direction bins, nearest horizontal distance to object bounds, and vertical relations requiring horizontal overlap and vertical separation. We retain complete four-heading groups with fixed target instances and avoid duplicate training records.
Q7: Figure 3 illustrates how spatial relations vary across headings rather than enumerating the full relation vocabulary. The alignment data include eight horizontal directions, including diagonal relations, as well as above/below relations. We now explicitly identify the task as egocentric spatial relation in §3.3, with the geometric definitions provided in Appendix B.
We thank the reviewer for these suggestions, which have helped us correct the baseline comparison, establish alignment transfer to PoseAlign, and strengthen the controlled evidence for EgoAlign's representation design.
Thank you for the update on various topics, which address some of my concerns as below:
-
W1, W4, and Q6 are solved through updated experimental results, many revised tables and analyses, revised introduction, and adding a yaw noise experiment in appendix Figure 11.
-
W4 is further addressed in A2 and A3. Your second answer (A2) establishes the general applicability of your training method across two architectures in revised Table 4. Your third answer (A3) adds an interesting experiment by using LEO agent token on your model. The experimental results presented in A3 are slightly concerning since it showed that your Cat., Dir., Dist., Rel., metrics can improve without necessarily gaining pose sensitivity. This is a minor issue, but I do recommend the authors dig into it later. What about defining multilateral metrics, i.e., a metric on your alignment data but calculate the percentage of questions that are answered correctly (or distance error is lower than a certain threshold) on all four directions? All the metrics used in the current paper, except for pose, are unilateral metrics (EM, Cat., Count, Acc, etc.), or are unilateral metrics averaged across different headings (VRS), instead of a genuine multilateral metric according to the above definition. It is helpful to discuss the distinctions of how unilateral metrics perform compared to multilateral scores and what they indicate.
-
Q1, Q2, and Q4 are answered. The authors are suggested to add your dataset curation effort details into the next revision since they will provide valuable information to the community.
The following items are not addressed or are only partially addressed:
-
W2 is partially solved. The claim that SQA3D and Real-3DQA remain nontrivial after removing situation descrption and supplying ego reference frame, needs further proof. E.g., how long are the questions on average after removing the situation descriptions? What are their question type composition and whether do they follow a fixed template? The authors are suggested to prove that these two datasets are nontrivial in the EgoAlign setting.
-
W3 reply missed the point. I understand that your claim in Contribution 4 is accurate, but contribution claims should be precise even when viewed in isolation. I suggested W3 from a presentation aspect instead of a factual concern aspect, where a first-time reader which skims through your paper would misunderstand this sentence, or even get confused by the seemingly bizzare implication that your model answers 'without' pose information; It would only be clear until the contribution is read along with other text. Therefore, as also pointed out by Reviewer 3ahU, the Contribution 4 is still a slight overstatement. I suggest the authors to revise this point into, e.g., "EgoAlign demonstrates superior performance with explicit pose input compared to counterparts recieving textual situation descriptions, exhibiting strong pose dependence." The significance of this finding can still be established nontheless. For example, you may discuss that letting the VLM implicitly find its ego pose is beyond its capability and practially should be handled in a different enginering solution, rather than cramming all tasks together.
-
Q3 needs examples. The authors are suggested to provide examples of your data distinguishes same-category items by adding "permitted attributes such as color or distance". The mechanics for generating such data is explained in your reply to Q4, so I lean towards believing that these are handled properly.
-
Q7 needs examples. All examples in the current revision still contain only 4 directions in the answer text. Providing some additional examples through simple pictures should be easy.
Overall, the author response solved the majority of my concerns while minor concerns remain. I keep the rating at weak accept and I keep the right to change according to other discussion results and further revisions.
Thank you for the constructive feedback and for highlighting the PoseAlign update. We appreciate your recognition of the multi-heading alignment supervision and agent-centric metric representation. We have rerun the affected experiments and added a controlled pose-interface ablation.
1. Updated PoseAlign evaluation and fairness (W1, W4, Q6; Revised: Abstract, Fig. 1, §1, Fig. 4, Table 1, Table 4, App. A)
Following your suggestion, we reran PoseAlign using the official code and checkpoints updated during the review period, which corrected SQA3D pose handling.
Updated Figure 1(b): SQA3D without situation descriptions
We agree that the corrected PoseAlign exhibits meaningful pose dependence and we revise our original characterization accordingly. EgoAlign retains a 4.1-point advantage in correct-pose EM and a larger pose-shuffle drop. Our revised interpretation acknowledges PoseAlign's effective pose conditioning while highlighting EgoAlign's stronger situated QA performance and functional pose dependence. Note that we use the original multi-dataset training recipe for PoseAlign, only remove the situation descriptions used in SQA3D.
We verified that both baselines receive numerical pose through their native interfaces during alignment and fine-tuning. LEO uses its embodied-agent token without rotating the point cloud, while PoseAlign applies its native pose-conditioned scene transformation, respecting its coordinate and elevation conventions. We also acknowledge that LEO's original mixed-dataset training does not consistently provide pose, which helps explain its limited pose sensitivity.
2. Alignment transfer to PoseAlign (W4, Q6; Revised: §5.3, Table 4, App. C.3)
The updated experiments reveal a further positive finding: our egocentric alignment supervision transfers effectively to PoseAlign.
Updated Table 4 and Appendix A: No-situation SQA3D
Changed (%) denotes the percentage of predictions that differ between correct-pose and shuffled-pose inputs after answer normalization. Compared with direct fine-tuning, the evaluated alignment + FT configuration achieves +1.31 EM, +4.01 points in pose-shuffle drop, and +7.93 percentage points in changed predictions for PoseAlign.
For clarity, direct FT follows PoseAlign's official multi-dataset recipe with situation descriptions removed from SQA3D, whereas alignment + FT uses our 300K alignment examples followed by SQA3D-only no-situation fine-tuning. These results demonstrate successful transfer under the evaluated protocols, although the fine-tuning recipes are not identical.
In updated Table 4, PoseAlign also achieves strong alignment validation, including 63.8% direction accuracy and 47.8% relation accuracy. This demonstrates that our alignment supervision is not architecture-specific. We therefore revise our previous interpretation of Table 4 to emphasize transferability rather than an inability of other architectures to learn the alignment tasks.
We also extended Figure 7's yaw-perturbation evaluation to both baselines after alignment + FT in Appendix C.3. From 0° to 90° yaw error, PoseAlign decreases from 57.6 to 47.4 EM, EgoAlign from 60.4 to 49.8, while LEO remains nearly unchanged. Both EgoAlign and the corrected PoseAlign thus exhibit substantial orientation dependence.