acceptodds
Under review as a conference paper at ICLR 2027

EgoAlign: Situated 3D Understanding without Situation Descriptions

Abstract

Situated 3D Question Answering requires models to interpret a 3D scene relative to an agent’s pose. Existing methods commonly convey that pose through a natural-language situation description, which may be unavailable or costly to obtain in real-world deployments. While some methods additionally incorporate numerical pose, we find that randomly shuffling the agent pose causes little performance degradation, indicating limited functional dependence on the numerical pose. To address this gap, we introduce EgoAlign, a simple 3D language model that represents a scene in an agent-centric coordinate frame, together with a dedicated egocentric alignment stage comprising four complementary tasks: object category recognition, egocentric direction prediction, metric distance estimation, and spatial relation prediction. This stage teaches the model to consistently interpret scene geometry relative to the agent’s pose before task-specific fine-tuning. EgoAlign achieves the strongest performance on SQA3D, Real-3DQA, and MSQA among compared methods while remaining highly efficient. More importantly, removing textual situation descriptions yields comparable or better downstream performance, whereas shuffling EgoAlign’s numerical pose causes substantial degradation. These results show that EgoAlign establishes agent-centric spatial context directly from numerical pose without requiring textual situation descriptions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision-language models, embodied AI, spatial reasoning, point cloud understanding, situated question answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t5GSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoAlign, a 3D language model for situated question-answering. The contributions are three-fold: (1) Proposed a model by combining Concerto with Qwen2.5, following Liu et al. 2026 to enable ego centric information by both transforming the point cloud to an agentic pose, and adding ego-centric position embedding on the input tokens. (2) Extended the idea of Ma et al. 2026 to use egocentric alignment for improving situated directional reasoning, which is done by actually rotating the ego pose by multiples of 90 degrees and determining the expressions of [front, left, back, right] in-situ. This method proved to be the single most important design to achieve sota (Table. 3). (3) A scale-up experiment covering multiple dimensions, signaling a scalable benchmark performance (SQA3D, Real-3DQA, MSQA) but some metrics on the egocentric alignment data remain stagnating across 1.5B to 7B (relation acc, relation consistency, etc.)

Critical strengths
  1. The problem is well-stated, where a numerical pose should be input to the model rather than sending the model a situated text descriptions, especially in realistic application scenarios. (Note: the flip side of this formulation is also discussed in W2)
  2. Presentation is clear, paper is generally well-written and easy to follow. Visualization of Figure 3 is well-drawn.
  3. The results are consistent and strong on all datasets, with a clean scaling curve of model capability with the model weight.
  4. The biggest, most valuable findings are the counterfactual group training on rotated ego poses and the synergetic effect with aligned point cloud and fusing the metric coordinates in the superpoint tokens. The results are rigously validated in Table 3 and provide valuable direction for development of future methods.
Critical weaknesses
  1. Figure 1(b) phenomenon and L74 claim are not precisely fair. Pose insensitivity of PoseAlign on SQA3D is because it used random poses for training on SQA3D, which is fixed recently in its official code. The authors are suggested to follow the fixed code and weights. LEO is trained with optional ego pose encoding only enabled on SQA3D and MP3D but with no pose on other mixed datasets, therefore its ego pose insensitivity is expected. Also see Q6.
  2. Ego-aligned evaluation on situated VQA benchmark kind of defeats its own purpose: The ego pose no longer needs inference and is provided as model input, yet implicit inference of ego pose from language description is the central idea of these datasets. From my perspective, these datasets mainly focus on a benchmark/offline-training aspect to improve model performance, rather than defining their problem formulation as the system model to use during actual deployment, and it is incorrect to discredit their formulation in L068. Furthermore, ego-aligned evaluation on SQA3D and Read-3DQA degrades to simple question-answering tasks such as “What is to your left” while the situation description is less relevant to the task (because it's known at input). This leaves only one task as sufficiently complex, which is MSQA. An extended experiment suite is highly recommended, e.g., on other benchmarks used by 3D-LLaVA or LEO.
  3. Contribution 4 “without requiring textual situation descriptions” is misleading, since you did use the ego situation (and in a stronger, explicit form) as model input.
  4. Table 4 experimental setting is not clearly stated and possibly unfair. Specifically, the numbers in Table 4 are even lower than original performance for LEO and PoseAlign in Table 1, which goes beyond simply 'learn the alignment tasks substantially less effectively' (L424) and raises further questions about implementation fairness. For LEO, it is unclear if the authors either (1) rotate the input to the embodied token, or (2) rotate the input point cloud as did in EgoAlign, where the latter could bring in significant distribution shift which can also cause deterioration. For PoseAlign, it uses a slightly different ego coordinate convention (camera with human height and elevation) compared to egocentric alignment data (floor height pose with zero elevation) which can also cause distributional shift in training data. If these questions are unanswered, the fairness of Table 4 can be compromised. Finally, if these questions persist, this will in turn lead to a new question of the general applicability of your training method.
Questions for authors
  1. The relation task deviates from dataset norms. In your definition, given a ‘bed to the left of a desk’ when the agent is facing them, the agent should answer the opposite direction, i.e., ‘bed is to the right of desk’, when the agent turns to its back. While I agree that this definition is consistent and clear under an agent-centric reference frame, this is not what a normal person would answer; an average person would prefer turning back first and answer from an-object-facing direction, and therefore human-annotated benchmarks would give a different answer from that defined by your relation task. Given this discrepancy maybe it’s better to identify this task as a new task, named ‘egocentric relation’ rather than regular ‘relation’, which most people would assume the latter with a target-facing ego pose.
  2. How are the poses selected? I would assume it’s (4,262+493)*4=19,020 positions according to L689. The authors report 18,750 training and 250 validation poses which sum up to 19,000. Under what rule are the remaining 20 poses pruned?
  3. How are multi-instances handled for your alignment data? For example, there are many cases in ScanQA where a wall has many pictures, or a conference room has many chairs, or a storage room has many boxes. Will two objects of the same class be clearly pinpointed to the exact one you meant in the relation task? Do you have a way of specifying the very target you are referring to in the direction and distance tasks?
  4. Are there human-in-the-loop quality checks for the alignment data?
  5. To further support your core claim that you ‘address pose insensitivity’, can you provide the pose-shuffling performance as an additional column on your ablation experiment (Table 3)?
  6. Following W1, the authors are suggested to conduct the experiment in a fairer setup where the ego poses in SQA3D used by the authors are equally added to training data of all methods, including Figure 1(b), Table 1, Table 2, and Table 4. Figure 7 is also recommended to add the two baseline models, LEO and PoseAlign.
  7. The authors are suggested to display some samples of diagonal relationships, e.g. front-left, front-right, and vertical relationships, e.g., above and below, in the Relation row of Figure 3, because the claimed 8-direction and two vertical relationships (L209) are not fully visualized.
5 comments

Thank you for the constructive feedback and for highlighting the PoseAlign update. We appreciate your recognition of the multi-heading alignment supervision and agent-centric metric representation. We have rerun the affected experiments and added a controlled pose-interface ablation.

1. Updated PoseAlign evaluation and fairness (W1, W4, Q6; Revised: Abstract, Fig. 1, §1, Fig. 4, Table 1, Table 4, App. A)

Following your suggestion, we reran PoseAlign using the official code and checkpoints updated during the review period, which corrected SQA3D pose handling.

Updated Figure 1(b): SQA3D without situation descriptions

MethodCorrect EMShuffled EMΔpose
LEO48.348.10.2
PoseAlign (updated)56.350.65.7
EgoAlign60.449.510.9

We agree that the corrected PoseAlign exhibits meaningful pose dependence and we revise our original characterization accordingly. EgoAlign retains a 4.1-point advantage in correct-pose EM and a larger pose-shuffle drop. Our revised interpretation acknowledges PoseAlign's effective pose conditioning while highlighting EgoAlign's stronger situated QA performance and functional pose dependence. Note that we use the original multi-dataset training recipe for PoseAlign, only remove the situation descriptions used in SQA3D.

We verified that both baselines receive numerical pose through their native interfaces during alignment and fine-tuning. LEO uses its embodied-agent token without rotating the point cloud, while PoseAlign applies its native pose-conditioned scene transformation, respecting its coordinate and elevation conventions. We also acknowledge that LEO's original mixed-dataset training does not consistently provide pose, which helps explain its limited pose sensitivity.

2. Alignment transfer to PoseAlign (W4, Q6; Revised: §5.3, Table 4, App. C.3)

The updated experiments reveal a further positive finding: our egocentric alignment supervision transfers effectively to PoseAlign.

Updated Table 4 and Appendix A: No-situation SQA3D

MethodCorrect EMΔposeChanged (%)
PoseAlign, direct FT56.265.7122.33
PoseAlign, alignment + FT57.579.7230.26
EgoAlign, alignment + FT60.3910.8833.05

Changed (%) denotes the percentage of predictions that differ between correct-pose and shuffled-pose inputs after answer normalization. Compared with direct fine-tuning, the evaluated alignment + FT configuration achieves +1.31 EM, +4.01 points in pose-shuffle drop, and +7.93 percentage points in changed predictions for PoseAlign.

For clarity, direct FT follows PoseAlign's official multi-dataset recipe with situation descriptions removed from SQA3D, whereas alignment + FT uses our 300K alignment examples followed by SQA3D-only no-situation fine-tuning. These results demonstrate successful transfer under the evaluated protocols, although the fine-tuning recipes are not identical.

In updated Table 4, PoseAlign also achieves strong alignment validation, including 63.8% direction accuracy and 47.8% relation accuracy. This demonstrates that our alignment supervision is not architecture-specific. We therefore revise our previous interpretation of Table 4 to emphasize transferability rather than an inability of other architectures to learn the alignment tasks.

We also extended Figure 7's yaw-perturbation evaluation to both baselines after alignment + FT in Appendix C.3. From 0° to 90° yaw error, PoseAlign decreases from 57.6 to 47.4 EM, EgoAlign from 60.4 to 49.8, while LEO remains nearly unchanged. Both EgoAlign and the corrected PoseAlign thus exhibit substantial orientation dependence.

3. Controlled pose-interface ablation (W4, Q5)

To complement the cross-architecture comparison, we added a matched-backbone ablation using the same Concerto/Qwen2.5-1.5B backbone and training schedule.

SettingCat. ↑Dir. ↑Dist. ↓Rel. ↑SQA EMSQA ΔposeReal EMReal Δpose
Default coords28.512.51.1511.648.6—16.4—
Default + agent token33.912.51.1712.050.40.1119.1−0.20
Full EgoAlign61.859.00.4742.256.89.7230.210.30

Adding a LEO-inspired pose token improves downstream accuracy but leaves direction accuracy at chance level for eight directions. The small pose-shuffle drops further indicate that simply exposing this backbone to pose through an agent token does not establish effective egocentric alignment. In contrast, EgoAlign's agent-centric representation substantially improves alignment accuracy and downstream pose dependence.

Regarding Q5, the original Table 3 already reports Δpose for component ablations. Removing alignment reduces SQA3D/Real-3DQA Δpose from 9.72/10.30 to 0.14/0.07.

4. Numerical pose and benchmark setting (W2, W3)

Our contribution is situated 3D understanding without situation descriptions, not without situation information. Numerical pose is explicitly provided throughout our problem formulation. We agree that supplying pose removes the need to infer the agent's reference frame from linguistic descriptions, an important component of the original SQA3D formulation. We study a complementary setting relevant to embodied agents, where localization supplies position and orientation directly. Even with a known reference frame, recognizing objects, grounding references, and reasoning about agent-relative spatial relationships remain nontrivial.

SQA3D, Real-3DQA, and MSQA evaluate these capabilities across different question distributions and scene settings. We agree that additional general-purpose 3D benchmarks would broaden the evaluation scope, while our current experiments focus specifically on pose-conditioned situated reasoning.

5. Data construction and relation semantics (Q1–Q4, Q7; Revised: §3.3)

Q1: We agree with the terminology egocentric spatial relation and now adopt it explicitly. Relations use the supplied agent frame without an implicit turn toward the objects. Consequently, a 180° heading change reverses horizontal egocentric relations.

Q2: The 4,262/493 counts refer to eligible training/validation source positions, whereas 18,750/250 refer to sampled four-heading question groups per task. Multiple groups can originate from one position. Four tasks × 18,750 groups × four headings produce 300K training examples, with 4K validation examples. No 20 source positions are missing.

Q3: Each referring expression must uniquely identify its target instance. Repeated categories are disambiguated using permitted attributes such as color or distance; ambiguous candidates are excluded. For relation questions, both references are checked independently, and same-category pairs are excluded.

Q4: We manually inspected generated examples during development before scaling up data construction. For the full dataset, we apply deterministic quality controls: (1) object references must uniquely identify annotated instances, with ambiguous candidates excluded; (2) color attributes are used only when the dominant color is supported by at least 35% of an instance's RGB points; (3) object centers use coordinate-wise medians and bounding boxes use the 5th–95th point percentiles to reduce outlier sensitivity; and (4) spatial labels follow explicit geometric rules, including eight 45° direction bins, nearest horizontal distance to object bounds, and vertical relations requiring horizontal overlap and vertical separation. We retain complete four-heading groups with fixed target instances and avoid duplicate training records.

Q7: Figure 3 illustrates how spatial relations vary across headings rather than enumerating the full relation vocabulary. The alignment data include eight horizontal directions, including diagonal relations, as well as above/below relations. We now explicitly identify the task as egocentric spatial relation in §3.3, with the geometric definitions provided in Appendix B.

We thank the reviewer for these suggestions, which have helped us correct the baseline comparison, establish alignment transfer to PoseAlign, and strengthen the controlled evidence for EgoAlign's representation design.

Thank you for the update on various topics, which address some of my concerns as below:

  1. W1, W4, and Q6 are solved through updated experimental results, many revised tables and analyses, revised introduction, and adding a yaw noise experiment in appendix Figure 11.

  2. W4 is further addressed in A2 and A3. Your second answer (A2) establishes the general applicability of your training method across two architectures in revised Table 4. Your third answer (A3) adds an interesting experiment by using LEO agent token on your model. The experimental results presented in A3 are slightly concerning since it showed that your Cat., Dir., Dist., Rel., metrics can improve without necessarily gaining pose sensitivity. This is a minor issue, but I do recommend the authors dig into it later. What about defining multilateral metrics, i.e., a metric on your alignment data but calculate the percentage of questions that are answered correctly (or distance error is lower than a certain threshold) on all four directions? All the metrics used in the current paper, except for Δ\Deltapose, are unilateral metrics (EM, Cat., Count, Acc, etc.), or are unilateral metrics averaged across different headings (VRS), instead of a genuine multilateral metric according to the above definition. It is helpful to discuss the distinctions of how unilateral metrics perform compared to multilateral scores and what they indicate.

  3. Q1, Q2, and Q4 are answered. The authors are suggested to add your dataset curation effort details into the next revision since they will provide valuable information to the community.

The following items are not addressed or are only partially addressed:

  1. W2 is partially solved. The claim that SQA3D and Real-3DQA remain nontrivial after removing situation descrption and supplying ego reference frame, needs further proof. E.g., how long are the questions on average after removing the situation descriptions? What are their question type composition and whether do they follow a fixed template? The authors are suggested to prove that these two datasets are nontrivial in the EgoAlign setting.

  2. W3 reply missed the point. I understand that your claim in Contribution 4 is accurate, but contribution claims should be precise even when viewed in isolation. I suggested W3 from a presentation aspect instead of a factual concern aspect, where a first-time reader which skims through your paper would misunderstand this sentence, or even get confused by the seemingly bizzare implication that your model answers 'without' pose information; It would only be clear until the contribution is read along with other text. Therefore, as also pointed out by Reviewer 3ahU, the Contribution 4 is still a slight overstatement. I suggest the authors to revise this point into, e.g., "EgoAlign demonstrates superior performance with explicit pose input compared to counterparts recieving textual situation descriptions, exhibiting strong pose dependence." The significance of this finding can still be established nontheless. For example, you may discuss that letting the VLM implicitly find its ego pose is beyond its capability and practially should be handled in a different enginering solution, rather than cramming all tasks together.

  3. Q3 needs examples. The authors are suggested to provide examples of your data distinguishes same-category items by adding "permitted attributes such as color or distance". The mechanics for generating such data is explained in your reply to Q4, so I lean towards believing that these are handled properly.

  4. Q7 needs examples. All examples in the current revision still contain only 4 directions in the answer text. Providing some additional examples through simple pictures should be easy.


Overall, the author response solved the majority of my concerns while minor concerns remain. I keep the rating at weak accept and I keep the right to change according to other discussion results and further revisions.

Reviewer_4B8POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EgoAlign, a point-cloud language model for situated 3D question answering that uses numerical agent pose without requiring natural-language situation descriptions. The method transforms scene geometry into an agent-centric coordinate frame and explicitly combines pooled metric coordinates and color information with semantic point features. A dedicated alignment stage uses 300K examples covering object recognition, egocentric direction prediction, distance estimation, and spatial relation prediction. Four-heading counterfactual groups teach the model how spatial interpretations should change with orientation while scene content and agent position remain fixed. Alignment is followed by task-specific fine-tuning using projection layers and LoRA adapters. Experiments on SQA3D, Real-3DQA, and MSQA report competitive performance with and without situation descriptions. Pose-shuffling, yaw-perturbation, and consistency analyses further indicate that EgoAlign’s predictions depend on the supplied numerical reference frame.

Critical strengths
  1. A meaningful diagnosis of functional pose use. The paper identifies pose insensitivity as a failure mode that ordinary QA accuracy can overlook. Pose-shuffling, controlled yaw perturbations, and four-heading consistency evaluations provide complementary evidence about whether predictions depend on the numerical reference frame. This is consequential because using the agent’s pose is a defining requirement of situated understanding, and these analyses assess that requirement more directly than leaderboard performance alone.

  2. A coherent combination of representation and targeted supervision. EgoAlign explicitly preserves agent-centric metric coordinates and introduces multi-heading alignment examples that teach how spatial interpretations should change with orientation. Fixed-data-budget ablations support the contributions of the alignment tasks and counterfactual heading construction. The resulting ability to maintain competitive QA performance without situation descriptions provides practical evidence that numerical pose can serve as an effective interface for situated reasoning.

Critical weaknesses
  1. Limited controlled validation beyond the four-heading construction. The paper provides useful four-heading consistency tests and pose-corruption analyses. However, the latter primarily establish dependence on pose: the original answer is retained while an incorrect pose is supplied. More direct evidence would come from counterfactual pairs with correctly updated answers across continuous heading changes and position changes within the same scene. This would help establish whether the learned spatial transformation generalizes beyond the alignment construction, which is consequential to the claim of numerical-pose-grounded understanding.

  2. Confounded attribution in the cross-architecture transfer comparison. Table 4 applies the same alignment data to LEO, PoseAlign, and EgoAlign, but these systems differ in language backbones as well as scene representations and integration mechanisms. Their unequal performance therefore does not isolate the representation as the cause of unsuccessful alignment transfer. The within-model coordinate ablations are informative, but a matched-backbone comparison would better support the broader architectural conclusion. This matters because the paper attributes effective pose grounding to the combination of its representation and alignment supervision.

Questions for authors
  1. Could the authors provide a targeted counterfactual evaluation beyond the four-heading alignment construction, using non-cardinal headings and changes in agent position within the same scene? Keeping the scene and question fixed while updating the reference answer for each valid pose would help distinguish correct spatial transformation from sensitivity to corrupted pose inputs.

  2. Could the authors clarify how the Table 4 comparison controls for differences in language backbones and optimization settings? Although the alignment data are shared, a matched-backbone comparison or focused control under a common training budget would help isolate the contribution of the scene representation to successful alignment transfer.

2 comments

Thank you for recognizing the value of evaluating functional pose use and the combination of agent-centric metric features with targeted supervision. We conducted additional experiments addressing both concerns.

1. Generalization beyond the four-heading construction (W1, Q1; Revised: App. C.4)

The four training headings are offsets relative to each source SQA3D pose, whose orientation varies across scenes; they are not four fixed global directions. Nevertheless, we agree that evaluation beyond the 90°-spaced construction is important for establishing generalization.

We therefore conducted a new counterfactual evaluation on 53 held-out scenes, using 250 four-heading groups per task under each condition:

  • Original headings: The original four 90°-spaced headings at the annotated position.
  • New headings: Each original heading independently perturbed by a signed angle of 15–30°, producing non-uniform angular separations.
  • New position: The original headings evaluated at an agent position shifted by 1 m.

We keep the scene, question, and target objects fixed across conditions and recompute ground-truth answers for every pose. The evaluation uses the aligned 7B checkpoint before downstream fine-tuning.

Evaluation conditionDir. Acc.Dir. 4/4Rel. Acc.Rel. 4/4
Original headings64.143.654.039.2
New headings57.838.249.535.2
New position65.448.053.841.6

Here, 4/4 consistency requires all four predictions to match their pose-specific reference answers. These results demonstrate generalization beyond the exact training heading pattern.

2. Separating representation effects from backbone differences (W2, Q2; Revised: §5.3, Table 4)

We agree that Table 4 compares complete systems and cannot independently attribute performance differences to their scene representations. For transfer experiments, all models receive the same 300K alignment examples, while retaining their native pose interfaces and model-specific optimization recipes. LEO continues from its released alignment checkpoint with its official fine-tuning learning rate; PoseAlign follows its official configuration. We apply one alignment epoch rather than imposing identical hyperparameters across different architectures.

Following PoseAlign's official code and checkpoint update during the review period (commit), we also reran the affected evaluations.

Updated cross-architecture transfer (Table 4):

MethodCat. Acc.Dir. Acc.Dist. MAERel. Acc.SQA3D EMΔpose
LEO30.912.70.9512.546.63−0.06
PoseAlign (updated)63.263.80.4747.857.579.72
EgoAlign63.261.30.4246.660.3910.88

The corrected PoseAlign learns the alignment tasks effectively, demonstrating that our supervision transfers beyond EgoAlign. We have revised the previous architectural interpretation of this experiment accordingly.

To isolate representation design, we additionally conducted a matched-backbone comparison under the same Concerto/Qwen2.5-1.5B backbone, 300K alignment corpus, and training schedule:

SettingCat. ↑Dir. ↑Dist. ↓Rel. ↑SQA EMΔposeReal EMΔpose
Default coords + agent token33.912.51.1712.050.40.1119.1−0.20
Full EgoAlign61.859.00.4742.256.89.7230.210.30

The agent-token control supplies numerical pose but achieves only 12.5% direction accuracy, with almost no downstream pose dependence. Agent-centric transformation without coordinate projection performs better, but still underperforms the full model.

Together, these controls support complementary contributions from agent-centric scene geometry and explicit metric coordinate features, independently of backbone differences.

Thank you again for the suggestions, which motivated more direct counterfactual evaluation and a stronger controlled analysis of our representation design.

I thank the authors for the additional counterfactual evaluations and matched-backbone controls, as well as the clarification and revision of the cross-architecture transfer interpretation. These additions address most of my concerns. After considering the rebuttal, I will maintain my original rating of 3 (Weak Acceptance).

Reviewer_3ahUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoAlign, a point-cloud language model for situated 3D question answering without requiring textual situation descriptions. The method transforms scene geometry into the agent’s coordinate frame, adds explicit coordinate and color projections to point features, and introduces an alignment stage covering category recognition, direction prediction, distance estimation, and spatial relations. Training examples are expanded across four headings. Experiments on SQA3D, Real-3DQA, and MSQA show competitive performance, while pose-shuffling experiments indicate substantially greater dependence on numerical pose than the evaluated baselines.

Critical strengths
  1. The paper examines a meaningful issue beyond benchmark accuracy. Testing whether a model actually uses its supplied pose is useful, particularly when language shortcuts can support relatively high QA scores. The pose-shuffle experiments and yaw perturbation analysis provide a more informative evaluation than reporting aggregate accuracy alone.

  2. The approach is simple and achieves strong practical results. EgoAlign reaches 61.2 EM on SQA3D and 60.0 overall on MSQA without situation descriptions in the latter setting. The reported inference efficiency is also attractive. These results make the method a potentially useful baseline for pose-conditioned 3D QA.

  3. The ablations cover several relevant design choices. The paper evaluates coordinate projection, alignment tasks, heading diversity, and model scale. Keeping the alignment corpus at 300K examples when removing individual tasks is a useful control. Applying the alignment data to existing architectures also helps investigate whether the data alone explains the gains.

Critical weaknesses
  1. The novelty over prior work is not sufficiently established. Pose-related supervision has already been explored in SIG3D, and SPARTUN3D uses situation-conditioned spatial data and explicit alignment supervision. MSR3D and PoseAlign also transform scene representations according to the agent’s pose. These methods are not identical to EgoAlign, but they cover much of its motivation and basic design. The contribution therefore seems to lie in the particular combination of metric coordinate features and multi-heading training. I would like to see stronger evidence that this combination offers something beyond existing pose-conditioned representations and spatial supervision. Simply showing that pose information is useful does not establish this.

  2. The fourth contribution is better viewed as a useful experimental result than a new finding. The paper shows that its model performs similarly with and without situation descriptions when numerical pose is available in both settings. This does not establish that pose is generally more effective than text. On SQA3D, removing text actually lowers EM from 61.2 to 60.4. More fundamentally, an annotated pose directly specifies the reference frame, whereas a textual description may require resolving ambiguity and localizing the agent first. The result is reasonable and useful, but its broader significance is overstated. A controlled comparison of text-only, pose-only, and combined inputs would be needed to support a stronger conclusion.

  3. The benefit of the proposed counterfactual construction remains unclear. In Table 3, single-heading alignment already achieves 55.6 EM on SQA3D, compared with 56.8 for the full method; the corresponding Real-3DQA scores are 27.0 and 30.2. The additional gains are meaningful, but the current comparison does not tell us whether they come from constructing matched counterfactual groups or simply exposing the model to more varied headings. Since the training objective treats examples through standard autoregressive supervision, independently sampled headings could plausibly provide a similar benefit. A comparison with matched heading coverage would help establish the value of the proposed construction. Table 4 is useful, but differences in encoders and language backbones also make it difficult to attribute the gap specifically to EgoAlign’s representation.

  4. The interpretation of the pose-shuffling results is stronger than the evidence supports. An accuracy drop after corrupting pose shows that the model depends on that input. It does not necessarily show that the model changes its answers correctly when the viewpoint changes. Likewise, unchanged aggregate accuracy does not mean that a baseline’s predictions are unchanged: different errors can produce the same accuracy. The significance tests in Appendix A do not resolve this distinction. The Real-3DQA results provide some supporting evidence, but the central claim would be more convincing with a breakdown of correct answer changes and cases where answers should remain unchanged, together with joint accuracy across viewpoints.

Questions for authors
  1. In the “default coordinates” ablation, does the model still receive the agent’s position and heading through another input? If not, this comparison removes pose information altogether rather than testing how it should be represented.

  2. How is the single-heading training set constructed? Does it have the same overall heading distribution as the four-heading set? Are the four variants processed jointly during training, or simply treated as separate examples?

  3. For the LEO and PoseAlign experiments in Table 4, are agent-relative coordinates preserved through preprocessing and feature extraction? Their direction accuracies are close to chance, so it is important to establish whether the necessary pose information reaches the trainable modules.

  4. What fraction of each model’s predictions changes under pose shuffling? Do the existing multi-view results show that EgoAlign changes its answers correctly when required while preserving them when the answer should remain the same?

4 comments

Thank you for recognizing the importance of evaluating functional pose use beyond aggregate QA accuracy. We appreciate the detailed suggestions and have conducted additional controlled experiments to clarify the contributions of our representation and supervision.

1. Contribution beyond existing pose-conditioned methods (W1, Q1)

Prior work has explored agent-centric representations and spatial supervision, but incorporating pose does not necessarily ensure that a model effectively uses it , as evidenced by our pose-shuffling evaluations. Moreover, methods such as SIG3D and SPARTUN3D focus on spatial grounding involving textual situation information, whereas we study situated understanding directly from numerical pose without textual situation descriptions . To address this setting, EgoAlign combines explicit agent-centric metric features with structured multi-heading alignment supervision, teaching the model to interpret the same scene configuration under different agent orientations.

You correctly point out that our original Default coordinates ablation does not supply pose through another interface. We therefore added a LEO-inspired agent-token control, using the same Concerto/Qwen2.5-1.5B backbone, alignment data, and training schedule:

SettingCat. ↑Dir. ↑Dist. ↓Rel. ↑SQA EMΔposeReal EMΔpose
Default coords28.512.51.1511.648.6—16.4—
Default + agent token33.912.51.1712.050.40.1119.1−0.20
Full EgoAlign61.859.00.4742.256.89.7230.210.30

Although adding an agent token improves downstream accuracy, direction accuracy remains 12.5% (uniform chance level for eight directions), with negligible downstream pose dependence. EgoAlign substantially improves both alignment and situated QA. Together with the original No coordinate projection ablation, these results support the complementary contributions of agent-centric geometry and explicit metric coordinates under a matched backbone.

2. Scope of the no-situation claim (W2)

We would like to clarify that our paper does not claim that numerical pose is generally more effective than textual situation descriptions. Rather, our claim is that strong situated 3D understanding can be achieved without textual situation descriptions when numerical pose is available. These are fundamentally different claims.

Indeed, we explicitly report that removing situation descriptions slightly decreases SQA3D EM from 61.2 to 60.4, and characterize the results as “comparable or better”, rather than consistently better. Real-3DQA and MSQA further show improvements without situation descriptions.

The significance lies in the deployment setting and input interface: a localized embodied agent already has access to its position and orientation, and our approach enables situated reasoning directly from this information without requiring an additional textual description. We recognize that inferring the agent's situation from language is a valuable but distinct problem, which our work does not aim to replace.

3. Four-heading construction versus heading diversity (W3, Q2)

Our original Table 3 already includes a 300K-example single-heading ablation, with the same task balance and training budget as the full four-heading construction. Importantly, the single-heading corpus already covers the full orientation range. Measured relative to each scan's coordinate axes:

Training corpus0–90°90–180°180–270°270–360°
Single heading24.54%25.25%24.66%25.55%
Four headings25.00%25.00%25.00%25.00%

Thus, the single-heading baseline already has broad, approximately balanced global heading coverage. The difference is that our four-heading construction systematically covers the same scene–position–target configuration across four orientations, rather than sampling only one heading per question instance.

The four variants are treated as separate autoregressive training examples, without a joint group loss. The improvement therefore comes from the structure and coverage of the supervision rather than a specialized optimization objective. Our results show that the four-heading construction improves over the tested single-heading alternative despite matched training size and similar global heading distributions.

4. Pose processing and updated cross-architecture results (W3, Q3; Revised: §5.3, Table 4)

We verified that both transfer baselines retain their native pose interfaces. LEO receives numerical position and orientation through its embodied-agent token without rotating the point cloud. PoseAlign uses its own pose-conditioned geometric transformation and coordinate conventions. Thus, the required pose information reaches each architecture through its intended processing path.

During the review period, PoseAlign's official code and checkpoints were updated to correct SQA3D pose handling (commit). We reran the affected experiments and revised Figure 1, Table 1, Table 4, and Appendix A. Importantly, updated Table 4 demonstrates that our alignment supervision transfers successfully to PoseAlign. It reaches 63.8% alignment direction accuracy and 57.57 SQA3D EM with a 9.72-point shuffle drop. We therefore interpret Table 4 as evidence of alignment transferability, rather than a controlled architectural attribution; the matched-backbone ablations above provide the latter.

5. Prediction changes versus correct spatial transformation (W4, Q4; Revised: App. A, C.4)

We agree that pose shuffling establishes dependence on numerical pose but does not by itself demonstrate correct viewpoint-conditioned reasoning. To address your concern, we measured how frequently predictions change after pose shuffling in the no-situation SQA3D setting:

SettingMethodCorrect EMShuffled EMΔposeChanged (%)
Direct FT (-sit.)LEO48.2848.110.176.65
Direct FT (-sit.)PoseAlign (updated)56.2650.555.7122.33
Alignment + FT (-sit.)LEO46.6346.69−0.065.54
Alignment + FT (-sit.)PoseAlign (updated)57.5747.859.7230.26
Alignment + FT (-sit.)EgoAlign60.3949.5010.8833.05

Following alignment, its changed-prediction rate increases from 22.33% to 30.26%, while EgoAlign reaches 33.05%. These rates measure behavioral sensitivity rather than correctness.

To evaluate correct viewpoint adaptation, we additionally tested the aligned 7B model on 53 held-out scenes, with non-uniform heading perturbations of ±15–30° and position shifts of 1 m. We keep the scene, questions, and target objects fixed while recomputing ground-truth answers for each valid pose.

We separate direction and relation examples according to whether the ground-truth answer should change:

Prediction change rate conditioned on ground-truth behavior (%)

Ground-truth answerNew headingsNew positions
Changes66.266.3
Remains unchanged28.215.4

The model changes its predictions substantially more often when the reference answer changes than when it remains unchanged. This indicates selective responsiveness rather than indiscriminate prediction changes. Nevertheless, prediction changes alone do not establish correctness.

We therefore also evaluate 4/4 consistency, requiring correct answers under all four valid poses, each with its own ground-truth reference answer:

Evaluation conditionDir. Acc.Dir. 4/4Rel. Acc.Rel. 4/4
Original headings64.143.654.039.2
New headings57.838.249.535.2
New position65.448.053.841.6

Unlike pose shuffling, this evaluation measures correctness under multiple valid reference frames. These results demonstrate generalization beyond the original construction.

Together, the conditional prediction-change analysis and multi-view correctness evaluation provide complementary evidence of selective pose use and correct spatial interpretation.

We appreciate the reviewer's suggestions, which helped us distinguish pose dependence from correctness, strengthen the controlled representation comparison, and clarify the scope of multi-heading supervision.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer 3ahU,

Thank you again for your detailed and constructive review. As the discussion period is approaching its end, we would greatly appreciate your feedback on our response and the revised manuscript.

In particular, we have provided additional controlled experiments on pose representation, updated the transfer experiments and analysis, and conducted new counterfactual evaluations under valid heading and position changes to address the distinction between pose sensitivity and correct spatial reasoning.

We would be grateful to know whether these results address your main concerns, or if there are any remaining questions we could clarify.

Thank you for your time and consideration!

Thank you for the additional experiments. The agent-token control addresses the missing-pose issue in the original ablation. The new evaluations with updated answers also provide evidence of correct spatial reasoning beyond pose sensitivity.

However, the corrected PoseAlign results change the original explanation. PoseAlign already uses pose effectively and learns successfully from the proposed alignment data. This supports the usefulness of the data, but weakens the argument for EgoAlign’s particular representation.

My main concern remains what new insight the paper provides. I accept that the authors do not claim pose is universally better than text. Still, numerical position and orientation directly specify the reference frame, while situation descriptions require inference. Finding that text becomes largely unnecessary in this setting is useful, but not surprising. The multi-heading supervision is more promising, yet its contribution beyond existing spatial supervision and ordinary heading augmentation remains unclear.

The rebuttal improves my confidence in the results, but does not sufficiently address my concern about novelty and insight. I will maintain Weak Reject.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author–Reviewer Discussion

We sincerely thank all three reviewers for their constructive feedback and thoughtful discussion.

During the rebuttal, we reran the affected PoseAlign experiments using its corrected official implementation updated during the review period (commit), and revised the corresponding results and claims. The new experiments also demonstrated that our egocentric alignment supervision successfully transfers to PoseAlign. We further conducted a matched-backbone pose-interface ablation and additional evaluations under valid heading and position changes, addressing concerns about architectural attribution and correct spatial reasoning beyond pose sensitivity. We also clarified the benchmark setting and expanded the alignment-data construction details and examples in the revised manuscript.

We are encouraged by the reviewers' feedback. Reviewer t5GS explicitly confirmed that all concerns were fully addressed. Reviewer 4B8P acknowledged that the additional experiments and revisions addressed their concerns. Reviewer 3ahU also recognized that the new experiments addressed the pose-input issue and provided evidence of correct spatial reasoning, while maintaining a concern regarding novelty.

We believe the discussion has helped clarify EgoAlign's contribution: an effective agent-centric representation combined with egocentric alignment supervision that also benefits existing pose-conditioned architectures. We hope the revised manuscript and additional evidence will assist the AC in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.